Problems running conda environment with knime-full:r-5.4.0-62

[tescnovonesis]

Hi,

I am trying to test python with knime-full:r-5.4.0-62.

Conda environment:
conda create -y --name test_conda_py3.11_k5.2.0 -c knime -c conda-forge python=3.11 knime-python-scripting=5.2.0

Execute failed: Failed to execute Conda: /home/knime/miniconda3/lib/python3.12/argparse.py:2006: 
FutureWarning: `remote_definition` is deprecated and will be removed in 25.9. 
Use `conda env create --file=URL` instead. action(self, namespace, argument_values, option_string) 

Traceback (most recent call last): File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 18, in __call__ return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/cli/main.py", line 61, in main_subshell exit_code = do_call(args, parser) ^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/cli/conda_argparse.py", line 205, in do_call result = getattr(module, func_name)(args, parser) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/notices/core.py", line 132, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/cli/main_env_create.py", line 178, in execute result[installer_type] = installer.install( ^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/env/installers/conda.py", line 62, in install unlink_link_transaction.download_and_extract() 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/core/link.py", line 257, in download_and_extract self._pfe.execute() 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/core/package_cache_data.py", line 900, in execute raise CondaMultiError(not_cancelled) conda.CondaMultiError: 
Error with archive /home/knime/miniconda3/pkgs/scipy-1.11.4-py311h64a7726_0.conda. You probably need to delete and re-download or re-create this file. 

Message was: failed with error: [Errno 28] No space left on device: '/home/knime/miniconda3/pkgs/scipy-1.11.4-py311h64a7726_0/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py' 

Error with archive /home/knime/miniconda3/pkgs/statsmodels-0.14.0-py311h1f0f07a_3.conda. You probably need to delete and re-download or re-create this file. 
Message was: failed with error: [Errno 28] No space left on device: '/home/knime/miniconda3/pkgs/statsmodels-0.14.0-py311h1f0f07a_3/lib/python3.11/site-packages/statsmodels/compat/scipy.py' 
Error with archive /home/knime/miniconda3/pkgs/pandas-2.0.3-py311h320fe9a_1.conda. You probably need to delete and re-download or re-create this file. 
Message was: failed with error: [Errno 28] No space left on device: '/home/knime/miniconda3/pkgs/pandas-2.0.3-py311h320fe9a_1/lib/python3.11/site-packages/pandas/core/reshape/__pycache__/concat.cpython-311.pyc' 
During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/knime/miniconda3/bin/conda", line 13, in <module> sys.exit(main()) ^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/cli/main.py", line 105, in main return conda_exception_handler(main, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 388, in conda_exception_handler return_value = exception_handler(func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 21, in __call__ return self.handle_exception(exc_val, exc_tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 62, in handle_exception return self.handle_application_exception(exc_val, exc_tb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 78, in handle_application_exception self._print_conda_exception(exc_val, exc_tb) 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exception_handler.py", line 84, in _print_conda_exception print_conda_exception(exc_val, exc_tb) 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/exceptions.py", line 1265, in print_conda_exception exc_val.dump_map(), indent=2, sort_keys=True, cls=EntityEncoder ^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/__init__.py", line 136, in dump_map errors=tuple(error.dump_map() for error in self.errors), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/home/knime/miniconda3/lib/python3.12/site-packages/conda/__init__.py", line 136, in <genexpr> errors=tuple(error.dump_map() for error in self.errors), ^^^^^^^^^^^^^^ AttributeError: 'InvalidArchiveError' object has no attribute 'dump_map

[tescnovonesis]

If try with tag knime-full:r-5.3.3-666

I get

Execute failed: Conda process terminated with error code 137.

[tescnovonesis]

Not sure if this is relevant
Conda Propagation Node fails with error 137 - KNIME Business Hub - KNIME Community Forum

[thor]

@tescnovonesis

failed with error: [Errno 28] No space left on device

Seems your host system ran out of disk space.

[tescnovonesis]

So, on the Kubernetes that is running KNIME Business Hub for enterprise, where is the setting that will give the executor image some storage?

[tescnovonesis]

Increasing ram to 8GB for 5.4.0 solved this one.

For tag r-5.3.3-666, I am double checking if the Dockerfile is running with
USER knime.

[tescnovonesis]

Closing this one.

Upon inspection of the EC2 that run the KNIME Hub executors, we saw the disk was capping out at 150 MiB/second.

Something somewhere must have been stuck in a loop.
Replacing the EC2 and new volume restored stability.

1: Increase disk size EC2 that runs the executors. From 50GB to > 100 GB.
It seemed conda environments would take up space on this machine.

2: Changes to executor dockerfile needs new tag
During testing, I rebuilded the executor dockerfile with some additional linux libraries for Barcode reading (zbar-tools). Even though the SHA fingerprint at the Container Registry changed, KNIME HUB would cache on naming:tag.

3: New conda env needs to run 2 times
First time running a new conda environment, it would fail. Re-running would be OK. Inspecting this further…

4: 8 GB RAM for executors?
Maybe/maybe not?, the executor RAM should be 8 GB. But now I am not sure about this, since it might be the other observation that tricked me with the need for re-running