We have issues when trying to install our custom built Python node extension.
We have closely followed this guide: Create a New Python based KNIME Extension to create and bundle our extension.
Bundling was successful, we used --include-packages and --knime-version 5.2 flags during bundling.
When trying to install extension that we just bundled to KAP 5.2.3 on Windows machine, we get an error:
Python Extension Labs is installed.
Anaconda is installed and configured in KAP settings.
Long paths are enabled.
Running KAP with admin rights does not help either.
We are out of ideas, why it fails to (what it seems like) create a script for markdown package.
nice that you try it out and that you got this far.
Thanks for the very good troubleshoot beforehand.
The path does not seem to be too long (at least as you have long paths enabled).
Some questions to hopefully nail it down:
Can you confirm that these files exist which it is trying to access?
What does this Scripts folder contain?
Is the Scripts folder directly in the root of your environment?
Does anything in that Scripts folder look familiar to you, i.e. could it be from your development?
Let’s also have a look at the bundling environment, which you used to build and bundle that extension.
a. could you please in a terminal/Powershell/anaconda prompt conda activate <your bundling environment>
b. and conda list knime
c. and paste the output here to determine which version of knime-extension-bundling you have exactly?
They do not exist. During the installation, some kind of KNIME script in the background is creating files in /bundling/envs/<insert environment name>. I believe, as soon as it starts working on markdown related package:
it just crashes with erratic exception.
It’s not being created. As far as I can understand, mamba is trying to build anaconda environment which will be used to execute the extension, but something in the process fails, therefore Scripts folder is not created.
What KNIME extension installer creates before it crashes:
P.s. we tried installing bundled extension on one of my colleagues machine and it was correctly installed on Kubuntu 23.04 x86_64, so this may be somewhat related to Windows platform.
I would also be happy to take this offline if necessary, as we have enterprise license for our PoC period.
Let us know, when you find out something, because we are blocked by this issue (we can’t share extension with the clients, who use Windows based machines).
New findings:
We did further investigation and found out, that commenting out our private pip packages from env.yml, in the end allowed to install the extension.
These private packages are used in Python based nodes source. We are trying to make our proprietary functions available as KNIME nodes for the clients.
So maybe the libmamba solver cannot correctly collect pip packages from the extra repository… or any other issue, we have no idea at this point…
But then it’s strange why this fails on Windows platform, but works on Linux…
Our env.yml:
If you have any idea, how we can help to speed up the investigation process, let us know (my colleagues are following this thread).
indeed, that is strange. Instead of trying to get the functionality via pip package, could you copy the files next to your node and access it from there?
While it’s doable it’s a workaround and not an acceptable solution as it would require us to copy multiple dependencies from the dependency chain with each of the extensions.
you are using --include-packages which should download all conda and pip packages while bundling
bundling succeeds (on which OS do you do this?)
installation on Windows fails
If you are willing to debug some more:
Could you please add the command --render-folder SOME_DIRECTORY to build_python_extension.bat? That will put the bundled extension into SOME_DIRECTORY so that the contents can be easily analyzed.
If you run that, there should be a folder SOME_DIRECTORY\\com.your.extension.name.channel.bin.win32.x86_64\\pkgs\\pip. Does that contain the pip packages that are coming from your private pip repository?
Maybe there is an issue in our bundling scripts that prevents fetching pip packages from private repositories in the --include-packages mode.
I’ll ask someone who’s running windows to do it, but yes --include-packages does in fact bundle the wheels for different distros. We confirmed this by inspecting the jar.
We managed to bundle and install our extension on Windows, with some trouble.
One issue that some of us ran into on Windows 10 is the pip-repositories section of the .yml config and pip.conf/ini files being seemingly ignored, which would fail the process due to inability to find the pypi dependencies from our private repo.
I had to manually add a line to conda_lock/package pypy_solver.py/_prepare_repositories_pool:
repos.append(factory.create_legacy_repository(source, config))
Where source was our private repo url. After this, I could build the extension on my Windows 10 machine.
After looking at the --render-folder output directory, nothing seems missing. The required pip dependencies appear to be included with the expected versions. This was done on Windows 10, but all the dependencies were also present in the earlier Linux builds so there’s no difference here between OS.
The pip_pkg_urls.txt file in the same output directory also lists our packages, with most coming from the common https://files.pythonhosted.org/... source and the rest from our private repository url. So this part seems correct.
So the remaining issue would be figuring out why our extension built from the same source on Linux or WSL does not install in a Windows environment, but one built on Windows installs on both. Not all of us work on Windows, so it would be great if the builds were shareable and interchangeable.
