Redfield Updates goes wrong

goodvirus · December 14, 2022, 9:38am

Hi,

I´m using KNIME 4.7 and when I update the redfiled extentsion I got an error:

The error:

Did I anything wrong?

Best regards,

Paul

mlauber71 · December 14, 2022, 9:52am

@carstenhaubold would it be an idea to bundle the necessary packages with the bundled Python version and make them aware? I had such questions from colleagues also.

carstenhaubold · December 14, 2022, 10:11am

@mlauber71 I don’t fully understand the question, could you elaborate please? Which files should be bundled?

The case @goodvirus is seeing is confusing. We just checked, this was just a rebuild of the community extension. The problem that @goodvirus reported yesterday in a separate thread about the Spacy extension not working was fixed by us locally yesterday, but the changes are not merged and thus not available yet.

python.exe should be part of the channel plugin. @goodvirus could you have a look at KNIME’s “Error Log” (in the menu under “View”) whether there were already problems before that? We’d be expecting to see some lines like ShellExec command: ./create_env.bat and ShellExec directory: ....

goodvirus · December 14, 2022, 10:18am

Hi @carstenhaubold,

here is the error-log:

!ENTRY org.knime.product 1 0 2022-12-14 10:35:42.930
!MESSAGE ShellExec command: C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/bin/create_env.bat "C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/pkgs"

!ENTRY org.knime.product 1 0 2022-12-14 10:35:42.931
!MESSAGE ShellExec directory : C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/bin

!ENTRY org.knime.product 1 0 2022-12-14 10:35:46.230
!MESSAGE Standard output: 
C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>REM Assumes to be called from the parent of the directory the script is located in 

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>SETLOCAL ENABLEDELAYEDEXPANSION 

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>SET unescaped="C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/pkgs" 

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>SET escaped=!unescaped: =%20! 

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>ECHO "C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/pkgs" 
"C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/pkgs"

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>.\micromamba.exe create -f ../env.yml -p ../env -r ../../../bundling/root -c "C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244/pkgs" --override-channels --platform win-64 --no-rc --no-env -y  1>.\create_env.out 2>.\create_env.err 

C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin>IF EXIST "..\requirements.txt" ..\env\python.exe -m pip install --no-warn-script-location --no-index --find-links ../pip_pkgs/ -r ../requirements.txt 


!ENTRY org.eclipse.equinox.p2.engine 4 4 2022-12-14 10:35:46.246```

Hope it helps

carstenhaubold · December 14, 2022, 10:22am

Thanks @goodvirus, I have one more inquiry:
could you please also look at the text file C:\Users\spo\Desktop\KNIME\KNIME\plugins\se.redfield.textprocessing.channel.bin.win32.x86_64_1.1.1.202212131244\bin\create_env.err and paste the contents here?

goodvirus · December 14, 2022, 10:29am

Hi @carstenhaubold,

it seems we found the problem:

"error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libzlib-1.2.13-hcfcfb64_4'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libsqlite-3.40.0-hcfcfb64_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\snappy-1.1.9-hfb803bf_2'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\openmp-5.0.0-vc14_1'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\xz-5.2.6-h8d14728_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libutf8proc-2.8.0-h82a8f57_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\re2-2022.06.01-h0e60522_1'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\lz4-c-1.9.3-h8ffe710_1'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\zstd-1.5.2-h7755175_4'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\bzip2-1.0.8-h8ffe710_4'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libbrotlicommon-1.0.9-hcfcfb64_8'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libcrc32c-1.1.2-h0e60522_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libffi-3.4.2-h8ffe710_5'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libbrotlienc-1.0.9-hcfcfb64_8'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libbrotlidec-1.0.9-hcfcfb64_8'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libflang-5.0.0-h6538335_20180525'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libcurl-7.86.0-heaf79c2_1'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libblas-3.9.0-16_win64_openblas'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\liblapack-3.9.0-16_win64_openblas'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\libcblas-3.9.0-16_win64_openblas'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\numpy-1.23.5-py39hbccbffa_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\pandas-1.5.2-py39h2ba5b7c_0'
error    libmamba Error when extracting package: Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\vs2015_runtime-14.32.31332-h1d6e394_9'
critical libmamba Zugriff verweigert
    : 'C:\Users\spo\Desktop\KNIME\KNIME\bundling\root\pkgs\vs2015_runtime-14.32.31332-h1d6e394_9'

Which is related to McAfee blocks access to geospatial extension - #2 by steffen_KNIME

It seemed that this files were no probleme, but when I want to update, the scanner picks these up and blocks the access… So no problem on your side, just the virus scanner.

Thanks for you help! (I don’t know what to do, because my security team is not really helping me out here)

Thanks,

Paul

carstenhaubold · December 14, 2022, 10:43am

Hi Paul,

oh well, that’s sad. Too bad McAfee blocks the files in our Python environments. As this is a problem that will probably surface more often now that we ship more extensions based on Python… do you know whether there is any way that you can create and use Python environments on your machine that satisfies your company policies and McAfee @goodvirus?

We are trying to come up with workarounds / solutions for the future for you and all our users. We’d prefer not to have to go all the way and provide Docker images running our Python installation

Thanks,
Carsten

goodvirus · December 14, 2022, 10:58am

I can install any python package I want in an enviroment and there seems to be no probleme, I don’t know why McAfee seems to have a problem with the bundled version…

mlauber71 · December 14, 2022, 3:58pm

The idea would be that the bundled Python version would fully support the Redfield nodes and also the main language packages, if possible also on the server (currently not sure about bundled Python and KNIME server to be honest). I have not tested all this myself unfortunately- a colleague asked if the Redfiled nodes could be used (also for German language) packages without additional Python installations - that would make the handling much easier. If this is already the case and I have just not checked it that also would be OK

steffen_KNIME · December 14, 2022, 4:29pm

Hi mlauber71,

taking the packages necessary for the Redfield nodes into the bundled environment would after my very rough estimations add for MacOS 80-150MB to the 250MB big bundled environment and for Windows even more to an even bigger environment. Increasing a standard environment by the factor 1.5 for one use-case seems to be a bigger decision. I’ll leave that to Carsten.

Best regards
Steffen

carstenhaubold · December 15, 2022, 8:32am

Hi @mlauber71,

when you install the Redfield Spacy or BERT extensions, you will get a bundled Python environment tailored specifically for those nodes. No need for a user to set up a Python environment on their own. (The installation of this bundled env is exactly what failed for @goodvirus).

As Steffen said, putting all of those dependencies into the normal bundled environment that you get with the KNIME Python integration does not make a lot of sense, as it does add quite a substantial amount of MB to the download which is not needed for every user of the Python integration.

Cheers,
Carsten

mlauber71 · December 15, 2022, 9:14am

Would this also include different language packages? And would it also be possible to use this with the KNIME server?

nemad · December 16, 2022, 8:55am

Would this also include different language packages?

Yes, spacy’s variety in languages was one of the main motivators for the extension.
The bundled environment references more than 20 languages plus multi-language models in the model selector. It also allows you to bring your own model but in that case you might have to setup a dedicated Conda environment if this model has additional dependencies that are not part of the bundled environment. You can do that via the Redfield NLP Nodes preference page which is similar to our Python preference pages and allows you to create a new environment with the click of a button or select an already existing environment.

In case of the BERT extension, you also have a number of models for different languages from both TF Hub and Huggingface available.

And would it also be possible to use this with the KNIME server?

Yes, at least from the development perspective there is no reason why they shouldn’t run on KNIME server.

system · December 23, 2022, 8:56am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.