I'd like to download files from S3 to KNIME server

I’d like to download files from S3 to KNIME server.

I got below error message from KNIME server console.
“ERROR Download 4:105 Execute failed: Illegal char <:> at index 27: C:/Program Files/KNIME/http::”

I’d like to download files to below folder.
“knime://knime.mountpoint/Users/knimeadmin/ASX/input/eve/yyyymmdd”.
But failed .
Could you teach me how to indicate file path or other altanative method.

Thank you,
Taku

Hi,
Currently the Download node does not support the knime:// protocol for the target folder, but we already have an open ticket for changing that! In the meantime, you can use a workaround: using List Folders you can get a list of all subfolders in your current mountpoint. Here we can make use of the fact that the .metadata folder is always there, so we filter the list of folders with a Row Filter and the pattern *.metadata* on the URL column (wildcards enabled). Now you can turn that row into a variable using Table Row to Variable and replace .metadata with your subfolder of choice using a String Manipulation (Variable) node and the expression

replace($${SURL}$$, ".metadata", "Users/knimeadmin/ASX/input/eve/yyyymmdd")

The variable created in such a way can then be used in the target folder setting.

Kind regards,
Alexander

Hi, Alexander

Thank you very much for valuable information.
Really helpful for me.
Could I ask one more?
I got attached error.

“ERROR List Folders 7:1011 Execute failed: (“NullPointerException”): null”

I’d like to solve this error.

Best regards,


Taku

Hi Taku,
Hmm, no idea what causes this. But your KNIME Log might help me finding out. Can you go to View -> Open KNIME Log and send me the part related to the error? There should be a section that is indented and all lines start with “at”. Can you send me this section including the previous two lines? It should mention “NullPointerException” somewhere in there.
Kind regards,
Alexander

Hi Alexander,

Thank you very much for reply.
I tried to get KNIME log.

2020-05-05 16:26:32,719 : ERROR : KNIME-Worker-0-List Folders 0:1011 : : Node : List Folders : 0:1011 : Execute failed: ("NullPointerException"): null java.lang.NullPointerException at com.vernalis.nodes.io.listdirs2.ListDirs2NodeModel.execute(ListDirs2NodeModel.java:136) at org.knime.core.node.NodeModel.execute(NodeModel.java:737) at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571) at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236) at org.knime.core.node.Node.execute(Node.java:1016) at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557) at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95) at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218) at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124) at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334) at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123) at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

Best regards,
Taku

1 Like

Hi,
I just noticed that this node comes from the Vernalis plugin and I cannot look at the code of that. But I have a workaround you can try: Instead of the List Folders node, use List Files and select the folder knime://knime.mountpoint/.metadata and unter Filter select “Wildcard Pattern” and as pattern “version.ini”. The result is a table with a single file, so you can remove the Row Filter in your workflow. Then, in the String Manipulation node, use this expression:

regexReplace($${SURL}$$, "^(.+)\\.metadata/version\\.ini$", "$1Users/knimeadmin/ASX/input/eve/yyyymmdd")

Kind regards,
Alexander

2 Likes

Hi Alexander,

Thank you very much indeed for quick reply.
I tried to use ListFiles.
But I got “romete directories are not supported” message from List files.

Thank you,
Taku

Hi,
Can you try with the List Remote Files node?
Kind regards,
Alexander

Hi Alexander,

Thank you very much indeed for many advise.
I tried List Remote Files node.
It works good!
But on the other hand, I got error from download…
The error message is "ERROR Download 0:1016 Execute failed: Illegal char <:> at index 27: C:/Program Files/KNIME/http:
"



2020-05-05 18:37:22,372 : ERROR : KNIME-Worker-16-Download 0:1016 : : Node : Download : 0:1016 : Execute failed: Illegal char <:> at index 27: C:/Program Files/KNIME/http:
java.nio.file.InvalidPathException: Illegal char <:> at index 27: C:/Program Files/KNIME/http:
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsUriSupport.fromUri(WindowsUriSupport.java:165)
at sun.nio.fs.WindowsFileSystemProvider.getPath(WindowsFileSystemProvider.java:92)
at java.nio.file.Paths.get(Paths.java:138)
at org.knime.base.filehandling.remote.files.FileRemoteFile.mkDir(FileRemoteFile.java:278)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:451)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.remote.files.RemoteFile.mkDirs(RemoteFile.java:444)
at org.knime.base.filehandling.download2.DownloadNodeModel.execute(DownloadNodeModel.java:130)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
at org.knime.core.node.Node.execute(Node.java:1016)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

Thank you,
Taku

Hi,
I think we have to go a different route here because we cannot be sure that the paths are resolved properly. Can you create a temporary folder with the Create Temp Dir node, download the file into it (you have to disable “Create temp directory in workflow folder”) and then use the Upload node in conjunction with the KNIME Server Connection node to upload the file via REST API to the correct directory on the server. This is more robust because it will work for the RMI executor and distributed executors.
Kind regards,
Alexander

Hi Alexander,

Thank you very much many advise.
I tried “Create temp directory in workflow folder”.
It was successes.
But again, download error happened.
The message was ”ERROR Download 0:1020 Execute failed: Illegal char <:> at index 28: C:/Program Files/KNIME/knime:”

So, I’ll try to use “List remote file” and read files on S3 directory direct instead of download files to knime server.
The “List remote file” is alternative method and it was your valuable information.
I really appreciate your many advise.

Best regards,
Taku


Hello,
I am glad it worked! If you still have to download something: you have to disable the “Create temp directory in workflow folder” option because the Download node does not support the knime:// protocol yet. Then you can use the temp directory as target folder.
Kind regards,
Alexander

Alexander,

Thank you very much!
I’m looking forward knime download will support “knime:// protocol”.

Thank you,
Taku

1 Like

Hi Taku,

I can’t reproduce the List Folders behaviour, but I can see where and how it happens from your stack trace. It looks like the knime url protocol resolver is unable to resolve your `knime://kniI think @wiswedel might have written it. In the meantime, I will modify our code to show a sensible error if this situation arises in our next update.

Thanks to @ipazin for bringing this to our attention.

Steve

1 Like

Hi Steave,

Thank you very much for explain the issue reason.
I tried alternative method which Alexander taught me and it succeeded.

I’m looking forward future updates.
And I really exciting about KNIME many functions every day.
knimeGood
Best Regards,
Taku

3 Likes

The list files node should no longer bomb out in these circumstances, and should rather fail gracefully with an error message - see Update to v1.27.0 - New Nodes and Bug Fixes for details

Steve

This bug fix is now available in the stable builds (3.7, 4.0, 4.1 and 4.2) - Update to v1.27.0 - New Nodes and Bug Fixes

Steve

1 Like

Steve,

Thank you very much.
Let’me try.

Best regards,
Taku

2 Likes