Copy/Move node issue

Hi experts,

I have created a workflow that copy files form a Network drive to another. The workflow starts with a file list node. The workflow has to reconize which files are already copied an which files are new, a Little bit like a file observer. This Problem I’ve solved with tabel wirte node and table Reader node and it works fine Unfortunatelly all files have to be rename but that’s not the problem. when I start the workflow for testing with a cuple of files the workflow works fine. When I start the workflow for the whole network drive (round about 8.000 files initial) the copy/move node throws a JavaScript exeption: “filename or paht are incorrect”. When I run the workflow with 2.000 files it also runs fine. I tested the workflow on my local Workstation the same JavaScript exeption is thowen. When I run the workflow with a Wildcard Expression in the file list node the workflow runs fine.
Do you have any idea what the Problem could be? Network Connection isn’t it because the same exextion is thowen on local drive? Attached you can see my workflow.

DataLake4VAN_Sync Temp.knwf (68.7 KB)
thanks for any solution.
Regards, brotfahrer

Perhaps there is a problem with a specific path (at specific iteration after 2000th).
Can you post entire stack trace of error copying it from View->knime.log?
Thanks

Hello pigreco,

this is the corresponding Content of the knime.log:

2020-04-23 17:46:20,178 : ERROR : KNIME-Worker-22-Copy/Move Files 0:11 : : Node : Copy/Move Files : 0:11 : Execute failed: java.io.IOException: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch
java.lang.RuntimeException: java.io.IOException: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch
at org.knime.base.filehandling.copyfiles.CopyFilesNodeModel.doAction(CopyFilesNodeModel.java:323)
at org.knime.base.filehandling.copyfiles.CopyFilesNodeModel.execute(CopyFilesNodeModel.java:129)
at org.knime.core.node.NodeModel.execute(NodeModel.java:737)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
at org.knime.core.node.Node.execute(Node.java:1016)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.io.IOException: Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:1012)
at org.knime.base.filehandling.copyfiles.CopyFilesNodeModel.doAction(CopyFilesNodeModel.java:292)
… 15 more

Ok.
We have to debug the problem to isolate the row that give you the problem.
You can attach a row filter node after list files node and move “row window” to identify the wrong row.
In first case, filter by row number including only rows after 2000th and so on…

1 Like

Hello pigreco,

I found the Problem. In some cases the filenames in the source folders don’t follow the name convention. In this case one of my cell splitter nodes create missing values. I puted a missing value node after the cell splitter node. Now the workflow works fine.
By the way: the copy/move node is not very fast but I think this is known issue.

greetings, Brotfahrer

2 Likes

Hello again,

after a few week of stable workflow I get again a Java exeption like at the end of April. How can I get the information which filename or path is wrong? It’s very boring the click through a 100000 row table :sleeping: :sleepy:

Greetings, Brotfahrer

You could initiate a CSV file at the beginning of the loop and then add a line with the current filename at the beginning of each step.

Another idea would be to track which data you already have processed and only work with those that you have’t.

Maybe this entry could save as an inspriration:

Hello @mlauber71,

the trick is that I don’t use a loop. I feed the copy/move node with a URI table.

My original task is to do an asynchronous mirroring of my source folder. I only want to copy the new files which are stored in my source. The file in my target folder will be moved to an analytics server.
The error is located in the metanode “Rename files”. Due to the analytics server I have to rename al the files in a new nomenclature. If somthing is wrong with the source filename the metanode don’t crash but the copy/move node.
Here is a scrennshot of my metanode. I alwas have a missing value nodes an at the end a duplicate row filter node for “error handling”.

Is ist possible that I can update a CSV file via a flow variable at the ende of my workflow like this?
grafik

Greetings,
Brotfahrer

Hi there @Brotfahrer,

You can try to add logic (regex is one way to go) into your Rename Metanode to find rows (files) that do not follow the name convention and then either filter them out either make them follow the convention. This way you can identify files not following convention but also if properly designed should Copy/Move Files node should’t fail :wink:

Br,
Ivan

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.