Copy/Move Files Node

rfeigel · December 12, 2020, 1:40am

Why was this node deprecated? Its very useful and worked well for my purposes. I could pass a file location to it through a standard input port. I assume it was replaced by the Transfer Files node, although there’s no clear documentation that I can find. I can’t figure out how to input to the Transfer Files node. A flow variable doesn’t seem to work. Adding an input port produces a blue input square which won’t accept data from a standard output port in the previous node. I’m lost. Can someone help please.

izaychik63 · December 12, 2020, 1:57am

This could be an alternative

rfeigel · December 12, 2020, 3:40am

Thanks for quick reply, but as I said in my original post, the Transfer Files node is the one I’m having problems with. I can’t figure out how to configure it for a an input from another node.

Mark_Ortmann · December 12, 2020, 9:17am

@rfeigel,

you are totally right. The transfer files node currently is lacking of this functionality. If you could explain us exactly where the file names come from I can send you a workflow on how to do this. What you will need is a flow variable of type path or a cell of this type and use a loop. There nodes are allowing you to create them called String to Path (Variable)

Best
Mark

rfeigel · December 14, 2020, 5:02pm

Current workflow attached. If I change the Copy/Move Files node to Transfer Files node, I’d like to be able to input to the node so I could use a loop of file names in the future. Thanks.File Transfer Test 1.knwf (60.4 KB)

rfeigel · December 19, 2020, 5:41pm

Mark:
Have you had time to look at my problem?

Mark_Ortmann · December 19, 2020, 8:06pm

Sry somehow lost track. Gonna look into it on Monday.

Enjoy your weekend,
Mark

Mark_Ortmann · December 21, 2020, 8:53am

@rfeigel your workflow says you want to move the zipped files from the server/remote location to your local file system. Is this a KNIME server? Do you need the zipped files on your local FS? Do you want to delete the archives on the server?

Here you can find some documentation on the new framework. https://docs.knime.com/2020-12/analytics_platform_file_handling_guide/index.html

Mark

mlauber71 · December 21, 2020, 9:25am

This File Handling guide is a great addition to a quest in understanding how to work with KNIME. Maybe not always obvious from the start but once you really start working with KNIME it becomes essential to know where everything is stored and how to access it - I often see people puzzled where their stuff ends up; better to address this up front even if it is somewhat technical.

mlauber71 · December 21, 2020, 11:04am

@Mark_Ortmann @rfeigel on another note:

The logic and handling of the new path variables in accordance with the KNIME protocol is not always straightforward. I have put together some (new) examples how to handle (local) files with them - not everything seems to work and some things are not consistent (eg the old Excel Reader access a URI as path variable while the new one does not).

You have to toy around with these things, and it would be further complicated if you add ZIP / UNZIP functions to the fold (I assume). I would take a look path them also.

rfeigel · December 21, 2020, 3:57pm

The “Server” is not a Knime server. It could be a corporate server or simply another folder on my PC. My question is simple. How do I replace the current Copy/Move Files node with the new Transfer Files node in my current workflow?

Mark_Ortmann · December 22, 2020, 8:43am

@rfeigel,

here is the workflow File Transfer Test 1.knwf (18.1 KB).

I was asking those question in order to create a proper solution, but anyway here is one that works.

Currently transfer files does not support table inputs, we are working on that.
I removed the file compression. I assume you used it to save time when downloading, but actually in order to compress those files you have to download them. The compression is not being carried out on the remote machine.

Happy holidays,
Mark

zedleb · January 4, 2021, 10:41am

Hi all - Happy New Year!

I’ve been playing with using the @Mark_Ortmann (thank you!) workflow to create a way of backing up my knime-workspace.

I realised that I obviously needed to take out the actual workflow itself () but I also need to remove the .metadata folder.

It appears to work OK but will removing the metadata folder cause me any problems when using the backed up folder to re-instate workflows I’ve messed up?

Cheers
Zx

zedleb · January 6, 2021, 4:46pm

Update: the workflow fails on certain files. I remove the folder (via the filter) containing the file and the loop continues then fails on another file.

First time its a String to Date_Time node, second time its a Joiner

Here’s the console:

Here’s the log, if anyone has any suggestions very much appreciated. It’d be great to get this working as until we have Knime server (this month I hope) all my work is on my hard drive and I use this workflow to transfer to a OneDrive folder for a backup.

2021-01-06 17:34:13,817 : ERROR : KNIME-Worker-216-Transfer Files 0:1 :  : Node : Transfer Files : 0:1 : Execute failed: C:\Users\leXXXXXXXXX\00 Knime\MAGASIN workflows\Magasin_Orig_Dev\Joiner (#26)
java.nio.file.FileAlreadyExistsException: C:\Users\XXXXX\DOCUMENTS\000 Workspaces\00 Knime\MAGASIN workflows\Magasin_Orig_Dev\Joiner (#26)
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:81)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
	at sun.nio.fs.WindowsFileSystemProvider.createDirectory(WindowsFileSystemProvider.java:504)
	at org.knime.filehandling.core.connections.local.LocalFileSystemProvider.createDirectory(LocalFileSystemProvider.java:233)
	at java.nio.file.Files.createDirectory(Files.java:674)
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
	at java.nio.file.Files.createDirectories(Files.java:727)
	at org.knime.filehandling.core.connections.FSFiles.createDirectories(FSFiles.java:165)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier.createDirectories(PathCopier.java:218)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier.copyDirectories(PathCopier.java:253)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier.copyPath(PathCopier.java:237)
	at org.knime.filehandling.utility.nodes.transfer.TransferFilesNodeModel.copy(TransferFilesNodeModel.java:177)
	at org.knime.filehandling.utility.nodes.transfer.TransferFilesNodeModel.execute(TransferFilesNodeModel.java:139)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:576)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1245)
	at org.knime.core.node.Node.execute(Node.java:1025)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:558)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

mlauber71 · January 7, 2021, 5:20pm

@zedleb I created and example where the KNIME workflow (or workflow group) is being zipped into a file with a current timestamp. And can later be restored to a (separate) folder.

This restore can either be automated also or you could do it manually if you need it.

Please note you will have to configure the Path variables and workflow names to your needs. I use local paths that will be created dynamically.

kienerj · January 8, 2021, 7:26am

While this is cool it would be great if KNIME offered such an option by default. At least the possibility to reset other workflows inside a knime workflow would help a lot. Backup with data can become a space/time problem specially if backup up over network

mlauber71 · January 8, 2021, 9:19am

Concerning Reset there is a node that would allow a reset at the start of a workflow. I agree a ‘remote’ reset would be good. The KNIME server offers the possibility to reset workflows before they are scheduled to run.

zedleb · January 8, 2021, 3:23pm

I tried combining @mlauber71 and @Mark_Ortmann ideas

Putting the end of the loop after the compression node.

It works really well!
Haven’t tried restoring yet though.

zedleb · January 8, 2021, 4:31pm

Restoring not working

No indication what the error is.
I made sure to delete the workflow from my workspace before importing.

mlauber71 · January 8, 2021, 6:35pm

Are you trying to import from a .zip file or from a folder? Maybe you could put your idea of combining the approaches in a sample workflow (or workflow group) so we might have a look.