As KNIME Server users at my company, we have just upgraded to Server 4.16 and therefore KAP 4.7.
So far it went pretty smoothly but we noticed a strange behaviour after upgrading the deprecated Transfer Files to the newer version of the node. We are using this node mainly to move a processed file from its input folder on our Azure Blob Storage, to its outbox folder. The configuration is pretty standard and is the same as before as much as possible.
However, the node now leaves behind, when deleting the source file, a .directory-marker file, that can be manually deleted, but is otherwise hidden.
The main issue is that this file prevents the folder from being deleted once empty, whereas many of our workflows rely on a folder existing or not to proceed with the ETL process. That can be fixed by adding breakpoints, it’s annoying but doable. More importantly, it pollutes the Blob Storage by leaving folders that are empty except for this file, and we need to be able to quickly have a look at the folder structure to see if there is a file we need to process or not.
Do you guys have any idea on how to prevent the Transfer Files node from leaving behind that annoying marker ? We can’t rollback to the previous version of the node either as it’s deprecated and no longer available in the node repository.
Hi @Vonwen ,
Welcome to the KNIME community!
Would it be possible for you to share the workflow or a dummy workflow so that we could have a look and then get back to you?
I’m not sure I can do that without sharing access to our Azure Blob Storage. I just tested locally by transfering files on my own computer and that didn’t leave that marker, but that is a Windows system that allows empty folders to exist, which Azure does not, and I’m pretty sure that’s the root cause. The main question is why the previous node didn"t do this and how can we replicate that former behaviour.
Small update on my side, I did try, on another workflow, to leave the Transfer Files node as they were (deprecated then), and the same exact thing happened.
So my conclusion would be here that it’s not the new Transfer Files nodes that changed that behaviour, but the new KNIME version.
As a reminder, we upgraded from 4.3.3 to 4.7.3 when changing the KNIME Server.
Hello @Vonwen ,
I guess you created the source folder in Azure Blob Storage also using KNIME. Unfortunately Azure Blob Storage has no notions of folders which is why KNIME adds this mysterious file when creating the folder. Without the file there would be no folder in Azure Blob Storage. When you transfer a single file but not the whole folder the file remains and thus the folder. This is the same behavior as on a local driver where the folder also remains once all files within are transferred. So I would suggest you delete the folder (including the marker file) once you have processed all files in the folder using the Delete Files/Folders node or transfer the whole folder instead of each file separately.
Bye
Tobias
Thanks for replying. It’s still weird that it didn’t happen with 4.3 with the exact same workflows, but we’ll work around this and add a cleanup workflow to delete empty folders.
