In 5.3 and 5.4 sometimes a download from within the KNIME Platform does fail with strange error messages. I assume this could have to do with workflows being created in macOS and now to be downloaded in Windows.
Download. I assume this is one of the hard to track down problems. It happens frequently on my machines but is not always reliably to be repeated so it most likely will initially get dismissed.
Import from temporary location seems to be forbidden:
2024-11-18 16:11:43,945 : ERROR : ModalContext : : HubSpaceAsyncTransfer : : : Could not import downloaded item '/Users/mlauber71/Public/kn_example_r_excel_find_united_fields/m_001_import_r_excel_unite_fields': C:\Users\x123456\AppData\Local\Temp\KNIMEHubWorkflowImport17736\m_001_import_r_excel_unite_fields -> C:\Users\x123456\knime-workspace-54\hub\kn_example_r_excel_find_united_fields\m_001_import_r_excel_unite_fields
java.nio.file.AccessDeniedException: C:\Users\x123456\AppData\Local\Temp\KNIMEHubWorkflowImport17736\m_001_import_r_excel_unite_fields -> C:\Users\x123456\knime-workspace-54\hub\kn_example_r_excel_find_united_fields\m_001_import_r_excel_unite_fields
at java.base/sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at java.base/sun.nio.fs.WindowsFileCopy.move(Unknown Source)
at java.base/sun.nio.fs.WindowsFileSystemProvider.move(Unknown Source)
at java.base/java.nio.file.Files.move(Unknown Source)
at com.knime.explorer.server.internal.HubSpaceAsyncTransfer.importWorkflowLike(HubSpaceAsyncTransfer.java:249)
at com.knime.explorer.server.internal.HubSpaceAsyncTransfer.importItem(HubSpaceAsyncTransfer.java:225)
at com.knime.explorer.server.internal.HubSpaceAsyncTransfer.lambda$2(HubSpaceAsyncTransfer.java:183)
at com.knime.explorer.server.internal.HubSpaceAsyncTransfer.lambda$26(HubSpaceAsyncTransfer.java:684)
at com.knime.explorer.server.internal.HubSpaceAsyncTransfer.lambda$25(HubSpaceAsyncTransfer.java:655)
at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:124)
For some reason your Windows Filesystem does not allow to either write to the destination or remove the file from its temp source.
It’s a long shot…, but do you have the workspace or some subfolder open in Windows Explorer? Was suggested on StackOverflow to be a cause for this specific AccessDeniedException.
Strange, the problem is coming from the filesystem. Can you manually create a folder/file at the destination or try to move it from the tempfolder to somewhere else? Or check what the folder permissions look like?
Is there another file/folder at the destination that has the same name, but with different casing? Windows is a bit strange when it comes to case-sensitivity (AFAIK it’s case-insensitive, so forbids two things with the same normalized name, but it remembers the case of each character, or something like this).
@hotzm it is nothing of this sort. The permission work just fine and I can create folders there and if I manually download the file as KNWF and import it it does work. This is one of the funny things to debug.
I think you should come across it in the team if people are using larger workflows back and forth maybe some that have been created with macOS. You can just use my repository on the hub.
About the name: is it possible the KNIME code mixes something up or has something already written there before copying it to the destination?
@hotzm I now also get this when trying to upload something to the KNIME Hub. Will see if I can provide the log.
What is possible is that I have another KNIME version 5.3 open with a different knime-workspace. I wonder if this has something to do with it. I will check if the temp folders are different. Although I would assume such function should work anyway.
2024-11-19 14:10:38,291 : DEBUG : ForkJoinPool.commonPool-worker-12 : : APRestFacade : : : Request 'https://api.hub.knime.com/knime/rest/v4/repository/*ver4ryUGKEXKUkqY' took 1,048 seconds
2024-11-19 14:10:41,668 : DEBUG : Worker-8: Refresh remote content : : RepositoryFetcherImpl : : : Starting fetcher thread
2024-11-19 14:10:41,668 : DEBUG : Worker-8: Refresh remote content : : JobFetcherImpl : : : Starting fetcher thread
2024-11-19 14:10:41,668 : DEBUG : Worker-8: Refresh remote content : : ScheduleFetcherImpl : : : Starting fetcher thread
2024-11-19 14:10:41,673 : DEBUG : Fetcher (Fetching schedules list for 'mlauber71' from Hub) : : ScheduleFetcherImpl : : : Starting schedule fetcher for Hub at: https://api.hub.knime.com
2024-11-19 14:10:41,673 : DEBUG : Fetcher (Fetching repository list for 'mlauber71' from Hub) : : RepositoryFetcherImpl : : : Starting repository fetcher for Hub at: https://api.hub.knime.com
2024-11-19 14:10:41,673 : DEBUG : Fetcher (Fetching schedules list for 'mlauber71' from Hub) : : ScheduleFetcherImpl : : : Stopping schedule fetcher for https://api.hub.knime.com
2024-11-19 14:10:41,673 : DEBUG : Fetcher (Fetching job list for 'mlauber71' from Hub) : : JobFetcherImpl : : : Starting job fetcher for Hub at: https://api.hub.knime.com
2024-11-19 14:10:41,673 : DEBUG : Fetcher (Fetching job list for 'mlauber71' from Hub) : : JobFetcherImpl : : : Stopping job fetcher for https://api.hub.knime.com
2024-11-19 14:10:43,878 : DEBUG : Fetcher (Fetching repository list for 'mlauber71' from Hub) : : RepositoryFetcherImpl : : : Stopping repository fetcher for https://api.hub.knime.com
2024-11-19 14:10:44,133 : DEBUG : main : : APRestFacade : : : Request 'https://api.hub.knime.com/knime/rest/v4/repository/*ver4ryUGKEXKUkqY' took 0,240 seconds
2024-11-19 14:10:44,385 : DEBUG : main : : APRestFacade : : : Request 'https://api.hub.knime.com/knime/rest/v4/repository/*ver4ryUGKEXKUkqY?details=full' took 0,249 seconds
2024-11-19 14:10:44,716 : DEBUG : main : : APRestFacade : : : Request 'https://api.hub.knime.com/knime/rest/v4/repository/Users/mlauber71/Examples' took 0,329 seconds
2024-11-19 14:10:45,162 : DEBUG : ForkJoinPool.commonPool-worker-12 : : CatalogServiceClient : : : Request 'https://api.hub.knime.com/repository/*ver4ryUGKEXKUkqY?details=none' took 0,399s
2024-11-19 14:10:45,692 : DEBUG : ForkJoinPool.commonPool-worker-12 : : CatalogServiceClient : : : Request 'https://api.hub.knime.com/repository/Users/mlauber71/Examples?deep=true' took 0,528s
2024-11-19 14:10:47,101 : DEBUG : ModalContext : : CatalogServiceClient : : : Initiating upload of 1 items
2024-11-19 14:10:47,417 : DEBUG : ModalContext : : CatalogServiceClient : : : Request 'https://api.hub.knime.com/repository/*ver4ryUGKEXKUkqY/manifest' took 0,314s
2024-11-19 14:10:47,491 : DEBUG : KNIME-Hub-Transfer-1 : : HubUploader : : : Upload of 'DB SQL - H2 School of duplicates - and how to deal with them' has failed, cancelling parts and notifying Catalog Service
2024-11-19 14:10:47,728 : DEBUG : KNIME-Hub-Transfer-1 : : CatalogServiceClient : : : Request 'https://api.hub.knime.com/uploads/63939dd3-63e4-44b7-a870-6c000f87260a~3fkQX9EFOtF7gAPZ1Y0O5PsWIVjqOtuH.SUADNjew7aa81z5tLVUCYYAmL4XkWK0PrBYvQEwQN3yDpiqUqR8Qijt89N7c1iGPJCABsxig5Acg0Gd97Qg14B8_2vH9V2q' took 0,237s
2024-11-19 14:10:47,733 : DEBUG : main : : DesktopAPI : : : Desktop API function successfully called: copyBetweenSpaces
After resetting it did work. It is also possible that the ‘reset before upload’ setting did something.
I assume the workflows will be somewhere in the temp-folder so be prepared and uploaded. It is possible that a virus scanner is hindering them. It would be good to create an option to wait and check and maybe retry the whole thing.
Do you find more text in the log about “Unable to add file…” which is truncated in the screenshot? Or if you click on “show more”?
If the temp folder and workspace are different, I don’t think two instances should collide with each other.
Thanks for the hint about “reset before upload” and the virus scanner. How exactly the “reset before upload” is done, I’m not sure and would need to find out.
I think waiting and retrying is difficult to get right, because from the API we just seem to see “AccessDeniedException”, which is not clear if it’s temporary or not. I’ll discuss that in the team to see whether they have any good ideas to improve it.
@hotzm the full screenshot does not reveal much. Also the log is not very helpful. I could reproduce the error when selecting the ‘reset before upload’.
Yes, unfortunately the real exception is swallowed I created an internal ticket to log it (AP-23607). I was able to upload one of your python timeseries workflows as a test with the reset option enabled (on macOS), so the problem seems a bit harder to pinpoint.
Since we create parts (KNIMEWorkflowUpload....knwf.part files) in the temp folder, it is plausible that the opening (for upload) or deletion of these temp files gets forbidden, because some Virus scanner still has the new “suspicious” file open. Hence, the upload job is reported as failed. If it forbids the opening for upload, the file should not be on Hub, if it is the deletion that has an exception, the file should be visible on Hub after upload. Why it only occurs when checking the reset flag, I don’t know yet.
We were able to diagnose the upload issue: the running H2 process still locks the database file in the data directory. If you reset the workflow, the database process gets shut down and releases the lock on the file, so it can get uploaded. The checkbox in the dialog has nothing to do with it.
The full error message is:
java.io.IOException: Unable to add file "C:\Users\XXX\eclipse-workspaces\runtime-KNIME\DB SQL - H2 School of duplicates - and how to deal with them\data\database.h2.mv.db" to archive: The process cannot access the file because another process has locked a portion of the file
It will be logged to debug output in a future release to make it easier to debug :).
So the scenario is that you overwrite an existing workflow in your workspace with the download, but you still have an R process running that uses a file in the workflow you’re trying to overwrite? The workflow is closed, otherwise it would not allow to overwrite, but the R process still locks a file in it.
I created an internal ticket to log it (AP-23607). I was able to upload one of your python timeseries workflows as a test with the reset option enabled (on macOS), so the problem seems a bit harder to pinpoint.