Transfer files from KNIME to Azure Blob Storage

trj · September 30, 2024, 7:15am

Hi,

I need to transfer files from a local Storage - can be from my laptop, KNIME Server and KNIME Business Hub. The list of files are provided as relative paths to the Data Area of the KNIME Workflow (used as temp folder).

I then use a Transfer Files (Table) node to move/copy files from KNIME to an Azure Blob Storage. The transfer files fails. When executing again, 1st row works, 2nd fails. Executing again, 1st-2nd rows work, 3rd fails. And so on.
At each failing step, the file has been copied correctly on the Blob storage.

It looks like the file is being uploaded, but the return message from Blob is not correct.

Any advice on this one ?

Thanks
Jerome

Error is (details of the rest of the error below):

2024-09-30 16:33:27,343 : ERROR : KNIME-Worker-241-Transfer Files (Table) 3:1863 :  : Node : Transfer Files (Table) : 3:1863 : Can't transfer file:
RowID  |   ..   | Selected (J... |    Filename   
-------------------------------------------------
Row0   | ..     | true           | XXXXX...
                                  ^^^^^^^^^^^^^^^
The requested URI does not represent any resource on the server.

org.knime.core.node.KNIMEException$KNIMERuntimeException: Can't transfer file:
	at org.knime.core.node.KNIMEException.toUnchecked(KNIMEException.java:145)
	at org.knime.filehandling.utility.nodes.transfer.AbstractTransferFilesNodeModel.transfer(AbstractTransferFilesNodeModel.java:273)
	at org.knime.filehandling.utility.nodes.transfer.AbstractTransferFilesNodeModel.execute(AbstractTransferFilesNodeModel.java:241)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:588)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1297)
	at org.knime.core.node.Node.execute(Node.java:1059)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:98)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: org.knime.core.node.KNIMEException: Can't transfer file:
	at org.knime.core.node.KNIMEException.of(KNIMEException.java:136)
	at org.knime.filehandling.utility.nodes.transfer.table.TransferFilesTableNodeModel.wrapTransferException(TransferFilesTableNodeModel.java:159)
	... 15 more
Caused by: org.knime.ext.azure.AzureUtils$WrappedHttpResponseException: The requested URI does not represent any resource on the server.
	at org.knime.ext.azure.AzureUtils.toIOE(AzureUtils.java:147)
	at org.knime.ext.azure.AzureUtils.toIOE(AzureUtils.java:165)
	at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageFileSystemProvider.tryToDeleteBlob(AzureBlobStorageFileSystemProvider.java:439)
	at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageFileSystemProvider.removeDirectoryMarker(AzureBlobStorageFileSystemProvider.java:426)
	at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageSeekableByteChannel.copyToRemote(AzureBlobStorageSeekableByteChannel.java:95)
	at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageSeekableByteChannel.copyToRemote(AzureBlobStorageSeekableByteChannel.java:1)
	at org.knime.filehandling.core.connections.base.TempFileSeekableByteChannel.close(TempFileSeekableByteChannel.java:164)
	at org.knime.filehandling.core.connections.FSSeekableByteChannel.close(FSSeekableByteChannel.java:89)
	at org.knime.filehandling.core.connections.base.BaseFileSystemProvider$1.close(BaseFileSystemProvider.java:196)
	at java.base/java.nio.channels.Channels$1.close(Unknown Source)
	at org.knime.filehandling.core.connections.FSOutputStream.close(FSOutputStream.java:104)
	at org.knime.filehandling.core.connections.base.BaseFileSystemProvider$2.close(BaseFileSystemProvider.java:504)
	at java.base/java.nio.file.Files.copy(Unknown Source)
	at java.base/java.nio.file.CopyMoveHelper.copyToForeignTarget(Unknown Source)
	at java.base/java.nio.file.Files.copy(Unknown Source)
	at org.knime.filehandling.core.connections.FSFiles.copy(FSFiles.java:618)
	at org.knime.filehandling.utility.nodes.transfer.policy.TransferPolicy.overwriteIfNewer(TransferPolicy.java:172)
	at org.knime.filehandling.utility.nodes.transfer.policy.TransferPolicy.apply(TransferPolicy.java:95)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyFile(PathCopier2.java:313)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyPath(PathCopier2.java:260)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copy(PathCopier2.java:197)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyDelete(PathCopier2.java:167)
	at org.knime.filehandling.utility.nodes.transfer.PathCopier2.transfer(PathCopier2.java:140)
	at org.knime.filehandling.utility.nodes.transfer.AbstractTransferFilesNodeModel.transfer(AbstractTransferFilesNodeModel.java:267)
	... 14 more
Caused by: com.azure.storage.blob.models.BlobStorageException: Status code 400, "<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidUri</Code><Message>The requested URI does not represent any resource on the server.
RequestId:42d96011-501e-0001-0606-131daf000000
Time:2024-09-30T07:03:27.3350405Z</Message></Error>"
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)
	at com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)
	at com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:377)
	at com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:117)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
	at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.java:189)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:180)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:180)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172)
	at com.azure.core.http.okhttp.OkHttpAsyncHttpClient$OkHttpCallback.onResponse(OkHttpAsyncHttpClient.java:242)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
		at reactor.core.publisher.Mono.block(Mono.java:1707)
		at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:191)
		at com.azure.storage.blob.specialized.BlobClientBase.deleteWithResponse(BlobClientBase.java:1191)
		at com.azure.storage.blob.specialized.BlobClientBase.delete(BlobClientBase.java:1156)
		at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageFileSystemProvider.tryToDeleteBlob(AzureBlobStorageFileSystemProvider.java:435)
		at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageFileSystemProvider.removeDirectoryMarker(AzureBlobStorageFileSystemProvider.java:426)
		at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageSeekableByteChannel.copyToRemote(AzureBlobStorageSeekableByteChannel.java:95)
		at org.knime.ext.azure.blobstorage.filehandling.fs.AzureBlobStorageSeekableByteChannel.copyToRemote(AzureBlobStorageSeekableByteChannel.java:1)
		at org.knime.filehandling.core.connections.base.TempFileSeekableByteChannel.close(TempFileSeekableByteChannel.java:164)
		at org.knime.filehandling.core.connections.FSSeekableByteChannel.close(FSSeekableByteChannel.java:89)
		at org.knime.filehandling.core.connections.base.BaseFileSystemProvider$1.close(BaseFileSystemProvider.java:196)
		at java.base/java.nio.channels.Channels$1.close(Unknown Source)
		at org.knime.filehandling.core.connections.FSOutputStream.close(FSOutputStream.java:104)
		at org.knime.filehandling.core.connections.base.BaseFileSystemProvider$2.close(BaseFileSystemProvider.java:504)
		at java.base/java.nio.file.Files.copy(Unknown Source)
		at java.base/java.nio.file.CopyMoveHelper.copyToForeignTarget(Unknown Source)
		at java.base/java.nio.file.Files.copy(Unknown Source)
		at org.knime.filehandling.core.connections.FSFiles.copy(FSFiles.java:618)
		at org.knime.filehandling.utility.nodes.transfer.policy.TransferPolicy.overwriteIfNewer(TransferPolicy.java:172)
		at org.knime.filehandling.utility.nodes.transfer.policy.TransferPolicy.apply(TransferPolicy.java:95)
		at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyFile(PathCopier2.java:313)
		at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyPath(PathCopier2.java:260)
		at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copy(PathCopier2.java:197)
		at org.knime.filehandling.utility.nodes.transfer.PathCopier2.copyDelete(PathCopier2.java:167)
		at org.knime.filehandling.utility.nodes.transfer.PathCopier2.transfer(PathCopier2.java:140)
		at org.knime.filehandling.utility.nodes.transfer.AbstractTransferFilesNodeModel.transfer(AbstractTransferFilesNodeModel.java:267)
		at org.knime.filehandling.utility.nodes.transfer.AbstractTransferFilesNodeModel.execute(AbstractTransferFilesNodeModel.java:241)
		at org.knime.core.node.NodeModel.executeModel(NodeModel.java:588)
		at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1297)
		at org.knime.core.node.Node.execute(Node.java:1059)
		at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
		at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:98)
		at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
		at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
		at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
		at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
		at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
		at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

ipazin · September 30, 2024, 9:08am

Hello @trj,

as a workaround you can try putting it in a loop. Maybe it could work.

Unfortunately can’t help much with root issue. It might be due to Azure settings, might be due to Transfer Files node or most probably combination of both

Br,
Ivan

trj · October 1, 2024, 7:13am

Hi,

The thing is that using Python to upload files, it works.

Seems to be related to the node itself.
The node seems to handle a wrong return message. Because the files are being uploaded, even with errors.

Cheers
Jerome