Hello everyone,
Our team is working on a manual read and write into sharepoint (excel) workflows. However, the data being generated is becoming in the order of gigabytes (note that we are not using any big data connectors). So, we are considering to “write” it to an azure storage space.
Wanted to know if KNIME upload rate is faster in sharepoint or blob/adls.
Also any information regarding how to increase the speed of KNIME.
Edit1 : It is also helpful if someone can share how to use blob/adls node , authentication, security issues etc as they are new. A simple workflow using the nodes is much appreciated.
Upload rate is time taken to write the excel files into the azure storage using the excel writer
Fun questions. I’m not sure if KNIME would contribute to the upload rates for either one as we usually just use the Microsoft APIs for both applications. I believe blob has a higher throughput upper limit and would think that would be faster.
Thanks, @wkhan. That makes sense . I just wanted the workflow to be more optimized. I guess I wanted to know how excel writer works with azure nodes.
Now I have one more doubt. as you know azure storage account is protect by network rules and only some IP addresses, Virtual Nets will be allowed to access the storage account.
I am using a corporate VPN to connect to internet
So my doubt is which IP address does the ADLS gen 2 / Blob connector Node use to request the storage access ?
Like will it use my Public IP or the VPN IP address to hit the request?
Asking because it says Authentication failed, or the account doesn’t have enough permissions to list containers and I am unable to see the containers, files inside the storage account
but I am sure I have the access.
Sorry if this question doesn’t make sense as I am a Non-CS guy
No problem at all! Networking can get messy, but in general If you’re connected to VPN all of your traffic should route through the VPN. KNIME will use the VPN. I would double check the access rights on the Azure Service!
Another follow up question .
What happens if I use two different VPNs at the same time ? I think I got 2 VPNs. One is for my laptop and the Other is for client and both are running at the same time
I’m definitely no networking expert, but in general two VPNs do not work together. You can be connected to two different VPNs but your request will only go through one of the VPNs.
Your VPN client is probably smart enough to know which VPN to route traffic to use based on where you are trying to connect to. You can definitely try to disconnect from the one you don’t need and try again!
. I just wanted the workflow to be more optimized. I guess I wanted to know how excel writer works with azure nodes.
