Connection to Google Cloud Storage

KNIME Analytics Platform
1

Are there any ways to connect to Google Cloud Storage?

1 Like
2

Hi,

I think you can use REST Web Services nodes (under the “Tools & Services” category in Node Repository) in KNIME and the Google Cloud Storage APIs:
https://cloud.google.com/storage/docs/apis

Best,
Armin

2 Likes
3

Hey Armin,

thanks for your answer, but I’ve solved it already.

I decided to use External Tool node, it calls a batch (.bat) file with gsutil cp command and it works just fine.

2 Likes
4

Pedro,
Can you share how you did this? What does the raw workflow look like? I’m trying to build a workflow that uploads folders and files into GCP storage from a local folder that gets S3 downloads daily. Thanks!

Tony

1 Like
5

Hi there @t2jr0385,

with KNIME version 4.1 there is dedicated node for this connection - Google Cloud Storage Connection.

Br,
Ivan

2 Likes
6

Thanks @ipazin!

I could use some more direction, however.

What I’m trying to do…maybe this can be mocked quick?

  1. Have local csv files in folders labeled as ‘/yyyymmdd/…’
  2. Want to create same folder name in GCP storage bucket and upload the csv files from local folder to this new folder

Can someone help mock the workflow?

Thanks,
Tony

7

Hey @t2jr0385,

Using External Tool node can became a pain in the …!

I’m not sure which version of Knime you’re using, but if you are not using 4 I’d advice you to try to use a Python or Java node and issue a command to your OS to upload your file/folder. You must have Google Cloud SDK installed in your machine for it to work.

In Python it would be something like:

import os

os.system('gsutil -m cp -r gs:///

8

My answer got cut. It should be:

os.system(gsutil -m cp -r path_to_folder path_in_GCS)

2 Likes
9

Thanks @pedromarfel I’ll try this out.

10

Hi there @t2jr0385,

well it is something like this:

Upload

where you’'ll need to add proper connectors to Google and Cloud Storage.

You can check these workflows for inspiration:


Br,
Ivan

1 Like