I am new to KNIME and was wondering if there is a way to have a workflow to create a backfill dataset as such:
BigQuery Connection – > Run SQL Script (This has a DECLARE statement for date) – > Writes Results into a Table – > Changes the INTERVAL component of the date (For example from 1 week, to 2 weeks – > Re-run the SQL Script – > Append again – > Loop until a condition / date end.
I am basically trying to create a historical backfill over the span of 2 years, week by week. The result of the script is a weeks worth backfill, so I am trying to run this script 100 times, each time to change the script interval by 1 week.
The script works fine on BigQuery, I just don’t want to tediously run it 100x, and wanted a way to learn how to create historical backfills more easily for this, and for the future.
I believe you are on the right place, all the functionalities that you were mentioned are available in KNIME.
For taking your first steps in KNIME world our Spaces page would be a good start. I would recommend the Education part where you can find a lot of material to learn using KNIME.
On the Hub you can search among more than 11.000 workflows to get inspiration for your further projects.
If you decide to start playing around KNIME you can download our opensource tool the KNIME Analytics Platform from here.
I hope you will find what you need, but if you have further questions please feel free to ask us, all the Community here in Forum!