Simple Workflow to Append New Weekly Data To Prior Weeks

I’d like to read in seven days of CSVs from an S3 bucket and append the raw data read-in to prior weeks. I’ve seen 3-4 prior conversations on this topic but none of them have a good example workflow to learn from and apply to this problem.

Additional Issue…what do you when a variable Loop End requires the ‘allow variable column types’ due to a column being a string in one file and double in another? This variable column type is preventing the option for using a CSV Writer to save/append the new data to an exisiting CSV.

Hoping a screenshot will further explain the context of this problem:

I would recommend to store data not in CSV but SqLite DB. Then you do not need to read Toast POS to concatenate with new orders. DB Merge will do it for you.

1 Like

You could import all the data as string from CSV and later if you use the data convert the appropriate columns to numbers if necessary. In general CSV is widely used but has its downsides.

@izaychik63 I like you suggestion to use SqLite DB but can’t get into something unfamiliar right now. Looks like it’s worth exploring though, do you have a good suggested starting link resource you can share?

@mlauber71 to the rescue again…I followed your prior post on the readr suggestion you made to someone else and utilized their recommendation to ignore column headers so all is imported as string to start with.

Thank you for the assistance!

1 Like

See example below.
Update-existing-values.knwf (22.6 KB)

1 Like

SQLite is definitely worth exploring. I can offer you several examples and discussions about it in the context of KNIME:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.