In my table i have a first column with date and times and then colums with various measurements taken at this time. I’d like to be able to add some rows for time istances that are not already into my table.
Example:
i have:
I’d like to insert 2019-05-01T00:03:53
2019-05-01T00:04:53
Right now the problem is solved by creating a date table with the times to add and using joiner->Column merger->sorter to get the final table. I’d like to know if there’s a faster solution since this represents kind of a bottleneck. Was thinking about using a Java snippet but i can’t find a way to append rows to table with it.
Why using Joiner node and not Concatenate if you are adding time instances that are not in your table? Currently I don’t see how you can avoid sorting in your use case but definitely some worth thinking about… How many rows you have?
Some of the time instances might be in both tables so i need to avoid to have duplicated rows with same time.
The file i’m working right now is about 36k rows. The table with time instances to add if not present is about 7k rows.
My idea was to cicle through rows and to “compare” time instances to insert rows at their right place. That should get the job done in one cicle throught the table.
Maybe i should try to implement this in a custom node.
@ipazin is right, the Concatenate Node should be the final step before sorting.
You can use the Reference row filter in order to filter the potentially new times, which are already in your dataset. see workflow attached. Z_020_time-insert.knwf (10.7 KB)
data size doesn’t seem to big for Sorter node to be a bottleneck so yo.
Apart from @tommy’s idea with Reference Row Filter node another approach would be to first concatenate, then use Duplicate Row Filter node to remove possible duplicates all followed by a Sorter.
