Monitor table for new data

[kamtaot]

Hello Knimers,

I have a table with a data for the last 365 days. Every day, this table will be updated with previous day data and 366 day data will be deleted. I want to monitor this table if the previous day data is uploaded to the table and then initiate further workflow to process the data.

I searched the forum and googled as well but I couldn’t find any specific reference for it. Maybe I am not searching properly but any pointers in this regard will be great help.

[mlauber71]

@kamtaot you could try and use a switch where your workflow continued when the latest entry in your database is one day behind the current date.

You can read about the basics here.

KNIME — Cases, Switches and Catching Errors

[kamtaot]

Hello @mlauber71,

That is indeed a good suggestion. I will try to implement it and see how it proceed. Actually, I was expecting a node for this functionality. It really doesn’t matter if I have to implement a small workflow to address this.

Thanks a million for the advice.

Best

[HansS]

Hi @kamtaot

Maybe the Row Splitter Rule Based is an option.

If every day is a record, than with the $$ROWCOUNT$$ you can identify if day 366 has been added and proceed your workflow from the first (top) output table,

gr. Hans

[kamtaot]

Hi @HansS,

I am not clear with the suggestion. Could you please take little pain and explain in detail. I have few hundred to thousand records coming in every day in this table. I want to initiate the workflow if the table is populated. If not, just leave it as is.

The problem I am facing is the timing. There is no specific time at which this table gets populated. Every day, the timing is different. Roughly between 1 AM to 8 or 9 AM. Few times, it even went to 11 AM as well. However, I want to process the data as soon as it is available in the table and not wait until 11 AM. Similarly, on some days, if the other process fails, it uploads the previous two days data on the next day. In such cases, I do not want to initiate the process at all on the failed day.

Because of these combinations, I want to monitor the table for uploaded data. Hope, I could able to explain well this time.

The suggestion given by @mlauber71 looks reasonable to me as I can schedule the workflow to check every 10 min and if the data is uploaded, I process it else ignore. In case you have better approach, I am happy to explore it as well.

[mlauber71]

@kamtaot here are two examples of such checkes running with loops. You could also try and employ other methods.

Planning will be key.

Start a Workflow, e. g. a database query, after a certain time

@USCHUKN1ME I tried to build a system you might be able to expand and use. The idea is to have a workflow that would iterate a number of times (v_number_of_intervals) with a set interval (of seconds) between (v_seconds_interval) them. The workflow looks for a trigger_file.csv if this file has changed from the last time the loop has run a component would be executed. If the file has not changed nothing would happen. Admittedly not the most elegant way but you could build a small automatic execu…

Question about Wait Node

@cuimaple without a KNIME Business Hub you will face some kind of complexity. That being said I have created a sample workflow that would check the last date a file has been changed on a drive (here simulated using a local folder) - and if there has been a change compared to the last recored date a sub-workflow will be called and the new change date will be recorded in the CSV file. KNIME CSV writer has the option to append lines without writing the header. So it is suitable for recording such d…