How to handle the 15 minutes disconnect from twitter with knime twitter workflow

Hey friends, 

 

I have already create my workflow to get twitter tweets, but the problem is, that twitter quit the connection, and let me wait for 15minutes, is there a node to dial again after the 15minutes ?

We've just released a wait-for-time node in the Vernalis plugin (currently only in the nightly build, but should be in the trusted update sites by tomorrow morning too), in which you can set a time delay.  I think you could put it in a loop and keep going round until you have results

Steve

Attached is a workflow that retries the node in the middle until it succeeds waiting the configured time until retry.

Cheers,

Patrick

Hmmm... maybe somebody could develop a "Try Loop Start" and a "Catch Loop End" pair of nodes (hint-hint)! Not a terrible idea - it's a general purpose solution that could be applied to a number of situations. Maybe add it to the wish list? Configuration parameters might include: Retry Number of Times, Retry Forever, Retry After Wait Time.

 

== Retry Database Writer Until Success ==

I modified Winter's wait_and_retry workflow to solve my particular problem. My workflows loop for 24 hours and write to a not-so-reliable database. I needed a way for KNIME to keep retrying writing to the database until it is finally successful.

I haven't tried this workflow in production but it seems to work in a test environment. To test this I point the Database Writer to a single-column table. If the column is of the wrong type (int instead of float) then the write will fail and the retry will loop. While retrying the user can delete the bad table and create a good table:

CREATE TABLE tempTable (
	column1 int,
);

DROP TABLE tempTable;

CREATE TABLE tempTable (
	column1 float(24),
);

But I have some questions about this approach:

Question 1: This seems like a complicated solution to a conceptually easy problem. Do I really need all of these nodes (including a Loop Start / End, a Try / Catch, and an If Switch / End If)? Is there an easier way??

Question 2: What happens if the Database Writer fails after already writing some data? Presumably I'll end up with duplicate records as the Database Writer keeps trying. This is better than having the whole thing stop dead in the middle of the night, but not ideal.

 

Ok - things got a little more complicated...

When I tried to put this "Retry Database Writer Until Success" metanode into production I discovered two problems:

1. This workflow doesn't work if the input table is empty. A race condition is created whereby the retry loop iterates at full speed forever. Another pair of nodes needs to be added around the original set of nodes: (a) Empty Table Switch, and (b) End If. That makes a total of 12 nodes to do something I think could be done more simply with just 2 nodes.

2. The actual wait time depends upon the number of rows in the input table. If there are 100 rows then the Java Snippet will run 100 times and the retry time will be 100 times slower than expected.

I've tried to fix both of these problems in the updated workflow attached.