Executing a Query periodically

@thentangler welcome to the knime forum. I have once proposed an example where a job would execute every x minutes for a given amount of times.

Also you could try the recursive loop with a condition that is never met and a wait node.

2 Likes

Hi there @thentangler, welcome to the KNIME forum.

Let’s try to understand your need.

The idea is to execute the data extraction/query nodes every 15minutes, without manual intervention ?

And of course, if you want to “re-execute” the data extraction, you need to reset the workflow and execute it again.

I would recommend you to plan your workflow to do the following (without loops and waits):

  1. PG Connector
  2. DB Query Reader
  3. Changes/Transformations
  4. Write results into a file or database

You could achieve this executing your workflow with batch execution and Windows Task Scheduler, no splash screen, no visual interface, workflow resetting, etc. Automatic execution. Take a look here:

FAQ | KNIME ( Is there any way to run KNIME in batch mode, i.e. only on command line and without the graphical user interface?)

1 Like

I was able to kinda make it work with the iris example. Thank you!
Is there a way to se a counter so i know how many times the loop has run? This is for debugging purposes in case it fails an n’th time, I would like to know what n is.
I tried using math formula, but it takes the input from the previous node, which is a table… how do i initialize a value and have it iterate through the loops?
Thanks again!

You could just create a TXT file where the “n” is in the file name. Not very elegant but you could always check on the hard-drive (make sure to delete the old one …)

1 Like

Thank for that suggestion, I would actually like to use this method since it would not involve me having to manually trigger it and leave KNIME open on my machine. However i am an infant with regards to KNIME.
I tried running knime in command prompt, but I get the following screenshot below. I used the following syntax and changed the path where my KNIME application was installed.

knime -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir=“C:\Program Files\KNIME\Auto_testing”

Auto_testing is the name of my workflow. I tried with both front and backslashes

Could you point me to any examples or resources where they have integrated windows task scheduler with Knime along with the command prompt execution?
Thank You

Please refer to this discussion. The talk about a couple of error codes when executing in batch mode.

Let me know if this helps. Otherwise we can take a deeper look into your workflow.

1 Like

Hi @eamendola, my workflow is very simple, as shown in the screenshot below. It just has a PG connector, DB reader and it spits the resulting table to a csv file that is written to a path in my local drive.
All parameters are hardcoded inside the nodes.
What other parameters would I need?

image

Hi @thentangler , try to add the parameter -reset in your command line. It will make sure that the workflow is reset before it’s executed. You probably want to add -nosave also.

You may consider create a batch file (.bat file) to encapsulate your command, and then schedule your Task Scheduler to execute that batch file.

I would also recommend that you move your workspace outside of the installation path of Knime. Your Knime is most probably installed in C:\Program Files\KNIME\ and it looks like your workflow also resides there. It’s better to keep them separate so that you can update your KNIME without risking deleting your workflows.

Thank you for asking this question. I’m not a coder and I didn’t even know what batch processing was. This thread made me realize the potential of KNIME to be used differently as I would normally do.

Upon digging up this “topic” further, I was able to find another thread that might be of interest to some of the questions you asked here too - hope it helps!

1 Like

On another note, I’d like to take the opportunity to suggest to KNIME Developers to provide an overview of resources focusing on Batch Execution with KNIME as you’d already done with Text Mining as exemplified in the link above. It will certainly make things more organized! :ok_hand:

Also, to add this as a topic under the KNIME Documentation here: https://docs.knime.com/ to make it more searchable-friendly :grin:

Thank you for this suggestion!
I have moved my workspace.
It still doesnt seem to work. In the FAQ and other examples the workflowDir has “/” in the path. But windows directory structure has “”. Should i put it as WorkflowDir=“C:\Knime_Workspace/Auto_testing”? or “C:\Knime_Workspace\Auto_testing”?
Where Auto_testing the name of my workflow.

Hi @badger101 ,
Thank you. This worked.
I created a silly workflow variable like test, integer, 1 and now it seems to work!
I dont know why my workflow would require that!
Between yours and @bruno29a I was able to get it to work.
Thank you!

1 Like

I’m glad to hear that! Anyway, do you have any idea why in the third point of that thread, the author puts this code:

-workflow.variable=myvariable1,999,int -workflow.variable=myvariable2,foo,String

in such a way? Am I correct to assume that each -workflow.variable is for each setting of the nodes in the workflow containing a variable?

Yes @badger101 that is right.

If you have multiple defined variables, you should build your command with one -workflow.variable for each one of them with its name, value and type.

@thentangler
Glad to hear that you get it to work. Just to add information, I have multiple workflows running on batch mode, with a command template in a .bat file like this:

cd C:\Program Files\KNIME\

knime
-application org.knime.product.KNIME_BATCH_APPLICATION
-nosplash
-reset
-workflowDir=“C:\Users\user\Knime_Workspace\workflow_group\workflow_name”

<workflow_group> is optional, only add to the path if you encapsulate your workflows in a folder/group. I do not add a final “\” to the path

Alright @eamendola That would mean if my workflow doesn’t have any variable Nodes, I can totally disregard (i.e. discard) that line of command right?

That’s right.
Check my previous answer, that’s my command for workflows without a parameter initialization/override default. I do have variables inside some workflows, but it isn’t always the case that I need to override them via command line. Sometimes they could be calculated inside the workflow.

1 Like

Thank you @eamendola , This really helps!

I still dont know what KNIME means when it says “variables”.
I would be having many variables in my nodes, I’ll try to learn all that in the tutorials…hopefully!

@badger101 Jus tto clarify, the variable specification i did was in the KNIME workspace by right clicking on my workflow and setting the variables there. I did not use the -workflow.variable switch in the command line.

1 Like

Hi @thentangler @badger101 , the variables that you would pass on the command line would be for the Workflow variables, not for the variables that you declare/create in a node from your workflow.

Workflow variables are global and are set interactively when you run a workflow via Knime AP.

You can access them by doing a right click on your workflow from the Workflow Explorer, and from the “Workflow Variables…” from the sub menu:

Typically, when you have such variables, Knime will prompt you to enter their values interactively when you open a workflow via Knime AP. So, how would you pass these values if you ran your workflow via command line? Well, that’s what the option -workflow.variable is for

2 Likes