There should be around 100 group loops to do. If I choose only one part of the group loops to do, let’s say 2 groups, the execution finishes, and the system works really well.
If I take away the Rule-based row filter, where I filter 2 groups the result is the Loop End that is queued.
and all the Metanode “NEAREST PLANT SONDA” stops after the first iteration.
What can be the problem? Why the system executes only the first iteration?
I went through your explanations and I believe there can be several reasons why the end loop may freeze in a queuing state. Among those, the amount of memory made available as you mentioned in your last post.
Another possible problem is that you are appending too many columns and this can be problematic if the number of columns becomes too big while loopping. How many columns are supposed to be appended at the end of the process ?
If increasing the memory doesn’t help, I suggest to implement the following simple test to eventually check the final generated number of columns: Add a -Header Extractor- node before the -Loop End Column Append- node and connect the latter only the table with the column headers from node 1st output. This will eventually show how many columns are generated by your -Loop End- node and whether the number is reasonable and handable.
unfortunatelly also th eheader exctractor desn’t make the loop going to the ed.
So I think it is only a question of memory?
To have some numbers:
the table has 410 columns, for 26k rows. The loop groups are working every time only with maximum 10 columns (same number of rows). at the end, the table should have the same number of columns and rows as at the beginning plus 110 columns for a total of 520 columns and 26k rows…
What do you think I can do about it?
I tried also to limit the group loop on 4 groups and the first 2 loops went weel the third one got blocked.
I want to add some info maybe it could help. The group loop table is a table that gets the columns that in every loop are used like a filter on an initial table of 410 columns. I don’t know if this is a helpful info.
in this case, three NOME SONDA ORIG are the values that are going to be taken from the initial table of 410 columns and elaborated in the whole Group loop
Thanks you for your replies and further information. It is not easy to guess without having access to your workflow. I would recommend the following to check whether it is a problem of memory or not.
One way to know is as follow:
Open the Windows Task Manager utility on your computer. It should look like this:
Run the loop nodes one by one for first iteration starting by the -Group Loop Start- node and check how the memory usage evolves at every node execution.
Once you arrive to the -Loop End Column Appender- node, run it using the “Step Loop Execution” option in the configuration menu, as follows:
Execute and Monitor again “step loop” by “step loop” execution to check how the memory occupation grows. If the occupation goes up to your memory limit, it means that KNIME is using at least as much memory as your computer has and it goes into memory pagination. If it doesn’t, the problem is somewhere else.
You may also want to try the -Run Heavy Garbage Collector- node to as explain here:
and used here in this workflow :
Maybe this helps, even if KNIME already optimizes the use of the Java Heap.
Another thing is that the problem is only on the loop end.
During the whole loop, everything is working well.
The error that occurs in the Error log is not the java memory error but this:
Cannot invoke “org.knime.core.node.workflow.WorkflowContext.getCurrentLocation()” because the return value of “org.knime.core.node.workflow.WorkflowManager.getContext()” is null
java.lang.NullPointerException: Cannot invoke “org.knime.core.node.workflow.WorkflowContext.getCurrentLocation()” because the return value of “org.knime.core.node.workflow.WorkflowManager.getContext()” is null
Can anybody tell me what kind of error is this one?
thanks a lot for your help, you are the best!
Memory checking doesn’t get much more answers than before. the usage of the memory stays around 10 GB and step-by-step node execution gets him on max 11.4 GB.
The strange thing is that I have made the step by step execution → the first two loops ok, but when I arrive on the loop end of the 2. loop it doesn’t want to do the 3.loop. In some way, it doesn’t switch from the 2.loop Loop end to the 3.loop Group loop start.
I tried also the second suggestion but the same thing. the loop arrives at the end of the 2. loop and got queued.
I have posted in the last message also an error that I see when I let the loop executes. I don’t know if it can be.
I have read that it could be a problem of a corrupted workspace??
Thank you for the extra explanations too which highlight the way to find a solution.
I also guessed a problem of workflow corruption but wanted to explore further the posibility of a memory problem as stated from the beginning.
I wonder how complex is your workflow and how much it takes to fully run it to try the following:
Create a new fresh workflow using option:
On your original “corrupted” workflow, do a select all (Ctrl-A) to select all the nodes and on the newly just created workflow do a paste of all the nodes (Ctrl-V). This will make sure that all your workflow is identical to previous one but without inheriting any possible corruption from the original corrupted workflow.
Make sure that this new workflow has access to the required input data, and create new local data folder for local data if required by your workflow.
At this stage, please save the new workflow. If it correctly saves without problems, try to save the original one too and then close and open again KNIME.
Execute the new workflow to check whether this new fresh version works without errors.
After following these steps, please let us know whether you get the same errors and we will try to help you further from there.
The local data are on a server so I didn’t create e new folder → point 3. of your list, could it be a problem?
When I tried to save the old workflow it appears a warning: “we won’t safe the nodes that are in execution”, but no execution was working at that time.
ANd this morning I had some problem with the workflow when it has opened–> it gave me some errors but I can’t remember which one.
This option was only there in case you have stored data locally in the workflow in its “data” directory. From your comment, it sounds it was not the case, so please ignore step 3. Access to your data should be ok in the new workflow if you copy and paste the nodes as suggested in my list of steps.
This is normal. Nodes within a loop that is not fully executed (as in your case) are saved but will appear as reset when you open again the workflow. The important thing here is to save the original workflow before exiting KNIME.
If you follow points 1, 2, 4 (warning about executing nodes within a loop is normal) and 5, does the new workflow give same errors ?
The new workflow doesn’t have any warnings/errors in the saving phase.
When I execute the problematic part of the flow, the Loop continues not to work, it remains in queued status
In the Error Log I can’t find any Error logs, only this warning:
Good to know that the workflow copy is not anymore corrupted.
I would say that this is just a warning and should not be a reason for the loop to stop working or get stuck in queued mode.
Would be possible for you to upload here your workflow with non-confidential data so that we can investigate what is the problem and help further ? Thanks.
I guess it is 4 MB because it is partially run or reset but with temporary data.
If you have a KNIME HUB account, you could post it in your HUB public space and post here the link to it. I believe it is limited to a maximum of 50 MBytes per workflow.
Otherwise, you could export and upload the workflow here making sure that it is reset before. I will try to manage to understand how it works and what fails from there even if I do not get the data at first. Thanks.
I have it know. The workflow is quite complex and it will be difficult to guess what is going on without a minimum of data. Do you think you could post the two Excel files here in two separate posts if individually they are smaller than 4 Mbytes?
The other option is to post the XLS files with a sub-ensemble of data if this is still good to test your workflow, so that we are sure they are smaller than 4 Mbytes.
!
I am desperate …
but cannot say why this is happening