ERROR... Execute failed:... (Too many open files)

Nancyjay · February 2, 2021, 9:34am

A few times now, when I have left a a few workflows to execute over night, they all fail with the above error. Each workflow fails on a different node, for example this was the error message for an Objects to String node:
“ERROR Binary Objects to Strings 8:930 Execute failed: java.io.FileNotFoundException: /private/var/folders/l_/999cl7p931q_x93c3ptlrtxm0000gn/T/knime_3C_10_R10_ishop52253/fs-HTTP__8-929-52342/000/014/binaryObject-14421 (Too many open files)”

When I first got this message it was when I was trying to run 5/6 workflows in parallel, which is fair enough. But now it fails when i have only 3 workflows running, which is think is a bit ridiculous.

It also will now not let me re-execute any of the workflow, not a single one, and just spits out this “Too many files open error”… So I cannot re-execute any of the workflows, and I cannot CLOSE any of the workflows as I get a " Save Failed java.lang.NullPointerException" error. So no one is winning here.

I have colleagues who have ever seen these errors, so I don’t know why I’m getting them.

Thanks,
Nancy.

ztrubow · February 5, 2021, 11:32pm

Hi Nancy,

Can you answer a few questions for us to help us troubleshoot:

What version of KNIME AP are you running these on? Or if Server, what Server version?
When did this start happening? Or has it always happened when you ran multiple workflows in parallel?
What OS are you on, Windows,Mac,Linux?
Do you happen to have any antivirus software running on the machine?
Would you be able to send us the logs for us to take a look and see if we can figure out why the issue is happening from there?

This should help us get started on finding the root cause.

Thanks,
Zack

Nancyjay · February 19, 2021, 9:53am

Hi, here’s a copy of knime.log as a .txt file:

knimelog.txt (734.5 KB)

Nancyjay · February 19, 2021, 11:15am

Hi Zack,

Sorry for the late reply.

Knime is now very slow for certain workflows, I assume it’s connected. I’ve seen suggestions to delete the contents of the .metadata folder, but I’m reluctant without knowing exactly which files to delete.

Version 4.3.1
The error has happened twice; when I ran multiple workflows in parallel. It is slow for multiple workflows.
I’m on MacOS Catalina
No I don’t. (I do have two VPN apps)
I tried uploading the .log file in the knime folder but it’s too large (9.6MB…) so I’ve enclosed a copy as a .txt file.

Thanks for your help,
Nancy.

knimelog.txt (734.5 KB)

Nancyjay · February 19, 2021, 2:57pm

I tried a solution I found here: Continental nodes invisible - #12 by izaychik63 where I started Knime with -clean option in the .ini file. This solved the latency in the workflow, but now I’m getting this error for multiple node including Binary Objects to Strings, XPath, & GroupBy:

failed to apply settings: com/google/common/collect/HashBiMap$KeySet

I tried to run a pre-existing metanode, and I got the following errors:

Blockquote
ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR LoadWorkflowRunnable ERROR Column Filter Errors during load: Status: DataLoadError: 3C_10_R10_JD 5 loaded with error during data load
Status: DataLoadError: 3C_10_R10_JD 5
Status: Warning: Super Robust HTTP Retriever - Multi URL 5:1839
Status: Warning: Column Selection Configuration 5:1839:0:1648
Status: Warning: State has changed from CONFIGURED to IDLE
Status: Warning: Column Selection Configuration 5:1839:0:604
Status: Warning: State has changed from CONFIGURED to IDLE
Status: Error: Generic Price Formatting 5:1681
Status: Error: Column List Loop Start 5:1681:827
Status: Error: Loading model settings failed: Config for key “column-filter” not found.
Status: Warning: State has changed from CONFIGURED to IDLE
Status: Warning: State has changed from CONFIGURED to IDLE
Status: DataLoadError: Generic Price Formatting 5:1947
Status: Error: Column List Loop Start 5:1947:827
Status: Error: Loading model settings failed: Config for key “column-filter” not found.
Status: Warning: State has changed from CONFIGURED to IDLE
Status: DataLoadError: Column Rename (Regex) 5:1947:829
Status: DataLoadError: Unable to load port content for node “Column Rename (Regex)”: No BufferedDataTable with ID 19824693
Status: DataLoadError: Regex Split 5:1947:824
Status: DataLoadError: Unable to load port content for node “Regex Split”: No BufferedDataTable with ID 19824694
Status: DataLoadError: String Manipulation 5:1947:830
Status: DataLoadError: Unable to load port content for node “String Manipulation”: No BufferedDataTable with ID 19824695
Status: DataLoadError: String To Number 5:1947:831
Status: DataLoadError: Unable to load port content for node “String To Number”: No BufferedDataTable with ID 19824696
Status: DataLoadError: State has changed from CONFIGURED to IDLE
Status: DataLoadError: Generic Price Formatting 5:1948
Status: Error: Column List Loop Start 5:1948:827
Status: Error: Loading model settings failed: Config for key “column-filter” not found.
Status: Warning: State has changed from CONFIGURED to IDLE
Status: DataLoadError: Column Rename (Regex) 5:1948:829
Status: DataLoadError: Unable to load port content for node “Column Rename (Regex)”: No BufferedDataTable with ID 19824708
Status: DataLoadError: Regex Split 5:1948:824
Status: DataLoadError: Unable to load port content for node “Regex Split”: No BufferedDataTable with ID 19824709
Status: DataLoadError: String Manipulation 5:1948:830
Status: DataLoadError: Unable to load port content for node “String Manipulation”: No BufferedDataTable with ID 19824710
Status: DataLoadError: String To Number 5:1948:831
Status: DataLoadError: Unable to load port content for node “String To Number”: No BufferedDataTable with ID 19824711
Status: DataLoadError: State has changed from CONFIGURED to IDLE
Status: DataLoadError: State has changed from CONFIGURED to IDLE
5:1946 failed to apply settings: com/google/common/collect/HashBiMap$KeySet

Nancyjay · February 19, 2021, 3:09pm

Just came across another error when trying to run a Chunk Loop (it seems the loop body has executed, but the Chunk Loop Start node & Loop End node remain as “queued”:

ERROR ThreadPool                      An exception occurred while executing a runnable.

mlauber71 · February 20, 2021, 6:06am

I am experiencing problems with the chunk loop start node as well.

The problem seems to occur when I use a chunk loop start loop and then further up in the workflow another loop (maybe another chunk loop). The first chunk loop seems to stay frozen or queued but somehow does execute but the second one stays stopped. I will have to further investigate. It happens in connection with a wait node.

It seems possible something is wrong with the “wiring” of a cascade of loops involving chunk loop.

Also on McOSX latest version.

@ztrubow maybe you could check this out.

ztrubow · February 25, 2021, 8:39pm

Hi @Nancyjay ,

Could you also provide us with the workflow you are using? We currently have a ticket open to potentially fix a bug with Chunk Loop and I am curious if your workflow fits into that bug. What I am seeing in the log you sent is a bunch of disconnects and handshake errors for SSL.

Thanks,
Zack

mlauber71 · February 26, 2021, 6:44am

@ztrubow I run several tries on Windows and MacOS with cascading loops with a chunk loop at the start. I was not able to reproducde the problem. I might have to get back to my workflow where it originally appeared and reconstruct that (other components have been involved, it was this thread) - unfortunately instead of reporting the (possible) bug at the time I just made a workaround without saving the problem in a reproductible way.

It does not seem to be just cascading (chunk and additional) loops but something else inbetween them that makes tthe chunk loop start ‘think’ it is still running and freezes the whole thing.

Nancyjay · February 26, 2021, 10:47am

Hi @ztrubow

Is it the whole workflow you need? If so, do you have an email address I can send it to?

Thanks.

ztrubow · February 26, 2021, 8:39pm

If you email the support@knime.com with the workflow, I can pick up that ticket.

Thanks,
Zack

Nancyjay · February 28, 2021, 4:46pm

Thanks, I’ve sent the workflow to the above email.

system · August 30, 2021, 4:46am

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.