Premature EOF when reading gz files on KNIME Server

I have a KNIME Workflow that reads a few large gzipped CSV files (largest is 700mb zipped, 35 million lines).

The workflow works locally in KNIME AP. The machine running KNIME AP has 16 GB memory, I do not see excessive RAM usage.

I deployed the workflow to KNIME Server, the executor has 20GB Memory. I constantly run into errors in the file parser nodes. The error says Premature EOF but I checked the gz files on the server, they are complete, can be unzipped and show OK on gzip -t -v.

Maybe I’m missing something obvious. Can this be a memory problem, maybe because the memory/caching policy on KNIME Server is different from KNIME AP?

The full log is attached, any help would be appreciated.

knime.log (36.2 KB)

The first things I would look at would be the memory settings for the KNIME Server and KNIME Executor.

Tomcat has a small default memory setting 1 or 2 GBs. Depending on how the server was installed the executor could have a low memory setting.

Where is the file stored? Are you reading it from the KNIME Server?

The file is downloaded to the KNIME Server, the download is part of the workflow. I just checked the files manually (with gzip) to see if they are ok.

I changed memory settings for the executor in the knime.ini file and set it to 20 GB (was actually higher before). Only one executor is configured and it runs on the same machine.

I installed KNIME Server as per this support doc: KNIME Server Installation Guide

To check if it is memory related I changed the memory policy in the file reader to “Write tables to disc”. The “Cache tables in memory” seems to be the default. Now it works in KNIME Server as well, I retried a couple of times.

I never had issues with memory on large files with KNIME AP and the description of the “Cache tables in memory” setting suggests that this still writes to disc when running out of memory. Are the caching mechanisms or policies different between AP and Server?

Thanks for your pointer @jeffgullick-knime, I’m new to KNIME Server and still trying to figure out the basics :slight_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.