Increase Time out

[rafaelfre]

Hello team,

I’m facing this error: Execute failed: Read timed out when I runing a loop of arund 40 web files by line read Node. How can I increase its time out?

Timeout.knwf (60.2 KB)

[Daniel_Weikert]

Hi maybe it makes sense if you share your error message as well. (like is it a timeout because the requested url could not be fetched?)
bR

[rafaelfre]

@rafaelfre

Execute failed: Read timed out

Hi Daniel, this is the error message: “Execute failed: Read timed out” this error happen after around 30 minutes of loop execution. I don’t think the problem is with fetching URL data.

[mlauber71]

@rafaelfre it is possible that the website has a mechanism to prevent too many requests from a single IP or something.

One way to deal with this is to store your list of URLs and download the files individually and store them in a local location. With the new Transfer Files node you could just tell the system to skip files that are already there. You could then deal with the CSV files later.

Demonstrates how to download CSV data from a list of URLs – mlauber71

An example how to make a list and the check if all items have been done is here:

"CSV Writer" node cannot write the word COM?

Not exactly sure what happened but again I have this modified solution to offer that would first download the files and later import them all into CSV and even collect them in a KNIME table in the end. If one download fails you could resume the download, that might be faster and more robust than to use the file reader directly. In the hub, there are all files contained from your example. [image] As a side note, you might want to think about using the relative KNIME paths like: knime://knim…

[mlauber71]

Another note. You might have to switch the Encoding in the CSV Reader to ISO-8859-1

[rafaelfre]

@mlauber71, tks for ideas, I’ll try it later it is a good workaround. But I made other way.
I’ve included the partitioning node after nominal filter 50% for each part, and perform the line reader node for both part at same it and it’s work

thank you so much

[mlauber71]

@rafaelfre good if this workaround -well- works. From my perspective it could be better to set up a mechanism that has a high chance of succeeding every time. It should be robust. Often this would involve something where you could pick up at a place the system left off.

And also the Line Reader seems to do its job. I personally would like to keep the original CSV files in case something goes wrong with my further analysis - so I could revert back to the original source without having to do the whole download again. Also downloading again might put stress on the originating site and it is always possible that they decide to disable mass-downloads. So you might want to use these resources wisely.