Increase Time out

Hello team,

I’m facing this error: Execute failed: Read timed out when I runing a loop of arund 40 web files by line read Node. How can I increase its time out?

Timeout.knwf (60.2 KB)

Hi maybe it makes sense if you share your error message as well. (like is it a timeout because the requested url could not be fetched?)
bR

Hi Daniel, this is the error message: “Execute failed: Read timed out” this error happen after around 30 minutes of loop execution. I don’t think the problem is with fetching URL data.

@rafaelfre it is possible that the website has a mechanism to prevent too many requests from a single IP or something.

One way to deal with this is to store your list of URLs and download the files individually and store them in a local location. With the new Transfer Files node you could just tell the system to skip files that are already there. You could then deal with the CSV files later.

An example how to make a list and the check if all items have been done is here:

2 Likes

Another note. You might have to switch the Encoding in the CSV Reader to ISO-8859-1

2 Likes

@mlauber71, tks for ideas, I’ll try it later it is a good workaround. But I made other way.
I’ve included the partitioning node after nominal filter 50% for each part, and perform the line reader node for both part at same it and it’s work :slight_smile:

thank you so much

1 Like

@rafaelfre good if this workaround -well- works. From my perspective it could be better to set up a mechanism that has a high chance of succeeding every time. It should be robust. Often this would involve something where you could pick up at a place the system left off.

And also the Line Reader seems to do its job. I personally would like to keep the original CSV files in case something goes wrong with my further analysis - so I could revert back to the original source without having to do the whole download again. Also downloading again might put stress on the originating site and it is always possible that they decide to disable mass-downloads. So you might want to use these resources wisely.

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.