Obtain CSV file from URL

Hi all,

I have Knime desktop installed only but moving to server soon. I’m new to Knime and loving it but have a problem. I wish to carry out the retrieval of data from a web URL which contains the query question mark within the address. For instance:

www.webpage.com/subpage/anothersubpage/filename.csv?v=2021-08-06-13-27-04

It seems I am unable to achieve this with the file reader or excel reader as the question mark seems to throw an exception. I’ve searched for other solutions but they always seem to include either extensions that I cannot download due to business restrictions or deprecated nodes. Is there a way to pass this kind of URL to a node to gather the CSV using currently active nodes ?

Appreciate any help you can provide.

thank you

1 Like

Hi not sure what your page is as I can’t test it but for web requests you could try a get request with the Get request node
br

2 Likes

Hi @d_skweez , what configuration are you specifying on the CSV Reader.I’m able to use a CSV Reader to read the following location:

https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv?x=1
(the ?x=1 is added by me as it wasn’t part of the actual URL, but the reader doesn’t complain)

Actually, given that you say it’s complaining about the “?”, I’m going to make a guess that you have two things wrong.

(1) You are probably specifying Read from as “Local File System” (it needs to be Custom/KNIME URL), and
(2) You haven’t added the “http://” at the start of your URL. A valid URL has to have a valid protocol identifier (e.g. http://, file:// etc), otherwise it doesn’t know how to make the request.
e.g.

3 Likes

Here’s a demo… I think CSV reader can probably handle it with the correct config. :wink:

Sample CSV from URL.knwf (6.7 KB)

2 Likes

Hi @d_skweez ,

I agree with what @takbb said:

If this still did not fix the issue, can you show what you are using (File Reader? CSV Reader? Web Request (GET, POST…)?) and its configuration? Or even share your workflow? Please also share the error message too.

4 Likes

ok, thank you all,

@bruno29a and @takbb. You are right, I am able to use the URL with (?) within the address with CSV reader and in my confusion this probably went un-noticed. However, whilst I set the timeout to a considerable time it doesn’t seem to want to connect/download. Also, the file is in fact a .xlsx file (apologies). Here is the link that works within a browser, and should continue to work even as the file name changes each month given this link is months old already perhaps.

https://www.rba.gov.au/statistics/tables/xls/c01-2-hist.xlsx?v=2021-04-30-10-05-34

So if it is an xlsx, and I cannot use the csv reader I may be back at square one. Any ideas how to get at this file :slight_smile: ?

thanks again

Hi @d_skweez , a CSV file is basically a text file, and is read as a text file, while xlsx files are Excel files and cannot be read as a text file. So you cannot open xlsx files by the CSV Reader. You have to use the Excel Reader instead.

As you can see below, I am able to open it:

Try it with Excel Reader, and configure as what I showed in the red boxes.

4 Likes

Thank you. I tried the above, but I still got errors:

I took to downloading Knime to my personal computer to test it there and it worked as you would expect; without a problem. Therefore it’s likely that the security at the company is preventing access so I’ll need to take up the issue there…

Thank you for your help.

2 Likes

Hi @d_skweez , it looks like it indeed. It looks like it’s being blocked.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.