GET fails if the response does not have Content-Type set

[bedo2991]

I am trying to retrieve a JSON string from here.
The response does not have any Content-Type set, unfortunately.

This is what Knime returns:

Any chance to fix it (without having access to the Server that must respond, of course)

[armingrudd]

Hi @bedo2991 and welcome to the KNIME forum,

This alternative approach will work for you:

Pass the URLs to HTTP Retriever and extract the result using HTTP Result Data Extractor and finally the String to JSON node will give you the JSON you are looking for.

Here is an example workflow for you:

Palladian_Nodes_JSON.knwf (294.8 KB)

[Iris]

You can use the JSON Reader for this.

https://hub.knime.com/knime/extensions/org.knime.features.json/latest/org.knime.json.node.reader.JSONReaderNodeFactory

I just tried and it works like a charm

[bedo2991]

Thanks Iris, that would work if I could provide an input to the node, but it looks like it does not have any input port and I can only hard code a path into it (I need multiple data with different parameters, not just tha example).

[Iris]

Hi @bedo2991

in this case you need to surround it with a loop.

I made you a small example and shared it in my hub. You can get it from here:

https://hub.knime.com/iris/space/ExamplesForTheForum/2019_08_02_LoadTonsOfJsons

Cheers, Iris

[Iris]

Acutally, I wanted to use the following node from @Vernalis or @s.roughley

Load text-based files

But it unfortunately can not read directly from the URL. Maybe @s.roughley can get this working

[s.roughley]

Hi @Iris - could you give me an example URL which it fails for?

Thanks

Steve

[Iris]

Hi @s.roughley

I added the node to the workflow. It only produced missing values for the read file

https://hub.knime.com/iris/space/ExamplesForTheForum/2019_08_02_LoadTonsOfJsons

[s.roughley]

Hmm, that definitely doesn’t work! I tried the Get Request node too - and that also fails. I will have to dig a bit deeper to try to figure out what the problem is.

Steve

[Vernalis]

Hi @Iris - I have a temporary fix to get it working. If you change the file encoding to assume UTF-8, then the node works:

I will be able to fix this so that the ‘Guess’ option works properly - will post again when that is done.

The technical stuff as to why… The ‘Guess’ option needs to read the first few bytes of the response to see if it can detect and interpret a Byte-Order Mark, and then return to the start of the response. Depending on the sort of InputStream returned from the URL or filepath, this may not be supported, which is what happens here. I will change the code to check first, and if it is not supported then wrap the response in a different InputStream implementation which does support it.

Steve

[Iris]

Good catch! Thank you @Vernalis
I updated my example workflow so it is now working well.

@bedo2991 I now would recommend using the lower path of the workflow as a solution. It will be faster than going over the list of url via a loop.

[Vernalis]

As an extra observation, the Load text-based files node will parallelise across multiple CPU cores, and if needed will run in streaming mode. (Incidentally, that info is not visible in the KNIME hub page for the node, as far as I could see, but is visible at https://nodepit.com/node/com.vernalis.nodes.io.txt.LoadTxtNodeFactory - maybe that’s another hub feature to add?)

Version 1.20.5 of the Vernalis nodes has now been release on trunk and stable build channels, and fixes this issue when ‘Guess’ is selected.

Steve

[danielesser]

@Vernalis

maybe that’s another hub feature to add?

Well, now you are hurting my feelings, Steve

[s.roughley]

Hey, you have lots of other cool features too

Steve