HTTP Retriever fails to fetch URL's containing a "?" outside of the context of query parameters.

I am using the HTTP Retriever node to fetch multiple product page URL’s. 3 requests return 403 Error responses. These failed requests share the commonality of containing a “?” in the path component of the URL. These question marks do not appear to indicate the start of query parameters - they are possibly a mistake. I’ve never actually seen a URL like this before. Nonetheless, they are successful in the browser.

I’ve included a small demonstration workflow with 3 regular URL’s and 3 URL’s containing the mystery question mark.

Thanks for your time.

Failed_Requests_HTTP.knwf (9.0 KB)

Hi Nacyjay,

Many thanks for this. We’ll have a look.

Would you be willing to give the pre-release build a try? This already fixes the issue you described here and probably this one will be resolved as well. If not, we’ll try to get this fixed for the next release as well. If you would give the pre-release a try, please contact us at mail@palladian.ws

Thanks,
Philipp

I am not sure if there’s anything to fix. All of these URLs (no matter whether there’s a ? or not) give me a 404 when I try to load them in the browser:

Hi,

that’s strange because all 6 URL’s included in the workflow I uploaded here do work in the browser, and then only 3 work (return 200 status code) in the workflow.


and yes I will give the prerelease a go, thanks.

Hi Nancyjay,

received your email – thanks. I will be in touch regarding the prerelease for the next version, this will still take a bit, as we’re currently working on some outstanding issues.

Regarding what I wrote below – my apologies. I just realized that I’d need to click on “English” on the website, and only then the links work (that’s why you set the cookie at all, I totally overlooked this :grimacing: ). So I can confirm that all the links work fine in the browser, but the ones with ? return 404 in the workflow. I think we have addressed this issue in the recent Palladian lib update as well.

Thanks for the feedback, and I’ll be in touch about the pre-release (hopefully next week).

–Philipp