I again believe the proxy config for the HTTP Retriever Node is not working as it should. I initially thought it was working because the response code was “200”, but I cannot explain the behaviour of the following workflow other than the explanation that it is an issue with the proxies.
When I make a request without using a proxy (so when the useProxy bool is false) the requests are successful and respond with the desired page. However the second and consecutive iterations do use a proxy, and while the response code is “200”, the page is actually a “maintenance” page even though the site is not under maintenance. This only happens under two conditions:
- the proxy config is enabled and uses the IP Address and Proxy flow variables
- the “accept” header is not included.
The site has a lot of anti-scraping mechanisms, so maybe I’m wrong, but I think it is the proxies. Here is the workflow:
Proxy_Example.knwf (52.6 KB)