Hi,
I am trying to use selenium nodes with proxy settings and they are ignored.
These are the options I set
And this is the result when I access https://whatismyipaddress.com/ with Start Web driver
I also tried passing the settings as JSON input, but the result is the same
Instead, when I set the proxy settings in the Firefox browser, not the Webdriver,
the result is this one:
I have also tried setting the proxy in KNIME settings, but they are also ignored by Webdriver.
Any ideas?
I’d expect httpProxy to only work for non-HTTPS, but the example page you mention uses HTTPS.
– Philipp
Thanks for the feedback, @dasturiol
To clarify for me: You were not able to get it to work using the included “Snippets”, neither with Firefox nor Chrome?
To clarify for you: The proxy settings in KNIME are not considered by the Selenium Nodes. This is a conscious decision, as for crawling, scraping, it sometimes makes sense to use a dedicated proxy setting which differs from the default, system-wide settings.
– Philipp
Hi Philipp,
Yes, I was not able to get it to work using the “templates” included in the WebDriver node. Neither Firefox nor Chrome.
Then, googleing, I found out the option I posted above. Tried and it worked.
I think what happened is that, at least chrome, changed the way to set the proxy settings in some version (I am currently using v83) and I think this is the reason why it was not working.
Regarding the KNIME proxy settings not being considered for Selenium, it makes perfect sense. Actually I was hoping it was like this. Mainly because it gives the possibility to change the proxy depending on where you are scraping or crawling.
cheers.
Many thanks for the feedback! We’ll definitely investigate this in detail! I’ll keep this thread updated.
– Philipp
Again, thanks for noticing this @dasturiol! There were indeed two bugs which we fixed, available now in the 4.4.3 release!
Changelog:
