Selenium WebDriver alwaus return the same page despite update URL in flow variable

Community Extensions Palladian & Selenium
#1

I have created a workflow, which supposes to scrap several URLs, which contains dynamically generated data.
I’ve prepared a list of URLs and convert them to the flow variables. But when I ran this workflow WebDriver did not update the content of the page, only URL. I was trying Navigate node, but it did not like my URL strings. What am I doing wrong?

Screenshot 2020-05-13 at 11.36.14
Screenshot 2020-05-13 at 11.36.143188×776 449 KB

#2

Hi lptolik,

  1. When you use the “Start WebDriver” (or the “pooled” counterpart), make sure that you set the flow variable urlInput (not remoteUrl) for navigation. (remoteUrl is for connecting to a remote Selenium server):

    image
    image1564×750 157 KB

  2. When you use the “Navigate” node you must specify a “dummy” URL, even if you override it through a flow variable. (this is a KNIME-specific annoyance peculiarity)

Please see this workflow on my NodePit Space for an example:

Does this solve your issue?

– Philipp

#3

Thank you! It was useful about Navigate. As for WebDriver it seems the problem is with Pooled one, as plain WebDriver manage to return appropriate content. Thank you, once again.

#4

Thanks for the feedback!

In fact this should of course also work with the “Pooled WebDriver” node – so if you still encounter the issue, I’d be grateful about a test workflow so that I can see whether there’s a bug!

Best,
Philipp