I have created a workflow, which supposes to scrap several URLs, which contains dynamically generated data.
I’ve prepared a list of URLs and convert them to the flow variables. But when I ran this workflow WebDriver did not update the content of the page, only URL. I was trying Navigate node, but it did not like my URL strings. What am I doing wrong?
Hi lptolik,
-
When you use the “Start WebDriver” (or the “pooled” counterpart), make sure that you set the flow variable
urlInput
(notremoteUrl
) for navigation. (remoteUrl
is for connecting to a remote Selenium server): -
When you use the “Navigate” node you must specify a “dummy” URL, even if you override it through a flow variable. (this is a KNIME-specific
annoyancepeculiarity)
Please see this workflow on my NodePit Space for an example:
Does this solve your issue?
– Philipp
Thank you! It was useful about Navigate. As for WebDriver it seems the problem is with Pooled one, as plain WebDriver manage to return appropriate content. Thank you, once again.
Thanks for the feedback!
In fact this should of course also work with the “Pooled WebDriver” node – so if you still encounter the issue, I’d be grateful about a test workflow so that I can see whether there’s a bug!
Best,
Philipp