Selenium Feature Request - Start WebDriver Retries After Error

Hi Philipp:

At your convenience, can you please add a "Retries After Error" option to the Selenium Start WebDriver node? Similar to the "# retries after error" option provided by the Palladian HttpRetriever node.

I've attached a picture of an attempted reliability wrapper to the Start WebDriver node but it isn't working very well. In the meantime, I'm experimenting with an AutoHotkeys Script that sends the "SHIFT-F7" keys to my KNIME Workflow every 10 seconds to keep it running.

I'm using the Selenium nodes more and more and having a lot of fun. Thanks again for the push in the right direction!

 

Hi Edlueze,

yes, I'll put it on the growing feature list :)

In the meantime, maybe the following workaround could be helpful: Use the "Start WebDriver" node only for starting the browser, and do not enter any URL into the configuration window. To load the page, rather use the "Navigate" node and enter your URL there. You should be then able to surround the "Navigate" node with KNIME's try/catch/looping nodes to keep retrying.

Hope that helps in the meantime.

Best,
Philipp

Ah - yes - good suggestion! That should definitely work better.

Ok - I've implemented your suggestion and will try it out. I've attached a picture for those who are following along.

BTW I captured an error message from earlier that I forgot to send you from when the Start WebDriver failed to load the URL. Probably nothing too surprising but thought I would send it to you just in case:

ERROR Start WebDriver      0:1602:1514:1115:1606:1139 Execute failed: {"errorMessage":"URL 'https:...' didn't load. Error: 'timeout'","request":{"headers":{"Accept-Encoding":"gzip,deflate","Connection":"Keep-Alive","Content-Length":"78","Content-Type":"application/json; charset=utf-8","Host":"localhost:25230","User-Agent":"Apache-HttpClient/4.5.1 (Java/1.8.0_60)"},"httpVersion":"1.1","method":"POST","post":"{\"url\":\"https:..."}","url":"/url","urlParsed":{"anchor":"","query":"","file":"url","directory":"/","path":"/url","relative":"/url","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/url","queryKey":{},"chunks":["url"]},"urlOriginal":"/session/03a25550-ffd7-11e5-8ee9-d50cebfad3ee/url"}}
Command duration or timeout: 11.86 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'France', ip: '192.168.0.101', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_60'
Driver info: org.openqa.selenium.phantomjs.PhantomJSDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, handlesAlerts=false, databaseEnabled=false, version=2.0.0, platform=XP, browserConnectionEnabled=false, proxy={proxyType=direct}, nativeEvents=true, acceptSslCerts=false, driverVersion=1.2.0, locationContextEnabled=false, webStorageEnabled=false, browserName=phantomjs, takesScreenshot=true, driverName=ghostdriver, javascriptEnabled=true, cssSelectorsEnabled=true}]
Session ID: 03a25550-ffd7-11e5-8ee9-d50cebfad3ee

 

Thanks for the details, Edlueze! Would be great to know whether the retry workflow works like this for you.

Kinds regards,
Philipp

Hi Philipp:

The reliability wrapper around the Selenium Navigate node seems to be working better, but now I have a reliability problem with the remote browser:

ERROR Navigate             0:1119:1504:849:1140 Execute failed: Error communicating with the remote browser. It may have died.
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'France', ip: '192.168.0.101', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_60'
Driver info: driver.version: RemoteWebDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, handlesAlerts=false, databaseEnabled=false, version=2.0.0, platform=XP, browserConnectionEnabled=false, proxy={proxyType=direct}, nativeEvents=true, acceptSslCerts=false, driverVersion=1.2.0, locationContextEnabled=false, webStorageEnabled=false, browserName=phantomjs, takesScreenshot=true, driverName=ghostdriver, javascriptEnabled=true, cssSelectorsEnabled=true}]
Session ID: 01f18600-0161-11e6-a6cb-4dd3961690b5

Do I need to try and reliably wrap the WebDriver Factory node itself to tackle this kind of problem or do you think it would be sufficient to have another try wrapping the Start WebDriver node?

 

Ok - I think I found something that should work. I had been overlooking the use of a second "Catch Errors" node (the first for Flow Variables and the second for Data Ports). This allows me to wrap all of the Selenium nodes. See the attached example for getting the cookies.