Thank you for the information and the feedback, @umutcankurt
The background, in a nutshell: The verification box which you show above is from Cloudflare, and colloquially called the “Turnstile” to verify that you are “human”. What makes it different from the more traditional CAPTCHAS which waste your time by giving you stupid tasks (and which can be worked around using the Solve CAPTCHA node): All you need to do is click a checkbox - seemingly easy.
Except when you automate a browser using WebDriver or CDP. Then you can keep clicking and clicking and clicking and you will not pass.
How can you interact with a website in such a case? There’s a simple way: Stop the auto pilot temporarily, let the human side take over (if needed) to submissively proof your unconditional good intentions, and then continue with your workflow.
The good news: The brand new Selenium Nodes v5.5 provide an additional plugin for that - you can install it through the “Install KNIME Extensions” dialog by selecting “Selenium Nodes for KNIME: Undetected Chrome Utilities”. The plugin contains a modified WebDriver (select Chrome for Testing (bundled, undetected) [BETA] in the Start resp. WebDriver Factory node). You can then make use of three new nodes which you find under Selenium → Utilities → Undetected [BETA].
During my experiments I found two approaches:
-
For sites which show the Turnstile when you navigate to a specific page: Schedule the navigation using a JavaScript
setTimeout, disconnect the WebDriver, let the navigation happen, reconnect the WebDriver. -
For sites which insist on clicking the Turnstile: Navigate, disconnect the WebDriver, pause the workflow, confirm the Turnstile (you must do that manually based on my current knowledge), then reconnect the WebDriver.
Not sure if these approaches will work for any imaginable scenario and they might require some additional MacGyver skills - yet they are hopefully a valuable addition in your toolbox to regain the control over your workflows.
