In my workflow, it occasionally gets stuck at the “Navigate” node, but I’d like to implement a solution to guide the workflow like this: “Hey! If it’s taking too long, refresh the URL!” How can I achieve this?
The workflow doesn’t crash and generally functions, but occasionally the website server stops loading the requested URL. This causes the “Navigate” node to take a long time to load the page. However, if I can refresh it after about five minutes, it’s fine, and the loop will proceed.
Is there any general solution for this like to PAUSE/RUN the loop if it takes long time.
Let’s look at this issue from a different perspective. I’ve observed that the “Navigate” node on certain web pages keeps loading endlessly, whereas when I open the same page directly in a browser, it loads quickly and without issues. Why does the same page load so slowly when using Selenium nodes? How can I make it load faster? Are there specific options in the “Navigate” node that should be disabled to improve performance?
Hey mmrostami,
That’s in fact valuable information. I had this case with another customer, so I’m copying here my email reply from back then - the relevant part is (2):
There’s two potential explanations:
(1) An issue with the browser, system, or (unlikely) node version. To rule this out, can you try running this with a different browser (e.g. with Firefox) and/or on a different system? Are you using the most recent version of the Selenium Nodes?
(2) The site uses some bot/automation detection mechanism which intentionally slows down the performance. You could try if using the “undetected” nodes help - see following links:
Selenium - Help with Cloudflare verification issue - #7 by qqilihq
For a test, you could use the “Timed Navigate” node to perform the navigation and see if the login page loads faster.
Best regards,
Philipp
1 Like
Philipp, your “Timed WebDriver Navigate” node is incredible! 
I’ve noticed Selenium has a ton of nodes, but I’m struggling to find documentation to help me learn how to use them! I came across https://seleniumnodes.com/, but is there a comprehensive resource or detailed documentation with examples I can follow? I’m super eager to dive in and master all of them. Could you please point me to a complete resource? I’d really appreciate it!
Philipp, I let the workflow run and observe it, but I get error like this after scraping many pages:
Execution failed in Try-Catch block: You are currently not connected to the browser - reconnect using the “Reconnect WebDriver” node.
in which point I need to reconnect WebDriver again? 
I tried this but it works slowly.
for each iteration it takes about 1 minute and 20 seconds, especially on second “Timed WebDriver Navigate” it loads slowly.
I tried this but after passing many pages , in “Reconnect WebDriver” node stuck.
@qqilihq I tried this but didn’t work also
Hi mmrostami,
-
Foreword: These nodes (undetected) were originally built for one-off use cases, e.g. logging in to a protected website once and then performing some actions - often after you passed the initial bot check, you have a “free ride”. Your workflows show some loops with repeated usage of the “Timed Navigate”, so I assume that you execute them repeatedly (probably in hundreds, thousands of times?). This might work or not - I have not verified it myself.
-
Evading bot detection is a brittle topic - what works today might no longer be working tomorrow - this is the nature of the things. Due to this, the nodes are explicitly labeled as BETA and not officially advertised in our product description.
-
If you continuously restart the browser within a loop, things will become slow - so try to avoid it.
-
Based on screenshots it’s difficult to give any advice. I can have a look to rule out any bugs in the nodes - for this a need a minimized, self-running workflow which helps me to reproduce the problem. Please reduce it only to the absolutely essential nodes and make sure that I can execute it without further assumptions or configurations.
Best regards,
Philipp
1 Like
Yes, Philipp, you’re right—there are over a thousand pages 
in my loop that I need to scrape. Could I send the workflow to your email for you to review?
I just tried the workflow you sent locally - it runs fine for me since a few dozens of iterations.
When you still run into the error, can you provide the details logs and stack traces please?
1 Like
After approximately more than 40 iterations, the process fails at the “Timed WebDriver Navigate” node.