Hey Umut,
difficult to give any universal advice here. Passing the Cloudflare checks is definitely no easy and will always stand on shaky grounds and involve trial and error. Some ideas given in the article might work in conjunction with the Selenium Nodes. I would try as follows, starting easy:
-
Switch to a different browser (e.g. if you’re currently using the included “Chrome for Testing”, switch to Firefox to see if it makes a difference)
-
Switch off headless mode
-
Run the workflow until the challenge occurs - try to solve it manually - continue execution
-
You might see if the Solve CAPTCHA node can help with the Cloudflare challenges - I’m not sure to be honest, but it’s worth a try
-
Use rotating proxies - personally, I have recently used Crawlbase successfully for a larger crawling project. The proxies were reliable and the pricing seems fair. There’s plenty of other providers, too.
Let me know how it goes!
Best regards,
Philipp