Hi guys,
what is the best practice to get the full content of a website when the elements are generated dynamically by once clicking on a show more button and after that you have infinte scroll down to get all the elements?
Regards.
1 Like
I assume the content on the website you are crawling is JavaScript-based, which makes it tricky to crawl. The standard HttpRetriever node will not work for this. You could try arbitrary crawler packages with Java, R or Python snippet nodes or give Selenium nodes (https://seleniumnodes.com/) a try.