I used to perform web scraping with Selenium, thanks to a generous license provided during my studies. However, my license recently expired, and since I am currently seeking a job, I cannot afford a paid subscription.
I would like to continue using my workflows that rely on Selenium, and I recently discovered the free Web Interaction (Labs) extension. I am curious about the capabilities of this extension for web scraping tasks.
Specifically, I would like to know if the following actions are possible:
Logging in with username and password
Navigating through multiple pages
Applying filters
Additionally, I have encountered websites that require more advanced handling, such as:
Looping over the last element and repeating the process because the DOM only displays part of the data
Handling infinite scrolling to extract all available information
Does the Web Interaction (Labs) extension support these kinds of interactions, or is it more suitable for static pages without complex dynamic elements?
I appreciate any insights or experiences you can share.
yes, those things should be well possible with the Web Interaction extension. You can find some workflows interacting with websites on the Hub, for example:
Regarding the looping, the Clicker node offers a setting for repeating an interaction.
I tried to reproduce a workflow that requires infinite scrolling to display all elements. Unfortunately, there isn’t a “show more results” button to click. Additional results only appear when reaching the bottom of the page.
This is exactly the challenge I’m facing because, for websites like this, I used to rely on the Execute JavaScript node to scroll down to the bottom of the last element and repeat this operation. However, this node isn’t compatible with the Web Interaction extension.
As a result, I’m stuck because I can’t figure out how to scroll to the bottom of the page. It would be so much easier if there was simply a “show more results” button!
If you have any ideas on how to handle this situation, I’d really appreciate it.
Hello @Foxyellow ,
thank you for bringing up this topic. I opened a Ticket (AP-23797) for your request. We are collecting feedback regarding the extension right now, so if you have any more suggestions, please let us know here.