Web Scraping

Community Extensions
1

Hi there!

Any new examples/workflows on how to scrape websites using Palladian/Knime?

Best

2

Hi Fabio,

afair there are some examples on the KNIME example server. Also you'll find some workflows on the Palladian KNIME community pages.

However, if you're targeting modern websites and web apps, I would strongly recommend to have a look at the Selenium Nodes. These have dedicated functionality which closer resembles a web browser (in contrast to the relatively "dumb" HttpRetriever/HtmlParser). This allows you to scrape practically any website, as you're running an actual web browser from KNIME.

We will dedicate all future developments concerning the web scraping functionality to the Selenium Nodes. (this does not mean, that the Palladian Nodes are dead -- as Palladian is much more than only web scraping.)

Best,
Philipp

PS: Tutorials on the Selenium Nodes will follow. I know that this is currently a blocker for many users.

1 Like
3

Hey Philipp!

Thank you!

Looking forward to learn more about the Selenium Nodes!

Best

4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.