Web Scraping

Hi there!

Any new examples/workflows on how to scrape websites using Palladian/Knime?

Best

Hi Fabio,

afair there are some examples on the KNIME example server. Also you'll find some workflows on the Palladian KNIME community pages.

However, if you're targeting modern websites and web apps, I would strongly recommend to have a look at the Selenium Nodes. These have dedicated functionality which closer resembles a web browser (in contrast to the relatively "dumb" HttpRetriever/HtmlParser). This allows you to scrape practically any website, as you're running an actual web browser from KNIME.

We will dedicate all future developments concerning the web scraping functionality to the Selenium Nodes. (this does not mean, that the Palladian Nodes are dead -- as Palladian is much more than only web scraping.)

Best,
Philipp

PS: Tutorials on the Selenium Nodes will follow. I know that this is currently a blocker for many users.

1 Like

Hey Philipp!

Thank you!

Looking forward to learn more about the Selenium Nodes!

Best