Web Scraping

Hi Fabio,

afair there are some examples on the KNIME example server. Also you'll find some workflows on the Palladian KNIME community pages.

However, if you're targeting modern websites and web apps, I would strongly recommend to have a look at the Selenium Nodes. These have dedicated functionality which closer resembles a web browser (in contrast to the relatively "dumb" HttpRetriever/HtmlParser). This allows you to scrape practically any website, as you're running an actual web browser from KNIME.

We will dedicate all future developments concerning the web scraping functionality to the Selenium Nodes. (this does not mean, that the Palladian Nodes are dead -- as Palladian is much more than only web scraping.)

Best,
Philipp

PS: Tutorials on the Selenium Nodes will follow. I know that this is currently a blocker for many users.

1 Like