Web scraping with Knime on infinite scroll

KNIME Analytics Platform
1

I am trying to scrape a site using Knime to get the 2024 news articles only (though some of 2025 is OK too). I scraped it last year but can’t remember how to go about scraping this year. Site is in Russian but новости is the news tag. It is here: Новости - Войсковое казачье общество Черноморское казачье войско 15. Earlier for other sites I was setting up a list of URLs to scrape (i.e. “newspage=” and then using counting loop to have the program iterate 43 times (for example) through the main directory pages for 2024, but that won’t work for infinite scroll. Any ideas?

2

Hi @drrarnold22,

depending on what extension you use:

  1. Knime Web Interaction
  2. AF Utilities
  3. Palladian Nodes
  4. Selenium Nodes (my personal favorite)

You could:

  1. Send key like space, down arrow or end
  2. Write a JavaScript to scroll down

Best
Mike

3 Likes