Read a table on a webpage that does not show up in HTML?

[cbirch]

I want to read the data in the table on this webpage:

Micro WTI Crude Oil Overview - CME Group

I have tried using Webpage Retriever and HTTP Retriever (Palladian), however the data does not show up in the HTML output at all. I’m not sure exactly how it’s loaded into the page. Are there any other methods of data scraping that can successfully read this table? Download the page to PDF automatically and read it from there? I am hoping to do this without having to pay for the internal API or data scraping tools. (Maybe Selenium could do it but I don’t want to pay.)

[sanket_2012]

Hello @cbirch ,
Could you please try to follow the approach mentioned in this thread?

Thanks,
Sanket

[cbirch]

Hi Sanket,

The Selenium nodes require an annual paid subscription to use, so that is not an option for me.

However I can access Selenium through Python for free, so I am working on a solution using this tutorial: Python Web Scraping Tutorial – How to Scrape Data From Any Website with Python

If there are any other solutions that do not require Python nodes or Selenium nodes I would be very interested to know, since it is a lot of work to install all the required software to make the Python solution work.

Thank you,
Clara Birch

[ScottF]

Someone else had a very similar question just a few days ago:

Get PDF from table on website

Hi @brandy71 - In the latest version of KNIME AP 5.1, under KNIME Labs, we released our own Selenium nodes. This is very new so we don’t have a ton of documentation on it yet, but there is an example workflow (which if you’re already familiar with Selenium, maybe you can adapt): There is also the fancy new Selenium Recording Player node, which allows you to record and save your clicks on a webpage. I haven’t tried it out myself but hope to soon. Maybe it could also help you?