How to extract specific words from web search with their links

[moretzar]

Hi to all
I’m working on a literature review paper and I want to know is there any workflow to search a word like “contractual risk” in for example Google Scholar to find articles and collect search results title and links and then filter final result for more accuracy?

[qqilihq]

Hi Moretzar,

I suggest to have a look at the workflow “Search Results Scraping” on the Selenium Nodes page to get some ideas: Selenium Nodes — Sample workflows

Selenium Nodes — Put your web browser to work with KNIME

Probably you could also achieve the same with Palladian nodes as well:

Palladian — NodePit

For general intro regarding differences b/w Palladian vs. Selenium web scraping, see this thread:

Scraping an AngularJS Website Post Load

Thanks for the pointer @ScottF! @mpfeifer14 The HTTP Retriever (or the GET Request node) or any other techniques will just download the static HTML will thus not be of any help, as AngularJS requires a JavaScript environment, which is just available in a JavaScript-capable web browser. Without that, you’ll see the website as shown in an ancient browser without JavaScript (which means, placeholders instead of the actual content). So at the end this is not a question of the right XPaths, but the …

Hope this helps to get you started!

–Philipp