Problem Scroll down with Execute JS node (Selenium suite)

[Chris_88]

Test uniqlo.knwf (27.9 KB)
Hi everyone,

As part of a project, I want to analyse and retrieve information from the following html page with the selenium node:

Pantalons Homme | Pantalons slim, stretch & plus | UNIQLO FR

There are 114 results. But if I try to retrieve the information with the find element node, I only get the first 12 results.

I introduced a javascript execute node to scroll down the page ( window.scrollTo(0, document.body.scrollHeight)

It works but I can still only get the first 12 results.

I tried afterwards to scroll down the page step by step with the code with the following execute js node :
function getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
getElementByXpath("/html/body/div[1]/div[3]/div[2]/article[12]/div[3]/div[1]/span/a").focus()

I had fun manually changing the value of the article tag with X = 12; 24;36… to reach the bottom of the page. It works well but I can’t introduce a loop to automate the change of value of this tag.

Please, Does anyone have an idea?

Thanks in advance,

Best Regards,

Chris

[victor_palacios]

Unfortunately, I can’t download those nodes from the knime hub, but I noticed your workflow doesn’t have an end for the loop.

Please see this workflow jKi-5 – KNIME Hub which seems similar to what you are trying to accomplish.

[Chris_88]

[quote=“Chris_88, post:1, topic:40803”]

Hi victor,

Thank you for your answer.

I know that I did not put the end loop node and the workflow.

My problem is that in the execute js node, I want to change the html tag in each loop.

1st loop :

getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
getElementByXpath("/html/body/div[1]/div[3]/div[2]/article[12]/div[3]/div[1]/span/a").focus()

2nd loop

getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
getElementByXpath("/html/body/div[1]/div[3]/div[2]/**article[24]/**div[3]/div[1]/span/a").focus()

3rd loop:
getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
getElementByXpath("/html/body/div[1]/div[3]/div[2]/article[36]/div[3]/div[1]/span/a").focus()
.
.
.
n th loop :

getElementByXpath(path) {
return document.evaluate(path, document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue;
}
getElementByXpath("/html/body/div[1]/div[3]/div[2]/article[X]/div[3]/div[1]/span/a").focus()

Do you have any idea how to proceed?

Thank you in advance

Best regards,

Chris

[victor_palacios]

Have you tried column expression? It also uses javascript and you can introduce flow vairables:

[Chris_88]

Thank you very much Victor. I will look at your solution.

Thanks again for your help

Best regards,

Chris