FİND ELEMENTS NODE

I want to get reviews from a web page,

My workfow contains find element , click nodes. Firstly i started workflow. There was no problem. After a while ,

Web page started to slow . find elements node takes a long time . i want to take 50 000  reviews. İs there any solution for this?

qqilihq ? have you got any solution?

Sorry, without more details it's hard to provide any help. Can you supply your workflow?

i want to get 50 000  reviews but chrome doesnt load reviews. i can get 4480 reviews .

 

Maybe you can see much better, this wokflow works good. i used different browsers but i can get maximum 4480 reviews. Google play store reviews

Soner, please attach the actual KNIME workflow, so that I can open and try locally in KNIME. You can export it from KNIME as ZIP file through the "File" menu.

okey how can i send zip file to you? 

Either use the "File attachments" function here in the forum, or send it to mail@seleniumnodes.com

Soner,

the issue is, that during scrolling the browser will simply accumulate and keep all shown reviews in the DOM. That's why after having processed several thousand reviews, the process will become very slow. I had a similar issue recently, which happened when trying to scrape all Tweets from a Twitter profile.

One solution to that issue is to manually take care of cleaning the DOM after having extracted the necessary data in each loop iteration, and thus keeping the memory consumption constant. You can do that using a JavaScript snippet, similar to the following:

var element = arguments[0];
var parent = element.parentNode;

parent.removeChild(element);

If that's of any help, I can sent you the workflow for Twitter. It will of course require some adaptation to your specific use case.

Philipp

Philipp,

Thank you for for answer. i understood the problem.Can you sent me your twitter workflow .Maybe i can manage to solve for my workflow.İf i solve the problem, i will inform you.

Soner.

Hey Soner,

yes, I'll send you the workflow in a second.

Philipp