Hello everyone, I am currently working on a project that involves extracting information from numerous websites using the Selenium node. With thousands of URLs to process, I am facing a challenge in terms of the efficiency of my workflow. I’ve come across the Chunk Loop Start node, which seems promising for handling multiple rows simultaneously. However, when I connect it to the Webdriver and create a flow variable, it opens three webdrivers with the same URL. I believe there might be a misconfiguration as I am new to using this method. Ultimately, I aim to process three URLs simultaneously, each with its own unique webdriver instance. There is an exemple what I try to do [image] You can also check the workflow in attachment Any guidance or suggestions on how to achieve this would be greatly appreciated. Thank you in advance for your assistance. Optimization Selenium whith Chunk Loop.knwf (2.0 KB)

Thank you very much for your help. After a few tries, I managed to get what I wanted. I took inspiration from your example. I replaced the Table Row to Variable node with the Table Row to Variable Loop Start node, added a Loop End just before the Parallel Chunk End node. It seems to be working; I have 5 web drivers running and finishing, and then another 5 start, and so on. I hope my data won’t get mixed up using this technique, but in any case, it’s going to save me a considerable amount of time. I used to think I had to leave my PC running for several days to process all the URLs. [image]

Help to Optimize Workflow for Processing Multiple URLs

KNIME Analytics Platform

ArjenEX November 13, 2023, 11:14am 3

Maybe you saw this one?