How to make an useragent loop in a larger loop

[Joma1000]

Hi!

How can I rotate an useragent value and put it after a counting loop?

I would like to have a new user agent assigned every time the counting loop starts over.

I made the following, but I can’t get it into the loop within the large workflow:

Thanks in advance!

[HansS]

Hi @Joma1000

If have Selenium not installed, so I can’t verify my solution. But I was thinking of something like this;

Make use of the Flow Variable “Current Itaration” created by the counting loop. Match the Useragent to this Flow Variable in the Row Filter node.

gr, Hans

[qqilihq]

Hey Joma,

(Philipp writing - we’ve just been in touch via email) Here’s an example workflow which sets a random user agent string taken from a given list within each loop iteration. To verify that it works, it will take a screenshot of a whatsmyuseragent page.

The key parts:

1. Ensure that the capabilities get recreated upon each loop iteration by moving them into the loop.

2. To set the capabilities dynamically, you can pass them as flow variable as JSON-encoded string. This is what the Java Edit Variable node is doing. The Capabilities JSON for a Chrome browser looks like this for example:

   {"chromeOptions":{"args":["--user-agent=Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.152 Safari/537.36 LBBROWSER "]}}
   

If you’d rather assign the UAs in a deterministic way, @HansS’s solution is the way to go!

By the way, instead of manually curating the UA list, if you would like this more dynamic, this could be extended to pull UA strings from a source like this using Palladian’s HTTP Retriever.

You can download the workflow from my NodePit Space:

Selenium_Rotate_Capabilities_77269 — NodePit

Hope that helps!

Best regards,
Philipp

[Joma1000]

Thanks! wanted to try it out @qqilihq but got this error at the Java Edit varable:

Bundle “org.glassfish.javax.json” required by this snippet was not found.

I think this one is not in version 5.1.0+ anymore?

Bug Notification (affects KNIME Analytics Platform 4.7.0 - 4.7.3 / 5.0.0)

Thanks @wiswedel I now recall doing the exact actions that you described before the workflow group loss on my end as well. Thanks for fixing the issue!

[qqilihq]

@Joma1000

Thanks! wanted to try it out @qqilihq but got this error at the Java Edit varable:

Ah - my sample workflow is built with KNIME 4.7 - if you’re using a different version you’ll probably need to update the JAR dependencies in that node.

Should work as follows:

1. Open the configuration of the Java node
2. Go to “Additional Bundles”
3. Remove org.glassfisch.javax.json
4. Search for jakarta.json and add this instead (this is the new namespace afair)
5. Replace javax.json with jakarta.json in the import section of the Java script

[edit] btw, stictly speaking the JSON dependency wouldn’t be needed at all and you could build the JSON as a plain string - I used the JSON lib just for pure convenience, and the pure string version would be even shorter here:

out_capabilitiesJSON  = String.format("{\"chromeOptions\":{\"args\":[\"--user-agent=%s\"]}}", v_user_agent);

[edit2] I’ve updated the example workflow as described above - you can simply re-download it and should be good to go!

[Joma1000]

Thanks for your help @qqilihq and @HansS it works now!