Selenium Web Automation via Java Snippet

[JLD]

Hi all & @knime,

tl;dr: There might be something wrong with the Java Snippet. This thread is about web automation. Tried to use Selenium (Java) which worked. Problem is that I have to restart KNIME every time I change the code in my Java Snippet.

Full story: I found this workflow which makes the web automation library Selenium available inside a Java Snippet by @Nuke_Attokurov via KNIME Hub. This is really awesome It is great that KNIME Hub exists and that people share their code/workflows with the community.

I used it and it worked. But I have a problem with the fact that the Java Snippet inside the workflow relied on version 3 of Selenium, which is now deprecated. It wouldn’t matter to me, if the old docs were available, bot they are not. I only find resources for Selenium V4. I want to do specific things and without documentation I can’t make it work.

So I decided to update the workflow, even though being far from fluent in Java.

Here is the result. This workflow runs and has Selenium 4.18 on board. I downloaded the newest library code and inserted all dependencies into the Java Snippet’s additional libraries.

The workflow does what it is supposed to: Navigate to Google.de, wait, then further navigating to Spiegel.de. Then check, if the Spiegel.de was reached. Then extract the browser cookies (name & value). So far, so good.

Now my problem. I can re-run the workflow multiple times as long as I don’t change anything inside the Java Snippet. As soon as I change something, I get the following error:

WARN Java Snippet 3:4173 Evaluation of java snippet failed for row "Row0".The exception is caused by line 38 of the snippet. Exception message (SessionNotCreatedException): Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. Host info: host: 'XX-XXXX.local', ip: 'XXXXXXXXXXXXXXXX' Build info: version: '4.18.1', revision: 'b1d3319b48' System info: os.name: 'Mac OS X', os.arch: 'aarch64', os.version: '14.2.1', java.version: '17.0.5' Driver info: org.openqa.selenium.chrome.ChromeDriver Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [], binary: /Applications/Google Chrome..., extensions: []}}]}]

I experienced this problem with the current KNIME version (5.2.1) as well as with my PROD KNIME version (4.7.4). I am using the newest Google Chrome browser (122.X) having Selenium Manager choosing the fitting chromedriver.

This question is addressed to @knime because I think, it might be a Java Snippet problem. But I welcome every answer or improvement of the workflow. Also I think that having a free alternative to SeleniumNodes is more than welcome. I also know about the new KNIME Web Interaction Nodes. I am really looking forward to using them. Right now, my use case is cookie extraction, which is not yet supported. So I tried to do it in my own. Maybe you can help me with improving the development experience: Not having to restart KNIME multiple times while failing hard at writing working Java snippets

[k10shetty1]

Hello @JLD ,

Thank you for bringing this to our attention, I have raised a ticket to address it promptly. Attaching the ticket number here for internal reference: AP-22181.

Best,
Keerthan

[JLD]

Update from my side: The problem still exists and I was not able to fix it via Java Snippet code. I found out another thing I want to share. I began using the Call Workflow Service feature. I wrapped my Java Snippet as a workflow service so that I can trigger it externally.

Let’s say I have 2 workflows which should trigger the Java Snippet: A & B

Java Snippet workflow service is closed.
Workflow A => works
Workflow B => error, need to restart KNIME

Now, if I have my Java Snippet workflow service open/loaded in the AP the behavior is different:
Workflow A => works
Workflow B => works

I think it might be a cleanup issue. Any news from your side @k10shetty1?

[k10shetty1]

Hi @JLD,

"Correct me if I am wrong, but based on your example, you’re utilizing three workflows. Firstly, Workflow C contains capture workflow nodes to capture a specific part of your workflow, including the Java Snippet node. Then, there are Workflow A and Workflow B, which call this captured segment.

From what I understand, when Workflow C is open or loaded, both Workflow A and B run fine. However, if you close Workflow C, issues arise with Workflow B, given that both workflows utilize the same captured segment.

Does this interpretation accurately capture the setup and behaviour you’re experiencing?"

Best,
Keerthan

[JLD]

Hi @k10shetty1

not exactly. I am not using the capture workflow nodes. What you call Workflow C is a small workflow with my Selenium web automation Java Snippet and a table writer (output result).

My workflows A & B trigger this workflow. I noticed that this is possible, when C is loaded. But when C is not loaded I can only run A or B.

I hope this helps.