Selenium Web Automation via Java Snippet

Hi all & @knime,

tl;dr: There might be something wrong with the Java Snippet. This thread is about web automation. Tried to use Selenium (Java) which worked. Problem is that I have to restart KNIME every time I change the code in my Java Snippet.

Full story: I found this workflow which makes the web automation library Selenium available inside a Java Snippet by @Nuke_Attokurov via KNIME Hub. This is really awesome :slight_smile: It is great that KNIME Hub exists and that people share their code/workflows with the community.

I used it and it worked. But I have a problem with the fact that the Java Snippet inside the workflow relied on version 3 of Selenium, which is now deprecated. It wouldn’t matter to me, if the old docs were available, bot they are not. I only find resources for Selenium V4. I want to do specific things and without documentation I can’t make it work.

So I decided to update the workflow, even though being far from fluent in Java.

Here is the result. This workflow runs and has Selenium 4.18 on board. I downloaded the newest library code and inserted all dependencies into the Java Snippet’s additional libraries.

The workflow does what it is supposed to: Navigate to Google.de, wait, then further navigating to Spiegel.de. Then check, if the Spiegel.de was reached. Then extract the browser cookies (name & value). So far, so good.

Now my problem. I can re-run the workflow multiple times as long as I don’t change anything inside the Java Snippet. As soon as I change something, I get the following error:

WARN Java Snippet 3:4173 Evaluation of java snippet failed for row "Row0".The exception is caused by line 38 of the snippet. Exception message (SessionNotCreatedException): Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure. Host info: host: 'XX-XXXX.local', ip: 'XXXXXXXXXXXXXXXX' Build info: version: '4.18.1', revision: 'b1d3319b48' System info: os.name: 'Mac OS X', os.arch: 'aarch64', os.version: '14.2.1', java.version: '17.0.5' Driver info: org.openqa.selenium.chrome.ChromeDriver Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [], binary: /Applications/Google Chrome..., extensions: []}}]}]

I experienced this problem with the current KNIME version (5.2.1) as well as with my PROD KNIME version (4.7.4). I am using the newest Google Chrome browser (122.X) having Selenium Manager choosing the fitting chromedriver.

This question is addressed to @knime because I think, it might be a Java Snippet problem. But I welcome every answer or improvement of the workflow. Also I think that having a free alternative to SeleniumNodes is more than welcome. I also know about the new KNIME Web Interaction Nodes. I am really looking forward to using them. Right now, my use case is cookie extraction, which is not yet supported. So I tried to do it in my own. Maybe you can help me with improving the development experience: Not having to restart KNIME multiple times while failing hard at writing working Java snippets :wink:

1 Like

Hello @JLD ,

Thank you for bringing this to our attention, I have raised a ticket to address it promptly. Attaching the ticket number here for internal reference: AP-22181.

Best,
Keerthan

2 Likes

Update from my side: The problem still exists and I was not able to fix it via Java Snippet code. I found out another thing I want to share. I began using the Call Workflow Service feature. I wrapped my Java Snippet as a workflow service so that I can trigger it externally.

Let’s say I have 2 workflows which should trigger the Java Snippet: A & B

Java Snippet workflow service is closed.
Workflow A => works
Workflow B => error, need to restart KNIME

Now, if I have my Java Snippet workflow service open/loaded in the AP the behavior is different:
Workflow A => works
Workflow B => works

I think it might be a cleanup issue. Any news from your side @k10shetty1?