[Selenium] Logger logs with huge delay (cache at play?)

[mwiegand]

Hi @qqilihq,

the new logger node seems to not work properly as:

1. It only creates zero byte files
   
   
   
   
   
2. Logs data with quite significant delay (only after several minutes I noticed the data)
3. Does neither clean up data upon reset nor closing Knime
   
   
4. Does not factor in temp file setting from Knime preferences
   
   

Maybe I forgot to hit F5, going to pay more attention and let you know in case I indeed made a reporting mistake.

Best
Mike

[qqilihq]

Hey Mike,

(1) + (2) - I’d never trust the Windows any GUI file manager to update sizes in anything which even gets close to realtime The logs should get updated as soon as there’s content, probably with some line buffering delay of milliseconds. On Unix/Mac you can do tail -f to see how the logs are written as soon as interactions happen in the browser (not sure about the Windows equivalent - sorry)

(3) Indeed, that can be improved

(4) True - we currently use the system temp path for intermediate storage - not sure if KNIME provides API access to the temp setting and that would be worth considering.

Best,
Philipp

[mwiegand]

Yes, the windows explorer is a nightmare. The temp path seems exposed:

PS: Knime ran for quite some time now and still no log for Chrome. I just opened that folder and refreshed it several times.

The selenium log file is almost 600 MB in size. Do you want it?

[qqilihq]

@mwiegand

The selenium log file is almost 600 MB in size. Do you want it?

That will likely not help I think.

But how about if you use other browsers - will they produce any logs? (Edge, Firefox, IE).

Point (3) and (4) are fixed in the next 0.0.1 update.

-Philipp

[mwiegand]

I am cleaning the selenium log atm but could not spot anything odd so far. Not even about the “Oh snap” of Chrome for which I wanted to open a topic sharing the thread dump … shall I?

I am trying to push a project over the line. Going to check other browsers if I find some time.

PS: Maybe there is no chrome log because of the pooled web driver node I am using since just recently?

[qqilihq]

@mwiegand

PS: Maybe there is no chrome log because of the pooled web driver node I am using since just recently?

That could be - can you check without?

I think it’s documented in the “Logger” node docs already, but here’s a general warning: The “Logger” is definitely not meant to be used in any “productive” scenarios, but rather for very targeted, isolated, and short term debugging.

The reason:

(1) Logging bears a performance overhead,

(2) The Logger node modifies global state (in Java speak, static) - i.e. settings which are applied when running the node will not only effect the current workflow, but the entire currently running KNIME instance incl. potentially already running workflows.

So, if the main reason for using the node is “feeling adventurous”: You’ve been warned about the side effects

-Philipp

[mwiegand]

Adventurous … call me “captain data-beard”

Curtesy of ChatGPT

No, in all honesty. I am using it for three reasons:

1. Scraping a nefariously annoying Ajax website (slow server is to blame too) to extract data for a client
2. Debugging the workflow to make the scraping as reliable as possible
3. Testing Selenium to support you as well

The logger node will be removed or disabled. Though, I have an idea how to keep it but conditionally en-/disable it using a debugging flag.

PS: Checking the temp dir, related to another Knime issue I reported about data not being cleaned up, I noticed a Chrome log being written from a later run after Chrome snapped.

Two suggestions:

1. Time Stamp: Convert to Date&Time format vs. ms after start (?)
2. Remove line breaks to improve processibility

Cheers
Mike