Regex split or another solution

[sm0lda]

Hello guys,

I would like to ask you how to parse some event log with floating fields.

Normally I can do that with regex split, but the log structure needs to in regular order.

Example - part of the log:
app=US-CLI cnt=1 proto=TCP dst=192.168.149.22 dhost=test dmac=00:0c:0d:e7:00:c1

But those bold fields (with the value after “=”) could be in random order or in same cases are not there at all. How to deal with this?

It is total chaos from the vendor, but I need to deal with it somehow.

Thank you for your help.

Jiri

[bruno29a]

Hi @sm0lda , so if I read you correctly, the issue for you is that the log structure is not always the same, and the pairs of variable=value could be in random order, but if it was always the same, you can use regex split to parse the logs, correct?

If that’s the case, I can help to enforce that the logs structure is the same by re-ordering the order.

As input, I have this:

It’s based on the example that you provided:
1st line: Same as what you have
2nd line: dhost and dmac moved at the beginning
3rd line: dst moved to 2nd position

Results:

You can now use regex split as you intended to do.

Here’s the workflow: Restructure log entries.knwf (10.2 KB)

[bruno29a]

Hi @sm0lda Just in case you struggle with regex split or that the logs might also not always have all of the elements on top of being ordered randomly, which would be a problem for regex split, I put something together for parsing the logs.

As some of you would know from my past posts, I would use JSON in this case, for “dynamic” variables, as Knime would dynamically create the proper columns automatically when converting from JSON to table.

Here’s the sample logs:

The first 3 lines are the same as before. For the new 3 lines:
Line 4: dhost is missing
Line 5: dst is missing
Line 6: everything is missing except for dst

Results:

I added the workflow to parse along with the restructure one (did not want to create 2 separate files on my local):

Here’s the workflow: Restructure or parse log entries.knwf (16.0 KB)