Dear All,
it’s over one year that i’m using (probably misusing) Knime for elaborate log in variuous text form (i.e. custom) from a lot of different application.
Knime is very usefull to correlated different sources using for example timestamp as cornerstone/linchpin and pivoting in order to give me a pattern of the event.
Sometimes i faced out with log file that mix
1)a well structured line (i mean date, time, code, description, ended by CRLF) (in blue)
2)with some following lines (one or more) with only description or something like that (in red)
3)followed by other well structured lines (in blue)
In first instance i’m reading them with a text reader node in a one column without apply any kind of logic (delimiter or fixed size cell) and eventually using a row filter to exclude point 2.
But it may be interesting to
A)let unchanged the lines of point 1 exclued the last one
B)take the last one line of point 1 and concatenate all the lines of point 2 in order to have a well structured line with a very long description and eventually in a next node splitting the content using some logic
C) then let unchanged the lines of point 3
Before open this discussion i evaluated forum, blog and other article but i don’t find anything that i can use.
I suppose that also other people faced with same issues and not strictly related to log management; have you some suggestion for a workflow that can achieve this task?
Thanks.
Kind Regards
Hi @fscardellato and welcome to the KNIME forum,
Are you looking for something like this:
26293-1-1.knwf (21.8 KB)
1 Like
Dear armingrudd,
thanks for prompt reply.
ASAP i’ll have some result i’ll update you all.
Thanks again
BR
F.
Dear armingrudd,
I tested it.
Yes, it works!
Nice also the backend logic, probably i’ll reuse it also in other circumstances.
Thanks a lot
BR
F.
1 Like
Dear armingrudd,
just a little further considerations.
It’s frequent for me manage large log files (txt about 100Mbyte and more also 500 Mbyte).
When i’m analyzing one of 125 Mbyte all it’ ok; but during saving an generic error has been risen and save it’s impossible.
I retried with new one workflow, execute, saved; but on the openning…
*** Welcome to KNIME Analytics Platform v4.1.3.v202005121100 ***
*** Copyright by KNIME AG, Zurich, Switzerland ***
Log file is located at: C:\Users\xxxxxxxx\knime-workspace.metadata\knime\knime.log
WARN GroupBy 0:13 Table spec file is 136 MB large - this may result in increased memory consumption (path ‘C:\Users\xxxxxxxx\knime-workspace\YYYYYY\20200903_log2\GroupBy (#13)\port_1\spec.xml’)
WARN WorkflowCoachView Could not update node recommendations statistics.
ERROR LoadWorkflowRunnable Errors during load: Status: Error: 20200903_log2 0 loaded with errors
ERROR LoadWorkflowRunnable Status: Error: 20200903_log2 0
ERROR LoadWorkflowRunnable Status: Error: Unable to load settings for node with ID suffix 14: Unable to read settings file C:\Users\xxxxxxxx\knime-workspace\YYYYYY\20200903_log2\String Manipulation (#14)\settings.xml
ERROR LoadWorkflowRunnable Status: Warning: Cell Splitter 0:15
ERROR LoadWorkflowRunnable Status: Warning: State has changed from CONFIGURED to IDLE
ERROR LoadWorkflowRunnable Status: Warning: State has changed from CONFIGURED to IDLE
I tried several times with same result, as long i limit the number of row in the file reader node; so i imagine that it could be matters of file size.
In this scenario have you further suggestions?
Thanks
BR
F.
What’s the error message when you are saving the workflow?
Try changing memory policy settings in nodes’ configurations from memory to disc and see what happens.
Hi,
unfortunately i tried to replicate the error on saving but…
In any case changing the configuration from “Memory” to “Disk” saving + opening + resetting and saving again works.
Thanks again.
BR
F.
1 Like
