Log File Creation during Knime Batch Execution

KNIME Analytics Platform
1

This is the simple batch code I am using to execute Knime Workflow
"C:\Program Files\KNIME_New\knime.exe" -consoleLog -nosplash -reset -noexit -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="C:\Users\knime-workspace\B11"

Suppose I required to store a log file for to my specific folder that has been generated during the execution of workflow B11 let’s say, then how to proceed with this.

It will be the great help for me if anyone guide me on this

2

Hey @adt_pranab,

what does this log file you’re talking of is supposed to store?

Kind regards,

Patrick.

3

Workflow Execution Log file, so that I can see between the execution process upto what node my process has been executed. Actually my flow consist of around 200 nodes, so I want to know after half an hour how many nodes are executed so for that I need to check my Knime workflow log

4

Ok. Got it. Check out that thread: https://forum.knime.com/t/how-to-show-the-result-in-batch-modle/10930

In summary, you need to add file writer nodes to your work flow or manage to redirect stdout to a file / console. Using Windows, I haven’t managed the second.

Kind regards,

Patrick

5

@Patrick1974 : Actually we can even see the Knime Log file by opening the Knime and View Knime Log but that log file is having some size limitation, due to which after certain size it will start clearing the line from the top, for resolving this issues I want to write log to some user defined files. For this do you have any idea how we can achieve this ?
It will be great help for us !

6

Hey @adt_pranab,

I’m not sure if using the console log is a feasible approach since the log has (a) another purpose (e.g., workflow developing and debugging) and (b) may be subject to change of behavior from release to release (i.e., it’s not under your control, hence unsuitable for using in a productive environment). Hence, my suggestions stays add additional writer nodes into your workflow.

If you, however, like to follow the path further down the aisle, have a look at knime.ini in your KNIME folder. May be there is an option helping you. You also may consider to check the logging level - maybe lowering the log level reduces the log.

image
image1220×503 131 KB

Finally I’d like to point you to this NodeLogger. Maybe it solves your challenge.

Keep me posted about your solution!

Kind regards,

Patrick.

7

I have a similar problem in my current use of the batch application from the command line. My use case is for the desktop, not the server. When I run the execution statement from the command line, the only value returned to stdout is a single 6-digit code. According to the documentation, this is the expected result. What surprised me is that there is also nothing written to the standard knime.log file that is populated when I run the same workflow from the GUI.

It appears that this is because KNIME_BATCH_APPLICATION runs as a sidecar Java program that doesn’t interact with the process that the GUI uses for logging. When ‘-consolelog’ is included in the execution, it appears that the batch application calls a separate System.Out object (from Java) and sends the logging information there. In Windows, this System.Out is handled by the Windows Terminal application. As far as I know, there is no way to capture the output of the Windows Terminal object from the command line.

Is there any easy way to re-direct the System.Out that is called by the KNIME_BATCH_APPLICATION to a log file or just get it to integrate with the same default log file used by the GUI?

My only other idea is write a Java wrapper program to try controlling the System.Out of the Java environment, but I’m not a Java expert. If I get it to work I will post here.

8

I think I finally managed to figure this out.

In your batch script (.bat file) add the following at the end of the command line. %WORKFLOW_DIR% is the path to your log4j3 config file (we’re going to modify the stock KNIME one and place it somewhere safe - in my case my workflow folder):

-Dlog4j.configuration=%WORKFLOW_DIR%\log4j3.xml

So your execution line might look something like this:

%KNIME_DIR%\knime.exe -consoleLog -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -reset -nosave --launcher.suppressErrors -workflowDir=%WORKFLOW_DIR% -preferences=%WORKFLOW_DIR%\batch_preferences.epf -vmargs -Xms8095m -Xmx12288m -Dlog4j.configuration=%WORKFLOW_DIR%\log4j3.xml

Now go and copy the KNIME default log4j file to your workflow. The file will be located in your workspace folder: .metadata\knime\log4j3.xml

Now edit the copied file so that the batchexec appender looks something like this:

<appender name="batchexec" class="org.apache.log4j.FileAppender">
    <param name="file" value="WorkFlow.log"/>
    <param name="append" value="false" />
    <layout class="org.knime.core.node.NodeLoggerPatternLayout">
        <param name="ConversionPattern" value="%d{ISO8601} : %-5p : %N : %m%n" />
    </layout>
    <filter class="org.apache.log4j.varia.LevelRangeFilter">
        <param name="levelMin" value="WARN" />
    </filter>
</appender>

Change the log level, file name and layout to your liking.

Now add the following line to the bottom of the file into the “root” section:

<appender-ref ref="batchexec" />

It should resemble the following:

<root>
    <level value="info" />
    <appender-ref ref="stdout" />
    <appender-ref ref="stderr" />
    <appender-ref ref="logfile" />
    <appender-ref ref="knimeConsole" />
    <appender-ref ref="batchexec" />
</root>

Now when you execute the batch script it will log to the log file specified.

Hope this info helps someone!

3 Likes