Enable to set a custom system property accessible from KNIME workflow

I’ve been dealing with some configuration challenges related to nested Wrapped Metanode recently which led me to an idea that it would be great if I had an option to set a system property from either command line (batch mode) or KNIME Analytics Platform.

I’ve investigated this a bit and I’ve learnt that it is impossible to set a custom system property using -vmargs -DmyPropertyName=myPropertyValue in the command the KNIME’s batch mode is started by. KNIME simply complains it doesn’t know the option you’re trying to set. Also, I haven’t found a way how to get custom system property available within a workflow while it is running in KNIME Analytics Platform.

Hi Jan,

your feature requests are always pretty special :wink: This shows to me that you are using KNIME extensively and I you are enjoying it. I guess :smiley:

Anyways can you explain what a system property is and how that affects workflow running and/or configuration of Wrapped Metanodes?

Br,
Ivan

This isn’t true, you can pass vmargs to KNIME. Using the attached workflow, running without and with (the logging line ERROR KNIME-Worker-1 ldq getProperty for 'hoopty' is are the salient lines)

without:

algebraic:~ loki$ /Applications/KNIME\ 3.7.2.app/Contents/MacOS/knime -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="/Users/loki/arbeit/knime/official_370_ws/jan_lender_batch_props/" -reset
CompilerOracle: exclude javax/swing/text/GlyphView.getBreakSpot
Jun 18, 2019 8:21:56 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensions
INFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http (122) 
...
log4j:ERROR Unexpected char [J] at position 28 in conversion patterrn.
INFO 	 main BatchExecutor	 ===== Executing workflow /Users/loki/arbeit/knime/official_370_ws/jan_lender_batch_props =====
2019-06-18 08:21:58.791 knime[76631:18336973] [JRSAppKitAWT markAppIsDaemon]: Process manager already initialized: can't fully enable headless mode.
ERROR	 KNIME-Worker-1 ldq	 getProperty for 'hoopty' is: null
INFO 	 main BatchExecutor	 Workflow execution done Finished in 2 secs (2014ms)
INFO 	 main BatchExecutor	 ============= Workflow executed sucessfully ===============
Jun 18, 2019 8:22:01 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener unregister
INFO: Removing the extensions for bundle 122
algebraic:~ loki$ 

with:

algebraic:~ loki$ /Applications/KNIME\ 3.7.2.app/Contents/MacOS/knime -nosplash -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="/Users/loki/arbeit/knime/official_370_ws/jan_lender_batch_props/" -reset -vmargs -Dhoopty="swoopty"
Jun 18, 2019 8:22:07 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensions
INFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http (122)
...
log4j:ERROR Unexpected char [J] at position 28 in conversion patterrn.
INFO 	 main BatchExecutor	 ===== Executing workflow /Users/loki/arbeit/knime/official_370_ws/jan_lender_batch_props =====
2019-06-18 08:22:10.282 knime[76636:18337244] [JRSAppKitAWT markAppIsDaemon]: Process manager already initialized: can't fully enable headless mode.
WARN 	 main Node	 Output file '/tmp/jlp.csv' exists and will be appended
WARN 	 KNIME-Worker-0 Node	 Output file '/tmp/jlp.csv' exists and will be appended
ERROR	 KNIME-Worker-1 ldq	 getProperty for 'hoopty' is: swoopty
WARN 	 KNIME-Worker-1 Node	 Output file '/tmp/jlp.csv' exists and will be appended
INFO 	 main BatchExecutor	 Workflow execution done Finished in 2 secs (2023ms)
INFO 	 main BatchExecutor	 ============= Workflow executed sucessfully ===============
Jun 18, 2019 8:22:12 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener unregister
INFO: Removing the extensions for bundle 122
algebraic:~ loki$ 

jan_lender_batch_props.knwf (8.1 KB)

2 Likes

You can add parameters which show themselves as flow variables visible throughout. The parameter needs too be called e.g.

-Dknime.constant.double.mypropname=1.2
-Dknime.constant.integer.myint=4
-Dknime.constant.string.mytext= "Hello World"

They will appear as flow variables named

knime.mypropname
knime.mint
Knime.my text

This I think undocumented feature comes from https://github.com/knime/knime-core/blob/master/org.knime.core/src/eclipse/org/knime/core/node/workflow/FlowObjectStack.java

Steve

2 Likes

You’re right quaeler, it is possible to pass system properties to KNIME workflow running in batch mode. For me, it worked as follows:
knime -nosplash
-application org.knime.product.KNIME_BATCH_APPLICATION
-workflowDir=/Users/jan/temp/“System Properties Elaboration”
-reset
-vmargs -Dcz.lender.integration.configFileLocation=/etc/knime/workflow.properties

I don’t know why did I get the error result yesterday. I must have made some mistake I can’t reproduce anymore. I ended up with eclipse error message dialog and a line in terminal that stated that:

ERROR main BatchExecutor Unknown option ‘-Dcz.lender.integration.configFileLocation’

Sorry for wasting your time.

Well, no it remains to solve how to pass vmargs to a workflow running in KNIME IDE. Is this possible?

1 Like

Thank for your point s.roughley. If I understand correctly, this does not solve my issue. Even if you pass vmargs to Workflow Variables you won’t see them within a Wrapped Metanode unless you configure the Wrapped Metanode to accept these variables (regardless of a particular way you do it).

Or, is this the way you can make Flow Variables accessible even within Wrapped Metanodes that either explicitly or implicitly filter out that particular Flow Variable?

I’ve just checked, and they are passed automatically into wrapped metanodes in the same way the the knime.workspace flow variable is.

Steve

1 Like

Aside from that great undocumented feature Steve pointed out, the workflow i attached to my post featured a Java Snippet node putting the value into a column; you could use the Java Snippet Variable to put it into a flow variable, alternatively - but i like Steve’s idea.

2 Likes

Hi @jan_lender,

with all this being said do you find your issues resolved or not? :slight_smile:

Br,
Ivan

Hi Ivan,

My idea was to make some data accessible from (possibly nested) Wrapped Metanodes even if the Metanodes don’t obtain the data from WrappedNode Input node either because their WrappedNode Input is not configured to make visible this data or their outer workflow / Wrapped Metanode doesn’t see a corresponding Workflow Variable. A particular example was a configuration parameter holding a name of properties file the Wrapped Metanode is supposed to process when all the other Wrapped Metanodes and even the workflow itself is completely unaware of the properties file and I don’t know an exact name of the properties file at design time.

Well, we figured out we can do it introducing vmargs when the workflow is invoked in batch mode. Also, we can use vmargs either to set System Properties variable or a Flow Variable which is accessible from (nested) Wrapped Metanodes consistently with what I’ve written down above.

What I still do not know how to make it possible, is an option to do the same when the workflow runs within KNIME Analytics Platform’s IDE. This remains an open issue from my point of view.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.