Recommendation for KNIME Development

Hello - I work with wbpaley and am working on refining our intellij build.working for KNIME extension plugins.

I have one big goal with the below sub bits.

1 - Build the plugin class files. (done)
2 - create the plugin .jar file. (done - I believe)
3 - deploy the plugin .jar file properly (into dropins?)
4 - run eclipse/KNIME so it reloads the plugin for testing
eventually run with remote debugging but first simply running it.
not having to reload eclipse/KNIME from scratch every time desirable to speed
up build/test cycle but this is not needed initially
(ie: dynamically reload plugin)

(repeat)

Note the above is done from the “command line” from our build system not via the Eclipse GUI. We use intellij but have it spawn our own build scripts.


so far I have #1 and #2 working. #3 maybe but can’t test,

I created the jar file to the Eclipse plugin spec for what can be put in “dropins” ( I think, eclipse wiki and google information is a bit all over the map and noisy )

Well eclipse did not load the plugin I built when it was put in the eclipse/dropins folder, and no errors are reported in the .log file on eclipse or KNIME startup.


So to back off to the simple case, I loaded the sample NumberFormatter project as is and used the

Export > Deployable Plug-ins and fragments" feature for that project and it writes out a .jar file that is supposedly proper to put in dropins to have it loaded.

I copy the resultant jar file to the “dropins” folder, and restart eclipse.

It does not load the plugin and no errors are reported

So I am now stuck on #3 as I can’t get the virgin example project to load “manually” as specified you can do by putting plugins in the “dropins” folder.

Should I start a new topic for this or continue with this thread?

A recent “primary doc” I have been looking at in terms of using jar files and dropping/copying them to the “dropins” folder. Note via filesystem copy but not through Eclipse UI.

https://wiki.eclipse.org/Equinox/p2/Getting_Started#Dropins

I have the Eclipse and KNIME installation as instructed in the knime github doc. It works and I can create the NumberFormatter project and use it as instructed in the posts above.

Peter K

Seems like this deserves its own thread.

What does the OSGI console say about the state of your plugin?

2 Likes

How does one open an osgi console with the Eclipse/KNIME dev kit install ?
And once open, the command to get the plugin state ?
I thought Eqinox etc was part of the base out of the box Eclipse RCP install ?

This is turning into a whirling dervish of loosely related topics; to hopefully curtail the expansion:

  • Equinox is part of Eclipse, though i don’t know why this came up
  • i was suggesting interrogation through the console of a release version of KNIME, but you’re bringing up the dev kit install. This makes me think you’re doing something involving dropins and launching KNIME from Eclipse?
  • rather than me giving you the command (ss,) which is going to lead to more questions about what the command response means, it would be more fruitful to learn how to use the console yourself.

Equinox is the base for osgi which is the basis for Eclipse Bundle dependency checking and loading - but some posts hint that the console is a separate plugin one must manually install but it appears to have been a moving target with different releases of Eclipse so knowing what is “right” is important…

Neither eclipse or Knime show any information about the plugin in their plugin lists or via the search feature in the “Plugin Repository”, and no errors were reported to any of the log files.

I tried it with and without the “-clean” argument when invoking eclipse.

True on using the console and having a pointer to a “man page” or docs about it.
At this point I do not even know how to launch one from (the latest) eclipse or knime.

A number of posters out there (stack overflow etc) think the osgi console has been removed from later eclipse.releases.

I’ll do some cliff notes to try to salvage a Saturday… as example:

algebraic:knime-workbench loki$ /Applications/KNIME\ 4.0.1.app/Contents/MacOS/Knime -console 1234
algebraic:~ loki$ telnet localhost 1234
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
osgi> ss org.knime.core
"Framework is launched."


id	State       Bundle
364	ACTIVE      org.knime.core_4.0.1.v201908131254
365	ACTIVE      org.knime.core.data.uritype_4.0.0.v201905311239
366	ACTIVE      org.knime.core.pmml_4.0.0.v201905311127
367	ACTIVE      org.knime.core.ui_4.0.1.v201908131226
369	STARTING    org.knime.core.wizard_4.0.0.v201906032109
444	ACTIVE      org.knime.core.util_5.12.1.v201909242005
osgi> 
telnet> quit
Connection closed.
algebraic:~ loki$ 

You keep talking about Eclipse in addition to KNIME knowing about your plugins. If you’re building these things over via IntelliJ (or IntelliJ + stuff, but not using Eclipse as your IDE,) i don’t know why you keep bringing Eclipse up. Build the plugin jar, drop it in your target KNIME release, run KNIME.

Im sorry for my Eclipse/Knime noob perspective.

I Thank you for the info on using the osgi console to the knime/(eclipse library) runtime environment, which from your cliff notes appears to be a server knime runs if given the host and port arguments. I will check it out and look for documentation on the console. And try to find a doc that explains how to run knime with the console argument on a windows machine via a shell script or by modifying the “launcher” file.

FYI if interested - what led me to here.

I followed the setup instructions for installation of eclipse and then knime inside of eclipse, for a windows PC installation.

The main way I am aware of to launch knime at present is via eclipse Project Explorer, using the org.knime.sdk.setup project folder added as part of the knime installation.by right clicking on the KNIME Analytics Platform.launch file and selecting “Run as -> 1 Knime Analytics Platform” context menu entry.

My plugin I built outside of eclipse, and copied using the file system outside of eclipse/knime to eclipse/dropins folder did not show up in the “Node Repository” area of Knime.after closing and restarting both eclipse and knime.

So to investigate and eliminate variables from my non-eclipse build and just test how to deploy a plugin via copying to the dropins folder, I used the knime NumberFormatter example as is, imported the project to eclipse in my eclipse workspace, built it and exported the plugin .jar file and saved it… ( It does work if I leave the project there as it is automagically loaded internally and I want to test explicit use of dropins folder deployment )

I then removed the NumberFormatter plugin project from eclipse using the eclipse UI, and closed eclipse.

then I copied the plugin jar file to the plugins folder, and launched eclipse again with the “-clean” argument , then launched knime. The NumberFormatter plugin did not show up in the Node Repository window/list.

I could not find any reasonable docs for what to do next or why it did not work so asked here.

Thanks for your help - I will continue searching for a pathway for using the eclipse dropins folder ( or another folder if knime uses a different path ) to dynamically “deploy” by external script a knime plugin jar file

Thanks Peter K.

(No apologies necessary!)

Some pedantry and general thoughts:

  • technically the OSGI telnet console is something that Eclipse is supplying and KNIME gets for free
  • unless you’re actually working on the KNIME application itself (for example, you’re writing new functionality for the application’s internal table storage to be used across the application) i would recommend to, and have in the past when i was doing only node and plugin development to have internal users install into KNIME, treat KNIME as a black box.
    • Knowing that we would deploy KNIME version X internally, i would develop my node in Eclipse, build the jar using the Eclipse plugin editor, put the jar in the dropins folder of a local KNIME-version-X-downloaded-from-knime-dot-org, and then run that KNIME just like i was launching any other application on my machine; for debugging purposes, i changed the knime.ini to launch the application allowing a remote debugger to attach and then attached to it from my Eclipse.
  • it’s super-bad-juju to mess directly with the content in plugins folder of an Eclipse RCP
2 Likes

Thanks - hopefully I’m not being too high maintenance - just finished a week of family holiday madness

I agree with the pedantry and it is what I had determined from the docs. /plugins is managed and one should only go through the eclipse manager library ( or eclipse UI ) to do things there,

Placing plugins in dropins from the docs I saw is the way to do it and it is supposed to “just work”.

I built the NumberFormatter example, exported the .jar file, and placed the result in dropins, and on my install it did not show up in KNIME when I re-launched it. The jar file appeared to have the right manifest, contents, etc. I didn’t touch the example project so I assume it is supposed to be correct.

Next step for me to sus it out is use the osgi console which KNIME conveniently loads and runs at startup. And try to determine why it is not loading.

Being on a windows 10 box ( I do have mingw/portablegit bash installed ) I need to know what command line script is needed to run KNIME so I can add the -console and port# arguments and get to the osgi console.

Can one derive a startup script (or export) from a “.launch” file ie: “KNIME Analytics Platform.launch” ?

thanks
Peter K.

Again, don’t launch KNIME from Eclipse (and don’t put plugins in the Eclipse dropins folder and then launch KNIME from Eclipse, if that’s what you’re doing.)

Download KNIME 4.0.2 from knime.org, put your built jar in that dropins folder, and if you need the console, you just cd into that directory and run knime.exe from there.

1 Like

Interesting- Brad and I did what I guess most people would do:

found the link Create a New KNIME Extension: Quickstart Guide

To set up your KNIME SDK, we start with an “Eclipse IDE for RCP and RAP Developers” installation (this version of Eclipse provides tools for plug-in development) and add all KNIME Analytics Platform dependencies. In order to do that, please follow the SDK Setup instructions. Apart from giving instructions on how to set up a KNIME SDK, the SDK Setup will give some background about the Eclipse infrastructure, its plug-in mechanism, and further useful topics like how to explore KNIME source code.

Then went to:

and followed the instructions for installing eclipse with the KNIME sdk which includes installing all the dependencies (managed by eclipse as eclipse bundles/plugins) into the workspace.

No mention about alternative or additional installs, and the SDK install I did does not include an executable launcher for knime :slight_smile:

The KNIME that install created is version 4.1.0

Then I would wonder -
Are there any potential conflicts between the two?
Do I need to uninstall the other one?
Or does the other installer update or overwrite the later version install ?
Do I need it as a prerequisite as it loads all the dependencies and jar files that I need to build the plugin into the eclipse workspace ?
Or do I use both - one for maintaining the SDK build prerequisites, and the other for testing ?

Anyway will archive the old install and try the other one. :slight_smile:

Yes - it’s on the to do list to make a better version of this document in the first month or two of the new year with @gab1one.

The problem here, though, is you’re mixing development worlds by using IntelliJ and an outside plugin build. You coooouuuulllldddd do just node development like the document says - use Eclipse, make a new plugin project, launch KNIME from the IDE which then embeds all your projects into the running KNIME - so you’d see your node, and whatever else. (Even then, i stick to my version that it is more sane to treat KNIME as a black box application on your desktop if you’re doing only nodes.)

… but this doesn’t work for building a plugin jar elsewhere and then getting it to show up when running KNIME from Eclipse (and there’s really no reason to run KNIME from Eclipse - it’s just become a big rube-goldberg-application launcher at this point.)

There’s no conflict between having multiple application versions of KNIME on your desktop (unless they all share the same workspace, in which case a newer version might write stuff that the older version doesn’t deal with error-free.) The in-your-IDE-version-of-KNIME can be whatever frankenstein mix of versions that happen to work - it has no effect on your ‘normal’ desktop installs of KNIME (workspace sharing caveat of course.) I not sure whether this answered all of your conflict / install / uninstall questions…

It seems you keep talking about using the Eclipse IDE for something, but i thought that you were writing your plugin code in IntelliJ and building it with something (mvn? not Eclipse though, i’m under the impression) - so i’m still confused as to why you’re bringing this up.

I can see you confusion probably my not expressing it obviously.

I tried building my own plugin with our own build. But the plugin jar file failed to load, so to eliminate variables for testing deployment only I decided to back off and use the provided sample “NumberFormatter” as is, build it in eclipse as instructions were deigned for, assuming that should work, and then try to deploy it eliminating any variables caused by our build outside of eclipse. For this “deployment test” using eclipse.

when doing so and using the Export > Deployable Plug-ins and fragments on the eclipse project which exports a jar file of the right type to be deployed as a plugin I assume.

It did not load with or without the -clean argument to eclipse/knime either when launched by eclipse UI or launched by the shell script you provided to Brad (with or without the -clean argument)

I did some poking at eclipse docs and saw that by adding -console localhost:8099 to the knime launcher shell script, or to the eclipse launch the osgi console server was run, and I could see the plugin status report. In all cases osgi did not even list the numberformatter plugin I put in dropins.

I next installed the generic non-sdk knime as you suggested and dropped the plugin jar file I created with eclipse-rcp using the knime plugin wizard for your NumberFormatter example by importing the git cloned plugin project as done purely in Eclipse above.

I dropped that jar file into the new generic knime 4.0.1 install “dropins” folder and launched knime.exe with the -console host:port option and could connect to the osgi console. It still did not load the plugin and it was not in the osgi “ss” listing after knime launched, but knime startup reported a bunch of loader errors. (below)

ERROR Worker-0 RepositoryManager Node org.knime.examples.numberformatter.NumberFormatterNodeFactory’ from plugin ‘org.knime.examples.numberformatter’ could not be created: Can’t load factory class for node: org.knime.examples.numberformatter.NumberFormatterNodeFactory The corresponding plugin bundle could not be activated!
java.lang.IllegalArgumentException: Can’t load factory class for node: org.knime.examples.numberformatter.NumberFormatterNodeFactory
at org.knime.workbench.repository.RepositoryFactory.createNode(RepositoryFactory.java:121)
at org.knime.workbench.repository.RepositoryManager.readNodes(RepositoryManager.java:386)
at org.knime.workbench.repository.RepositoryManager.readNodes(RepositoryManager.java:355)
at org.knime.workbench.repository.RepositoryManager.readRepository(RepositoryManager.java:171)
at org.knime.workbench.repository.RepositoryManager.getRoot(RepositoryManager.java:644)
at org.knime.workbench.repository.RepositoryManager.getRoot(RepositoryManager.java:657)
at org.knime.workbench.workflowcoach.ui.WorkflowCoachView$3.run(WorkflowCoachView.java:268)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)
Caused by: org.eclipse.core.runtime.CoreException: Plug-in org.knime.examples.numberformatter was unable to load class org.knime.examples.numberformatter.NumberFormatterNodeFactory.
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.throwException(RegistryStrategyOSGI.java:194)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:176)
at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:905)
at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:55)
at org.knime.workbench.repository.RepositoryFactory.createNode(RepositoryFactory.java:118)
… 7 more
Caused by: java.lang.ClassNotFoundException: An error occurred while automatically activating bundle org.knime.examples.numberformatter (444).
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:112)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:529)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:328)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:368)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:446)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:564)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:174)
… 11 more
Caused by: org.osgi.framework.BundleException: Error loading bundle activator.
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:721)
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:933)
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309)
at org.eclipse.osgi.container.Module.doStart(Module.java:581)
at org.eclipse.osgi.container.Module.start(Module.java:449)
at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:468)
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:103)
… 21 more
Caused by: java.lang.ClassNotFoundException: org.knime.examples.numberformatter.NumberFormatterNodePlugin cannot be found by org.knime.examples.numberformatter_1.0.0.201911292318
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:762)
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:716)
… 27 more
ERROR Worker-0 RepositoryManager Node org.knime.examples.numberformatter.NumberFormatterNodeFactory’ from plugin ‘org.knime.examples.numberformatter’ could not be created: Can’t load factory class for node: org.knime.examples.numberformatter.NumberFormatterNodeFactory The corresponding plugin bundle could not be activated!
java.lang.IllegalArgumentException: Can’t load factory class for node: org.knime.examples.numberformatter.NumberFormatterNodeFactory
at org.knime.workbench.repository.RepositoryFactory.createNode(RepositoryFactory.java:121)
at org.knime.workbench.repository.RepositoryManager.readNodes(RepositoryManager.java:386)
at org.knime.workbench.repository.RepositoryManager.readNodes(RepositoryManager.java:355)
at org.knime.workbench.repository.RepositoryManager.readRepository(RepositoryManager.java:171)
at org.knime.workbench.repository.RepositoryManager.getRoot(RepositoryManager.java:644)
at org.knime.workbench.repository.RepositoryManager.getRoot(RepositoryManager.java:657)
at org.knime.workbench.workflowcoach.ui.WorkflowCoachView$3.run(WorkflowCoachView.java:268)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56)
Caused by: org.eclipse.core.runtime.CoreException: Plug-in org.knime.examples.numberformatter was unable to load class org.knime.examples.numberformatter.NumberFormatterNodeFactory.
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.throwException(RegistryStrategyOSGI.java:194)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:176)
at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:905)
at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:55)
at org.knime.workbench.repository.RepositoryFactory.createNode(RepositoryFactory.java:118)
… 7 more
Caused by: java.lang.ClassNotFoundException: An error occurred while automatically activating bundle org.knime.examples.numberformatter (444).
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:112)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:529)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:328)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:368)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:446)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:564)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:174)
… 11 more
Caused by: org.osgi.framework.BundleException: Error loading bundle activator.
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:721)
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:933)
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309)
at org.eclipse.osgi.container.Module.doStart(Module.java:581)
at org.eclipse.osgi.container.Module.start(Module.java:449)
at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:468)
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:103)
… 21 more
Caused by: java.lang.ClassNotFoundException: org.knime.examples.numberformatter.NumberFormatterNodePlugin cannot be found by org.knime.examples.numberformatter_1.0.0.201911292318
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387)
at org.eclipse.osgi.internal.framework.BundleContextImpl.loadBundleActivator(BundleContextImpl.java:762)
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:716)
… 27 more

At this point I guess either the plugin jar is invalid, or some compile time versionitis problem between the knime-eclipse-sdk wizard, the sdk jars versions I have (nightly build) and the generic knime runtime.

PK

Ok - thanks for the clarifications. Thoughts (& pedantry) :

  • (i didn’t provide a launch script to anyone… ? )
  • you mentioned the ‘built the plugin jar’ before the step where you launched KNIME from Eclipse - so just to be clear, building the plugin jar is not needed if you are developing in Eclipse and not treating KNIME as a black box. The facet of having your plugin project be in the project explorer will suffice in having the plugin exist for KNIME when it is running. Presumably you cannot see it for the same reason you can’t see it when building a jar and doing the black-box-4.0.1 route - which is …
  • the bundle load exceptions… playing Kreskin, i would guess that there is something specified in the plugin manifest pointing to a class or package which is incorrectly named ? (though i would have thought the Eclipse plugin/fragment builder would have had pre-flight checks for that…) could you attach the built jar to this thread?

Yes it WILL Load the pluigin if I leave the project in place. BUT I am testing deployment so I was using it to build the deployable jar file not for runtime.

I used this for instructions on deployment.

https://docs.knime.com/2019-06/analytics_platform_new_node_quickstart_guide/index.html#_option_2_dropin

I the deleted the project from eclipse as I was testing to see of the jar file was deployable.

I closed eclipse after deleting the project and it’s source from my workspace.

I then dropped the jar file into the dropins folder as stated in the prior message.

I then tried invoking knime in the various ways I knew about ( via eclipse or a shell script I was provided), it did not load the plugin via any of them.

When I tried using it in the generic 4.0.1 non-sdk version you suggested for testing deployment it displayed startup errors and did not load the plugin.

PK

I would recommend not dropping it into your IDE’s dropins hoping that it shows up when the Eclipse-launched-KNIME is launched; stick to the blackbox for built plugin jars.

Please attach the jar to the thread.

myPlugin.zip (7.6 KB)

ok jar file built with eclipse inside a zip

Note I am using Java\jdk1.8.0_112 for runtime and builds.

Ok - thanks - will take a look.

(more pedantry: KNIME ships with its own JRE (which is Java8 currently) - so it doesn’t use whatever JDKs you may have installed on your machine.)

so if built with Java\jdk1.8.0_112 should be no problem.

Yep! (and a bunch of babble here to meet the 20 char minimum on posts…)