Building KNIME with Buckminster

miguelalba · August 14, 2018, 11:02am

Hi all,

I have looking into Buckminster and Jenkins in the last weeks and tried to follow the tutorial from Thorsten, https://www.knime.com/blog/creating-knime-update-sites-with-buckminster. While I managed to build the example from the tutorial I could not build my own nodes. I guess some extra configuration files are needed and I decided to post my installation as it may help others.

First I installed the Buckminster plugins from http://download.eclipse.org/tools/buckminster/updates-4.5. I could not find update sites for newer Eclipse releases.

Buckminster Core
Buckminster - PDE Support
Buckmnster - Git Support

The first problem with the tutorial is that the required SVN plugins are not supported anymore. Or at least the versions needed by Buckminster. I solved it by copying the example projects to a public Git repository and using the Git reader type in resource map file. The Git repository can be accessed at https://gitlab.bfr.berlin/miguel/nodes4knime and only contains the three example projects:

my.example.feature
my.example.plugin
my.example.update

The command query file is left unchanged. It even has the SVN properties which are not used now.

<?xml version="1.0" encoding="UTF-8"?>
  <cq:componentQuery xmlns:cq="http://www.eclipse.org/buckminster/CQuery-1.0" resourceMap="knime.rmap">
  <cq:rootRequest name="my.example.update"/>
  <cq:property key="svn.password" value="knime"/>
  <cq:property key="svn.user" value="anonymous"/>
  <cq:advisorNode namePattern="^my\.example\..+" sourceLevel="DESIRE"/>
</cq:componentQuery>

In the resource map file all that changes is the search path that now points to my Git repository instead of the KNIME community one.

<rm:searchPath name="my-stuff">
  <rm:provider readerType="git" componentTypes="buckminster,osgi.bundle,eclipse.feature">
    <rm:uri format="{0}/nodes4knime,{1}">
      <bc:propertyRef key="workspace.root" />
      <bc:propertyRef key="buckminster.component" />
    </rm:uri>
    <rm:property key="git.remote.uri" value="https://gitlab.bfr.berlin/miguel/nodes4knime.git" />
    <rm:property key="git.remote.name" value="nodes4knime" />
  </rm:provider>
</rm:searchPath>

With these changes now I can resolve and materialize which seems to work as it makes a builds in /tmp.

The problems come with my extension node plugins at https://github.com/SiLeBAT/FSK-Lab. The resource map seems to work and they are retrieved but Buckminster fails to build them. Maybe I skipped some Buckminster configuration file. How should I proceed? I am attaching the command query and resource map files for my plugins.

Thanks,
Miguel

fsklab.cquery.xml (270 Bytes)
fsklab.rmap.xml (1.4 KB)

thor · August 14, 2018, 12:19pm

What does “fails to build” exactly mean?

miguelalba · August 14, 2018, 12:49pm

Hi Thorsten,

Buckminster pulls the projects but then throws the following errors:

Cloning remote repository GitHub - SiLeBAT/FSK-Lab: FSK Lab into D:\buckminsterWS\FSK-Lab
ERROR [0002] : No suitable provider for component de.bund.br.knime.nodes.util:osgi.bundle was found in resourceMap file:/D:/bm-example/fsklab.rmap
ERROR [0002] : No suitable provider for component de.bund.br.knime.nodes.util:osgi.bundle was found in searchPath knime-stuff
ERROR [0002] : Rejecting provider p2(http://update.knime.org/analytics-platform/3.4[http://update.knime.org/analytics-platform/3.4]): No component match was found
Errors and Warnings
E [0002] : No suitable provider for component de.bund.br.knime.nodes.util:osgi.bundle was found in resourceMap file:/D:/bm-example/fsklab.rmap
E [0002] : No suitable provider for component de.bund.br.knime.nodes.util:osgi.bundle was found in searchPath knime-stuff
E [0002] : Rejecting provider p2(http://update.knime.org/analytics-platform/3.4[http://update.knime.org/analytics-platform/3.4]): No component match was found
TAG-ID 0002 = Query for de.bund.bfr.knime.fsklab.feature, path: de.bund.bfr.knime.fsklab.feature:eclipse.feature$1.0.0.qualifier

BTW, something weird I noticed is that next to my plugins the example projects were also downloaded but they are not included in the resource map.

thor · August 15, 2018, 7:17am

Apart from your rmap not being correct or the project not being a plug-in I cannot see any other potential problems.

Vernalis · August 15, 2018, 11:10am

One suggestion, looking at our Jenkins job and rmap is that we check out the source code for our nodes in to the Jenkins job workspace, and then use a local provider rather than the git/svn providers, e.g.:

<rm:locator
	pattern="^com\.vernalis\.knime\.internal(\..+)\.feature"
	searchPathRef="feature"
	failOnError="false" />
<rm:locator
	pattern="^com\.vernalis\.knime\.internal(\..+)"
	searchPathRef="local"
	failOnError="false" />

and

<rm:searchPath name="local">
	<rm:provider
		componentTypes="osgi.bundle,eclipse.feature"
		readerType="local">
		<rm:uri format="{0}/sources/{1}/">
			<bc:propertyRef key="workspace.root" />
			<bc:propertyRef key="buckminster.component" />
		</rm:uri>
	</rm:provider>
</rm:searchPath>

<rm:searchPath name="feature">
	<rm:provider
		componentTypes="osgi.bundle,eclipse.feature"
		readerType="local">
		<rm:uri format="{0}/sources/{1}/">
			<bc:propertyRef key="workspace.root" />
			<bc:propertyRef key="buckminster.component" />
		</rm:uri>
	</rm:provider>
</rm:searchPath>

Steve

sonnenburg · August 20, 2018, 10:32am

When I took over our knime package from a colleague I was also confronted with buckminster and found it very hard to understand. I then switched to maven / tycho, maybe it’s worth to have a look at it: https://www.eclipse.org/tycho/

Since I am using eclipse (as well as knime itself) as ide, I also find tycho much better integrated into eclipse, which makes depeloping much easier for me.

Best, Frank

miguelalba · September 7, 2018, 1:19pm

Thank you all. Sorry for answering so late. I was on holidays leave.

@thor the project de.bund.bfr.knime.fsklab.feature is not a plugin but a feature project like in the example from the tutorial, my.example.update. In this example project there are also two configuration files: bm.properties and buckminster.json. Maybe there are needed in my project?

@Vernalis thanks for your suggestion! I have seen it in other Buckminster configurations but my idea was to use the mutable property of the provider. If I am not wrong, this only pulls the changes and not the whole repository on each build.

@sonnenburg currently I a looking into Tycho but so far I have not been able to debug the Tycho build with a target as an RCP application. Following the Vogella tutorial at http://www.vogella.com/tutorials/EclipseTycho/article.html#removing-compiler-warning-messages-from-the-build I created a build for an update site but I ignore how to launch it from Eclipse. The only idea I have so far is to create a product configuration.

Best,
Miguel

gab1one · September 10, 2018, 7:56am

Hi @miguelalba,
did you have a look at https://github.com/3D-e-Chem/tycho-knime-node-archetype yet? It is a community maintained maven prototype for developing KNIME nodes. It should be relatively easy to adapt it to your needs.
best,
Gabriel

miguelalba · September 10, 2018, 9:35am

Hi @gab1one

yes I know that project. I started with it but I found it confusing since you get the whole structure at once. Following the vogella tutorial I created the same RCP project structure with the example KNIME node and uploaded to GitHub at https://github.com/miguelalba/my.example. Pretty glad with Tycho so far:

Dependencies are retrieved from a KNIME target.
Only tests I included so far are unittests. In the maven archetype they are including test workflows which I will look into.
CI with Travis. The single example node is taking 2 minutes to build and half a minute when caching the dependencies. It will be interesting to see with many extension nodes and test workflows.
I failed to launch as RCP application from a product project but succeded from the bundles project (nodes project).
I ignore how to use 3rd libraries as Maven dependencies in Tycho. There seems to be some workarounds.

Best,
Miguel