How to Create a Single Update Site for Multiple Pure-Python Extensions?

DerMaxdorfer · September 17, 2023, 7:10am

Hi KNIMErs!

When I currently bundle my pure-Python extension for KAP 5.1 and add them as a local update site, this is what I see in in the Install KNIME Extensions... dialog:

KNIME & Extensions
|_ ...
KNIME Big Data Extensions
|_ ...
...
[My extension]
|_ [My extension]
...
More extensions

As you can see in the excerpt of the extension tree above, my extension automatically appears in a separate section (Great!) and has the same name as the section that it is grouped under (Not so great!). Their shared name seems to be taken from the line description of the knime.yml file.

Now, what I am after is rather something like the following to better accommodate future extensions of my own:

KNIME & Extensions
|_ ...
KNIME Big Data Extensions
|_ ...
...
[My organization]
|_ [My extension 1]
|_ [My extension 2]
...
More extensions

Here go my two questions:

How can I bundle multiple pure-Python extensions into a single update site?
How can I define where they appear in the extension tree?

Cheers!

steffen_KNIME · September 18, 2023, 6:45am

Maybe @gab1one has an idea how to put them together.

gab1one · September 19, 2023, 12:01pm

Hi @DerMaxdorfer,

You want to create a composite update site, we have an example here:

In your case, you need to define the update sites created by the individual extension builds as repositories in the pom.xml file and then arrange them using the category.xml file.

best,
Gabriel

DerMaxdorfer · September 21, 2023, 10:48am

Thanks for the example, @gab1one!

I understand your .xml files, but am not very clear on where to include them and how to merge multiple extensions into a single update site.

My bundled pure-Python extension has this folder structure:

[my extension]
|_ features
   |_ [group_id].features.[name]_1.0.0.202309150750.jar
|_ plugins
   |_ [my extension].channel.bin.linux.x86_64_1.0.0.202309150750.jar
   |_ ...
|_ artifacts.jar
|_ artifacts.xml.xz
   |_ artifacts.xml
|_ content.jar
|_ content.xml.xz
   |_ content.xml
|_ p2.index

What should the combined folder structure look like?

Option 1

Combine extensions as-is in a common parent folder:

[my composite update site]
|_ [my extension 1]
   |_ ...
|_ [my extension 2]
   |_ ...
|_ category.xml
|_ pom.xml

Option 2

Copy features and plugins into joint folders:

[my composite update site]
|_ features
   |_ [group_id].features.[name 1]_1.0.0.202309150750.jar
   |_ [group_id].features.[name 2]_1.0.0.202309150750.jar
|_ plugins
   |_ [my extension 1].channel.bin.linux.x86_64_1.0.0.202309150750.jar
   |_ [my extension 2].channel.bin.linux.x86_64_1.0.0.202309150750.jar
   |_ ...
|_ artifacts.jar
|_ artifacts.xml.xz
   |_ artifacts.xml
|_ content.jar
|_ content.xml.xz
   |_ content.xml
|_ p2.index
|_ category.xml
|_ pom.xml

If it is option 2, I assume that the files artifacts.jar, artifacts.xml, content.jar, content.xml, p2.index, which are generated automatically during bundling, require some modifications too. How should I go about this?

gab1one · September 21, 2023, 12:56pm

Hi @DerMaxdorfer,

The idea with the provided example is as follows, you create your custom extensions as before, lets call them extension1 and extension2.
For distribution, you now create a third project (composite), based on the linked example, you configure it so that it knows about extension1 and extension2:

you replace the url here with the path to the output folder of extension1 https://github.com/knime-community/composite-update-site-example/blob/master/pom.xml#L23
you replace the url here with the path to the output folder of extension2 https://github.com/knime-community/composite-update-site-example/blob/master/pom.xml#L28
You add the features of extension1 and extension2 here: in the category.xml https://github.com/knime-community/composite-update-site-example/blob/master/category.xml#L11

Now you can create the composite update site by running mvn package in the composite project. This will pull the features and put them into a joint update site.

DerMaxdorfer · September 22, 2023, 6:45am

I see what you are getting at, @gab1one. The Java world really is not my home turf, so I have got a few more questions if you do not mind.

In the properties node of pom.xml, you specify a version for KNIME and Tycho. Your example is for KAP 5.1 and the Tycho version that it uses. I assume that this will change in future KAP releases. Is there a way to see for myself which Tycho my KAP requires?
Where do I create the composite project in and run the command mvn package from? Do I need to install any additional software on my Windows machine, or can I simply point the knime-extension-bundling conda package to the location of my category.xml and pom.xml?
Will the joint update site contain full copies of the features of extension1 and extension2 or only references to them? Put differently, will it be enough if I give my end users access to the joint update site only, or do they need to be able to access the original extensions as well because the “pulling in” happens every time when their installation is triggered?

gab1one · September 22, 2023, 2:57pm

Hi @DerMaxdorfer,

The Tycho version is not so important, tycho is a maven plugin to build eclipse based software with maven. In this case, we only use the update site generation portion of it, which should work for the foreseeable future.
You need to have maven + java installed on your computer to run this. You can create the project anywhere you want, it is not related to the knime-extension-bundling conda package.
Yes, it will contain everything needed to install your extensions.

best,
Gabriel

DerMaxdorfer · September 25, 2023, 9:06am

Thanks, @gab1one! I will try it out once I have gotten Java and Maven installed on my company’s laptop. This will take some time unfortunately. I will let you know here if it worked for me.

DerMaxdorfer · September 29, 2023, 10:11am

Hi @gab1one!

While still waiting on the installation, one thing has kept me thinking:

In your pom.xml, you specify the download URLs of your update sites’ repos. These URLs could easily be replaced with the one of a public GitHub resource, right?

Unfortunately, I am not allowed to have public repos in my corporate environment, but I really do not want to manually download my extensions to my local machine before I can bundle them in a composite update site. I therefore created this one-liner that can be executed from the Windows command line:

FOR /F "usebackq tokens=*" %A in (`curl -L -H "Accept: application/vnd.github+json" -H "Authorization: Bearer <TOKEN>" https://api.github.com/repos/<OWNER>/<REPO>/releases/latest | jq '.assets[] | select(.name == "<EXTENSION NAME>.zip").id'`) DO curl -OJL -H "Accept: application/octet-stream" -H "Authorization: Bearer <TOKEN>" https://api.github.com/repos/<OWNER>/<REPO>/releases/assets/%A

Here are a few more details about it:

My extension’s update site lives in a non-public GitHub repo as a release asset.
The command consists of two cURL requests against the GitHub API: the first one to get its asset ID from the latest release, the second one to download said asset to the current directory.
The first cURL call requires jq to be installed so that the returned JSON can be filtered for the asset ID by the asset name.
The FOR DO loop is necessary to squeeze it all into a one-liner.

My question is the following:
How can I run this command from within the pom.xml? I read about the exec-maven-plugin, but this seems to go into the <plugins> section, i.e., I cannot simply substitute the <url> tag of each <repository> with it.

If it is even possible, could you please extend your example to accommodate this use case too?

If you recommend a different approach, feel free to share it!

gab1one · October 2, 2023, 2:08pm

Hi @DerMaxdorfer,

I assume your update site is a ziped archive? In that case, you will need to create a shell script that downloads and extracts those zip files to a predefined location, which you reference in the url section of the repository definition.
You could look into triggering that script directly in the maven build. In that case, you will need to tell the exec-maven-plugin to run early enough in the build. This way tycho will not complain that there is not yet anything at the location you will extract the update sites to.

To do so modify the <phase> option in the execution definition, take a look at this example:

github.com

knime/knime-product/blob/master/org.knime.update.product/pom.xml#L65


      
          					<finalName>org.knime.update.product_${qualifiedVersion}</finalName>
          				</configuration>
          			</plugin>
          			<plugin>
          				<artifactId>exec-maven-plugin</artifactId>
          				<groupId>org.codehaus.mojo</groupId>
          				<version>1.6.0</version>
          				<executions>
          					<execution>
          						<id>Fix Windows launcher</id>
          						<phase>package</phase>
          						<goals>
          							<goal>exec</goal>
          						</goals>
          						<configuration>
          							<executable>${project.basedir}/fix-windows-launcher.sh</executable>
          						</configuration>
          					</execution>
          				</executions>
          			</plugin>
          		</plugins>

In your case consider seeting this to the validate phase, which is the first in the build (Maven – Introduction to the Build Lifecycle)

FYI, you do not need to install maven or the jdk system wide to get started experimenting. Both tools work fine if you just download the archives, extract them and add them to your PATH environment variable.

best,
Gabriel

DerMaxdorfer · October 10, 2023, 3:01pm

Hi @gab1one,

I finally have the JDK and Apache Maven on my machine! My first attempt was in fact to install it only for my local user, but for some reason, the installer would not let me outsmart it.

Thanks for the additional information on how to use the command line with Apache Maven. You guessed correctly that my update site is a zipped archive. The reason behind this is that “normal” directories cannot be attached to GitHub releases as assets. Given the extra extraction step needed, I replaced my one-liner with a full-fledged Python script in the end to improve readability – and also leave the door open for concurrent/parallel downloading in the future if necessary.

With everything set up now, I ran mvn package, but it failed:

C:\Users\[MY USER]\Desktop\KNIME_Update_Sites>mvn package
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/eclipse/tycho/tycho-maven-plugin/3.0.4/tycho-maven-plugin-3.0.4.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Unresolveable build extension: Plugin org.eclipse.tycho:tycho-maven-plugin:3.0.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.eclipse.tycho:tycho-maven-plugin:jar:3.0.4 @
[ERROR] Unknown packaging: eclipse-repository @ line 17, column 13
 @
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR]   The project [groupId]:updatesite.[groupId]:1.0.0 (C:\Users\[MY USER]\Desktop\KNIME_Update_Sites\pom.xml) has 2 errors
[ERROR]     Unresolveable build extension: Plugin org.eclipse.tycho:tycho-maven-plugin:3.0.4 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.eclipse.tycho:tycho-maven-plugin:jar:3.0.4: The following artifacts could not be resolved: org.eclipse.tycho:tycho-maven-plugin:pom:3.0.4 (absent): Could not transfer artifact org.eclipse.tycho:tycho-maven-plugin:pom:3.0.4 from/to central (https://repo.maven.apache.org/maven2): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 2]
[ERROR]     Unknown packaging: eclipse-repository @ line 17, column 13
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginManagerException

It seems that the process fails due to an SSL error, but I already had the website https://repo.maven.apache.org unblocked before. Otherwise, I would not be able to bundle my pure-Python extension, since the KNIME bundling script accesses these two files from Apache Maven:

I therefore tried replacing the tycho.version property in your example pom.xml with 2.7.5 and 2.7.4, but both attempts failed with SSL errors too.

What do you suggest to work around this?

gab1one · October 10, 2023, 6:50pm

Hi @DerMaxdorfer,

It looks like your company is routing all internet access through a proxy which modifies ssl content. You will need to import the CA certificate of that proxy into the JVM used by maven. Your IT should be able to help you with this.

best,
Gabriel

DerMaxdorfer · October 13, 2023, 7:22am

The SSL error is resolved and the composite update site packaged, @gab1one.

However, I can’t get it to work in KAP… Here is what I did:

Created a folder with the following structure:

C:\Users\[my user]\Desktop\KNIME_Update_Sites
|_ [my unzipped extension]
|_ category.xml
|_ pom.xml

Ran in the Windows command line:

cd C:\Users\[my user]\Desktop\KNIME_Update_Sites
mvn package

This is the output created from 2.:

C:\Users\[my user]\Desktop\KNIME_Update_Sites
|_ [my unzipped extension]
|_ category.xml
|_ pom.xml
|_ target
   |_ p2agent
      |_ ...
   |_ repository
      |_ artifacts.jar
      |_ artifacts.xml.xz
      |_ content.jar
      |_ content.xml.xz
      |_ p2.index
   |_ targetPlatformRepository
      |_ content.xml
|_ category.xml
|_ local-artifacts.properties
|_ p2.artifacts.xml
|_ p2content.xml
|_ [my composite update site's artifactId]-1.0.0.zip
   |_ artifacts.jar
   |_ artifacts.xml.xz
   |_ content.jar
   |_ content.xml.xz
   |_ p2.index

I am not sure which part of the output from mvn package I should register as an update site with KAP. I therefore tried various options, but none worked:

target: No repository found error
target/repository: repository recognized, but neither category nor features displayed in Install KNIME Extensions…
target/[my composite update site’s artifactId]-1.0.0.zip: repository recognized, but neither category nor features displayed in Install KNIME Extensions…

From what mvn logged during packaging, I take that my extension was identified correctly:
[INFO] Adding repository file:/C:/Users/[my user]/Desktop/KNIME_Update_Sites/[my unzipped extension]

Any advice on why nothing shows up in Install KNIME Extensions…?

gab1one · October 13, 2023, 11:58am

Hi @DerMaxdorfer

Glad to hear that the maven build now runs through, I am sure we are very close to your goal.
target/repository: / target/*.zip are indeed the update sites that you should be able to register in your KNIME AP. However it looks like there is nothing inside your update site. Can you check your category.xml file? It should list the features you want contained on the update site.

best,
Gabriel

DerMaxdorfer · October 13, 2023, 12:29pm

Yes, one step at a time, @gab1one!

This is my category.xml:

<?xml version="1.0" encoding="UTF-8"?>
<site>

<!-- Feature Category -->
   <category-def name="org.[my company]" label="[my company]">
      <description>
      	This category contains KNIME extensions published by [my company].
      </description>
   </category-def>

<!-- Features -->
   <feature id="org.[my company].features.[my extension's name]" version="1.1.0">
      <category name="org.[my company]"/>
   </feature>

</site>

I took the feature tag’s id attribute from the bundled extension’s directory:

[my unzipped extension]/features/org.[my company].features.[my extension's name]_1.1.0.202310050715.jar

gab1one · October 16, 2023, 7:35am

Hi @DerMaxdorfer,

If you used the wrong feature name in the category.xml, the build would not work, so that looks ok.
Can you also please post your pom.xml here?

best,
Gabriel

DerMaxdorfer · October 16, 2023, 8:08am

I see, @gab1one. Here goes my pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>org.[my company]</groupId>
	<artifactId>updatesite-org-[my company]</artifactId>
	<version>1.0.0</version>
	<name>[my company] Update Site</name>

	<properties>
		<tycho.version>3.0.4</tycho.version>
		<knime.version>5.1</knime.version>
		<base.directory>${env.USERPROFILE}/Desktop/KNIME_Update_Sites</base.directory>
	</properties>

	<packaging>eclipse-repository</packaging>

	<repositories>
	<!-- Plugin Update Sites -->
		<repository>
			<id>p2-knime-update-site</id>
			<layout>p2</layout>
			<url>https://update.knime.com/analytics-platform/${knime.version}</url>
		</repository>
		<repository>
			<id>[my unzipped extension]</id>
			<layout>p2</layout>
			<url>file:///${base.directory}/[my unzipped extension]</url>
		</repository>
	</repositories>

	<build>
		<plugins>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-maven-plugin</artifactId>
				<version>${tycho.version}</version>
				<extensions>true</extensions>
			</plugin>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-packaging-plugin</artifactId>
				<version>${tycho.version}</version>
			</plugin>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-p2-director-plugin</artifactId>
				<version>${tycho.version}</version>
			</plugin>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>tycho-p2-plugin</artifactId>
				<version>${tycho.version}</version>
			</plugin>
			<plugin>
				<groupId>org.eclipse.tycho</groupId>
				<artifactId>target-platform-configuration</artifactId>
				<version>${tycho.version}</version>
				<configuration>
					<environments>
						<environment>
							<os>linux</os>
							<ws>gtk</ws>
							<arch>x86_64</arch>
						</environment>
						<environment>
							<os>win32</os>
							<ws>win32</ws>
							<arch>x86_64</arch>
						</environment>
						<environment>
							<os>macosx</os>
							<ws>cocoa</ws>
							<arch>x86_64</arch>
						</environment>
						<environment>
							<os>macosx</os>
							<ws>cocoa</ws>
							<arch>aarch64</arch>
						</environment>
					</environments>
				</configuration>
			</plugin>
		</plugins>
	</build>

</project>

The p2-knime-update-site is part of my pom.xml because it seems to be a dependency of my own extension. I guess it is required for the KNIME Python API? mvn package would not run successfully without it at least.

gab1one · October 16, 2023, 11:42am

ok that looks all good. When you run the build you do see any Warning messages that relate to your plugin? If your build complains about missing dependencies and you can fix it by adding the extra update site, that means maven you have configured the repositories correctly.

I suspect that this might be an issue with platform specific features. You can also experiment with only keeping the windows specific environment block in the pom.xml:

<environment>
	<os>win32</os>
	<ws>win32</ws>
	<arch>x86_64</arch>
</environment>

best,
Gabriel

DerMaxdorfer · October 16, 2023, 12:49pm

Well, it is already good to hear that nothing seems to be wrong with my files, @gab1one – although this would have been more obvious and easier to fix.

I ran the build again with the full pom.xml as shared earlier and only the Windows-specific environment block as you suggested, but the output and logs do not differ. I do see some warnings in both cases, however:

C:\Users\[my user]\Desktop\KNIME_Update_Sites>mvn package
[INFO] Scanning for projects...
[INFO] ### Using TychoRepositoryTransport for remote P2 access ###
[INFO]     Cache location:         C:\Users\[my user]\.m2\repository\.cache\tycho
[INFO]     Transport mode:         online
[INFO]     Update mode:            cache first
[INFO]     Minimum cache duration: 60 minutes
[INFO]       (you can configure this with -Dtycho.p2.transport.min-cache-minutes=<desired minimum cache duration>)
[INFO] Adding repository https://update.knime.com/analytics-platform/5.1
[INFO] Adding repository file:/C:/Users/[my user]/Desktop/KNIME_Update_Sites/[my unzipped extension]
[INFO] Resolving dependencies of MavenProject: org.[my company]:updatesite-org-[my company]:1.0.0 @ C:\Users\[my user]\Desktop\KNIME_Update_Sites\pom.xml
[INFO]
[INFO] -----------------< org.[my company]:updatesite-org-[my company] >-----------------
[INFO] Building [my company] Update Site 1.0.0
[INFO]   from pom.xml
[INFO] -------------------------[ eclipse-repository ]-------------------------
[INFO]
[INFO] --- tycho-packaging:3.0.4:build-qualifier-aggregator (default-build-qualifier-aggregator) @ updatesite-org-[my company] ---
[INFO] The project's OSGi version is 1.0.0
[INFO]
[INFO] --- clean:3.2.0:clean (default-clean-1) @ updatesite-org-[my company] ---
[INFO] Deleting C:\Users\[my user]\Desktop\KNIME_Update_Sites\target
[INFO]
[INFO] --- target-platform-configuration:3.0.4:target-platform (default-target-platform) @ updatesite-org-[my company] ---
[INFO]
[INFO] --- resources:2.4.3:resources (default-resources) @ updatesite-org-[my company] ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\[my user]\Desktop\KNIME_Update_Sites\src\main\resources
[INFO]
[INFO] --- tycho-p2-publisher:3.0.4:publish-osgi-ee (default-publish-osgi-ee) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO] Published profile IUs: [a.jre.javase 17.0.0]
[INFO]
[INFO] --- tycho-p2-publisher:3.0.4:publish-products (default-publish-products) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO]
[INFO] --- tycho-p2-publisher:3.0.4:publish-categories (default-publish-categories) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO]
[INFO] --- tycho-p2-publisher:3.0.4:attach-artifacts (default-attach-artifacts) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO]
[INFO] --- tycho-p2-repository:3.0.4:assemble-repository (default-assemble-repository) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO]
[INFO] --- tycho-p2-repository:3.0.4:archive-repository (default-archive-repository) @ updatesite-org-[my company] ---
[WARNING]  Parameter 'qualifier' (user property 'buildQualifier') is read-only, must not be used in configuration
[INFO] Building zip: C:\Users\[my user]\Desktop\KNIME_Update_Sites\target\updatesite-org-[my company]-1.0.0.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17.791 s
[INFO] Finished at: 2023-10-16T13:46:16+02:00
[INFO] ------------------------------------------------------------------------

gab1one · October 17, 2023, 11:55am

I was able to identify the issue in a call with @DerMaxdorfer:

The feature version in the category.xml needs to be 0.0.0, then everything works.
The working version of that file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<site>

<!-- Feature Category -->
   <category-def name="org.[my company]" label="[my company]">
      <description>
      	This category contains KNIME extensions published by [my company].
      </description>
   </category-def>

<!-- Features -->
   <feature id="org.[my company].features.[my extension's name]" version="0.0.0">
      <category name="org.[my company]"/>
   </feature>
</site>