Headless Junit Testing

Hi,
 
I have developed a set of KNIME nodes. 
 
As part of a CI (Continuous Integration) process, every day (indeed every night) the source code of these KNIME nodes is downloaded from the SCM, compiled, and a p2 site is built (in a more or less laborious process along CI software like Jenkins and Sonar). This procedure works fine using a combination of maven and tycho for Ecplipse plugin development. 
 
Unfortunately I am stuck in a problem that I have no idea of how to overcome. I want to add some Junit testing in the CI process. Thus the process will be as following: download from SCM, compile, test, mount p2 site with the nodes. I have started with a very simple Junit test following the indications described here
 
http://tech.knime.org/community/developers
I have tried to do my best coding a very simple junit test in a fragment, but the following error is raised:
 
java.lang.SecurityException: class "org.knime.core.node.InvalidSettingsException"'s signer information does not match signer information of other classes in the same package

 

Surfing the KNIME forum I found a message where thor explained the reason of this error:

http://tech.knime.org/forum/knime-developers/securityexception-is-triggered-by-the-following-test-which-previously-succeed

I wonder if anybody know some kind of workaround or solution to tackle this problem. I firmly believe in JUnit headless testing as a very important part of CI. How could I include headless testing (I really love JUnit but I am willing to use another testing framework if necessary).

Thanks in advance

Oscar

The JUnit tests must be run inside Eclipse, otherwise the whole classloading does not work as expected (therefore the SecurityException). If you follow the instructions from the community developers page, you can use the Eclipse application org.knime.testing.UnittestRunner. It will search for JUnit tests in all plug-ins/fragments that have an AbstractTestcaseCollector. The application will create a XML file with the test results that can be read by Jenkins.

Hi,

I managed to use "org.knime.testing.NGUnittestRunner" to run my JUnit tests in my testing feature from eclipse.

But how do i run this runner in a command line. Need this for jenkins. Thanks!

Cheers,

Ken

  1. Install KNIME including the plug-ins you want to test and the KNIME Testing Framework
  2. Execute the org.knime.testing.NGUnittestRunner application, e.g. ./knime -application org.knime.testing.NGUnittestRunner ...

I copied the command line from the Debug Configurations. Seems to work too.

java -Xms40m -Xmx384m -Declipse.pde.launch=true -Dfile.encoding=Cp1252 -classpath C:\eclipse_knime_2.9.4\plugins\org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -application org.knime.testing.NGUnittestRunner -data C:\Users\xxx\workspace/../runtime-RunUnitTests -configuration "file:C:/Users/xxx/workspace/.metadata/.plugins/org.eclipse.pde.core/Run Unit Tests/" -dev "file:C:/Users/xxx/workspace/.metadata/.plugins/org.eclipse.pde.core/Run Unit Tests/dev.properties" -os win32 -ws win32 -arch x86_64 -nl en_SG -consoleLog -xmlResultDir D:\testresults

Hi thor,

So my feature project containing the fragment project that contains the test cases have to be exported as a jar, and dropped into the drops-in folder of KNIME, before i run the NGUnittestRunner?

Cheers

Or how else do i install my plugins, fragments and features.

Please don't use the dropins folder! This is outdated and causes lots of problems. Instead create an update site with your feature(s) and install them from there into KNIME using the p2 directory (you can even create an installation from scratch with the p2 director). This is the standard Eclipse approach and documented in the Eclipse help.

Ok i will try that for installation. Even my test features have to be installed this way?

With regards to running the tests, i will also need to provide the parameters "-configuration" and "-dev" right?

Copied from the launch configuration in eclipse IDE.

knime -application org.knime.testing.NGUnittestRunner -nosplash -xmlResultDir C:\testresults -configuration "file:C:/Users/xxx/workspace-testing/.metadata/.plugins/org.eclipse.pde.core/KNIME Unit Test Runner/" -dev "file:C:/Users/xxx/workspace-testing/.metadata/.plugins/org.eclipse.pde.core/KNIME Unit Test Runner/dev.properties"

 

Yes, also the test feature can be installed this way. You don't need the -configuration and -dev options if you are running the test in a standalone product. They are mainly used for starting an Eclipse instance out of the SDK (which is set up quite differently regarding plug-ins, features, etc.).

I have installed both plugin and plugin test fragment via a local update site. I can see my plugin when i start up the knime analytics workbench.

However when I ran the following in command line:

C:\knime_2.9.4>knime -application org.knime.testing.NGUnittestRunner -nosplash -xmlResultDir C:\testresults

Nothing happened.

I dunno why.

If i put and registered my tests in a fragment, the tests are not picked up.

If i put and registered my tests in the plugin itself, the tests are picked up.

So you put a subclass of AbstractTestcaseCollector into your fragment and registered this class in the fragment at the org.knime.testing.TestcaseCollector extension point? Then this has to work.

It worked. Now when i run in linux. I am getting this SWT error.

[xxx@dev01 eclipse_knime_testing]$ ./eclipse -application org.knime.testing.NGUnittestRunner -xmlResultDir ~/tests -consoleLog

!ENTRY org.eclipse.osgi 4 0 2014-08-13 20:52:47.438
!MESSAGE Application error
!STACK 1
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
        at org.eclipse.swt.SWT.error(SWT.java:4308)
        at org.eclipse.swt.widgets.Display.createDisplay(Display.java:909)
        at org.eclipse.swt.widgets.Display.create(Display.java:897)
        at org.eclipse.swt.graphics.Device.<init>(Device.java:157)
        at org.eclipse.swt.widgets.Display.<init>(Display.java:500)
        at org.eclipse.swt.widgets.Display.<init>(Display.java:491)
        at org.eclipse.ui.internal.Workbench.createDisplay(Workbench.java:716)
        at org.eclipse.ui.PlatformUI.createDisplay(PlatformUI.java:161)
        at org.knime.testing.core.ng.UnittestRunnerApplication.start(UnittestRunnerApplication.java:55)
        at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
        at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1386)

I see that UnittestRunnerApplication#start is creating the display.

PlatformUI.createDisplay();

 

How do i suppress this in linux.

You cannot. You should start and use a virtual X server (Xvfb) instead. Testcases should in principle be able to open dialogs, windows, etc. We do the following:

Xvfb :$$ &
export DISPLAY=:$$
./knime ....
pkill -P $$

Got the following error when i try your command.

[xxx@dev01 eclipse_knime_testing]$ ./eclipse -application org.knime.testing.NGUnittestRunner -xmlResultDir /home/xxx/tests
Xlib:  extension "RANDR" missing on display ":5042".
Invalid option: 'org.eclipse.platform'

Valid arguments:
    -xmlResultDir <dir_name>: specifies the directory into which the test results are written.
Valid arguments:
    -xmlResultDir <dir_name>: specifies the directory into which the test results are written.
6 XSELINUXs still allocated at reset
SCREEN: 0 objects of 232 bytes = 0 total bytes 0 private allocs
DEVICE: 0 objects of 96 bytes = 0 total bytes 0 private allocs
CLIENT: 0 objects of 144 bytes = 0 total bytes 0 private allocs
WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
PIXMAP: 2 objects of 16 bytes = 32 total bytes 0 private allocs
GC: 4 objects of 16 bytes = 64 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 6 objects, 96 bytes, 0 allocs
2 PIXMAPs still allocated at reset
PIXMAP: 2 objects of 16 bytes = 32 total bytes 0 private allocs
GC: 4 objects of 16 bytes = 64 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 6 objects, 96 bytes, 0 allocs
4 GCs still allocated at reset
GC: 4 objects of 16 bytes = 64 total bytes 0 private allocs
CURSOR: 0 objects of 8 bytes = 0 total bytes 0 private allocs
DBE_WINDOW: 0 objects of 24 bytes = 0 total bytes 0 private allocs
TOTAL: 4 objects, 64 bytes, 0 allocs

The error has nothing to do with the virtual X server (only the other message come from it). You must have mistyped the command somehow.

Downloaded a new copy of KNIME and seemed to work, although i still get the xlib error above, but the XML result files are generated.

Another 2 questions i have.

  1. How do we configure Jenkins to interpret the xml result files that are generated after running the test application.
  2. How can we measure code coverage based on the unit tests run by the test application.

1. Just follow the standard approach in Jenkins, it's described in its documentation.

2. You have to use a Code Coverage library, e.g. EcmlEmma/JaCoco

The Xlib message is only a warning that the resolution in the virtual X server cannot be changed (what the RANDR extension is used for).