1.x to 2.x migration

Hello all,

we have a knime plugin with several nodes we had written for 1.3.5, which seem to be working with 2.0.3 (we are always using the version bundled with Schrodinger, to guarantee compatibility with their extensions, so no 2.1 yet).

However, since we saw the concept of flow variables we’ve been thinking that many custom nodes we had created will not be needed anymore (for instance, we have reader and writer nodes which are copies of the included nodes, with the addition of extra ports that enabled us to pass the original filename to the end, so that if we had read something.input, we would write something.output, configuring just the first node of the workflow). I see that the flow variable allows the original filename to be accessible by the writer node, but will we be able to transform it (e.g. s/input/output) ?

Also, is/will it be possible to convert data cells to flow variables?

In any case, our first doubt is whether to create a new plugin or adapt the old one. I see that there are some differences between a fresh node extension of 1.3.5 and 2.0.3 (e.g. in the xml, dataIn/Out became in/outPort, and it seems some classes are now generic, taking the NodeModel as type parameter (?) ). If we opt to adapt the old code, should we make this changes?

Is there some other issue with migrations that we should be aware of?

Sorry if there are too many questions for a single post :slight_smile:

Thanks in advance, and congratulations on the excellent work you’re doing.

Miguel

Hi Miguel,

You can check the diffs for my project around r58 - r62. The changes not mentioned in your questions were: change in row id, the values in comboboxes now StringIcons, not Strings, unHiliteAll got a KeyEvent parameter.
Bests, gabor

Hi Miguel, As for the other questions: It’s save to upgrade your existing plugins, i.e. updating the NodeFactory to use java generics (though not required) and also the make other changes mentioned by Gabor. The important API changes are made with v2.0.0 (see changelog.) … and it is possible to convert between cells and flow variables. There are nodes suchs as “TableRow to Variable”, “Variable to TableRow” and “Variable to TableColumn” in the “Loop Support” category.

Thank you both! I still have some deprecated warnings but I’m able to compile and launch from the 2.x sdk.

Regarding the flow variables, I had seen those nodes, but I must be missing something.

After converting to variables with “TableRow to Variable”, it seems I need a variable connection to put those variables in the workflow. In order to use those variables with the other knime nodes, I would need to customize them and add a variable input port ?

Cheers,
Miguel

Also, what about transformation of variables?

Imagine a simple workflow: SDFReader -> … -> SDFWriter

With flow variables, I can pass the original filename of the reader to the writer, but is there a way to transform it (e.g. sed ‘s/input-pattern/output-pattern/’ ), so that I can write to a different filename derived from the original one ?

If this is not possible, is it a planned feature? This seems to be a very general issue, no?

Miguel

It is possible to add more columns to the flow variables, and those columns might be the modified versions of the file names for example. You can do this outside of the flow variable handling (I usually do this with Java Snippet). Here is an example workflow (it requires HCDC, and HiTS).

The usual procedure of using/modifying variables is: use the “Extract Variables” node (red variable output), then use a “Variable to TableRow” node (ouptut is a data table), modify the variable with some nodes in knime, connect it to a “TableRow to Variable” node (again, red variable output) and finally injecting the modified variables into the data pipeline using the “inject variables” node. This is still somewhat complicated … we are working on nodes that allow for easier handing of variables in the data stream already. You should get everything done with it, though.

Bernd, I would argue with the “everything” part. The limitations I know:

  • embedded loops, well I know it is possible generate the table with the Direct Product node, but it is not part of the base KNIME nodes.
  • flow variable sources cannot be combined,
  • cannot add different columns in different iterations,
  • practically cannot use source or sink nodes, as the views are usually sink nodes you cannot open/save them in a loop.
I would be really happy if I were wrong, so if there are solutions these problems I apologise. Thanks, gabor

(This thread isn’t really about 1.x to 2.x migration anymore… eh.)

With everything I meant to say that you can do everything with the current setup that you will be able to do with the “Edit Variable in Data Stream” node (once we have something like that).

As for the open problems:

  • The embedded loops (or nested loops) are possible. You just need to make sure that your inner start node is a successor of the outer start node.
  • I don't understand what you mean by flow variable sources cannot be combined. If you are saying that there is no node with two variable input ports, ... you are right.
  • The different column sets in different iterations are still a problem, yes. It's not a framework limitation, it's just that we don't have such a flexible loop end node in the release.
  • Source and Sink nodes are a problem, in particular the source nodes. The sink (e.g. view) nodes are executed as part of the loop, though.

    • Cheers, Bernd

Hello all.

Sorry, I have been away from this issue.

Following gabor’s suggestion, I had tried to modify a variable with a java snippet - I could access it, but didn’t know how to modify it. I see now, from what bernd said, that it must be transformed to data in order to be modified, and then back to variable again.

Just tested it and it seems to be working, thanks!

I suppose the “Edit Variable in Data Stream” would be similar to a Java Snippet that also allowed modification of variables?

Cheers,
Miguel

I suppose the “Edit Variable in Data Stream” would be similar to a Java Snippet that also allowed modification of variables?

Yepp, that will be of similar kind. We also got funds that allow/require/force us to implement that for 2.2. So it’s not just on the feature request list.