Getting excited about KNIME 5.3...

takbb · May 24, 2024, 8:36am

A lot of times the feedback I and others give is (rightly) noting problems and issues, or is suggesting future features that we’d like to see.

But today I’m excited by a couple of things I just spotted in the upcoming KNIME AP 5.3 release. I know it is a little way away yet (July 2024, I believe)… but one specific thing is close to my heart, so close in fact that it caused me to download the nightly build (first time I’ve done that) to try it out.
I’ll mention a couple of other “goodies” in a moment, but firstlythe stand-out bug fix that has got me so excited is this one:

AP-21669: The component output port gets activated in an inactive branch in case of missing flow variable

This has been the bane of my KNIME life for the past year or so, featuring in several forum posts, including recently:

Naturally, as soon as I saw this, I just had to give 5.3 Nightly a try, and I am pleased to report that the bug is resolved in that build. Woohoo! Thank you and well done KNIME team

Meanwhile, in other positive news… while I was there I had a play with a couple of other things.

Loving the new " Workflow monitor" panel containing details of nodes that are in error or giving warnings! That is going to be a game changer for me - especially with some of my spaghetti workflows! Being able to click on the message and have the workflow immediately scroll across to the node in error (or giving the warning) will make such a difference.

Hey that reminds me of something …

Wow - is it my birthday!

This to me will be a massive step forward in usability and efficiency on large workflows.

Mind you, I would still like to find a place where console messages produced by such nodes as Java Snippet could appear, as I use this for generating progress messages, e.g. logWarn("Hello world"); , but one thing at a time, and I think for many use cases the Workflow monitor will be far more friendly, especially to new users, than the classic “console log”.

Brilliant though it is, I don’t think as it stands it can completely replace Console Log… mostly… but not completely… just putting that out there!

I also noticed in the changelog, mention of a new “Expression” node. I see there is a KNIME Expressions (labs) extension in addition to the familiar “KNIME Expressions” extension.

I installed this and the (labs) Expression node looks to be pretty good. I may be wrong, but my take on it is that this is not a replacement to Column Expressions or Variable Expressions, but instead will be a modern alternative to both the String Manipulation and Math Formula nodes. It provides a modern UI for a single expression, but provides a more user friendly syntax, including conditional statements. It’s still in the labs but looks promising!

One thing I stumbled across (didn’t see it in the changelog) was a much-needed enhancement of the Modern UI.

Right-clicking on the tab for an open workflow now gives an option to “Reveal in space explorer”

Clicking this takes you to where the workflow is in Space Explorer.

This is the direct equivalent of the Classic UI button to select active workflow in the explorer:

This very small feature is another big thing that has been on my personal wish-list.

Now, as a bit of additional feedback, I would still like for the full name of the current workflow to be displayed when I hover over the workflow title bar, as it does when I hover over the name in Space Explorer:

Because currently when I have “My fantastic workflow that does something really amazing” and “My fantastic workflow that does something really amazing2” open side by side, there isn’t an easy well to tell which is which, without revealing in space explorer and then hovering there to find out.

There are plenty more things I could comment on, and yet more suggestions I can and will make in future, but I want to take this moment to say thanks to the KNIME team for listening and keeping the improvements coming. The move to Modern UI continues to be a journey (admittedly longer and tougher for some than for others!) but it’s great to see the direction of travel.

For me, KNIME 5.3 cannot come soon enough, and I strongly believe that KNIME 6 will be awesome!

Cautionary notes
I should be responsible here and add that of course what I was looking at was a nightly build of KNIME 5.3. Not being involved in the release process, I understand that there are no absolute guarantees that anything I’ve talked about here will be in the 5.3 final release product, and functionality is still subject to testing and alteration. Also, if you do decide to try out the 5.3 nightly build, best to use the zip archive (as I did) so it is side-by-side with your installation. You must accept the risks, and ideally use a non-critical machine!

The nightly build is not for production use, and you should note the warnings on the download page. Don’t overwrite any current KNIME installation with it, if it is an environment you currently rely on!

tomljh · May 24, 2024, 11:11am

Yes, this is also the feature I need: reverse navigation from the current workflow to the workflow of the spatial browser.
Thank you to the development team for their work.

iCFO · May 24, 2024, 12:59pm

I also noticed that they marked the filtering of flow variables as solved for 5.3! I will definitely check out that new Expressions node later today. Hopefully it can execute multiple expressions in a series as well. That has been on my wishlist forever, and would do wonders for reducing the number of nodes in my giant “spaghetti workflows”!

takbb · May 24, 2024, 1:00pm

Sadly I don’t think we have multiple expressions as yet. It currently looks like a simple direct replacement for String Manipulation and Math Formula.

One node → one output

takbb · May 24, 2024, 1:05pm

Good spot @iCFO ! I just had to go back in and take a look and sure enough

This will make a lot of people very happy

Even more excited now!

Additionally… not mentioned on changelog, but it is possible in the nightly build to show heap memory in Modern UI (as it currently is in Classic) if turned on in preferences (under General)

Adding links to forum references for this feature so they’ll track here if in future
people have the same questions:
https://forum.knime.com/t/variable-filter-after-merge-variable-node/29292
https://forum.knime.com/t/trying-to-delete-some-floating-variables/71835
https://forum.knime.com/t/delete-flow-variable/6641
https://forum.knime.com/t/components-dont-encapsulate-variables/31816

HaveF · May 24, 2024, 1:25pm

@takbb I believe it can eventually handle multiple expressions. This seems to have been mentioned in “What’s cooking”, if I remember correctly.

Workflow monitor" panel is great! Thanks for mention it!

iCFO · May 24, 2024, 1:31pm

We will get there one day… An easy to order / review series Column Expressions node is definitely on the top of my list (similar to the Formula node in Altery). I really hate all of the hassle / wasted time of digging through dozens of nodes in a row to edit and re-order a series of calculations, when it would be seconds in a single cleanly laid out node that calculates in series…

Christopher41 · May 26, 2024, 7:00pm

The new Exprssion node has the joinSep but not join. Will this be added?

Column Renamer has a lot of wasted space. I hope the dropdown list would be larger. I work with large data tables with a lot of columns. The legacy Column Rename displays a lot more columns.

takbb · May 26, 2024, 8:01pm

@Christopher41 , I admit I’m not a big fan of the 5.x Column Renamer node, but for other reasons…

I have a large number of components that if I accidentally modify using 5.x instead of 4.7, they cease to remain backward compatible with 4.x, and it’s all because of the way that one node has been implemented. For most 4.x nodes, even if the component is saved in 5.x, the component will work with the older KNIME. But not this one and it’s frustrating.

I have often wondered why it got implemented as an incompatible replacement for the old Column Rename node, instead of allowing the old one to coexist alongside as a legacy node. It’s a real pain when supporting components that work with both 5.x and 4.7, and I’ve taken to using Table Manipulator for column renaming instead.

[ Edit: I just re-read what I wrote and realised the above is actually not quite accurate. The old Column Rename in 4.x has become Column Rename (deprecated) when the component is opened in 5.x. The problem comes when I mistakenly edit in 5.x having then included the new “Column Renamer” node, and if the component is opened in 4.x, it is broken. That’s actually my fault of course, but it is easily done!

It is for this reason I try to remember to use Table Manipulator instead - but sometimes forget - as there is no obvious option for dropping the legacy/deprecated version onto such a workflow as it doesn’t appear in the node palette, and cannot be dragged from the hub. ]

On your point about wasted space, efficiency of screen usage was (and remains) probably one of my biggest concerns with modern UI. Hopefully over time we will see a reduction in the space being used unnecessarily for many dialog.

The Expression node is still in labs so I’d like to think that all the functions of both String Manipulation and Math Formula will be included at some point (and hopefully more besides!), but of course I don’t know.

rfeigel · May 26, 2024, 9:31pm

For whatever my two cents are worth I think the Table Manipulator node is superior. It does what the Column Renamer does plus other functionality.

szawadski · May 27, 2024, 9:53am

Another missing feature in Table Manipulator is to show by default current column name when editing (like in old node) so that it’s faster to prefix/suffix current name.

szawadski · May 27, 2024, 9:55am

As soon as they add support for multiple expressions in the new Column Expression node, it would be good to be able to reuse a created column in a next one to avoid having to create 2 nodes…

iCFO · May 27, 2024, 2:48pm

Unfortunately using the Table Manipulator instead of the Column Rename is only a temporary fix, as a redesign / replacement of the Table Manipulator node in the same fashion would cause all of these same issues again.

The real solution is for KNIME AP to have the ability to manage and implement node upgrades across a workflow and transfer settings from the older node versions to newer ones. This seems very feasible (at least for KNIME issued nodes) if they incorporate a settings transfer map in the node production process when node redesigns would cause old nodes to become deprecated.

takbb · May 27, 2024, 4:09pm

@iCFO , I’ve been thinking more about the Column Rename change since I wrote the above (which I have now edited as I was wrong in part of my earlier comment).

I think the reason why it didn’t get the usual upgrade path to Column Renamer was because the new node has reduced functionality over the deprecated Column Rename node. (i.e. in Column Rename you could change the datatype of the column being renamed). As a result, whilst I’m sure most workflows never used that function, there was the potential that a flow would no longer work if it was simply migrated to the new node.

I agree though that there ought to be a straightforward upgrade available where the node is only renaming columns, and yes Table Manipulator could very well have the same issues, as it also has that datatype change capability built in currently, and I suspect it won’t be there in a future version.

carstenhaubold · May 28, 2024, 6:26am

Hey everyone,
Thank you very much for trying out the nightly build and discussing the new features! That’s exactly the kind of early feedback we need. Keep it coming

I can answer a few of your speculations about the new expression node:

Thanks for your input! You are quite right, in the first iteration this shall be a modern replacement for String Manipulation and Math Formula nodes. We’re currently curating the list of available functions in the node and ironing out some kinks in the UI/UX. In the future we’d like it to be a full replacement for the Column (and Variable) Expression node and more.

The vision is that we’ll extend this from a single to multiple expressions very soon, but after the summer release. And yes, we want subsequent expressions to be able to use results from previous expressions.

We hear you, that’s very much aligned with our plans for the new expression node. We want to allow editing multiple expressions, as well as providing a way to easily review the results. Stay tuned

takbb · May 28, 2024, 7:54am

Thank you @carstenhaubold , it is good to see this vision of the future, and I think this direction of travel will excite a good many people.

I may be wrong as I’ve only played with it briefly, but at the moment an “expression” appears to be “single statement” similar to the way String Manipulation and Math Formula operate rather than a multi-statement “script” as it is with Column Expressions, and there are no “intermediate results” in the form of script variables. So if in future it is looking to replace the other Expressions nodes, presumably that’s something that will be coming?

Something I’d very much like to see in such a node in future would be the ability to build up a set of user defined functions , and not have to rekey these on each new workflow; perhaps be able to supply one or more “libraries” of such functions to the node via an optional port or by some other means. Extensibility of that nature would make it hugely powerful.

Thanks again for your insights.

iCFO · May 28, 2024, 3:28pm

Ditto on the “Expressions Library” for sure! A well thought single expressions node with the ability to stack multiple expressions, a flexible formula result view during formula editing, the ability to name / save / share / organize custom expressions, a wide array supported calculations, multi-row calc support, and simplified expression syntax would go a long way toward dramatically simplifying KNIME adoption and open up growth opportunities.

Very exciting!

carstenhaubold · June 10, 2024, 7:22am

Yes, exactly, we’re starting with a single-expression version but already have the multi-expression version in mind for a future KNIME AP release.

Thanks for stressing the importance of a custom expression/function library. I’ll add it to the list of feature requests for the new expression node

What exactly do you mean by wide array supported calculations?

Multi-row support will already be included in this first version, at least in the form that you can use a constant row offset when accessing values from a column (making the use of the Lag Column node unnecessary).

iCFO · June 10, 2024, 11:21am

After seeing the current options in the expression library of the nightly version, I would say the box is checked already for “a wide array of supported calculations”.

I didn’t see Multi-Column for the Expressions node, but it would be nice to see in the future.

Would it be possible to have an output column selection option that allows us to reference the output column in the expression itself rather than have a single defined output column? (Similar to Java snippet) So we would have column output options for: selecting a current column, creating a new one, or included within expression.

That way a single if statement expression could make changes to values in various columns simultaneously. Having a button that duplicated an expression below the current one would be helpful for speeding up expression building, but edits to multiple complex if statements would still need to be done to multiple expressions manually later and changing the order of grouped expressions would require moving them as a group… Single expressions that made multiple conditional changes across multiple columns at once would reduce complexity, clutter and error likelihood.

Sajjadmosaheb · August 11, 2024, 1:33pm

I think one big drawback is, that when I open to configure the metanode or component It’s not open a new tab and I cannot see the whole workflow and the component at the same time it is very frustrating for me to handle these two now.