I would like to share my findings that I obtained with Knime with others. Especially with people that do not have Knime and which are not that tech savvy. Is there a nice way? Or maybe someone can help me overcome the issues.
I would like to produce some simple visualizations (bar chart, line chart, pie chart), share them and also update them over time.
What I tried so far:
a) Export the data to a Google Sheet which I could share via a link. Export works fine using the “Google Sheets Updater”, but all Date, Int and Double fields are written as Strings. Therefore I can not use them to create charts.
b) Create a BIRT Report. There are hundreds of tutorials on how to create a BIRT reports. I tried it, and suceeded in creating the report. I can then look at it in the Browser ( Run → View Report → In Web Browser ) but I can not find out on how to make it accessible to others. How to export it to a Webserver.
c) Using the “normal” Bar Chart / Line Chart nodes in Knime. Nice, but how can people see them without Knime?
d) Export the Charts as static images. This works, but is obviously an ugly solution. It also does not allow for any kind of interactivity, like mose-over tool-tips.
At work, most users have a browser on their computers and usually some kind of access to a server. Therefore, two other options are available:
f) Use R or Python nodes in KNIME to generate interactive views (as a web page) or a set of static views (svg, png, pdf). Examples of libraries which can do this are e.g. the vegawidget package in R (with vega-lite and vega as backends) and the Altair library in Python, the latter also having vega and vega-lite as its backend. In R, you can easily generate embedded web pages by using knitr and rmarkdown.
I found this forum entry Google API: Sheets updater saves values as strings - #5 by oole
It shows how to make the Google Sheets Writer write numbers (instead of strings). Maybe this solves my problem.
The downside to the Google Sheets solution is, that I have to configure the visualization in Google sheets, which is not that powerfull, but for the current problem it might be good enough.
There is unfortunately no free lunch With R or Python, you can most certainly do all this, however, you will spend time on your side on assimilating the technology. The following gallery demonstrates what can be done with R’s markdown and knitr: Gallery
If you take a look at the source code of the examples, you will notice that they are essentially markdown reports with embedded R code. These reports can be transformed into fully working html pages using the knitr package. If you use R Studio, Rmarkdown and knitr are well integrated therein.
Regarding the interfacing with KNIME, given that you are doing all the processing in the latter:
outside of KNIME, the interface will probably be a set of CSV or JSON files deployed somewhere;
inside of KNIME using the R nodes, you will indeed be limited to (multiples of) static images (svg, png, etc.).
maybe there is something in the making for you. I’ve dealt with this issue for quite some time and have also talked to people who construct full HTML-Reports “manually” in KNIME.
To overcome this shortcoming I developed a set of nodes which leverage Asciidoctor for the output. Asciidoc is a human-readable markup language comparable to Markdown with a more strict definition and is supported by Asciidoctor, GitHub, Gitlab and a few others. This language allows to gradually add simple text sections of the final document in Asciidoc and finally render it to .html or .pdf (Asciidoctor also supports other formats like epub or Docbook but thats untested by me so far).
Let me give you a sneak preview from my development workflows:
The three Java snippets add building blocks of the document and the GroupBy Node concatenates everything to Asciidoc-Text in the bottom of this post. The two AsciiDocWriter Nodes export the raw text to either an .html-Document or a .pdf (Asciidoctor also supports epub and Docbook but I haven’t tested that so far).
For a more complex example of whats possible with Asciidoc see this file
That’s a rather short example but it definitively also works for longer documents. I’ve used it to construct a report about published preprints where the data fore each paper has been added in a loop and then a final document has been rendered. We used to run this workflow in a dockerized KNIME triggered by Jenkins and send out the report by email but it should be somewhat straightforward to also publish the generated files to a webserver.
So far the Nodes have only been used internally but I am currently working with Daniel and Philipp from NodePit to release these nodes for the general public. I hope to get out a beta version by the end of the year at the latest. So if that sounds interesting be sure to follow my posts and the NodePit website for further details…
And last but not least for reference the raw source text for both outputs:
= Asciidoc Nodes Example
:description: This document is a quick example about what's possible with the AsciiDoctor Nodes for KNIME
:keywords: AsciiDoc, Asciidoctor, pdf, theme, themeing
The AsciiDoc Nodes are a new way of reporting in KNIME. It is easy to construct reports in multiple blocks and then convert them to an .html or .pdf-Document
It's also easy to integrate images that have been previously rendered to .jpg, .png or .svg!
@Iris Knime Server is just waaaay out of our price range. I am not doing anything that could not be done in Excel. The Workflow is basically just join, join, filter, group by, sort and visualization. But I don’t like Excel and I am a fan of open source software.
About the Google Sheets thing. The option is already there in the node, but hidden in the variable section. Just make it a checkbox on the normal node-config page and maybe add a tooltip to explain what it does and you are good to go.
@Geo by now, I worked with dozens of visualization tools. R is very powerful for complex datasets but can get quite ugly. The beauty of Knime is, that you do something … then come back to it after a few months and it’s quite easy to understand what you did and where you need to change something. The ability to look at the results of each node are like an integrated debugger. I can not say the same about R.
It has one syntax - you don’t have to deal with a plethora of flavors or deal with the least common denominator CommonMark which still not includes tables although the idea ist discussed since 2014 in a (as of now) 190 post long thread. Coincidentally the 189s post from a month ago reads “It surprises me how often people want to reinvent AsciiDoc.”
Asciidoctor treats pdf output with the same priority as html, i.e. it has a dedicated processing toolchain comprising extensive theming options. All relevant markdown libraries that I’ve seen so far only render .html directly and rely on things like OpenHTMLtoPDF to convert html to pdf which has new challanges e.g. with pagination.
Nonetheless I already thought of leveraging commonmark-java or flexmark-java in a different set of nodes later on but as for many things this is a matter of time and priorities.
That’s not really inline with using Google software. Their software is far away from Open Software.
Maybe you mean Freeware. Even then you should have a deeper look on the contract you make with Google
A view to the license agreement that you agree to
Check what you need to give in order to use the software
Consider whether your partners/consumers may use the software without being bound too
Having said this I think there’s a huge defference between EXCEL (not even freeware), Google sheets (freeware but you will pay with data) and KNIME (Freeware or paid by money).
As you said: There’s no free lunch. All of us need to earn money.
At the end of the day it’s typically more than just (direct) costs of software that count:
Creditability of the company
Data security & ISEC concerns
Simplicity of the whole system architecture (esp. data interfaces)
Availability of (own and external) resources to maintain the solution
I don’t have interactive graphs, but a static website with graphs that update every day is good enough for this project.
Only caveat at the moment is the DB connection. Since the DB is only accessible through an SSH tunnel, I need to have a batch file which first starts the tunnel, runs KNIME and then stops the tunnel again. It would be really nice if KNIME had SSH tunneling built into the DB connectors.
I am personally settling with automated BIRT PDF reports distributed by the email node for my KNIME AP only solution. You can also do some “one off” free interactive html graphs and maps though datawrapper.de
If you settle on the HTML export it should be possible to include raw html/js blocks generated elsewhere (e.g. from plotly) although I have not tested that yet. Obviously that doesn’t work with a PDF-export.
Generally I don’t think that the creation of dashboard like reports would be a prime usecase for the nodes. These are more about “written” type of reports although you can also prepare reveal.js slideshows with AsciiDoc (see here for an example with AsciiDoc source).
For dashboards we need to wait for KNIME to also allow building nodes with R like it is now possible with Python. Then creating (at least simple) flexdashboards from an RMarkdown file would be breeze…
Right now BIRT is only accessed separately through the menu after a workflow is executed, so this workaround is the only solution that I am aware of at this point…
Hopefully someone can develop a “Report Output” node that allows us to create and further manage BIRT reporting files within a single workflow. Since BIRT can handle looped file outputs through a child workflow, it seems like all of the programming exists except for the return path from BIRT. We just need the workflow to “pass through” BIRT instead of end there.