KNIME Business Hub features in small steps

A new series of KNIME posts in the next weeks!

Coming up: KNIME Business Hub (KNIME Business Hub | KNIME)

The enterprise (cloud :cloud: or on-prem :office:) software for deploying your workflows :rocket: !

Productionization of workflows to achieve APIs / Service, DataApps, Schedules, Triggers in steps :computer_mouse: !

Stay tuned :satellite: !

Also on (click the icons):

  • LinkedIn :business_suit_levitating:

  • Twitter :bird: / :heavy_multiplication_x:

  • Mastodon :elephant:

  • Instagram :camera_flash:

8 Likes

STEP 1: Upload Workflow :arrow_up:

The first step towards deployment in KNIME is uploading one or more workflows to KNIME Business Hub:

  1. Add the KNIME Business Hub of your organization to your KNIME Analytics Platform (just needed if it your first time doing this)

  1. Create a workflow and select upload

  1. Select your “KNIME Team” and the project “Space”

If you find yourself performing the last 2 steps too often you should use the Workflow Writer node and the KNIME Hub Authenticator. Check the KNIME Integrated Deployment Blog Series to learn how to automate the capturing and upload of workflow segments.

Check more videos on the L3-CD course playlist: LINK

For a complete experience enroll in our free learning platform: LINK

3 Likes

STEP 2: Version Workflow :stopwatch:

After the workflow is uploaded a new version needs to be created before it can be deployed.

  1. Open the “Workflow History” panel from the workflow page

  2. Provide the version name and small description, then create the version

That’s it, you should be able then to select the new version for deployment!

In previous versions of KNIME Business Hub you had to version the entire space the workflow was deployed in. This is no longer the case and versioning is now required and only available at workflow-level.

2023-10-09_15h57_55

Different options are available for versions:

  • Show this version: display the workflow image and description of a previous version

  • Restore this version: roll back to a previous version as if you were uploading a new workflow with that previous version. That means that you will still need to create a new version to completely roll back!

2023-10-09_16h12_39

  • Delete this version: remove this version from the list available in the workflow history panel

You can automate this and more by creating a workflow that versions workflows :bulb:
Use nodes from the KNIME Hub Connectivity extension like:

Read more on versioning in the docs:

https://docs.knime.com/latest/business_hub_user_guide/index.html#versioning

Check more videos on the L3-CD course playlist: LINK

For a complete experience enroll in our free learning platform: LINK

4 Likes

STEP 3A: Deploy Workflow :rocket: as a Service :computer: :arrow_right: :cloud: :arrow_right: :computer:

When you deploy a workflow as a service you basically create an API endpoint for anything or anyone to execute the workflow. The workflow, using container nodes, can receive and send back data using the API. This is one of the most common types of deployments. Related topics are: inference service/deployment, RESTful services, microservices architecture, orchestration and containerization.

When deploying a workflow as a service on KNIME Business Hub you can select the version if more than one are available.

Check the documentation to learn more about the advanced settings.

https://docs.knime.com/latest/business_hub_user_guide/#_service

To automate the creation of the deployment service from an uploaded workflow you can also use the following node in another workflow:

Once the deployment is created you can find it listed under your KNIME Team deployments tab. There you should see the API endpoint (you can get the raw link that returns JSON data or browse the type of API requests via Swagger). You can generate an “application password” for personal use to call the deployment from external clients. It is recommended to have different application password per project so you can delete when the project is over.

Sharing options with other users are available for KNIME deployments. We are going to cover the “Step 4: Share Deployment” topic after we cover the other deployment types (data apps, schedules and triggers).

3 Likes

STEP 3B: Deploy Workflow :rocket: as a Data App :chart_with_downwards_trend: :bar_chart: :chart_with_upwards_trend:

A data app is a web app where data tasks can be performed: any use case usually performed by a Data Engineer, Data Scientist or Data Analyst could be theoretically executed via data apps.

Any workflow equipped with at least one component with inside Widget nodes or View nodes can be deployed as a data app.

Similarly to the deployment of services, you can deploy the data app from its workflow page, then select the version. The data app should be available listed under KNIME Team deployment tab. You can share with KNIME Business Hub users who are not in your team via the Data App portal and provide details upon deployment.

We will learn more about the Data App Portal in “STEP 4 Manage Access”

Check advanced deployment settings at:

https://docs.knime.com/latest/business_hub_user_guide/#_data_apps

Check examples of data apps at:

For data app building (workflow design/building) I recommend this reading list:

To read more on the “why” data apps can empower your organization, check:

1 Like

STEP 3C: Deploy Workflow :rocket: as a Schedule :spiral_calendar: :calendar: :date:

You can automate the execution of the workflow using a schedule: every minute? every hour? every day?

What is automation useful for? Many things! Below a few examples…

  • :gear: for data engineering to move data and update data sources, usually from and to data warehouses and data lakes → KNIME Database nodes
  • :page_facing_up: for reporting to generate a PDF with charts and KPIs and automatically send it via emails → KNIME Reporting (Labs) nodes
  • :brain: for machine learning batch deployments to score a chunk of new data (that is what the example workflow is doing in the video above) → many Predictor nodes are available
  • :musical_score: for more complex orchestration of workflows calling workflows calling workflows → KNIME Personal Productivity Tools nodes

Similarly to the deployment of services, you can deploy the schedule from its workflow page, then select the version. The data app should be available listed under KNIME Team deployment tab.

Check advanced schedule deployment settings at:

https://docs.knime.com/latest/business_hub_user_guide/#_schedule

1 Like

STEP 3D: Deploy Workflow :rocket: as a Trigger :zap:

A trigger is essentially setting up a workflow to execute when a certain event happens. The type of the event can be configured when the deployment is created. Let’s take this workflow deployed as a trigger for example:

2023-11-03_10h56_48

For this example workflow it might sense to trigger execution when:

  • See the Excel Reader: a new Excel file appears in a selected folder on KNIME Business Hub. For example the file is uploaded manually by a user and that would mean new predictions are to be scored. In a machine learning setting this means we are scoring new data asynchronously: we trigger the new predictions (unlke a schedule deployment) only when there is new data and we update the database.

  • See the Model Reader: a new model file appears in a selected folder on KNIME Business Hub. For example the model might have been created by another workflow (via a Model Writer node) that is scheduled to re-train the model every 24 hours or so.

For this other example workflow it might sense to trigger execution also when:

  • Remember “Step 2: Versioning”? (scroll up): a new version has been created for the workflow being called by the workflow we trigger.

As you can see it gets ‘meta’ pretty quickly. Especially if you use in the trigger workflows nodes from the KNIME Personal Productivity Tools and the KNIME Hub Connectivity extensions: back to the orchestration topic!

An orchestration example could be making a new version of a data app workflow trigger a second workflow which updates the deployment of the data app. This way the data app end-users get the newest updates on the fly! This would require the workflow deployed as a trigger to have a Data App Deployment Creator node.

You can check more advanced settings below and on KNIME Docs (KNIME Business Hub User Guide)

3 Likes

Excellent compilation of the steps needed for Business Hub for various scenarios. May be this could be rolled out as a cheat sheet . Keep up the good work of KNIMEing

Rgds. KRISH

2 Likes

Hi there,
Thanks for the feedback!

A cheat sheet should be out soonish, likely beginning of next year.

For now (besides this forum thread) you can check out this playlist on YouTube:

Short playlist: Execution and Deployment on KNIME Business Hub - KNIME TV

Full course playlist: L3-CD Continuous Deployment and MLOps - KNIME TV

Preview:

E-learning platform (also free): knime.learnupon.com/catalog/courses/3465418

1 Like

STEP 4: Share Deployment :people_hugging:

For a data app (see STEP 3B) and service (see STEP 3A) deployments there is one final step: sharing the access to more KNIME Business Hub users.

You can find the deployment listed below in the workflow page or in the list of deployments of the KNIME Team.

For service and data app deployments you can select “Manage Access”.

You can give access to all users on KNIME Business Hub, or you can give access only to selected users.

A data app deployment will become available in their user “Data Apps Portal” where tags can be used to browse all the data apps shared to them.

For service deployments, the API endpoint will become accessible to those users authenticated via their “Application Password”. The users can create or delete application passwords in their user profile settings. These application password can be used in other workflows (via the KNIME Hub Authenticator and Call Workflow nodes), or in a block of code, to call the deployed workflow service.

3 Likes

The guide “KNIME Business Hub features in small steps” finishes here.

Heads up: the same steps should also be valid if you get deployment access on KNIME Community Hub (once available)!

Find the workflows adopted in the examples here:

Feedback? Let us know here!

3 Likes

Hey, thanks for the guide :slight_smile:

I have just one quick question regarding Trigger Deployments.

I have run some tests sending multiple files using the API to my test space to trigger an extremely simple test workflow, and I noticed that the jobs is triggered once for each file.

I didn’t see a setting for this, is there a way to run that deployment only once if multiple files are added at around the same time ? I didn’t have the chance to check yet, if the first job is still running, will it run the second one anyway ?

Thanks ^^

1 Like

I can’t see to be able to edit my previous post, so just to mention that I did try adding a Wait node to make sure that the job couldn’t be finished immediately, and I still had 2 jobs executed when sending 2 files at the same time.

Did I miss an option somewhere to force that behaviour, that exists in the schedules ?

Thanks :slight_smile:

1 Like

Hi @Vonwen,
thanks for posting in here your questions.

After double-checking with the KNIME team, I confirm that at the moment there is no setting for the following:

triggering a workflow only after multiple files appear in the folder during a time window.

The trigger will execute always and only once per file :frowning:

A workaround for this, based on the assumption that you know that multiple files are always created at the same time, is to have your API, or workflow service, to merge these events into 1 file. For example those multiple files could be written directly in as a zip file OR concatenated in a single file. The trigger workflow can then wait for the single file/zip to appear.

The trigger then will work once each time this kind of multi-file-event happens.

On the other hand, I would also keep in mind that executing one job per file should not cause issues if the triggered workflow is designed correctly. For example you can trigger each time a DB Writer node node, then schedule another workflow to read from such DB and perform more actions on the data collected from such files.

Happy KNIMing
Paolo

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

KNIME Community Hub (hub.knime.com) offers now a payed subscription for your team to execute workflow directly there!
Learn more at:

Register for the webinar (or watch the recording) at:

KNIME Business Hub private installations are still available but it’s no longer the only option!

2 Likes