Workflow Schedule: Issues with configuration within the POST & PUT request

[RKromEX]

Hi all,

We have built a couple of workflows which are used by multiple of clients. Whenever a client executes the workflow, it will automatically determine which client is executing the workflow, what their cloud storage locations, preferred databases, etc. are.

This all works for us, but we have an issue since we want clients to be able to schedule these workflows, including choosing the filters & flow variables to use.
With the use of the POST, PUT and DELETE request API calls I’ve made a workflow which let our clients schedule a workflow on the set date and/or interval.

The issue comes when they need to configure schedule with flow variables (including filters and settings). In the API /rest/v4/repository/{path}:scheduled-jobs there is a possibility to set the configuration in JSON format to for example a String Configuration node.

To give you some background information, our clients need to execute the same workflows with different parameters/filters on different intervals (e.g. the workflow will loop through a list of assets and determines which assets to include in the current workflow execution: asset 1, 2 and 3 on weekly basis, asset 4,5,6 on monthly basis, asset 7,8 * 9 on a monthly basis).

When setting this String Configuration from within the KNIME AP, the flow variable is called Workflow Schedule JSON. However, this will not the way our clients will schedule workflows, as they will use the WebPortal.

Unfortunately in the API this String Configuration has the Node ID included in the configuration. In this example the Node ID is 10.

  "configuration" : {
    "workflow_schedule_JSON-10" : "Here comes the JSON configuration"
  },

In the POST request we cannot simply use the flow variable workflow_schedule_JSON, but we have to use workflow_schedule_JSON_10. This means we cannot automate this API configuration dynamically, because in every workflow this Node ID will most likely have another value.

My first question: Is there maybe another way to workaround this issue in the API configuration by not including the Node ID?
My second question: Is there a way to identify if a workflow is executed as a scheduled job (1) or manual execution (2)?

If we could identify if the workflow is executed as a schedule, we could read the configuration (specific for this scheduled job) from an external source (such as a database table). We can link these configuration via the Schedule Job ID/name (we’ve made the names unique) inside the workflow.

With the Extract Context Properties node we can extract the context.job.id variable, but this does not tell if the workflow is executed as schedule or manually executed.

Thanks!

RK

[julian.bunzel]

Hi @RKrom,

It is currently not possible to address the parameters of a Configuration node by just its name.
However, I have created a ticket for it.
As a workaround, you could use a Container Input node that takes REST requests and still keep the Configuration node to be able to configure the workflow via the AP.
For the configuration node, the parameter name would not matter anymore as the value that is shown in the Execution Options is the label of the Configuration node and not the parameter name.

The workflow could look like this:

You have a default value in the Table Creator (using the same parameter name in there) and pass it to the Container Input. The Container Input can be addressed by just using the parameter name. For the String Configuration we would use another parameter name then. Afterward, merge the variables
and come up with a rule for the final variable using a Rule Engine node.
For example: If the variable coming from the Container Input node is not the default value then take it as the final variable.

For the second question. You could have a small service workflow that retrieves the localhost logs from the server. You can send the job id to that service workflow and the workflow will then search the logs for entries like this:
21-Dec-2022 00:51:00.822 INFO [KNIME-Job-Loader_255] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.loadAndExecute Loaded job '9d5c9d96-687e-4bc9-8e91-487afe840153' for schedule 'Workflow....
Depending on whether an entry can be found or not, you can send either scheduled or unscheduled (or any other value) back to the caller workflow.

I hope this helps.

Best regards,
Julian

[RKromEX]

Hi @julian.bunzel,

Thank you very much for your reply. I’ve tried to build your logic in my test workflow, but at the moment I can’t find out how to use the Container Input (variable) in the API request. I was assuming I would see the new container input somewhere in the configuration tab as well.

In the request itself I’ve added the logic as well, but no success:

  "configuration" : {
    "workflow_schedule_JSON" : "test"
  },

Could you help me on how to use the container input?

[julian.bunzel]

Hi @RKromEX,

The Container Input is solely for REST requests (or if using the Call Workflow nodes) so it will not show up in the Configuration options.
The purpose of the workaround is to provide the option to either call the workflow using the Configuration option in the execution options or call the workflow via REST using the interface the Container Input provides. It is not using the configuration entry of the call but a different one. The Configuration part is only for Configuration nodes.
Documentation of how to invoke an execution via the Container Input node can be found here:
https://docs.knime.com/latest/analytics_platform_workflow_invocation_guide/index.html#workflow-invocation-nodes
There is an example for the Container Input (Variable) node specifically.

Best regards,
Julian

[RKromEX]

Hi @julian.bunzel,

At the moment I’m using the nodes POST Request, PUT Request, DELETE Request in the workflow. In the documentation I do some reference to being able to use the container in a rest, but I can’t make it work for my use case.

Below is an example of the PUT request (only changed the uuid)

{
{
  "@controls" : {
    "self" : {
      "href" : "hidden_for_forum_post",
      "method" : "GET"
    },
    "edit" : {
      "href" : "hidden_for_forum_post",
      "method" : "PUT",
      "title" : "Edit scheduled job"
    },
    "knime:delete" : {
      "href" : "hidden_for_forum_post",
      "method" : "DELETE"
    }
  },
  "id" : "hidden_for_forum_post",
  "numFailures" : 0,
  "workflowPath" : "/Admin/test",
  "reset" : false,
  "discardAfterSuccessfulExec" : false,
  "discardAfterFailedExec" : false,
  "targetName" : "Customer_Schedule_Weekly",
  "schedule" : {
    "startTime" : "2022-12-21T18:03+01:00[Europe/Amsterdam]",
    "nextScheduledExecution" : "2023-01-02T18:03+01:00[Europe/Amsterdam]",
    "disabled" : false,
    "skipIfPreviousJobStillRunning" : false,
    "delay" : 7,
    "delayType" : "DAYS",
    "filter" : {
      "times" : [ {
        "start" : "02:00:00",
        "end" : "23:59:59"
      } ],
      "daysOfWeek" : [ "MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY", "SATURDAY", "SUNDAY" ],
      "months" : [ "JANUARY", "FEBRUARY", "MARCH", "APRIL", "MAY", "JUNE", "JULY", "AUGUST", "SEPTEMBER", "OCTOBER", "NOVEMBER", "DECEMBER" ],
      "days" : [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32 ]
    }
  },
  "notifications" : {
    "recipients" : [ "hidden_for_forum_post" ]
  },
  "actions" : [ {
    "type" : "SendMail",
    "onSuccess" : true,
    "onFailure" : true,
    "onInactivityDiscard" : true,
    "appendNodeMessages" : true,
    "recipients" : [ "hidden_for_forum_post" ]
  } ],
  "executionRetries" : 0,
  "configuration" : {  },
  "user" : "nexusdi_admin",
  "@namespaces" : {
    "knime" : {
      "name" : "http://www.knime.com/server/rels#"
    }
  }
}
}

I’ve played around with the inputParameters section, but it doesn’t work yet. Do you mind assisting me with the setup?

[RKromEX]

In the meantime I’ve done some more testing, but so far no success.
I’ve tried several ways. The two ways which allow me to create the POST request are defined below:

   "workflow_schedule_JSON": {
	"workflow_schedule_JSON" : "pushed variable"
  },

  "workflow_schedule_JSON" : "pushed variable",

In both cases the jobs are executed successfully, but the Container Input (Variable) is always resulting in the set “default”, which is configured in the Container Input (Variable) (see exclamation mark in the node status) instead of the value “pushed variable” set in the POST request.

[julian.bunzel]

Hi @RKromEX,

I tried it out myself right now and it works fine for me.
Your request body needs to be either:

{
   "workflow_schedule_JSON": {
       "workflow_schedule_JSON" : "pushed variable"
   }
}

or

{
    "workflow_schedule_JSON" : "pushed variable"
}

However, the latter only works if you enable the Use simplified JSON format option in the Container Input node.

Additionally, you need to be careful in which request you are sending the request body.
There are two different ways of calling a KNIME workflow.
A one-step and a two-step approach.
The one-step approach uses the :execution endpoint of the workflow.
If you send a POST request to that endpoint of a workflow with one of the request bodies from above, it will take the input parameter, execute the workflow, and immediately discards the workflow afterwards.
With the two-step approach, you first create a new job by sending a POST request to the :jobs endpoint of the workflow. Afterwards, you can execute the workflow by sending a POST request to the individual job, i.e.:
http://<server-address>/knime/rest/v4/jobs/<job-id>

If you want to pass the parameter to the Container Input node of the workflow, you need to send the request body with the second request, so the one that starts the execution.
If you want to pass the configuration parameters via REST to the String Configuration node, you need to send the body with the first request which is loading and configuring the workflow.

I hope this helps.

Best regards,
Julian