Where is the logs file in the KNIME Business Hub?

Hello everyone.

In the KNIME Server, I can find all the knime logs file in the following location:

Question 1:
But in the KNIME Business Hub, where is the logs file which stores the all logs info like the above knime.log in the KNIME Server?

Question 2:
So I click the workflow in the console page, and click “download logs”


But no log info is shown:

Thanks in advance
Ryu

Hi Ryu,
That the log for the job is not available should not occur, as it is saved with the workflow. You can contact our support at support@knime.com to get help figuring out why that happens. Otherwise, you find the executor logs in the output of the executor Pod in the Hub’s Kubernetes cluster. You can get all executor pods by running

kubectl get pods -n hub-execution

And if you have found the right executor pod (executor ID is included in the pod name), you can run

kubectl logs <exec_pod_name> -n hub-execution

to print the logs.
Kind regards,
Alexander

2 Likes

Hello @AlexanderFillbrunn

Thanks for your answer.
Now I can find the logs by the above commands.

Best regards.
Ryu

1 Like

Hello @AlexanderFillbrunn

I am so sorry , the logs printed in the Java Snippet are not found in the above command.

Do you know where the logs in the Java Snippet print?

Thanks in advance.
Ryu

Hi,
It should be there. I just tested it and got the output (4th line from the bottom in the screenshot).
Kind regards,
Alexander

Hello @AlexanderFillbrunn

I am so sorry, I know the reason now. The executor pod (executor ID) is changed, so only logs from the last several hours can be displayed.

How to find previously printed logs(printed in the previous executor pod)?

Best regards.
Ryu

Hi Ryu,
If the pod changed, I think the logs are gone unless you save them using a separate system like Splunk. kubectl logs has an option to print logs from the previous container

-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.

But if the pod changed, the logs are probably gone.
Kind regards,
Alexander

Hello @AlexanderFillbrunn

Our Development server shutdowns during weeknights and holidays to reduce AWS EC2 instance’s cost , and our production server also shutdowns in the holidays automatically.

So after restarting the servers, all the logs will be lost because pod is changed.
It would be nice if this problem could be solved.

Kind regards,
Ryu

Hi,
I agree. I am in contact with our developers to get information what they recommend for persisting logs. In the meantime, I recommend this resource for logging options: Logging Architecture | Kubernetes.
Kind regards,
Alexander

1 Like

Hi @AlexanderFillbrunn

Do you get the information about persisting logs from developers?
I will appreciate if you could tell me how to do it step by step.

Best regards
Ryu

Hi @laughsmile,
We do not have step-by-step instructions yet how to do it, but I recommend using https://www.fluentd.org/, which should be pretty straigh-forward. You can either set up its Kubernetes integration, or you just use it as a normal Linux process to watch the logs in /var/log/containers. I was planning to try it out myself, but haven’t had time yet. I will update here when I did.
Kind regards,
Alexander

1 Like

Hi,
I now successfully set up log persistence with KNIME Business Hub on AWS with S3. It is pretty basic, but logs ended up in S3.
The most important links are:

Here’s what I did:

  • Install Helm on a machine that has kubectl access to the KBH cluster
  • Create a new private S3 bucket for the log files
  • Create a new IAM policy that allows putting files into the bucket
{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"s3:PutObject"
		],
		"Resource": "*"
	}]
}

Note: this is the policy from the fluentbit documentation, but you can most likely limit it to the bucket you created in the previous step.

  • Then create a new IAM role that uses the policy you created above.
  • Attach the newly created role to your EC2 instance where KBH runs by going to Actions → Security → Modify IAM role in the AWS console.
  • Download the default values.yaml file for the Helm installation of fluentbit.
  • Modify the file’s config → outputs section so instead of ElasticSearch it sends data to S3. It should look somewhat like this:
outputs: |
    [OUTPUT]
        Name s3
        Match *
        bucket my-knimehub-fb-logs
        region eu-west-1
        store_dir /home/ec2-user/buffer
        total_file_size 50M
        upload_timeout 10m

Region and bucket name can of course be chosen by you and have to correspond to the bucket you created above.

  • Add the fluent-bit helm repo to Helm:
    helm repo add fluent https://fluent.github.io/helm-charts
  • Install fluent-bit with the modified values.yaml file:
    helm install fluent-bit fluent/fluent-bit -f values.yaml

Now fluentbit will start a DaemonSet in your cluster so that one pod runs on each node. The container mounts /var/logs and can therefore read all log files that are produced on that node. Periodically, it will send the data to S3. To check if everything is alright, you can run kubectl get pod and check if there is a running fluentbit pod in the default namespace. Via kubectl logs <fluentbit-pod-name> -f you can see what fluentbit does and at some point it should write something like:

[2023/12/19 14:56:27] [ info] [output:s3:s3.0] Successfully uploaded object /fluent-bit-logs/kube.var.log.containers.knime-postgres-cluster-0_knime_postgres-34244af5f0202272746255d256c555161b8e460ecb3f60d29c083e6b6dd2e0a6.log/2023/12/19/14/46/19-objectL2ZwAhg0

Now you should find the logs in S3.
What I have not done yet, but what may make sense, is to set up S3 to delete files of a certain age automatically via lifecycle rules in order to not accumulate too much log data. You can also automatically let it move old logs to slower and cheaper storage.

I hope this helps!
Kind regards,
Alexander

5 Likes

Hi @AlexanderFillbrunn

Thank you so much.
I will follow the procedure to check the logs persistence with KNIME Business Hub on AWS with S3.

Best Regards
Ryu