LLM Prompter node - GenAI & Data workflow error

[mgirdwood]

Hi,
I am trying to run the workflow 02 Multimodal Prompting, contained in the GenAI & Data Workflows - Getting Started self passed course using an offline local LLM.

All the workflow details are “out of the box” but I am getting the following error:

Execute failed: Error code: 422 - {‘detail’: [{‘type’: ‘string_type’, ‘loc’: [‘body’, ‘OpenAICompatibleChatCompletionRequest’, ‘messages’, 1, ‘content’], ‘msg’: 'Input should be a valid string

Is this error caused by the specific model that I need to call or is there an issue with the way the data in the workflow is being input to the LLM Prompter node which is the combination of the Text +image details combined to a Message column ?

Thanks

[k10shetty1]

Hi @mgirdwood,

Could you share a bit more detail about how you’re accessing your local model?

Also, does your model support multimodal input (text + images)? The error might be related to whether the model can handle image content. Can you share some details on the local model that you are using.

Best,
Keerthan

[mlauber71]

@mgirdwood with Ollama and qwen3:8b there seems to be an answer given when modifying the example you mentioned (02 Multimodal prompting).

You will have to download the rest of the workflow group, install Ollama, expose this to the network and

ollama pull qwen3:8b

02 Multimodal prompting.knwf (140.4 KB)

---

–
Medium: Multimodal Prompting with local LLMs using KNIME and Ollama

[mlauber71]

For the fun of it I expanded the example and added a few tasks. Like letting the (local) model create a JSON file with the severity level of the accident and also assign a compensation based on the image and description. To be honest I am not 100% sure the local Quen3 model actually does look at the picture and it always agrees with the initial claim but the reasoning it displays are interesting and I think one can work on the prompts. It is interesting that the model will give back consistently a good JSON that you can then process further. And all this with a local LLM via Ollama. So no leakage to the web - run on a standard Apple M1 machine.

Mistral 3.1 ( mistral-small3.1 ) claims to actually ‘see’ the images. One would have to explore further if this is the case.

Ollama KNIME - Multimodal Prompting - Car Accident Severity Analysis – mlauber71

[mlauber71]

Another observation. The Mistral multi modal local model seems indeed to recognize images and also make a decision about the claim. Sometimes even opting for the higher one in the case of the broken window which it rates as a more severe damage then the initial claim.

Not bad for a purely local model. This indeed cloud help for a preparation. The calculation takes some minutes, but all the data stays strictly local.