Open AI connector - reopen of issue

[mgirdwood]

Hi
Sorry to reopen a closed Thread - Open AI connector - sort off - info - #8 by mgirdwood

Now I have the new 5.4.2 installed on the works PC I have been able to revisit this issue. (AI nodes were broken for me in the last version )

Using part of the the workflow Using KNIME LLM Nodes for Text Generation and Chat based Information Retrieval (RAG) created by Tosinlitics to test the connection to my companies local LLM again I believe that the initial connections are ok based on Green as shown in the image. However when the data is fed into the Prompter this fails with the info in the txt file.

LLM_Check.txt (8.9 KB)

In our documenation for the API there is the need to have the “/v1/chat/completions” added to the end of the API key but I can not see if this actually part of the Chat Model Prompter Node or if it should be added as part of the URL in the Open AI authenticator or elsewhere (i tried but same errors)

As mentioned previously I know when we are using Alteryx we needed to call a Powershell script in the workflow due to a proxy issue and this may be the actual problem here but I am trying to check my way through from the start. If i run the local LLM in Chrome it of course works so I had hoped that Knime might use these settings in its proxy call.

Is there any way I can provide additional details here to help work through the topic ?

Thanks as always

[MartinDDDD]

Hey there,

let me start with the general set up as it is intended in the KNIME Gen AI nodes:

• green: You set up your API key as password in credentials config node (or pull it using secrets manager if you use that in a KNIME Hub context). The requirement to add “/v1/chat/completions” to the end of your API key sounds really odd to me… have you tried “just” the API key in this set up?
• red: here you set up your companies endpoint. To use the OpenAI nodes you need to ensure that the endpoint you are using is OpenAI compatible. In your OpenAI Authenticator always point to the part until “/v1” so do not add “/chat/completions” here. That is what the Chat Model Connector / LLM Prompter etc. take care of automatically

So if your companie docs say your chat model endpoint is https://yourcompany.org/v1/chat/completions I’d try https://yourcompany.org/v1 in the authenticator node.

Another good way to check is what you see in your OpenAI Chat Model Connector node if you select “All Models”:

Above screenshot is from me using my local Ollama endpoint http://localhost:11434/v1
It lists the models that are available at the Chat Model endpoint [Preformatted text](http://localhost:11434/v1/chat/completions)

This might be another good way to verify if your set up is correct…

[mgirdwood]

Hi @MartinDDDD

I think I have the setting correct in the different nodes, as shown in the images below. As you can see I am going with only the V1 in the authenticator node, although I have tried it also with the chat/completion

[MartinDDDD]

Thanks for the additional information - can you tell me which model you are using?

In your Chat Model Connector it shows a mistral instruct model?

I think for instruct models you typically use the “old fashioned” completions endpoint… so should you be using that model this may be the cause of failure.

I see you already have an LLM Connector node set up - have you tried that in combination with LLM Prompter Node?

[mgirdwood]

@MartinDDDD - This is the model we are using at the moment along with Phi4. Could you point me towards the “old fashioned” endpoint ?

Connecting a LLM Prompter gives the same error messages with Instruct as well as the Phi4 models.

I just quickly checked both models in our Alteryx build and they do provide outputs.

[MartinDDDD]

The LLM Prompter should use the “completion” endpoint.

That’s why it is not encouraging to hear that the instruct model doesn’t work with Chat Model Connector => Chat Model Prompter nor with LLM Connector => LLM Prompter.

I take you are running this from your local computer and not from hub?

If it is a work computer - maybe a firewall issue can cause this or something on the server side…

You mentioned Alteryx - is that also going why local computer?

I have to admit that any firewall or server related issues are not my strong suit…

[mgirdwood]

@MartinDDDD
Thanks again Martin, Yes Alteryx is running from the same local machine.

What I wanted to check first was that the setup was correct and in theory ought to be working, which based on your knowledge, should have been the case.

So the next step will be to try to work out the “Connection” issue. I beleive it will be a Proxy problem - We had the same with Alteryx and had to include a Powershell call into the build up of the LLM coonector. Unfortunately I can not use LLama or GPT4All on the works PC to test any further here.

Probably better to close this post now and Raise a seprate one so as not to get confused in a long trail ?

[MartinDDDD]

Yes I think that’s the way to go and yes based on from what I can tell your general set up looks correct.

It’s odd that you get your connection error when prompting - typically if you authenticate with wrong API key or something you’ll already get an error in an earlier node as far as I know…

sorry for pestering you with questions and not finding a solution in the end…

[mgirdwood]

@MartinDDDD
No - Thank you for the time you gave on the topic, This looks now like a Network topic so new post