OpenAI Tokenizer

[anguslou]

I am playing with “GPT4All - Create your own LLM Vector Store” at GPT4All - Create your own LLM Vector Store – KNIME Community Hub

There are a few tokenizers to be selected at PDF Parser, how to work out if I want to use OpenAI tokenizers instead of what is provided by default?

[mlauber71]

@anguslou the idea of this example is to have an offline variant without sending data thru the internet to a third party.

There are examples how to build a vector store with OpenAI and knime

3.1 - Create a Vector Store – knime

Or using Azure

2.5 - Retrieval Augmented Generation ChatApp – knime

[anguslou]

Got it, I understand the purpose of this flow now.