OpenAI Tokenizer

I am playing with “GPT4All - Create your own LLM Vector Store” at GPT4All - Create your own LLM Vector Store – KNIME Community Hub

There are a few tokenizers to be selected at PDF Parser, how to work out if I want to use OpenAI tokenizers instead of what is provided by default?

@anguslou the idea of this example is to have an offline variant without sending data thru the internet to a third party.

There are examples how to build a vector store with OpenAI and knime

Or using Azure

1 Like

Got it, I understand the purpose of this flow now.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.