How to merge vector stores?

[MartinDDDD]

Hey there,

it looks like with KNIME 5.4 there is a new functionality that makes updating vector stores simpler:

Simplified GenAI model maintenance for retrieval-augmented generation (RAG)

Managing and updating vector stores in RAG workflows can be complex and time-consuming, requiring manual effort to ensure the knowledge base stays accurate and aligned with evolving data.

KNIME Analytics Platform now comes with a new Vector Store Data Extractor node for updating and migrating vector stores to help you simplify the maintenance of knowledge bases used in RAG workflows, ensuring your data remains up-to-date and workflows remain efficient.

There are two example workflows available that I recommend to take a look at:

Vector store creation – knime

RFI Q&A Chatbot – knime

I think a similar solution was provided here:

Load embeddings from file into FAISS Vector store creator

Hello @Ellison, In the example you shared the list was a collection of strings, I used a simple lambda function using a python script node to convert it to a list of doubles as below : df['column'] = df['column'].apply(lambda x: [float(i) for i in x]) Now you can see the column appear in the FAISS Vector Store Creator node. [image] You have to connect to OpenAI Embeddings connector to use it further as shown here. [image] If you want to download the updated workflow, you can find it here.…

Hope that helps you!