Merging many vector stores

[longoka]

What are strategies for merging large sets of FAISS vector stores? I’ve used the tool concatenator node for merging smaller sets of stores (see the example image attached below where I’ve connected to stores for use with a query to ChatGTP-4). But as my embeddings become more numerous, sizable and complex, I’m wondering if there is a judicious way to do this in KNIME (i.e., avoiding using the tool concatenator nodes thousands of times!). Or do I need to do this outside of KNIME for now (e.g., using the from_texts and merge_from functions from LangChain)? Or am I thinking about all of this too narrowly and missing better strategies entirely?

[longoka]

I think I can answer my own question here. It appears langchain has methods for merging vectorstores, described in merging FAISS vectorstores.

So this can be handled quite nicely with a python snippet.

[ScottF]

Thanks for posting what you found to benefit everyone!