I am looking to see if anyone is using KNIME to create and query Vector Databases that will jump me forward with a starting point. My goal is to create a local KNIME AP based component that allows the user to empower Chat GPT with custom data while minimizing the amount of data required (and token cost) in an API call. I am currently sending larger prompts with embedded ASCII tables in my component, but would like a more efficient and dynamic solution.
I have found python solutions for all of the steps, but I was hoping to minimize the python / external service dependencies so that components can more easily be shared with the wider community. Below is an explanation of the goal.
No sorry, so far only done it with code. There is no vectordatabase in KNIME is there? (referring to sth. like the H2 db in KNIME)
What is the use case in KNIME for you regarding that?
Curious to see what “KNIME community swarm intelligence” comes up with. (So my comment is rather a bookmark apologies for that)
br
there is nothing that works out-of-the-box, yet, but we are working on filling this particular gap.
I can’t reveal the details as of now but there will be news soon.
That saves me spinning my wheels on a workaround. I would much rather wait until there is a cleaner vector database integration than try to hack something together. Thanks for the info!
As far as the use cases, they seem nearly infinite on my end if you check out that article link.
Internally I would use this technique as a more efficient way to leverage Chat GPT during data prep and enhancement, build custom Chat GPT “chat bots” that could query Client Policies docs and return workflow guidances, provide my own internal procedures as guidelines, provide a wider depth of knowledge on certain specific platforms, etc.
Sorry for the confusion. I get the use case of vector dbs but not in which way it is useful within KNIME. To me it would be useful in combination with a frontend with user input to ask for infos about an interal document or sth (similar to openai interface).
But this might be due to seeing KNIME as an ETL Tool and I do not have access to KNIME Server
br
I already find Chat GPT to be useful in the ETL process even without the ability to customize its knowledge base with a Vector DB. I built a basic component with inputs to pass data tables into the prompt in ASCII format with separate outputs for text responses, table responses and Regex so that it can incorporate the responses downstream in the workflow. (The component is not shared yet, as it is still in its infancy.) I also use the component output to generate things like samples of format and table structure requirements which need to be matched in order to import into another program. I have especially found it useful for basic data enrichment tasks. Things like adding columns for the state, county and township associated with a zip code in the table, or adding population data for an area.