Hello, I’m a beginner in the knime tool and new here on the forum and I currently have a question about how to ingest data into elasticsearch.
The initial structure looks like this:
It consists of extracting data from the SQL Server database. I then read the data and then convert it from table to JSON. I then try to use String Manipulation to format them into Bulk format so that Elasticsearch will accept them. Next, I use the ElasticSearch Bulk Load node. Currently, only the ‘Index’ is being sent, but the data is not being included. I will leave here the configuration of each step of the workflow.
But the idea is to let this flow update automatically within Elasticsearch without repeating any records.
I’ll leave you with each configuration
“DB Query Reader”:
“Table to JSON”
“String Manipulation”
“ElasticSearch Bulk Load”
The structure at the moment is this, and with that it is only creating the “Index” without the data.
Could anyone help me with this?
Hi there,
I think the string concatenation is not quite correct and will not give you valid JSON. However, have you already seen this example workflow? Might make things easier:
-Philipp
Hello Philipp,
Thanks for the answer, in this case on this NodePit website can I download this Elastic workflow for free? and in relation to the structure of my workflow, do you think this flow would work? Or would you add more things?
Hey talo_Gabriel_de,
your workflow will probably require some structural changes to work with the nodes used in the example, which are the dedicated Elastic Search nodes:
I have been using these myself in the past successfully when I needed to connect KNIME with ES - for reading from ES indices and writing into ES. They are not free, but there’s a free trial of 30 days available. The node are built and maintained by @danielesser , who is also reading the forum here occasionally - so you can reach out to him in case of questions.
Hope that helps!
Philipp
1 Like
Hello Philipp,
Thank you very much again for responding, but at the moment I live in Brazil the currency is devalued, and my company wants something free to run with airflow later on the entire flow, so I believe that if there is no other free way through KNIME, I believe that I have to go to another structure route, if there is no other option than Elastic NodePit
Yours sincerely!
Ítalo
