ElasticSearch Index -- Loop Through Indexes

Community Extensions
2

Hi @CodiDixon1,

In case you don’t specify an index, the node will search across all available indices using _all (see documentation). Anyhow, ES will just ignore the index field and return matching documents according to your query from the pool of all documents. It will not query each index individually on its own.

From what I understand, the following is exactly what you want to achieve: Execute a query individually against 10+ indices and concatenate the results of each individual query (instead of a combined one).

The ES Nodes currently do not support this. Anyhow you can easily solve this with a loop:

  1. Prepare your ES search with Elasticsearch Connector, Elasticsearch Search and Elasticsearch Query nodes
  2. Fetch all available indices via the Elasticsearch Console node from endpoint _cat/indices?format=JSON
  3. Loop over each index, add the index via flow variable to Elasticsearch Index and execute the query via the Elasticsearch Reader

image
image888×266 26.1 KB

I uploaded an example workflow to my NodePit Space. You might not be able to run the workflow (as it uses a local cluster on my machine), but it should give you a good idea what I actually described above.

Does this help you? Please let me know if you struggle somewhere.

Best regards,
Daniel

1 Like