ElasticSearch Index -- Loop Through Indexes

[CodiDixon1]

@danielesser Hi Daniel. I am facing an issue where we have multiple indexes (10+) that I need to query but do not want to set up an “Index–Reader–JSON Path” flow for every index. I have tried both leaving the Index blank in the Index node, and omitting the Index node altogether and connecting straight to the Reader node; neither returns the correct number of records. Are you aware of a way to loop through available indexes and execute the query against all?

Thanks,
Codi

[danielesser]

Hi @CodiDixon1,

In case you don’t specify an index, the node will search across all available indices using _all (see documentation). Anyhow, ES will just ignore the index field and return matching documents according to your query from the pool of all documents. It will not query each index individually on its own.

From what I understand, the following is exactly what you want to achieve: Execute a query individually against 10+ indices and concatenate the results of each individual query (instead of a combined one).

The ES Nodes currently do not support this. Anyhow you can easily solve this with a loop:

1. Prepare your ES search with Elasticsearch Connector, Elasticsearch Search and Elasticsearch Query nodes
2. Fetch all available indices via the Elasticsearch Console node from endpoint _cat/indices?format=JSON
3. Loop over each index, add the index via flow variable to Elasticsearch Index and execute the query via the Elasticsearch Reader

I uploaded an example workflow to my NodePit Space. You might not be able to run the workflow (as it uses a local cluster on my machine), but it should give you a good idea what I actually described above.

elasticsearch-index-loop-through-indexes-25523 [Workflow] — NodePit

Does this help you? Please let me know if you struggle somewhere.

Best regards,
Daniel

[CodiDixon1]

This is exactly what I needed. Thanks so much!