# Parallel Processing of Multiple Excel Files Based on Dynamic Sheet Names from List Files/Folders Node in KNIME **URL:** \ **Category:** KNIME Analytics Platform\ **Created:** [October 23, 2024, 9:40am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034 "2024-10-23T09:40:44Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![san\_98](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/e47c2d/32.png) [@san\_98](https://forum.knime.com/u/san_98)\ **Post date:** [October 23, 2024, 9:40am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/1 "2024-10-23T09:40:45Z")
Hi, I want to process multiple Excel files in parallel without using loops in KNIME, based on dynamic routing from the **List Files/Folders** node. Each file will be routed to a specific workflow depending on its sheet name, allowing for parallel processing. The file order may change, and there can be more than 3 files at times. For instance, if there are 3 files, each file should be routed to a separate workflow running in parallel, ensuring that each file gets processed independently of the others. The file paths and sheet names should be dynamically handled without looping, and all workflows should run simultaneously. ![{13B4132E-09FD-40CA-958F-75DC6295C226}](https://forum-cdn.knime.com/uploads/default/original/3X/6/7/6731d4877f60e597da7cee894ea389fda76d0a7f.png) ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 23, 2024, 9:46am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/2 "2024-10-23T09:46:25Z")
@san_98 will the sub-[workflow(s) which are being called](https://hub.knime.com/s/-OjK8KaTPRdnoL_4) processing the Excel files all be the same or do they have different functions and if so will the input and output ports differ? Is there a specific reason not to use [loops](https://medium.com/low-code-for-advanced-data-science/knime-paths-and-loops-automate-everything-213455788685)? ---
**Author:** ![san\_98](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/e47c2d/32.png) [@san\_98](https://forum.knime.com/u/san_98)\ **Post date:** [October 23, 2024, 10:41am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/3 "2024-10-23T10:41:31Z")
Hi @mlauber71 , Thanks for the quick response. The sub-workflows being called for processing Excel files will be the same, but the SQL table used within the workflow will differ based on the specific Excel file being processed. For **Excel 1** , the workflow will query a certain SQL table for data insertion, and for **Excel 2** , it will query a different table for data insertion. Essentially, the parallel workflow logic remains consistent, but the table names will change dynamically depending on the input file. ### Reasons to Avoid Loops: **Parallelism** : When using loops, KNIME processes one file at a time, which can be slower for a large number of files. Parallel processing allows multiple files to be processed simultaneously. ---
**Author:** ![MartinDDDD](https://forum-cdn.knime.com/user_avatar/forum.knime.com/martindddd/32/69988_2.png) [@MartinDDDD](https://forum.knime.com/u/MartinDDDD)\ **Post date:** [October 23, 2024, 11:10am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/4 "2024-10-23T11:10:55Z")
Looking at your screenshot I want to make you aware of the following: If you use the Table Row to Variable node, it will turn the **first row** into a variable only. In order to process both paths you’d have to iterate over the table using e.g. Table Row to Variable Loop Start Node. I think if you indeed go with an approach that calls a different workflow (e.g. via Call Workflow Service node) you may be able to go with minimal “conditional” logic inside your loop (pretty much just the “routing logic” that determines which workflow has to be called) and provided that the Call Workflow Service Node does not have to return an output that is processed in your caller workflow, any looping delay may be negligible. Here’s a very minimal example… ![image](https://forum-cdn.knime.com/uploads/default/original/3X/9/e/9e5e53d98d196ddd93b8e50251a15e419f2306b8.png) [CallerCallee.knar](https://forum.knime.com/uploads/short-url/o4GwlbNRgsaOwtqLpzc2sM9HsbF.knar) (243.9 KB) Edit: I actually just tested this and it looks like even if you don’t expect something to be returned the loop still only continues once the workflow you call finishes… Right now I am actually not sure how to make it work that both processes happen in parallel. ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 23, 2024, 11:11am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/5 "2024-10-23T11:11:02Z")
> [@san\_98](#): > > **Parallelism** : When using loops, KNIME processes one file at a time, which can be slower for a large number of files. Parallel processing allows multiple files to be processed simultaneously. The problem you wil be facing is that you can process jops in parallel in KNIME but in your case the SQL commands might query the same database/database connection. The question will be how to make sure that this will accept parallel connections. What kind of database is it BTW? Also: why do you need sub-workflows? You might as well use Components or Metanodes that could run in parallel. ---
**Author:** ![san\_98](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/e47c2d/32.png) [@san\_98](https://forum.knime.com/u/san_98)\ **Post date:** [October 23, 2024, 12:25pm UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/6 "2024-10-23T12:25:54Z")
I used the **List Files/Folders node** to retrieve all the files available in the folder, regardless of the file name. Then, I used the sheet name to read all the available sheets in the Excel files. In my case, I will have more than three sheets in the Excel file. After retrieving the sheets, I applied the **Rule Engine node** to split the workflow based on conditions. - If the sheet names are the same, the workflow runs in a **loop** to process the sheets. - If the sheet names are different, the workflow runs in **parallel** to process them concurrently. This approach ensures that when the sheet names are identical, the process loops through them one by one. If the sheet names vary, the workflow executes in parallel for efficiency… Please refer the workflow attached. ![{BA2E987F-86C8-4AF0-B0CE-4FA26A796B9D}](https://forum-cdn.knime.com/uploads/default/original/3X/0/e/0ec1e44ae28e45ff574befe1281b527931b6d3fa.png) ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 23, 2024, 3:34pm UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/7 "2024-10-23T15:34:18Z")
@san_98 In your screenshot I do not see any Parallel Chunks and no database access, but if you are fine we are fine 🙂 For the fun of it I expanded on @MartinDDDD workflow and constructed a parallel access with a database node calling sub-workflow(s) that would then use this connection. So far it seems to work although any benefits in speed might depend heavily on the actual database(-connetion) supporting parallel access. ![image](https://forum-cdn.knime.com/uploads/default/original/3X/8/3/8334f581426fff5e7e0a489d4c3bbe3296a934b5.png) The callee workflow does look like this: ![image](https://forum-cdn.knime.com/uploads/default/original/3X/1/3/133ff5ae1a9c37329383f87c8f85d882998e1894.png) If you want you can explore this further > **[Caller Workflow with parallel Access to SQL Database – mlauber71](https://hub.knime.com/mlauber71/spaces/Examples/Caller%20Callee%20-%20Parallel%20Access%20to%20SQL%20Database/Caller%20Workflow%20with%20parallel%20Access%20to%20SQL%20Database~orI2FhxzLYEd4MBo/current-state)** > > Several examples of how to call a workflow. One exampel how to call workflows in parallel chunks and access a SQL database ---
**Author:** ![san\_98](https://forum-cdn.knime.com/letter_avatar_proxy/v4/letter/s/e47c2d/32.png) [@san\_98](https://forum.knime.com/u/san_98)\ **Post date:** [October 24, 2024, 5:24am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/8 "2024-10-24T05:24:55Z")
> [@mlauber71](#): > > In your screenshot I do not see any Parallel Chunks and no database access, but if you are fine we are fine 🙂 Hi @mlauber71 , Thanks for the solution! It seems that the workflow runs in parallel without the Parallel Chunk Start node in my case. Additionally, I couldn’t find the Parallel Chunk Start node in my KNIME 5.3.2 version. We are performing some Excel sanity checks using a Python script, and after that, we’ll add the MS SQL connection for further parallel processing. Is your KNIME version above 5.3.2? ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [October 24, 2024, 6:05am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/9 "2024-10-24T06:05:45Z")
@san_98 you might have to activate the All Nodes settings: > [@Where is the DB Selector Node?](https://forum.knime.com/t/where-is-the-db-selector-node/83816/2): > > @michael390 welcome to the KNIME Forum. The selector is here: You can also search for it in the Nodes section: You might have to activate “All nodes” in the Filter configuration. [image] If the extension is not installed knime should prompt you to do so once you load a workflow that requires it. You can also install additional extensions [![](https://img.youtube.com/vi/opGCHeWZ_p0/maxresdefault.jpg "How To Install Extensions in KNIME Analytics Platform") ](https://www.youtube.com/watch?v=opGCHeWZ_p0) ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [October 31, 2024, 6:06am UTC](https://forum.knime.com/t/parallel-processing-of-multiple-excel-files-based-on-dynamic-sheet-names-from-list-files-folders-node-in-knime/84034/10 "2024-10-31T06:06:42Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.