# Large volumes with Knmie **URL:** \ **Category:** KNIME Server\ **Tags:** database, big-data\ **Created:** [January 24, 2025, 5:52pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931 "2025-01-24T17:52:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![fmc00006](https://forum-cdn.knime.com/user_avatar/forum.knime.com/fmc00006/32/69156_2.png) [@fmc00006](https://forum.knime.com/u/fmc00006)\ **Post date:** [January 24, 2025, 5:52pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/1 "2025-01-24T17:52:37Z")
Good evening I am planning to use Knime, mainly, as an ETL tool to populate data into a company datawarehouse, either in local version or in server version mode. Do you know the kind of limitation on volume or performance you have experimented working with large volumes of structured data? I mean the volumes of data that your workflows have managed with some Server o local computer hardware characteristics? Thanks in advance ---
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [January 24, 2025, 6:37pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/2 "2025-01-24T18:37:30Z")
@fmc00006 I can offer this article with my experience. > **[KNIME Snippets (1): Collect and Restore — or how to handle many large files...](https://medium.com/low-code-for-advanced-data-science/knime-snippets-1-collect-and-restore-or-how-to-handle-many-large-files-and-resume-loops-c57795b65d7e)** > > “KNIME Snippets” is a series of short articles dwelling on some specific KNIME subjects. Follow me on Medium to get more. Mostly the question is how large is the data and what is the power of the system where knime is running and the connection to the server where the database is. Then with large files and knime - Do it in chunks / [loops](https://medium.com/low-code-for-advanced-data-science/knime-paths-and-loops-automate-everything-213455788685) - Use streaming when possible - check underlying data formats like parquet (external) or columnar backend (internal) ---
**Author:** ![PBJ](https://forum-cdn.knime.com/user_avatar/forum.knime.com/pbj/32/31549_2.png) [@PBJ](https://forum.knime.com/u/PBJ)\ **Post date:** [January 25, 2025, 1:30pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/3 "2025-01-25T13:30:40Z")
We are working with large structured data (more than 6.000.000 records) without any problems. (reading large Excel files into CSV files and ETL). ---
**Author:** ![fmc00006](https://forum-cdn.knime.com/user_avatar/forum.knime.com/fmc00006/32/69156_2.png) [@fmc00006](https://forum.knime.com/u/fmc00006)\ **Post date:** [January 25, 2025, 2:41pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/4 "2025-01-25T14:41:26Z")
Thank you very much for answering me. Your answers are very helpful. Kind regards Fernando ---
**Author:** ![Add94](https://forum-cdn.knime.com/user_avatar/forum.knime.com/add94/32/77533_2.png) [@Add94](https://forum.knime.com/u/Add94)\ **Post date:** [January 25, 2025, 2:49pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/5 "2025-01-25T14:49:05Z")
If you use duckdb jdbc with DB nodes: [https://mvnrepository.com/artifact/org.duckdb/duckdb\_jdbc](https://mvnrepository.com/artifact/org.duckdb/duckdb_jdbc) And if your hardware has plenty of ram and a fast SSD, you can seamlessly process even a billion+ rows locally in seconds. I often work with 100+ M rows on a Thinkpad with 16 gb ram and Ryzen 5 7535u cpu only - all queries (incl parametric queries , window functions etc) done within seconds. ---
**Author:** ![fmc00006](https://forum-cdn.knime.com/user_avatar/forum.knime.com/fmc00006/32/69156_2.png) [@fmc00006](https://forum.knime.com/u/fmc00006)\ **Post date:** [January 25, 2025, 3:05pm UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/6 "2025-01-25T15:05:48Z")
Wow!!! I’am very impressed. Thanks for your comments. ---
**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\ **Post date:** [February 8, 2025, 11:59am UTC](https://forum.knime.com/t/large-volumes-with-knmie/85931/7 "2025-02-08T11:59:39Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.