# Optimization needed List Files, Python Source, DB Writer **URL:** \ **Category:** KNIME Analytics Platform\ **Tags:** batch, database, python\ **Created:** [January 31, 2020, 4:45pm UTC](https://forum.knime.com/t/optimization-needed-list-files-python-source-db-writer/20765 "2020-01-31T16:45:12Z")\ **Posts on this page:** 1\ **Showing post:** 4
**Author:** ![mlauber71](https://forum-cdn.knime.com/user_avatar/forum.knime.com/mlauber71/32/73590_2.png) [@mlauber71](https://forum.knime.com/u/mlauber71)\ **Post date:** [January 31, 2020, 7:45pm UTC](https://forum.knime.com/t/optimization-needed-list-files-python-source-db-writer/20765/4 "2020-01-31T19:45:06Z")
Hmm difficult to judge from the screenshots. One idea could be to first collect all of your data locally and then load it into the DB in one single step. Also if you have 13 parallel sessions accessing the DB from one account it may well be that the configuration would slow that down (only accepting a certain number of connections at one point) - another question would be the connection. How is the DB you are trying to write to connected to your system. Then you mention you write to a MySQL DB. Is that a remote one or one on your machine. And what configurations are there. In general you should think about your setup and reduce the number of DB operations to a minimum. You could see if certain calculations in KNIME can be done in memory - depending on your systems RAM. You would have to identify which nodes cost the most time in oder to do something about it. And here we have a list of discussions and blog entries about KNIME an performance. You might want to take a look if something strikes you as an idea to explore further. * * * KNIME performance > **[Optimizing KNIME workflows for performance | KNIME](https://www.knime.com/blog/optimizing-knime-workflows-for-performance)** > > KNIME provides performance extensions such as the KNIME Big Data Connectors for executing Hive queries on Hadoop, or the KNIME Extension for Apache Spark for training models on Hadoop using Apache Spark. But sometimes it doesn’t make sense to run... Process 900+ CSV files > [@Processing hundreds of millions of records](https://forum.knime.com/t/processing-hundreds-of-millions-of-records/13593/5): > > A few remarks since it is difficult to judge from the screenshots: if you have to process that many files and lines it could make sense to do it in chunks and have some method in place to log the current status and be able to restart at a certain point. Eg. write results to disk after each 50 out of 900 processed files or something. Because if you do not have a very powerful infrastructure, compartmentalization might be a thing for you also this saving of steps in-between might function as som… > [@Large data tables missing?](https://forum.knime.com/t/large-data-tables-missing/13108/2): > > Will the large dataset be imported and the problems will occur later or do you experience problems also when loading data? If you have problems with import it might help to use the [R library Readr](https://forum.knime.com/t/error-with-file-reader-node-while-running-in-a-loop/11263/3) instead of KNIME file reader. I know of no formal restriction for data in KNIME but it depends on how much power your system might bring. These points might be able to help you check out the links below with hints from KNIME and other information about performance with large datasets from CSV it mig… > [@Knime 3.6 Crash when dealing with massive data](https://forum.knime.com/t/knime-3-6-crash-when-dealing-with-massive-data/12145/2): > > you could try and increase the assigned memory in the knime.ini configuration file but with only 4 GB there are limits to that ([https://www.knime.com/blog/optimizing-knime-workflows-for-performance](https://www.knime.com/blog/optimizing-knime-workflows-for-performance)). On other option is to set the “Memory Policy” of the (sorter) node to “Write tables to disc”. Obviously this increases the load on the hard drive but it might save on RAM. [image] Also you could try and close other workflows, restart KNIME an try to use the garbage collector. [image] [image] T… > [@KNIME Takes Long to Load?](https://forum.knime.com/t/knime-takes-long-to-load/12669/4): > > You might want to check your virus scanner. On certain systems with quite aggressive virus scanners I experienced KNIME to be slow. Since it can contain several 1,000 single files an ‘agressive’ virus scanner might scan all of them every time they are accessed/opened. When we changed the settings KNIME started faster. Of course there is a trade off between security and speed. But modern virus scanners might be able to have a flexible configuration. But of course the number of AddOns and the ove… > [@Knime is slowing down](https://forum.knime.com/t/knime-is-slowing-down/12619/2): > > That is certainly odd. To the best of my knowledge, there haven’t been any recent major changes to KNIME that are known to have a degrading effect on performance. From my experience, the most dominating performance factor in KNIME is the materialization of intermediate data tables on disk. Have you noticed the performance of your hard disk degrading independently of KNIME? While we have some performance improvements for KNIME in the pipeline, here are some small adjustments you can try out: Y… --- _[View the full topic](https://forum.knime.com/t/optimization-needed-list-files-python-source-db-writer/20765)._