# S3 List Files Performance

**URL:** <https://forum.knime.com/t/s3-list-files-performance/43052>\
**Category:** KNIME Analytics Platform\
**Created:** [May 17, 2022, 12:37pm UTC](https://forum.knime.com/t/s3-list-files-performance/43052 "2022-05-17T12:37:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aaron\_Hart](https://forum-cdn.knime.com/user_avatar/forum.knime.com/aaron_hart/32/7814_2.png) [@Aaron\_Hart](https://forum.knime.com/u/Aaron_Hart)\
**Post date:** [May 17, 2022, 12:37pm UTC](https://forum.knime.com/t/s3-list-files-performance/43052/1 "2022-05-17T12:37:16Z")

</div>

The new file handling nodes are really great for working in complex environments but I am running into a performance issue when recursively listing the contents of an s3 directory when compared to the same operation using the AWS CLI.

For example, if you take 1000 files distributed randomly in say 10 folders, each of which has 2-3 sub folders, I see a 10-100x performance difference between the list files node and the aws cli and the gap gets worse as the directory tree grows.

Does anyone have any insights on why this is the case and if anything might be possible to boos the performance of this node?

Thanks!

Aaron

---

<div class="post-metadata">

**Author:** ![nsas](https://forum-cdn.knime.com/user_avatar/forum.knime.com/nsas/32/30899_2.png) [@nsas](https://forum.knime.com/u/nsas)\
**Post date:** [May 25, 2022, 7:47pm UTC](https://forum.knime.com/t/s3-list-files-performance/43052/2 "2022-05-25T19:47:44Z")

</div>

Hi Aaron,

Just did a measurement from my computer:  
`time aws s3 ls s3:// ******--profile***** --recursive 2.84s user 0.14s system 39% cpu 7.624 total`  
`time aws s3 ls s3:// ******--profile***** --recursive 2.74s user 0.13s system 39% cpu 7.313 total`  
`time aws s3 ls s3:// ******--profile***** --recursive 2.70s user 0.15s system 38% cpu 7.441 total`

Using the List Files/Folders node the average runtime was: 11.323 ms

The bucket I used had 11.174 files and folders, the typical subfolder depth was 3-4.

Is this in line with your experience?

Norbert

---

<div class="post-metadata">

**Author:** ![Aaron\_Hart](https://forum-cdn.knime.com/user_avatar/forum.knime.com/aaron_hart/32/7814_2.png) [@Aaron\_Hart](https://forum.knime.com/u/Aaron_Hart)\
**Post date:** [June 16, 2022, 7:49am UTC](https://forum.knime.com/t/s3-list-files-performance/43052/3 "2022-06-16T07:49:23Z")

</div>

Thanks for the comparison, that is not at all our experience! Such an operation would never complete with the List Files node in our environment. I wonder what the difference could be?

---

<div class="post-metadata">

**Author:** ![nsas](https://forum-cdn.knime.com/user_avatar/forum.knime.com/nsas/32/30899_2.png) [@nsas](https://forum.knime.com/u/nsas)\
**Post date:** [June 17, 2022, 8:24am UTC](https://forum.knime.com/t/s3-list-files-performance/43052/4 "2022-06-17T08:24:58Z")

</div>

Hi Aaron,

Maybe a proxy or something that makes the network latency big?  
I am attaching the WF I used to measure the runtime:  
[aaron.knwf](https://forum.knime.com/uploads/short-url/ecDg2bUxNCW7L5Vs3Hgtl7p7vaU.knwf) (13.0 KB)

---

<div class="post-metadata">

**Author:** ![system](https://forum-cdn.knime.com/uploads/default/original/1X/ab3ccf34482a0329361734a18199390177204f15.png) [@system](https://forum.knime.com/u/system)\
**Post date:** [September 15, 2022, 8:25am UTC](https://forum.knime.com/t/s3-list-files-performance/43052/5 "2022-09-15T08:25:04Z")

</div>

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.