Writing file to S3 bucket

KNIME Analytics Platform
1

Currently when I write a file into S3 bucket, it gives output files in partitions. How do I write a file to S3 bucket in a single CSV file, rather than the files being divided into multiple parts? A single file which contains the contents from all the partitions.

2

Hello @tjmbakwe,

and welcome to KNIME Community!

What nodes do you use and how does you workflow design looks like?

Br,
Ivan

3

Hi,
if you are using the Spark to CSV node you can override the Spark DataFrame partition using the Overwrite partitions count option which results in a single csv file. By default Spark is simply writing out as many files as the DataFrame has partitions.
Bye
Tobias

3 Likes
4

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.