Problem with disk space during workflow execution.

@takbb, @dbolshev you can configure streaming options it in the tab “Job Manager Selection” of the component.

Also you might want to consider other options. Also a tweaking of data storage might be an option. Parquet and ORC file formats offer a good combination of compression and keeping of one’s column types. And also they allow to handle larger files in chunks but also treat them as one (big) table if necessary.

3 Likes