@wiswedel that is odd as none of the cores were even close to being fully utilized per windows task manager. The table Format was generated by the Test Data Generator Node configured using v1 instead of legacy.
I do recall that I mentioned before that this node, because it generates outdated data types, seems in need of an update. Maybe adding v2 or “most recent” would be an option.
Nevertheless, I also converted all special data types, including the outdated date&time as well as duration data to simple strings in order to use a CSV writer. Whilst being slightly faster, write speed drops over the course of hours to the same level and basically never finishes.
As far as I understand the nested data must get deserialized which would explain the fluctuating write speeds as seen in this screenshot of mine from before:
About the backend type, it is the default. I will run some benchmarks to narrow things down like:
- Removing legacy data
- Testing with current default row vs. future columnar backend type
Best
Mike