I am trying to export the resulting table to a “Writer node” using a preprocessing node using Spark (e.g. “Spark Normalizer”, “Spark Column Rename”, etc.).
In this case, it seems essential to use a node like “Spark to table” to transform the data.
Is there any way to skip this intermediate process of “Spark to table” and export directly to “Writer node”?
Because, Spark is useful for handling big data, and if I use a node like “Spark to table” to convert the resulting table, it takes too much time for the node to operate.
I know that if I want to load a large amount of data from DB or Local (I loaded 100 million data as a parquet in Local) and connect it with a Spark node, I must go into the “Table to Spark” node.
Also, after the “Table to Spark” node, I need to go through several preprocessing nodes related to Spark, and then I need to go through the “Spark to table” node to save this data as a csv or parquet to my personal PC as a “Writer node”.
These “Table to Spark” and “Spark to Table” nodes took about 15 minutes to convert 100 million data. Is there any way to reduce this time?
Are transformation nodes like “Table to Spark” mandatory? Are there any other options?
@JaeHwanChoi where is this Spark process being done? On your local machine or on a server?
You need to take into account how Spark works with Lazy evaluation - Wikipedia. That means the process will only start once you want something done like writing the data to a table. The other spark nodes which seem very fast are just ‘plans’ until execution starts.
Using Spark will always come at an additional cost by the machine setting it up and starting the workers - the benefit is you can process very large amounts of data by using the memory of your hopefully many distributed servers. The performance very much depends on which resources you have in the form of RAM and processor and how many …
You can also write the result to a big data System, a table with Hive or Impala - which would be a standard procedure.
The Table to Spark and Spark to Table nodes transfer the data between the Spark cluster and the KNIME AP/Server, this might take some time on large data.
There are dedicated nodes to read/write the data using Spark on the cluster, like Parquet to Spark and Spark to Parquet. You can read/write them in KNIME later on using the Parquet nodes.
@JaeHwanChoi basic question: do you actually need big data technology because your data is so large it would not fit in a ‘normal’ server/computer; and if yes: do you have the resources to do it. The Create Local Big Data Environment – KNIME Community Hub ist just there to demonstrate and develop not to be used in real production.
I think you might want to just do the transformations on your machine/sever with traditional KNIME (or Python) procedures. Or you could use a database.