Hi,
This is a continuation of the issue I raised here: AVRO file reader?
I’m afraid I still can’t load local AVRO files using the provided solution. The Avro to Spark node throws the following error:
2020-10-28 14:01:12,427 : ERROR : KNIME-Worker-12-Avro to Spark 0:3 : : Node : Avro to Spark : 0:3 : Execute failed: An error occured. For details see View > Open KNIME log.
java.lang.NullPointerException
at org.knime.bigdata.spark2_4.api.TypeConverters.getConverter(TypeConverters.java:121)
at org.knime.bigdata.spark2_4.api.TypeConverters.convertSpec(TypeConverters.java:162)
at org.knime.bigdata.spark2_4.jobs.genericdatasource.GenericDataSource2SparkJob.runJob(GenericDataSource2SparkJob.java:82)
at org.knime.bigdata.spark2_4.jobs.genericdatasource.GenericDataSource2SparkJob.runJob(GenericDataSource2SparkJob.java:1)
at org.knime.bigdata.spark.local.wrapper.LocalSparkWrapperImpl.runJob(LocalSparkWrapperImpl.java:127)
at org.knime.bigdata.spark.local.context.LocalSparkJobController.lambda$1(LocalSparkJobController.java:92)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I’m not sure I’ll be able to post the AVRO I’m trying to process here, because it’s semi-sensitive and is around 300MB. I might be able to post PyCharm’s interpretation of the schema portion.
As an aside, I still think it would be easier if we had a Read AVRO node that just uses the relevant apache libraries, rather than mucking about with spark.
Cheers,
Richard