Hi,
given a few Arrow features and integrations already present: any concrete plans on adding ADBC: Arrow Database Connectivity, primarily for workflows running in columnar backend mode?
Or is this already (partially) in place for DB <> Spark (or Databricks/BigDataEnvironments)?
I realize this is currently not implemented via “native” nodes, but you can easily connect to adbc compliant sources via duckdb’s extensions (they can also be installed once you import the jdbc to knime)
adbc_scanner – DuckDB Community Extensions
from what I grasped, Arrow is already somehow part of Knimes columnar table backend.
I might be able to grab data from a DB via ADBC into duckdb, but the last mile from DuckDB to Knime would still be JDBC / flight SQL and hence make a double conversion from Arrow (column) to row-wise (JDBC) and back to columnar.
I am mostly wondering if
a) Knimes columnar backend is actually arrow based
b) I can get DB data into Knime without double conversion then
(and at best with native nodes (no Java Snippet or Python Script node))
(maybe mixing up things with polars example now from Python - Arrow Polars Pandas – KNIME Community Hub)