Hi guys,
this is just to let you know: There is a problem I guess, and I could not find it in Python itself.
…hours later I switched from apache arrow backend back to serialized…and suddenly everything works.
AP: 4.5.1 windows
And yes, I am using a manual environment on purpose (no conda), since that will also be the case on knime server (out of my hands).
Python 3.9.9 with all packages up to date.
I tried uploading some data to bigquery.
With apache arrow:
with bigquery API directly:
‘index_level_0’
Traceback (most recent call last):
File “”, line 3, in
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\google\cloud\bigquery\client.py”, line 2635, in load_table_from_dataframe
job_config.schema = _pandas_helpers.dataframe_to_bq_schema(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\google\cloud\bigquery_pandas_helpers.py”, line 383, in dataframe_to_bq_schema
sample_data = _first_valid(dataframe[column])
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\core\frame.py”, line 3506, in getitem
indexer = self.columns.get_loc(key)
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\core\indexes\base.py”, line 3623, in get_loc
raise KeyError(key) from err
KeyError: ‘index_level_0’
with pandas_gbq: (essentially the same error, its just an easier wrapper)
‘index_level_0’
Traceback (most recent call last):
File “”, line 1, in
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\core\frame.py”, line 2055, in to_gbq
gbq.to_gbq(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\io\gbq.py”, line 212, in to_gbq
pandas_gbq.to_gbq(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas_gbq\gbq.py”, line 1142, in to_gbq
connector.load_data(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas_gbq\gbq.py”, line 559, in load_data
chunks = load.load_chunks(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas_gbq\load.py”, line 237, in load_chunks
load_parquet(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas_gbq\load.py”, line 129, in load_parquet
client.load_table_from_dataframe(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\google\cloud\bigquery\client.py”, line 2635, in load_table_from_dataframe
job_config.schema = _pandas_helpers.dataframe_to_bq_schema(
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\google\cloud\bigquery_pandas_helpers.py”, line 383, in dataframe_to_bq_schema
sample_data = _first_valid(dataframe[column])
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\core\frame.py”, line 3506, in getitem
indexer = self.columns.get_loc(key)
File “C:\Users\torsten.goedeke\knime_manual_env\lib\site-packages\pandas\core\indexes\base.py”, line 3623, in get_loc
raise KeyError(key) from err
KeyError: ‘index_level_0’
Once I switch back to serialized buffer in knime preferences…it suddenly works (both versions).
I can check for logs in a few days, if you can’t replicate the error.
best,
Torsten