Python Script unsupported DataType issue.

[JaeHwanChoi]

Hi KNIME Support

I am not getting any results from my Python Script because it is an unsupported DataType. The code is working successfully in Jupyter Notebook.

I need to put all the values that exist in the form of a dict in one column of a DataFrame. When I print inside the KNIME’s Python Script node, I get the same result as in the Jupyter notebook, but when I execute the node, I get the following error and the KNIME Python Script I used is shown below.

I’d be grateful for an answer.

[carstenhaubold]

Hi @JaeHwanChoi,

What format exactly would you expect to be stored inside the KNIME table if there’s a column containing a dict?

If you expect a JSON value, then there’s a simple solution: Upgrade to KNIME 5.1 We have added support for JSON (= Python dict) and XML (= Python’s xml.elementtree.ElementTree) cells in Python in the last KNIME release.

Best,
Carsten

[gonhaddock]

Hello @JaeHwanChoi

A solution using KNIME 4.7; You can alternatively take a look to this post:

Python Script node. How to work with output table?

@Maya Thanks for the additional information. If you want to bring the response into KNIME as a JSON object then you can convert the dictionary to a JSON string and return that as a cell in the KNIME table as follows. Note, you will need to import json and pandas as pd. I’ve included a dummy dictionary to demonstrate the workflow. Note: If you have the response as a JSON you do not need to convert it to a dictionary, you could just return it in the dataframe by changing json_string to response. …

Py Script node:

import json
import knime.scripting.io as knio

import pandas as pd

dict_df = {'model_1': {'indicator_1': 0.65
            , 'indicator_2': 0.63
            , 'indicator_3': 0.88}
        , 'model_2': {'indicator_1': 0.83
            , 'indicator_2': 0.76
            , 'indicator_3': 0.93}}

json_string = json.dumps(dict_df)

df = pd.DataFrame([{"dict_detail": json_string}])

knio.output_tables[0] = knio.Table.from_pandas(df)

Adapted @DiaAzul 's solution…

BR

[JaeHwanChoi]

Hi @gonhaddock

Is it possible to code this same situation in Pyspark?

Any answers would be appreciated.

[gonhaddock]

Hi @JaeHwanChoi
I’m not a Pyspark user. I don’t think I can support you with this request.
Let’s see if other knimers in forum can help.

BR

[Daniel_Weikert]

assume your data vol is to big to just convert it to pandas df and then back to pyspark?
br

[JaeHwanChoi]

@gonhaddock

Since it is a large data, I don’t think I can convert it to pandas, so I need to put it in one column in Json form within Pyspark Code.

Thanks