How to extract Livy's internal Context settings values

[JaeHwanChoi]

Hello KNIME support team.

When using Spark in connection with Livy, I want to receive the information of the internal Context settings created from the Create Spark Context(Livy) node as a Flow Variable.

For example, after executing Create Spark Context(Livy), when I open the Spark Context result window, I can see Livy’s Context settings information like below.

Here, I want to pull out the value of spark.app.id, livy-session-92, as a flow variable, is there any way?

Your answer will be appreciated.

[ScottF]

I don’t think this is possible with just the node itself. Maybe you could use a PySpark Script Source to get at the Spark properties that way? But I’ll defer to @sascha.wolke for a more educated answer

[JaeHwanChoi]

@JaeHwanChoi

Context settings

Hi, @sascha.wolke

Is there a way around this? I need to terminate a Spark session via the API, so I need to extract the values in the Context settings.

Thank you.

[sascha.wolke]

Hi @JaeHwanChoi,

Maybe the Destroy Spark Context node helps already?

Otherwise, you can create a new Dataframe inside a Dataframe or PySpark snippet, use Spark to Table and then convert it into a flow variable.

Cheers,
Sascha

[JaeHwanChoi]

Thank you for your response. @sascha.wolke

I have a situation where I terminated the Spark Session with the Destroy Spark Context node, but due to unexpected circumstances, the Destroy Spark Context could not be executed and the WorkFlow was terminated.

Therefore, I want to get the generated Session id & name and force termination externally via API, which requires the Session id & name value in Context Settings as shown above.

You mentioned that I can pull that value by creating a DataFrame and connecting it as a flow variable, but that doesn’t work for me. Do you have an example of this?

Any help would be greatly appreciated.

[sascha.wolke]

Hi @JaeHwanChoi,

You might use the PySpark Script Source node with something like this:

resultDataFrame1 = spark.createDataFrame( [ { 'appId': spark.sparkContext.applicationId } ] )

Cheers,
Sascha