Python script for loop error

I am using python script node for computing stock technical indicator computation using ta-lib library.

I am using for loop, for stock symbol that has different number of rows in the data frame. I am getting following error while executing code via python script (writing loop in node itself or in jupyter):

“Execute failed: ValueError: all batches of the table must have the same size, but batch 4 has size 67 (expected: 81)”

I checked the knime documentation for batch processing, it says that number of rows should be the same in each batch except for last batch.

I am able to perform my analysis using group loop node, however it’s very slow and want to have better solution.

How to resolve this issue? Knime is working on this issue?

Hi @MNVADGAMA,

Sadly, due to internal implementation details of the columnar backend the batches in the Tables all have to have the same size (except for the last one). This is also causing other issues so we have plans of getting rid of this limitation. However, this needs some work. I cannot give an estimate on when this will happen.

Can you somehow re-write your script to respect this limitation? The batch sizes given in the error message seem quite small. If you do not expect too many rows you could just use a single batch (and/or fill the actual table at the end of the script).

3 Likes

Thanks @bwilhelm for your response.

1 Like

What does your workflow look like? Just curious because of the jupyter notebook you mentioned
br

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.