Recursive loop not returning all data with GET Request

[mnguyen]

Hello,

I’m using the Recursive Loop with GET Request node to pass the next url with the right offset. It works if I have “Collect data from last iteration only” checked in the Recursive Loop End but it would only return the last iteration of data. If I unchecked it I get an error message “ERROR Recursive Loop End 3:112 Execute failed: Cell count in row “Row0_1_1#1” is not equal to length of column names array: 75 vs. 185”. I have searched on the forum but I could not find a resolution. Below is the flow diagram, hope someone can help me sort this out. Thanks,

[takbb]

Hi @mnguyen, are you able to upload a minimal sample of your exported workflow that exhibits the problem you are having, so that we can try running it for ourselves? Hopefully somebody will then be able more easily able to assist you. Without that it’s difficult trying to picture what it’s doing. Thanks.

[mnguyen]

Hi @takbb., Thanks for responding, I’m not able to load worklfow, but basically i’m passing back the from the recursive loop end the next_url for next iteration, and from the console log it shows “Cell count in row “Row0_1_1#1” is not equal to length of column names array: 74 vs. 184”. Below is log from the console:

WARN String Manipulation 3:103 Row “Row0_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_1_1_1_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN String Manipulation 3:103 Row “Row0_1_1_1_1_1_1_1_1_1_1_1” contains missing value in column “next” - returning missing (omitting further warnings)
WARN Column Rename 3:116 Invalid type index: 2
WARN Column Rename 3:116 Invalid type index: 2
WARN Column Rename 3:116 Invalid type index: 2
WARN String Manipulation 3:103 Row “Row0_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN Column Rename 3:116 Invalid type index: 2
WARN String Manipulation 3:103 Row “Row0_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
ERROR Recursive Loop End 3:112 Execute failed: Cell count in row “Row0_1_1#1” is not equal to length of column names array: 75 vs. 185
WARN Column Rename 3:116 Invalid type index: 2
WARN Column Rename 3:116 Invalid type index: 2
WARN String Manipulation 3:103 Row “Row0_2” contains missing value in column “next” - returning missing (omitting further warnings)
WARN Column Rename 3:116 Invalid type index: 2
WARN String Manipulation 3:103 Row “Row0_1_2” contains missing value in column “next” - returning missing (omitting further warnings)
ERROR Recursive Loop End 3:112 Execute failed: Cell count in row “Row0_1_1#1” is not equal to length of column names array: 74 vs. 184
WARN Column Rename 3:116 Invalid type index: 2

Here is the variables use to pass into recursive loop end.

[takbb]

@mnguyen , without seeing the config on the individual nodes I am working very much blind here. I see you use JSON Path and then Ungroup to presumably pull out information into a table, but then you are using JSON to Table, which having looked on forums doesn’t necessarily do what you might expect.

Let’s ignore the recursive loop for a moment. If you simplify your workflow and remove the recursive loop nodes, make your table a single row containing one URL, and do away with all the lower nodes in your diagram, so your JSON to Table node joins directly to the Column Filter at the end,
ie

do you get the expected result in your CSV Writer for one url?

Can you then do the same for what would be your expected second url?

I suspect that you aren’t actually getting what you expect at the end of each iteration.

[takbb]

My working assumption is that on different iterations, the data set that you are trying to collect contains a different number of columns. I can replicate this behaviour with a very simple workflow (attached)

KNIME_resursive_loop_fail.knwf (15.3 KB)

If you run this workflow then on the first iteration the top data set is used, but on the second iteration the lower data set is used. If you modify the Column Filter (that feeds into the Recursive Loop End for the subsequent iteration) to include a third column in the lower data set, you get an error similar to the one you are seeing.

If you set the recursive loop end to “Collect data from last iteration only” the error “goes away” because it no longer matters that the data sets on each iteration have different columns, since it’s only collecting the final one.

So my guess is that your workflow is having similar problems. Hope that helps.

[Iris]

@mnguyen Did you try to debug this? Maybe there are columns added.

i would recommend two thing.

1. You can step through the workflow manually, see here a video on how to step: https://www.youtube.com/watch?v=oNFUCZDHnew
2. Use a Table Validator node. it can be configured to fail when the input table is changing.

[mnguyen]

Thank you for your explanation, I now understand the error message and I’m able to get pass the issue.

[mnguyen]

Hi @Iris, Thank you your respond, I see you mentioned on other post to use the recursive loop to get the next pagination json result, would you have a sample workflow to share?