Choice for time series looping

Hello,

I could use some help identifying the proper type of loop to use from among the wide variety of choices.

Goal: To use the prior 250 rows (days) of data to calculate a confidence score and assign that to the next 30 rows (future days), then move forward 30 rows and recalculate the score for the prior 250, etc.

So, rows 1-250 would have no score. Rows 251-281 would have the score based on rows 1-250. The score for rows 282-313 would be based on data from rows 31-281, etc.

If somebody can point me in the direction of a workflow or component that has something like this already done, I’m happy to modify it from there. I just didn’t get the sense that any of the ones I looked at would do what I wanted (or I’m doing it wrong).

The input is a prediction table created with cross validation.

Thanks in advance!

Hi @cybrkup,
I would suggest to use the Group Loop Start Node, however in this case you need a set of columns (or just one column), which categorize your whole dataset in order to split it into rows 1-280, 281-560,…
The node should reference this set of columns as Group columns. Then each loop pushes the next set of 280 rows.
However, if this set-up isn’t possible, you could use the chunk loop start node and “Rows per chunk” -> 280.

After the loop start use a Row splitter in order to split the first 250 “history” rows and 30 “future” rows.

regards, Tommy

1 Like

Hi there @cybrkup,

if I got you right Recursive Loop Start/End nodes should do the trick as well. Start with 250 rows, then calculate next 30 days, do some manipulation to feed the End node with rows 31-281 and iterate.

Br,
Ivan

Thanks for the suggestions.

While looking at how to implement your ideas, I ran across “Window Loop,” which is exactly what I was looking for.

1 Like

That is the node I was trying to find! Though unsuccessfully :man_facepalming:

Actually, now that I’ve played with this more, it is somewhat different than what I was looking for. This breaks down the bars, but not in a walk-forward manner. So, back to looking at using the other ideas.

For completeness https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.time.node.window.LoopStartWindowNodeFactory :slight_smile:

1 Like

Hmmm,

maybe share an input data and desired output? Don’t have to be real data. Example data :slight_smile:

Br,
Ivan

Yep… here goes…

Row Input Result
1 10
2 15 12.5
3 20 17.5
4 10 15
5 5 7.5
6 20 12.5

The average Input value of rows 1 and 2 is 12.5. Then the Inputs of rows 2 and 3 are averaged for the Result of row 3, which is 17.5, and so on…

So each iteration a step is taken to include the next row and remove a prior row from consideration. In this case, there is a two-day look back period, with a one step iteration.

Ideally, I’d like to have a 250 row look back with a 30 step iteration, but I’d certainly settle for a 250/1 if it is easier to implement.

I hope that is a clear enough example.

I’ve thought about incrementing a variable and then using row spliter, but nothing is working as a solution just yet.

Hi @cybrkup

I was thinking of the Moving Average node. Did you try that node?
gr. Hans

I was using average values as an example, but the score I’m trying to calculate is much more complicated. I have the score calculation ready, but not the means by which to step through the data and pass the correct rows to my scorer.

Hi,

seems we are missing something…

So again why is Window Loop not working for you? In first iteration you take 250 rows and calculate your score (or score for 30 next rows). Then in second iteration you take 31-280 rows and do the same and so on…

Br,
Ivan

My bad, unsurprisingly. The Window Loop does work as long as you filter out all the extra rows created by the looping process. I had seen all the duplicate Result values an mistook what the loop was doing.

By adding a row splitter and only including Row #1 immediately before Loop End, this works.

Thanks again for the remedial help!

2 Likes

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.