I'm trying to implement the CuSum algorithm - https://en.wikipedia.org/wiki/CUSUM - which is basically a sequential calculation.
I tried many methods but I'm apparently missing something.
1) With a recursive loop I go through the ordered table line after line. (inspired by the elegant solution presented by Iris in her video).
2) I update the CuSum variable (using "replace" in the java node)
Problem is that the Java node doesn't really replace the variable but creates a second instance. From several contributions I understand that it's the intended behaviour to put the values in a stack. I force the updated variable to the Recursive end loop node but it clearly doesn't pass the value to the Recursive-Loop-start.
I'm not sure if it passes the bottom one or if the Recursive-Start takes the variables value from the predecessor node since both return value = 0.
In the above screenshot I expect 0.7865 to be passed as the value to the next step but it actually starts with a value = 0.
Any help is appreciated, either on finding the bias or with a different approach to calculate the sequential CuSum.
I'll reformulate a simpler question :)
in a Recursive loop, does the loop start node take its variables value from the predecessor node or from the loop end node ?
All the best,
I found a solution. It is not elegant but it works :
physically write the variable value into a file forces the expected top value of the stack to be written. It can then be read at the beginning of the loop.
tip : I had to connect the entry point of the read-variable input port so that the value is read from file at each step of the loop and not from the node cache.
very interesting solution!
About the last question, the variables are taken from the column filter node. So if you want to reuse the variables created in the loop end, you need to save them in a table and use the version of the recursive loop with two ports. The second port is also pushed back and can than be used as a "Variable Table".
Would you mind sharing the workflow with us?
I'm sorry I thought I would receive a notification in case of a reply to the thread.
Here is a simplified and annotated workflow.
The data is very simple : Week and Market Share
Goal of Cusum : detect a significant and sustainable increase in the data
Inelegant parts of the solution : (takes minutes to run where sub-second is expected)
- use a file to store variables
- java node for a max function is probably overkill
- R node to access the Z-score table is probably overkill
I'll try your approach with the double port recursive loop.
All the best,
I applied your suggestion to use the Double Port Recursive Loop in order to remove the variable file Read/Write and it works just fine !
Here is the workflow, it can now be used as a metanode to perform the CuSum algorithm (still slow).