Correlation Filter for Panel Data #THK-DSFB-2023


In the course of data reduction for our project, we would like to implement a correlation filter for our panel data set, which should act as follows:

All variables are to be sorted out that correlate below a certain threshold with a previously defined key figure. TimeLags should be considered, e.g. variable A at time (t-1) correlates strongly with the defined key figure but variable A at time (t-2) does not correlate strongly with the key figure. We would limit the lags to a maximum of t-4. The variables of t0 are not to be taken into account, cause they cant be used for a prediction.

So, after the correlation filter, we might be left with one column with variable A at time t-1 and at time t-4. On the one hand, we don’t even know how to incorporate a correlation filter, which can only calculate the correlation of all variables to a given variable and then sort it out. Secondly, we do not know how to reformat our panel data set for this purpose. In the picture you can see how our data set is structured. A simple column lag would mix the ids which is not allowed.

Thanks in Advance!

Hi @palterau

Have you checked out the Correlation Filter node?

If you scroll down to the Related workflows & nodes section, you’ll find example workflow that people uploaded which uses the Correlation Filter.


This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.