A/B Test - Frequentist

[Gavin_Attard]

Hi Folks

Trying not to re-invent the wheel here, so after any suggestions of nodes or node combinations.
Assume the below data, what nodes can i use to derive the statistical significance of my test result

I am looking to output:

Relative Lift in CR
Z-score
Observed Power
p value
Standard Error A
Standard Error B
Std Error of Difference

INput data:
Control: Users = 64299, Conversion=647, CR=1.01
Variation: Users = 63864, Conversion=699, CR=1.09

cheers

G

[thor_landstrom]

Hello @Gavin_Attard,

I believe the nodes you are looking for to calculate the statistics you want can be done through the math formula node and python script node.

The math formula can handle the std error of difference, zscore, relative lift, etc.

For example, if we wanted to calculate Standard Error A (which I assume would be either the first row, or maybe different table for control and variation), we can do something like this:

Where inside the math formula we do:

Now, this may not be the correct formula you are looking for in your case as I came down to two types, Standard Error and Standard Error for Proportions. I went with the proportions because it looks like you are dealing with conversions.

Also for more complex calculations, I would suggest using the python node.

Hope this helps,
TL