I watched the presentation on youtube by about these new node. Interestingly enough like the example Greg made, my interest would be around molecules. Anyway I pretty much took the example workflow with some minor adjustments and my own dataset (binary classification).
I played around with the results a bit as I’m not used to the new metrics and have no intuition about them. I tried a error rate of 0.05 and then wanted to verify if that error rate is actual true. There were 0 null predictions, all predictions with both labels means no error so all one needs to look at are the single-label predictions. And that is where I found a concerning issue, at least for my use-case. Simply said basically all of the error is on the minority class which in terms of molecules is almost always the class of interest. The total error was actually 7% (I’m ok with that small difference) but for the minority class this meant a 43% error rate.
Not sure what my actual question is but I considered that an important observation which was not intuitive to me. One really needs to remind oneself that the error is on the whole prediction set but within it is not uniformly distributed.
My question is more what I make of this, What would actually be interesting in the terms of molecules (eg making a suggestion to chemist what to make in lab) is being able to have a reliable error rate on the class of interest, eg. like precision but in this more statistical robust way and not just calculated from CV. Or what am I missing?