Quantcast
Viewing all articles
Browse latest Browse all 12

Answer by brethvoice for Why is accuracy not the best measure for assessing classification models?

After reading through all the answers above, here is an appeal to common sense. Optimality is a flexible term and always needs to be qualified; in other words, saying a model or algorithm is "optimal" is meaningless, especially in a scientific sense.

Whenever anyone says they are scientifically optimizing something, I recommend asking a question like: "In what sense do you define optimality?" This is because in science, unless you can measure something, you cannot optimize (maximize, minimize, etc.) it.

As an example, the OP asks the following:

"Why is accuracy not the best measure for assessing classification models?"

There is an embedded reference to optimization in the word "best" from the question above. "Best" is meaningless in science because "goodness" cannot be measured scientifically.

The scientifically correct response to this question is that the OP needed to define what "good" means. In the real world (outside of academic exercises and Kaggle competitions) there is always a cost/benefit structure to consider when using a machine to suggest or make decisions to or on behalf of/instead of people.

For classification tasks, that information can be embedded in a cost/benefit matrix with entries corresponding to those of the confusion matrix. Finally, since cost/benefit information is a function of the people who are considering using mechanistic help for their decision-making, it is subject to change with the circumstances, and therefore, there is never going to be one fixed measure of optimality which will work for all time in even one problem, let alone all problems (i.e., "models") involving classification.

Any measure of optimality for classification which ignores costs does so at its own risk. Even the ROC AUC fails to be cost-invariant, as shown in this figure.


Viewing all articles
Browse latest Browse all 12

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>