The challenges of ignorance

Abstract

The authors have previously argued that the infamous "No Free Lunch" theorem for supervised learning is a paradoxical result of a misleading choice of prior probabilities. Here, they provide more analysis of the dangers of uniform densities as ignorance models, and point out the need for a framework that allows for prior probabilities to be constructed in a more principled fashion. Such a framework is proposed for the task of supervised learning, based on the trend of the Bayes error as a function of the number of features employed. Experimental measurements on a number of standard classification tasks confirm the representational utility of the proposed approach.