Instance-Based and Model-Based Learning

One way to categorize Machine Learning systems is by how they generalize. There are two main approaches to generalization: instance-based learning and model-based learning.

Instance-Based Learning

For instance-based learning, the system learns the examples by heart, then generalizes to new cases by using a similarity measure to compare them to the learned examples or a subset of them.

Model-Based Learning

Another way to generalize from a set of examples is to build a model of these examples and then use that model to make predictions. This is called model-based learning.

For model selection, you can either define a utility function or fitness function that measures how good your model is, or you can define a cost function that measures how bad it is.

If all went well, your model will make good predictions. If not, you may need to use more attributes, get more or better-quality training data, or perhaps select a more powerful model.

It is crucial to use a training set that is representative of the cases you want to generalize to. If the sample is too small, you will have sampling noise, i.e., nonrepresentative data as a result of chance, but even very large samples can be nonrepresentative if the sampling method is flawed. This is called sampling bias.