Edited by Pier Giuseppe Giribone

Another way to categorize learning systems is based on how they are able to generalize.

Most of the expected goals of machine learning algorithms concern their ability to make predictions. This means that, given a number of examples on which to train, the system must be able to make good predictions on new, or previously unknown, data.

Finding good performance metrics on training data (“in-sample”) is certainly positive, but not sufficient, since the real objective is to obtain reliable results on the new instances.

The measures “out-of-sample” therefore represent the best tool to verify whether the system has been able to correctly generalize the knowledge acquired during training.

There are two main approaches to generalization:instance-based learning and model-based learning.

The simplest form of learning, which falls into the first category, is simply learning by rote. This means that if we were to design a spam filter that followed this heuristic, it would mark as spam all emails that were exactly identical to those already classified this way by other users. This is probably not the worst way to solve the problem, but certainly not the best.

Instead of flagging emails that are identical to those already recognized as spam, the algorithm could be programmed to also flag emails that are very similar to known spam. This design approach, while obviously more reasonable, requires a similarity measure between two emails.

A very simple measure of similarity could, for example, be represented by the number of words in common that two emails have: the system will flag them as spam if the comparison between the new instance and those already known to be spam reveals many words in common.

This way of learning is called instance-based learning: the algorithm learns the examples provided during training as faithfully as possible, then generalizes this knowledge to new cases using a similarity measure for comparison with already known elements (or a subset of these).

Another way to generalize from a set of examples is to build a mathematical model of a potential relationship between those examples and use it to make predictions. This form of generalization is called model-based learning.

Suppose we have a data set that can be represented using a scatter plot, that is, with points arranged on a Cartesian plane. This could be the case, for example, of the relationship between the OECD (Organization for Economic Cooperation and Development) Better Life Index (https://www.oecdbetterlifeindex.org) and GDP per capita (Gross Domestic Product).

While it is well known that money alone cannot bring happiness, statistics clearly show that it can be used as a tool to help people achieve this noble goal, while obviously considering ethics and mutual respect for other present and future inhabitants of our planet.

Therefore, a positive trend line would be noted between the dependent variable (index score, y-axis of the graph) and the independent variable (GDP per capita, x-axis of the graph). The points would be arranged with a reasonable approximation along this increasing trend line. The mathematical model that could appropriately represent this relationship is therefore linear. This can be expressed in the form Score = theta0 + theta1*GDP.

The phase described is called model selection: that is, among all the mathematical functions that could represent the relationship between the available data, the most suitable one is chosen. This choice may be the result of intuition, as in the case just discussed, or it may derive from a rigorous mathematical proof verified under certain hypotheses.

For example, the relationship that allows us to calculate the price of a bond from its yield is based on rigorous, well-known principles of financial mathematics, so it would make no sense to build an algorithm that learns it directly from empirical data or through an intuition of a representative model deduced from them.

Once the relationship that links the score with the GDP has been chosen, the most suitable values ​​must be established for theta0 (linear model constant) and theta1 (slope of the linear model), or those that best represent the trend present in the data.

In order to obtain optimal values ​​for the model parameters, it is necessary to specify a performance measure. Typically, one can either define a utility function (utility function, also called fitness function) which measures how good the model is, or a cost function (cost function) which measures how unfit the model is.

For a linear regression problem, such as the one presented, a cost function is typically used, which measures the distance between the theoretical predictions provided by the linear model and those that actually occurred in the data used to train the model: the objective is to minimize this distance.

At this point, the learning process comes into play: the training data is passed and the values ​​for the most representative parameters are found, that is, those that best generalize the relationship assumed to exist between the data.

It is worth noting that in the case of linear regression, there are analytical formulas that allow us to find the values ​​for the theta: it is in fact an OLS – Ordinary Least Squares.

In the case of nonlinear models, a numerical procedure is used to solve the model training problem.

Once the training phase is complete, predictions can be made using the calibrated model.

For example, suppose there is a country whose GDP is known, but whose score has not been calculated by the OECD.

You can use the linear relationship Score = theta0 + theta1*GDP, replacing the value of the independent variable (Gross Domestic Product) with the known value of the country, use the estimated theta and obtain as output (i.e. as prediction) the value of the dependent variable, i.e. the score interpreted by our model.

In conclusion, it can be observed that the most common approach used by traditional econometrics is the model-based approach, in which, once the existing relationship between the data has been chosen, the best estimate is made to determine the coefficients of that model.

The instance-based learning, by not making a priori hypotheses on the functional form to which the relationship existing between the data must necessarily adapt, has received particular emphasis with the advent of new Machine Learning paradigms.

We conclude by asserting that “There is no such thing as a free lunch”: if on the one hand instance-based learning presents greater flexibility in defining the relationship that links the data, as it does not have to be specified a priori, it is worth highlighting that models with this type of learning are more affected by a potential over-fitting of the data (overfitting) and tend to be “black-box”, with a consequent lack of explainability of the prediction obtained.