May 21, 2022

# Monotonicity constraints in machine learning

The design will a little overfit (due to little min_child_samples), which we can see from plotting the worths of X versus the anticipated worths of Y: the red line is not monotonic as we d like it to be.

#predicted output from the model from the very same input.
forecast = overfit_model. forecast( x.reshape( -1,1)).

Default model mse 37.61501106522855 Monotone design mse 32.283051723268265.

from sklearn.metrics import mean_squared_error as mse.

monotone_model = lgb.LGBMRegressor( min_child_samples= 5,.
monotone_constraints=” 1″)
monotone_model. fit( x.reshape( -1,1), y).

import lightgbm as lgb.
overfit_model = lgb.LGBMRegressor( silent= False, min_child_samples= 5).
overfit_model. fit( x.reshape( -1,1), y).

Because we understand that the relationship in between X and Y should be monotonic, we can set this restraint when defining the design.

Lets fit a fit a gradient enhanced model on this information, setting min_child_samples to 5.

What may easily happen is that upon building the model, the data scientist discovers that the design is behaving unexpectedly: for example the design forecasts that on Tuesdays, the customers would rather pay \$110 than \$100 for a room! The reason is that while there is an anticipated monotonic relationship between rate and the likelihood of reservation, the design is unable to (completely) capture it, due to noisiness of the data and confounds in it.
Too frequently, such restrictions are disregarded by practitioners, specifically when non-linear models such as random forests, gradient boosted trees or neural networks are utilized. And while monotonicity constraints have actually been a subject of scholastic research study for a very long time (see a study paper on monotonocity restraints for tree based approaches), there has been absence of support from libraries, making the problem tough to tackle for professionals.
Thankfully, recently there has actually been a lot of progress in various ML libraries to permit setting monotonicity restraints for the models, including in LightGBM and XGBoost, 2 of the most popular libraries for gradient enhanced trees. Monotonicity restrictions have also been built into Tensorflow Lattice, a library that carries out a novel approach for producing interpolated lookup tables.

Other techniques for implementing monotonicity.
Tree based techniques are not the only alternative for setting monotonicity constraint in the data. One recent advancement in the field is Tensorflow Lattice, which implements lattice based designs that are essentially interpolated look-up tables that can approximate arbitrary input-output relationships in the data and which can optionally be monotonic. There is a comprehensive tutorial on it in Tensorflow Github.
If a curve is currently provided, monotonic spline can be fit on the information, for instance using the splinefun bundle.

In practical maker knowing and information science jobs, an ML design is often used to measure an international, semantically significant relationship between two or more values. A hotel chain may want to use ML to enhance their rates method and use a model to estimate the likelihood of a room being reserved at a given rate and day of the week. What might easily take place is that upon building the design, the information researcher finds that the design is acting suddenly: for example the design forecasts that on Tuesdays, the customers would rather pay \$110 than \$100 for a space! The reason is that while there is an anticipated monotonic relationship between price and the likelihood of reservation, the model is unable to (totally) record it, due to noisiness of the information and puzzles in it.
One recent advancement in the field is Tensorflow Lattice, which implements lattice based models that are basically interpolated look-up tables that can approximate arbitrary input-output relationships in the information and which can additionally be monotonic.

Monotonicity restraints in LighGBM and XGBoost
For tree based techniques (decision trees, random forests, gradient boosted trees), monotonicity can be forced throughout the design learning stage by not developing divides on monotonic features that would break the monotonicity constraint.
In the following example, lets train too designs using LightGBM on a toy dataset where we know the relationship between X and Y to be monotonic (however loud) and compare the default and monotonic design.

print (” Default design mse”, mse( y, overfit_model. anticipate( x.reshape( -1,1)))).
print (” Monotone design mse”, mse( y, monotone_model. forecast( x.reshape( -1,1)))).

import numpy as np
size = 100
x = np.linspace( 0, 10, size).
y = x ** 2 + 10 – (20 * np.random.random( size)).

The parameter monotone_constraints=” 1 ″ states that the output should be monotonically increasing wrt. the very first functions (which in our case occurs to be the only function). After training the monotone model, we can see that the relationship is now strictly monotone.
And if we examine the design performance, we can see that not only does the monotonicity restraint supply a more natural fit, but the model generalizes much better as well (as anticipated). Determining the mean squared mistake on brand-new test data, we see that mistake is smaller for the monotone model.

size = 1000000.
x = np.linspace( 0, 10, size).
y = x ** 2 -10 + (20 * np.random.random( size)). 