RuleTree.ensemble package

RuleTree.ensemble package#

Submodules#

RuleTree.ensemble.EBMRegressor module#

EBMRegressor: explainable boosting machine for regression.

class EBMRegressor(n_iterations=100, base_estimator=RuleTreeRegressor(base_stumps=DecisionTreeStumpRegressor(), max_depth=6, random_state=42), learning_rate=0.04, val_size=0.15, patience=5, n_jobs=1, random_state=42)[source]#

Bases: RegressorMixin, RuleTreeBase

Explainable boosting machine for regression: a cyclic gradient boosting where each base estimator is fitted on a single feature or on a pair of features, so that the prediction is a sum of functions of one or two features.

At each iteration, a copy of the base estimator is fitted on the residuals for each feature and each pair of features. The boosting stops early when the error on a validation set does not improve for patience iterations.

Parameters:

  • n_iterations (int, default=100) – Maximum number of iterations.

  • base_estimator (RuleTreeRegressor, default=RuleTreeRegressor(max_depth=6, random_state=42)) – Estimator fitted on each feature and pair of features.

  • learning_rate (float, default=0.04) – Shrinkage of the contribution of each estimator.

  • val_size (float, default=0.15) – Fraction of the training set used for validation.

  • patience (int, default=5) – Number of iterations without improvement before stopping.

  • n_jobs (int, default=1) – Not used.

  • random_state (int, default=42) – Seed of the split between training and validation.

Variables:

  • base_prediction (float) – Initial prediction (0).

  • estimators (dict) – Feature or pair of features -> list of fitted estimators.

See also

GBoostRegressor

a boosting of trees on all the features.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from RuleTree import RuleTreeRegressor
>>> from RuleTree.ensemble.EBMRegressor import EBMRegressor
>>> X, y = load_diabetes(return_X_y=True)
>>> model = EBMRegressor(base_estimator=RuleTreeRegressor(max_depth=1, random_state=0),
...                      n_iterations=5, learning_rate=0.1, random_state=0).fit(X, y)
>>> len(model.estimators_)  # 10 features and 45 pairs of features
55
>>> len(model.estimators_[(0, 1)])  # trees on the pair of features 0 and 1
5
fit(X, y)[source]#

Fit the boosting.

Parameters:
Returns:

self

Return type:

EBMRegressor

predict(X: ndarray)[source]#

Predict the target: the initial prediction plus the contributions of all the estimators.

Parameters:

X (numpy.ndarray of shape (n_samples, n_features)) – Tabular data.

Returns:

y_pred

Return type:

numpy.ndarray of shape (n_samples,)

set_fit_request() → EBMRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_predict_request() → EBMRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → EBMRegressor#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

RuleTree.ensemble.GBoostClassifier module#

GBoostClassifier: gradient boosting of RuleTrees for classification.

class GBoostClassifier(base_estimator=RuleTreeRegressor(base_stumps=DecisionTreeStumpRegressor(), max_depth=4), n_estimators=100, learning_rate=0.1, loss='squared_loss', patience=5, val_size=0.15, n_jobs=4, random_state=42)[source]#

Bases: ClassifierMixin, RuleTreeBase

Gradient boosting for classification with RuleTreeRegressors as base estimators (one-vs-rest, logistic loss).

For each class, a boosting of regression trees is fitted on the residuals of the predicted probability of the class; the leaves of each tree are given the Newton step of the log-odds. The prediction is the class with the highest probability. The classes are fitted in parallel.

Parameters:

  • base_estimator (RuleTreeRegressor, default=RuleTreeRegressor(max_depth=4)) – Estimator fitted at each iteration.

  • n_estimators (int, default=100) – Maximum number of estimators per class.

  • learning_rate (float, default=0.1) – Shrinkage of the contribution of each estimator.

  • loss ('squared_loss' or callable, default='squared_loss') – Checked but not used.

  • patience (int, default=5) – Number of iterations without improvement before stopping.

  • val_size (float, default=0.15) – Fraction of the training set used for validation (stratified).

  • n_jobs (int, default=os.cpu_count()) – Number of processes (at most one per class); with 1, the classes are fitted in the current process.

  • random_state (int, default=42) – Seed of the split between training and validation.

Variables:

  • classes (list) – Classes of the target.

  • base_log_odds (numpy.ndarray) – Initial log-odds of each class.

  • estimators (list of list) – For each class, the fitted estimators with the value of each of their leaves.

See also

GBoostRegressor

the same boosting for regression.

Examples

>>> from sklearn.datasets import load_breast_cancer
>>> from RuleTree import RuleTreeRegressor
>>> from RuleTree.ensemble.GBoostClassifier import GBoostClassifier
>>> X, y = load_breast_cancer(return_X_y=True)
>>> model = GBoostClassifier(base_estimator=RuleTreeRegressor(max_depth=2, random_state=0),
...                          n_estimators=10, n_jobs=1, random_state=0).fit(X, y)
>>> round(model.score(X, y), 3)
0.919
fit(X: ndarray, y: ndarray)[source]#

Fit a boosting for each class.

Parameters:
Returns:

self

Return type:

GBoostClassifier

predict(X: ndarray)[source]#

Predict the class with the highest probability.

Parameters:

X (numpy.ndarray of shape (n_samples, n_features)) – Tabular data.

Returns:

y_pred

Return type:

numpy.ndarray of shape (n_samples,)

set_fit_request() → GBoostClassifier#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_predict_request() → GBoostClassifier#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → GBoostClassifier#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

RuleTree.ensemble.GBoostRegressor module#

GBoostRegressor: gradient boosting of RuleTrees for regression.

class GBoostRegressor(base_estimator=RuleTreeRegressor(base_stumps=DecisionTreeStumpRegressor(), max_depth=3), n_estimators=100, learning_rate=0.1, loss='squared_loss', patience=5, val_size=0.15, random_state=42)[source]#

Bases: RegressorMixin, RuleTreeBase

Gradient boosting for regression with RuleTrees as base estimators (squared loss).

Each estimator is fitted on the residuals of the previous ones; the boosting stops early when the error on a validation set does not improve for patience iterations.

Parameters:

  • base_estimator (RuleTreeRegressor, default=RuleTreeRegressor(max_depth=3)) – Estimator fitted at each iteration.

  • n_estimators (int, default=100) – Maximum number of estimators.

  • learning_rate (float, default=0.1) – Shrinkage of the contribution of each estimator.

  • loss ('squared_loss' or callable, default='squared_loss') – Checked but not used: the loss is always the squared loss.

  • patience (int, default=5) – Number of iterations without improvement before stopping.

  • val_size (float, default=0.15) – Fraction of the training set used for validation.

  • random_state (int, default=42) – Seed of the split between training and validation.

Variables:

  • base_prediction (float) – Initial prediction (mean of the target).

  • estimators (list of RuleTreeRegressor) – Fitted estimators.

See also

XGBoostRegressor

the boosting of the XGBoost trees.

EBMRegressor

a boosting of trees on single features and pairs of features.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from RuleTree import RuleTreeRegressor
>>> from RuleTree.ensemble.GBoostRegressor import GBoostRegressor
>>> X, y = load_diabetes(return_X_y=True)
>>> model = GBoostRegressor(base_estimator=RuleTreeRegressor(max_depth=2, random_state=0),
...                         n_estimators=20, random_state=0).fit(X, y)
>>> len(model.estimators_), round(model.score(X, y), 3)
(20, 0.531)
fit(X: ndarray, y: ndarray)[source]#

Fit the boosting.

Parameters:
Returns:

self

Return type:

GBoostRegressor

predict(X: ndarray)[source]#

Predict the target: the initial prediction plus the contributions of all the estimators.

Parameters:

X (numpy.ndarray of shape (n_samples, n_features)) – Tabular data.

Returns:

y_pred

Return type:

numpy.ndarray of shape (n_samples,)

set_fit_request() → GBoostRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_predict_request() → GBoostRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → GBoostRegressor#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

RuleTree.ensemble.XGBoostRegressor module#

XGBoostRegressor: gradient boosting with the trees of XGBoost.

class XGBoostRegressor(base_estimator=None, n_estimators=100, learning_rate=0.3, loss='squared_loss', patience=5, val_size=0.15, random_state=42)[source]#

Bases: GBoostRegressor

Gradient boosting for regression with the trees of XGBoost (RuleTreeXGBoostRegressor): the splits maximize the XGBoost gain and the leaves predict the regularized XGBoost values.

Parameters:

  • base_estimator (RuleTreeXGBoostRegressor, default=None) – Tree fitted at each iteration. If None, RuleTreeXGBoostRegressor with the XGBoost stumps and as many jobs as CPUs, built at fit time.

  • learning_rate (float, default=0.3) – Shrinkage of the contribution of each tree.

  • n_estimators – As in GBoostRegressor.

  • loss – As in GBoostRegressor.

  • patience – As in GBoostRegressor.

  • val_size – As in GBoostRegressor.

  • random_state – As in GBoostRegressor.

Variables:

  • base_prediction (float) – Initial prediction (mean of the target).

  • estimators (list of RuleTreeXGBoostRegressor) – Fitted trees.

See also

RuleTree.tree.RuleTreeXGBoostRegressor.RuleTreeXGBoostRegressor

the base tree.

GBoostRegressor

the same boosting with any RuleTreeRegressor.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from RuleTree.ensemble.XGBoostRegressor import XGBoostRegressor
>>> from RuleTree.stumps.regression import XGBoostStumpRegressor
>>> from RuleTree.tree.RuleTreeXGBoostRegressor import RuleTreeXGBoostRegressor
>>> X, y = load_diabetes(return_X_y=True)
>>> tree = RuleTreeXGBoostRegressor(max_depth=2, lam=1, base_stumps=XGBoostStumpRegressor(lam=1))
>>> model = XGBoostRegressor(base_estimator=tree, n_estimators=10).fit(X, y)
>>> round(model.score(X, y), 3)
0.569
set_fit_request() → XGBoostRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_predict_request() → XGBoostRegressor#

No-op.

Calling this method has no effect.

Returns:

self – The updated object.

Return type:

object

set_score_request(*, sample_weight: bool | None | str = '$UNCHANGED$') → XGBoostRegressor#

Configure whether metadata should be requested to be passed to the score method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

  • True: metadata is requested, and passed to score if provided. The request is ignored if metadata is not provided.

  • False: metadata is not requested and the meta-estimator will not pass it to score.

  • None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

  • str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

sample_weight (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for sample_weight parameter in score.

Returns:

self – The updated object.

Return type:

object

Module contents#