RuleTree.evaluation package

RuleTree.evaluation package#

API reference#

RuleTree.evaluation.evaluate_classification(...)

Accuracy, F1, precision and recall of the predictions of a classifier and, with the class probabilities, ROC AUC and average precision.

RuleTree.evaluation.evaluate_regression(...)

Errors and explained variance of the predictions of a regressor.

RuleTree.evaluation.evaluate_clustering(...)

Agreement between a clustering and known classes (external metrics).

RuleTree.evaluation.evaluate_clustering_internal(X, ...)

Quality of a clustering from the data alone (internal metrics).

RuleTree.evaluation.evaluate_interpretability(model)

Size of a fitted RuleTree or scikit-learn decision tree: the fewer leaves and the shallower the tree, the shorter its rules.

RuleTree.evaluation.evaluate_fairness(...[, ...])

Fairness of the groups of a partition (clusters, leaves or predicted classes) with respect to a sensitive attribute.

Submodules#

RuleTree.evaluation.classification module#

Metrics of a classifier on a test set.

evaluate_classification(y_true, y_pred, y_proba=None, labels=None) → dict[source]

Accuracy, F1, precision and recall of the predictions of a classifier and, with the class probabilities, ROC AUC and average precision.

F1, precision and recall are given with the micro, macro and weighted averages; f1_score, precision_score and recall_score are the ones of the positive class (the second one) for binary problems, and the micro averages otherwise.

Parameters:

  • y_true (array-like of shape (n_samples,)) – True classes.

  • y_pred (array-like of shape (n_samples,)) – Predicted classes.

  • y_proba (array-like of shape (n_samples, n_classes), default=None) – Predicted class probabilities (e.g. predict_proba), in the order of labels. If None, the metrics based on probabilities are not computed.

  • labels (array-like of shape (n_classes,), default=None) – Classes, in the order of the columns of y_proba (e.g. model.classes_). If None, the sorted classes of y_true.

Returns:

scores – Metric name -> value.

Return type:

dict

See also

sklearn.metrics

the definitions of the metrics.

Examples

>>> from sklearn.datasets import load_iris
>>> from RuleTree import RuleTreeClassifier
>>> from RuleTree.evaluation import evaluate_classification
>>> X, y = load_iris(return_X_y=True)
>>> model = RuleTreeClassifier(max_depth=2, random_state=0).fit(X, y)
>>> scores = evaluate_classification(y, model.predict(X), model.predict_proba(X), labels=model.classes_)
>>> round(scores["accuracy"], 3), round(scores["roc_macro"], 3)
(0.96, 0.98)

RuleTree.evaluation.regression module#

Metrics of a regressor on a test set.

evaluate_regression(y_true, y_pred) → dict[source]

Errors and explained variance of the predictions of a regressor.

Parameters:

  • y_true (array-like of shape (n_samples,)) – True values.

  • y_pred (array-like of shape (n_samples,)) – Predicted values.

Returns:

scores – r2, explained_variance, max_error, mean_absolute_error, mean_squared_error, median_absolute_error, mean_absolute_percentage_error and mean_squared_log_error (nan if some value is negative).

Return type:

dict

See also

sklearn.metrics

the definitions of the metrics.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from RuleTree import RuleTreeRegressor
>>> from RuleTree.evaluation import evaluate_regression
>>> X, y = load_diabetes(return_X_y=True)
>>> model = RuleTreeRegressor(max_depth=3, random_state=0).fit(X, y)
>>> round(evaluate_regression(y, model.predict(X))["r2"], 3)
0.501

RuleTree.evaluation.clustering module#

Metrics of a clustering: agreement with known classes, and quality without them.

evaluate_clustering(labels_true, labels_pred) → dict[source]

Agreement between a clustering and known classes (external metrics).

Parameters:

  • labels_true (array-like of shape (n_samples,)) – Known classes.

  • labels_pred (array-like of shape (n_samples,)) – Cluster labels.

Returns:

scores – adjusted_rand, rand_score, adjusted_mutual_info, mutual_info, normalized_mutual_info, homogeneity, completeness, v_measure and fowlkes_mallows.

Return type:

dict

See also

evaluate_clustering_internal

metrics that do not need the classes.

Examples

>>> from sklearn.datasets import make_blobs
>>> from RuleTree import RuleTreeCluster
>>> from RuleTree.evaluation import evaluate_clustering
>>> X, y = make_blobs(n_samples=300, centers=3, cluster_std=0.5, random_state=0)
>>> labels = RuleTreeCluster(max_depth=3, random_state=0).fit_predict(X)
>>> round(evaluate_clustering(y, labels)["adjusted_rand"], 2)
0.95
evaluate_clustering_internal(X, labels, metric='euclidean') → dict[source]

Quality of a clustering from the data alone (internal metrics).

Parameters:

  • X (array-like of shape (n_samples, n_features), or (n_samples, n_samples) if metric='precomputed') – Numerical data, or the matrix of the distances between the instances.

  • labels (array-like of shape (n_samples,)) – Cluster labels.

  • metric (str or callable, default='euclidean') – Distance used by the silhouette, as in sklearn.metrics.silhouette_score; ‘precomputed’ if X is a distance matrix (then only the silhouette is computed).

Returns:

scores – silhouette_score (the higher the better), calinski_harabasz (the higher the better) and davies_bouldin (the lower the better); nan with a single cluster.

Return type:

dict

See also

evaluate_clustering

metrics that compare the clusters with known classes.

Examples

>>> from sklearn.datasets import make_blobs
>>> from RuleTree import RuleTreeCluster
>>> from RuleTree.evaluation import evaluate_clustering_internal
>>> X, _ = make_blobs(n_samples=300, centers=3, cluster_std=0.5, random_state=0)
>>> labels = RuleTreeCluster(max_depth=3, random_state=0).fit_predict(X)
>>> round(evaluate_clustering_internal(X, labels)["silhouette_score"], 2)
0.59

RuleTree.evaluation.interpretability module#

Size of a tree, as a measure of how easy it is to read.

evaluate_interpretability(model) → dict[source]

Size of a fitted RuleTree or scikit-learn decision tree: the fewer leaves and the shallower the tree, the shorter its rules.

Parameters:

model (RuleTree or sklearn.tree.BaseDecisionTree) – Fitted tree.

Returns:

scores – n_leaves (number of rules), n_internal_nodes (number of splits) and max_depth (length of the longest rule; the root has depth 0). Empty for other models.

Return type:

dict

Examples

>>> from sklearn.datasets import load_iris
>>> from RuleTree import RuleTreeClassifier
>>> from RuleTree.evaluation import evaluate_interpretability
>>> X, y = load_iris(return_X_y=True)
>>> evaluate_interpretability(RuleTreeClassifier(max_depth=3, random_state=0).fit(X, y))
{'n_leaves': 5, 'n_internal_nodes': 4, 'max_depth': 3}

RuleTree.evaluation.fairness module#

Fairness of a partition of the instances (clusters or predicted classes) with respect to a sensitive attribute.

evaluate_fairness(labels, sensitive_attribute, ideal_distribution=None) → dict[source]

Fairness of the groups of a partition (clusters, leaves or predicted classes) with respect to a sensitive attribute.

Parameters:

  • labels (array-like of shape (n_samples,)) – Group of each instance, e.g. its cluster.

  • sensitive_attribute (array-like of shape (n_samples,)) – Value of the sensitive attribute of each instance.

  • ideal_distribution (dict, default=None) – Value of the sensitive attribute -> ideal fraction in each group. If None, the distribution in the whole data set.

Returns:

scores – balance: the smallest ratio between the fraction of a value of the sensitive attribute in a group and in the whole data set (or its inverse), from 0 to 1, the higher the fairer; max_fairness_cost: the largest mean absolute difference between the distribution of the sensitive attribute in a group and the ideal one, the lower the fairer.

Return type:

dict

See also

RuleTree.stumps.regression.FairTreeStumpRegressor

the stump that builds fair trees.

Examples

>>> import numpy as np
>>> from RuleTree.evaluation import evaluate_fairness
>>> labels = np.array([0, 0, 0, 0, 1, 1, 1, 1])
>>> sex = np.array(["F", "M", "F", "M", "F", "F", "F", "M"])
>>> evaluate_fairness(labels, sex)
{'balance': 0.6666666666666666, 'max_fairness_cost': 0.125}

Module contents#

Evaluation of the models: predictive performance, clustering quality, size of the trees and fairness, each as a dictionary of metrics.

evaluate_classification(y_true, y_pred, y_proba=None, labels=None) → dict[source]

Accuracy, F1, precision and recall of the predictions of a classifier and, with the class probabilities, ROC AUC and average precision.

F1, precision and recall are given with the micro, macro and weighted averages; f1_score, precision_score and recall_score are the ones of the positive class (the second one) for binary problems, and the micro averages otherwise.

Parameters:

  • y_true (array-like of shape (n_samples,)) – True classes.

  • y_pred (array-like of shape (n_samples,)) – Predicted classes.

  • y_proba (array-like of shape (n_samples, n_classes), default=None) – Predicted class probabilities (e.g. predict_proba), in the order of labels. If None, the metrics based on probabilities are not computed.

  • labels (array-like of shape (n_classes,), default=None) – Classes, in the order of the columns of y_proba (e.g. model.classes_). If None, the sorted classes of y_true.

Returns:

scores – Metric name -> value.

Return type:

dict

See also

sklearn.metrics

the definitions of the metrics.

Examples

>>> from sklearn.datasets import load_iris
>>> from RuleTree import RuleTreeClassifier
>>> from RuleTree.evaluation import evaluate_classification
>>> X, y = load_iris(return_X_y=True)
>>> model = RuleTreeClassifier(max_depth=2, random_state=0).fit(X, y)
>>> scores = evaluate_classification(y, model.predict(X), model.predict_proba(X), labels=model.classes_)
>>> round(scores["accuracy"], 3), round(scores["roc_macro"], 3)
(0.96, 0.98)
evaluate_regression(y_true, y_pred) → dict[source]

Errors and explained variance of the predictions of a regressor.

Parameters:

  • y_true (array-like of shape (n_samples,)) – True values.

  • y_pred (array-like of shape (n_samples,)) – Predicted values.

Returns:

scores – r2, explained_variance, max_error, mean_absolute_error, mean_squared_error, median_absolute_error, mean_absolute_percentage_error and mean_squared_log_error (nan if some value is negative).

Return type:

dict

See also

sklearn.metrics

the definitions of the metrics.

Examples

>>> from sklearn.datasets import load_diabetes
>>> from RuleTree import RuleTreeRegressor
>>> from RuleTree.evaluation import evaluate_regression
>>> X, y = load_diabetes(return_X_y=True)
>>> model = RuleTreeRegressor(max_depth=3, random_state=0).fit(X, y)
>>> round(evaluate_regression(y, model.predict(X))["r2"], 3)
0.501
evaluate_clustering(labels_true, labels_pred) → dict[source]

Agreement between a clustering and known classes (external metrics).

Parameters:

  • labels_true (array-like of shape (n_samples,)) – Known classes.

  • labels_pred (array-like of shape (n_samples,)) – Cluster labels.

Returns:

scores – adjusted_rand, rand_score, adjusted_mutual_info, mutual_info, normalized_mutual_info, homogeneity, completeness, v_measure and fowlkes_mallows.

Return type:

dict

See also

evaluate_clustering_internal

metrics that do not need the classes.

Examples

>>> from sklearn.datasets import make_blobs
>>> from RuleTree import RuleTreeCluster
>>> from RuleTree.evaluation import evaluate_clustering
>>> X, y = make_blobs(n_samples=300, centers=3, cluster_std=0.5, random_state=0)
>>> labels = RuleTreeCluster(max_depth=3, random_state=0).fit_predict(X)
>>> round(evaluate_clustering(y, labels)["adjusted_rand"], 2)
0.95
evaluate_clustering_internal(X, labels, metric='euclidean') → dict[source]

Quality of a clustering from the data alone (internal metrics).

Parameters:

  • X (array-like of shape (n_samples, n_features), or (n_samples, n_samples) if metric='precomputed') – Numerical data, or the matrix of the distances between the instances.

  • labels (array-like of shape (n_samples,)) – Cluster labels.

  • metric (str or callable, default='euclidean') – Distance used by the silhouette, as in sklearn.metrics.silhouette_score; ‘precomputed’ if X is a distance matrix (then only the silhouette is computed).

Returns:

scores – silhouette_score (the higher the better), calinski_harabasz (the higher the better) and davies_bouldin (the lower the better); nan with a single cluster.

Return type:

dict

See also

evaluate_clustering

metrics that compare the clusters with known classes.

Examples

>>> from sklearn.datasets import make_blobs
>>> from RuleTree import RuleTreeCluster
>>> from RuleTree.evaluation import evaluate_clustering_internal
>>> X, _ = make_blobs(n_samples=300, centers=3, cluster_std=0.5, random_state=0)
>>> labels = RuleTreeCluster(max_depth=3, random_state=0).fit_predict(X)
>>> round(evaluate_clustering_internal(X, labels)["silhouette_score"], 2)
0.59
evaluate_interpretability(model) → dict[source]

Size of a fitted RuleTree or scikit-learn decision tree: the fewer leaves and the shallower the tree, the shorter its rules.

Parameters:

model (RuleTree or sklearn.tree.BaseDecisionTree) – Fitted tree.

Returns:

scores – n_leaves (number of rules), n_internal_nodes (number of splits) and max_depth (length of the longest rule; the root has depth 0). Empty for other models.

Return type:

dict

Examples

>>> from sklearn.datasets import load_iris
>>> from RuleTree import RuleTreeClassifier
>>> from RuleTree.evaluation import evaluate_interpretability
>>> X, y = load_iris(return_X_y=True)
>>> evaluate_interpretability(RuleTreeClassifier(max_depth=3, random_state=0).fit(X, y))
{'n_leaves': 5, 'n_internal_nodes': 4, 'max_depth': 3}
evaluate_fairness(labels, sensitive_attribute, ideal_distribution=None) → dict[source]

Fairness of the groups of a partition (clusters, leaves or predicted classes) with respect to a sensitive attribute.

Parameters:

  • labels (array-like of shape (n_samples,)) – Group of each instance, e.g. its cluster.

  • sensitive_attribute (array-like of shape (n_samples,)) – Value of the sensitive attribute of each instance.

  • ideal_distribution (dict, default=None) – Value of the sensitive attribute -> ideal fraction in each group. If None, the distribution in the whole data set.

Returns:

scores – balance: the smallest ratio between the fraction of a value of the sensitive attribute in a group and in the whole data set (or its inverse), from 0 to 1, the higher the fairer; max_fairness_cost: the largest mean absolute difference between the distribution of the sensitive attribute in a group and the ideal one, the lower the fairer.

Return type:

dict

See also

RuleTree.stumps.regression.FairTreeStumpRegressor

the stump that builds fair trees.

Examples

>>> import numpy as np
>>> from RuleTree.evaluation import evaluate_fairness
>>> labels = np.array([0, 0, 0, 0, 1, 1, 1, 1])
>>> sex = np.array(["F", "M", "F", "M", "F", "F", "F", "M"])
>>> evaluate_fairness(labels, sex)
{'balance': 0.6666666666666666, 'max_fairness_cost': 0.125}