Evaluating the models#
RuleTree.evaluation computes the usual metrics of a model in one call, each as a dictionary
{metric name: value}, so that the results of several models can be collected in a table. The
metrics are the ones of sklearn.metrics.
Classification and regression#
evaluate_classification takes the true and the predicted classes and, optionally, the class
probabilities (with their order, classes_), for ROC AUC and average precision:
from sklearn.datasets import load_diabetes, load_iris
from sklearn.model_selection import train_test_split
from RuleTree import RuleTreeClassifier, RuleTreeRegressor
from RuleTree.evaluation import evaluate_classification, evaluate_regression
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
model = RuleTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
scores = evaluate_classification(y_test, model.predict(X_test), model.predict_proba(X_test),
labels=model.classes_)
print(sorted(scores))
print({name: round(scores[name], 3) for name in ["accuracy", "f1_macro", "roc_macro"]})
['accuracy', 'average_precision_macro', 'average_precision_micro', 'average_precision_weighted', 'balanced_accuracy', 'f1_macro', 'f1_micro', 'f1_score', 'f1_weighted', 'precision_macro', 'precision_micro', 'precision_score', 'precision_weighted', 'recall_macro', 'recall_micro', 'recall_score', 'recall_weighted', 'roc_macro', 'roc_micro', 'roc_weighted']
{'accuracy': 0.956, 'f1_macro': 0.955, 'roc_macro': 1.0}
For regression, evaluate_regression gives the errors and the explained variance:
X_reg, y_reg = load_diabetes(return_X_y=True)
X_reg_train, X_reg_test, y_reg_train, y_reg_test = train_test_split(X_reg, y_reg, test_size=0.3, random_state=0)
regressor = RuleTreeRegressor(max_depth=3, random_state=0).fit(X_reg_train, y_reg_train)
scores = evaluate_regression(y_reg_test, regressor.predict(X_reg_test))
print({name: round(scores[name], 3) for name in ["r2", "mean_absolute_error"]})
{'r2': 0.188, 'mean_absolute_error': 50.408}
Clustering#
evaluate_clustering compares the clusters with known classes, evaluate_clustering_internal
measures their quality from the data alone (silhouette, Calinski-Harabasz, Davies-Bouldin):
from sklearn.datasets import make_blobs
from RuleTree import RuleTreeCluster
from RuleTree.evaluation import evaluate_clustering, evaluate_clustering_internal
X_blobs, blobs = make_blobs(n_samples=300, centers=3, cluster_std=0.5, random_state=0)
labels = RuleTreeCluster(max_depth=3, random_state=0).fit_predict(X_blobs)
print("adjusted Rand index:", round(evaluate_clustering(blobs, labels)["adjusted_rand"], 3))
print("silhouette:", round(evaluate_clustering_internal(X_blobs, labels)["silhouette_score"], 3))
adjusted Rand index: 0.95
silhouette: 0.592
Size of the trees#
evaluate_interpretability measures how long the rules of a tree are: the number of leaves
(rules), of internal nodes (splits) and the maximum depth. It also works with scikit-learn trees,
for comparisons:
from sklearn.tree import DecisionTreeClassifier
from RuleTree.evaluation import evaluate_interpretability
print(evaluate_interpretability(model))
print(evaluate_interpretability(DecisionTreeClassifier(random_state=0).fit(X_train, y_train)))
{'n_leaves': 5, 'n_internal_nodes': 4, 'max_depth': 3}
{'n_leaves': 8, 'n_internal_nodes': 7, 'max_depth': 4}
Fairness#
evaluate_fairness measures how the values of a sensitive attribute are distributed among the
groups of a partition, e.g. the clusters or the leaves of a tree (see the fair stumps in
Choosing the splits: stumps): balance is 1 when each group has the distribution of the whole data set (and
0 when a group lacks a value, e.g. a group of one instance), max_fairness_cost is 0 when each
group has the ideal distribution (by default, the one of the whole data set). Here the groups are
the three blobs, with a sensitive attribute drawn at random, and with one that depends on the
position of the instances:
import numpy as np
from RuleTree.evaluation import evaluate_fairness
random_attribute = np.random.default_rng(0).integers(0, 2, size=len(blobs))
position_attribute = (X_blobs[:, 0] > np.median(X_blobs[:, 0])).astype(int)
print({name: round(value, 3) for name, value in evaluate_fairness(blobs, random_attribute).items()})
print({name: round(value, 3) for name, value in evaluate_fairness(blobs, position_attribute).items()})
{'balance': 0.913, 'max_fairness_cost': 0.043}
{'balance': 0.0, 'max_fairness_cost': 0.5}