Comparing Intrinsic Interpretability and Post-Hoc Explanations: An Empirical Study across Multiple Datasets

In recent years, considerable attention has been directed toward clarifying complex, opaque systems, commonly referred to as black-box models, including deep neural networks and random forests. So-called model-agnostic techniques generally approximate the outputs of these non-transparent systems using an interpretable architecture, ignoring the internal mechanisms of the primary black-box system itself. Consequently, it remains an important inquiry whether the straightforward training of transparent, white-box architectures ought to be favored over the post-hoc approximation of these opaque models. This study presents the findings of an empirical investigation comparing post-hoc explanatory methods against inherently interpretable architectures across multiple datasets, focusing on both rule-based and feature-based frameworks. The empirical outcomes indicate that directly trained transparent models frequently approximate black-box models with an accuracy that matches or exceeds that of post-hoc surrogate models, despite lacking direct access to the underlying black-box structure.

Machine learning techniques have been broadly deployed across a variety of fields and use cases, including healthcare and financial services. Frequently, the resulting architectures function as black-box models, meaning their internal representations lack straightforward interpretability. Consequently, the predictions generated by these systems are difficult for humans to comprehend.

The critical requirement for explanations to clarify the operational mechanics of these systems has driven extensive research into understanding trained models [1]. These approaches can be categorized into local explanations, which attempt to approximate the black-box model in the immediate neighborhood of a specific instance being analyzed (e.g., [2, 3]), and global models, which aim to represent the overall behavior of the entire black-box system through an interpretable surrogate. Recently, numerous frameworks have explored the synthesis of global models from sets of local explanations (e.g., [4, 5]). Additionally, a distinction can be made between model-specific explanation techniques designed for particular classes of black-box architectures like deep neural networks (e.g., [6]), and model-agnostic explanation techniques that operate independently of the underlying black-box architecture's characteristics (e.g., [2]).

Although the necessity of explaining black-box systems is undeniable in high-stakes decision-making scenarios, multiple difficulties and complications have revitalized interest in training inherently interpretable architectures, such as decision trees or rule induction systems, from the outset.

The primary limitation is that post-hoc explanatory techniques merely approximate the primary black-box system, meaning the generated explanations frequently fail to precisely map the true behavior of the target model. This phenomenon is commonly monitored by evaluating the fidelity of the surrogate system, which measures how closely it reproduces the target model's decisions. Furthermore, even in an ideal scenario where an explanation operates without error, it might utilize entirely distinct features, indicating that the explanation lacks fidelity to the actual computational path of the black-box model. Additionally, if the underlying black-box model contains flaws, debugging becomes highly complex because both the explanatory framework and the black-box architecture must be simultaneously debugged and maintained. Given these limitations, researchers have argued that greater emphasis should be placed on developing highly accurate, inherently interpretable models [7].

Prompted by these considerations, this study assesses the degree to which post-hoc explanations can be substituted by directly trained interpretable techniques that operate without knowledge of the underlying black-box systems. The objective is to determine if the predictive performance of an interpretable model is sufficiently robust to replace model-agnostic frameworks, or conversely, to quantify the information loss associated with such a substitution. To achieve this, the effectiveness of both local and global explanation techniques is examined empirically to evaluate the validity of this premise. We execute a sequence of experiments to analyze and compare the performance of multiple interpretable frameworks against black-box explanations. The evidence gathered from rule-based and feature-based explanatory frameworks appears to support our core hypothesis.

A substantial body of literature has explored the explainability and interpretability of black-box models. We point to [8-10] for comprehensive overviews and restrict our discussion to a brief summary of the literature most relevant to our investigation. Specifically, we focus on contrasting studies within two primary streams: the direct training of interpretable systems and post-hoc explanation frameworks.

Direct learning of interpretable models

Directly trained interpretable techniques comprise a category of algorithms that construct transparent models without relying on a primary black-box architecture. Linear regression, logistic regression, and decision trees represent the most conventional transparent frameworks. Our primary focus centers on the induction of logical rules [11], an area where Ripper (Repeated Incremental Pruning to Produce Error Reduction) continues to serve as a highly competitive, state-of-the-art benchmark [12]. In the domain of feature-based explanations, we utilize GA²Ms as an interpretable (white-box) framework derived from Generalized Additive Models. This approach is engineered to achieve competitive predictive performance relative to advanced machine learning models while preserving structural intelligibility and explainability [13].

 Local explanations of black-box models

Post-hoc explanation systems can either deliver a global overview of the entire black-box model (global surrogates) or provide a localized explanation targeted at a specific instance (local surrogates). Several prominent approaches are model-agnostic, meaning they are compatible with any underlying black-box architecture.

Shap [3] is a framework grounded in game theory that explains predictions by calculating post-hoc weights that quantify the significance of an input feature toward the final output. Ribeiro et al. [2] introduced Local Interpretable Model-agnostic Explanations (LIME), which centers on clarifying the specific predictions of a black-box model. While LIME is theoretically independent of the specific interpretable model selected for the explanation, it is predominantly deployed to generate feature-weight-based explanations. LOcal Rule-based Explanations (LORE) represents a variation of LIME that is specifically configured to output rules as local surrogate models [14]. It delivers interpretable and locally faithful explanations by training a local interpretable model on a synthetic neighborhood produced via a genetic algorithm. This pipeline then extracts structured rules from the local estimator to form a coherent explanation. Model Agnostic Supervised Local Explanations (MAPLE) operates as a hybrid framework capable of functioning both as a highly precise tree-based predictive model and as a feature-based local explanatory tool [15].

Combining local explanations into global models

While interpretable architectures generally operate globally—meaning they offer a mechanism to classify any arbitrary point within the data space—post-hoc explanations are typically local, meaning they apply exclusively to the specific instance for which they were generated. Rule-based frameworks are notably compelling in this domain because a global model typically manifests as a rule set or rule list built from discrete rules, which can themselves be conceptualized as local models [16, 17]. More broadly, a number of recent initiatives have concentrated on integrating local explanations into comprehensive global models, which simplifies the comparison between local and global systems.

For instance, Yang et al. [18] introduced Global Interpretation via Recursive Partitioning (GIRP), a framework designed to construct a global interpretation tree for diverse machine learning models using their local explanations. This approach sequentially divides the input variable space by optimizing the average contribution of input features derived from local explanations across the separated sub-spaces. The final output of this process is a binary interpretation tree that maps out a collection of decision rules approximating the target model. van der Linden et al. [19] presented Global Aggregations of Local Explanations (GALE) as a strategy to illuminate the global decision-making pathway. Their work evaluates the degree to which local explanations can provide comprehensive, macro-level insights into a black-box model. To accomplish this, local explanations generated by the LIME framework are examined and synthesized using multiple aggregation strategies to test their capacity to represent global behavior. Lastly, Setzu et al. [4] developed the GLobal to loCAL explainer (GLocalX), which functions as a "local-first" model-agnostic explanation framework. This methodology seeks to exploit local explanation tools and their inherent advantages to construct a global overview. The pipeline initiates from local decision rules and progressively builds global explanations through systematic aggregation.

Problem statement

This investigation addresses the empirical validation of the premise presented by Rudin [7], which argues that artificial intelligence research should prioritize the development of inherently transparent systems rather than the post-hoc interpretation of black-box architectures. To explore this, we select and systematically contrast pairs of model-agnostic post-hoc explanation frameworks and independent, directly trained interpretable algorithms that generate identical syntactic classes of models. Specifically, as illustrated in Figure 1, a black-box system ℳ is optimized using a designated training dataset. This training set contains n individual instances {(𝐱𝑖,),=1⋯𝑛}, where each instance is characterized by m distinct features and a corresponding target label 𝑦𝑖∈𝑌.

Subsequently, standard explainable AI methodologies are applied to approximate the decision boundaries of ℳ via an interpretable surrogate system ℐℳ. Concurrently, a syntactically comparable transparent model ℐ is trained directly from the source data to serve as a baseline for comparison against ℐℳ.

Consequently, the core research question investigated here is the extent to which an inherently interpretable model ℐ, learned directly from data, can successfully approximate an independently optimized black-box system ℳ, and to what degree fidelity degrades relative to an interpretable surrogate ℐℳ that possessed direct access to ℳ. Theoretically, one anticipates that ℐℳ will exhibit superior fidelity (and potentially higher predictive accuracy) compared to ℐ, given that ℐ ℳ explicitly fits the outputs of ℳ, whereas ℐ is optimized independently. Nevertheless, because both architectures are trained on the same data distributions, implicit correlations may materialize between them.

Furthermore, it is widely acknowledged that transparent models ℐ often display lower raw accuracy than complex systems ℳ because they fundamentally function as approximations of the underlying black-box model ℳ. This approximation quality is commonly quantified via fidelity, which measures the degree to which ℐℳ accurately mirrors the predictions generated by ℳ.

Thus, our objective is to analyze how the two architectures, ℐ and ℐℳ, compare across standard performance metrics—such as structural complexity and predictive accuracy—as well as across these specific fidelity measures.

Additionally, multiple distinct explanations can potentially account for a model's behavior due to the Rashomon effect [20]. This concept dictates that for structured representations like decision trees or rule lists, there frequently exist multiple distinct model configurations that fit the empirical data with equal efficacy. We aim to determine whether substantive differences emerge among the explanations yielded by our selected interpretation methodologies for a given model.

In general, our focus centers on rule-based and feature-based frameworks. We evaluate rule-based methods by examining the logical rule sets they generate, and feature-based methods by analyzing the feature weights assigned to individual variables. The subsequent sections outline the specific algorithms and techniques chosen for this study.

Rule-based interpretability methods

For our rule-based evaluation, GLocalX and JRip are selected to represent model-agnostic and inherently interpretable frameworks, respectively. Both techniques produce explanations formatted as rule sets, which are preferred due to their compact representation and strong alignment with human cognitive reasoning [11].

JRip [12] is a traditional inductive rule learner that constructs rule sets through a sequence of three core operations: growing, pruning, and optimization. Prior to generating each individual rule, JRip partitions the available training instances into two distinct subsets: a growing set utilized to formulate the initial rule, and a pruning set utilized to simplify and generalize the newly learned rule. The complete rule set undergoes further optimization by iteratively re-learning individual rules within the context of the wider rule base once an adequate volume of positive instances has been successfully accounted for.

GLocalX [4] constructs comprehensive global explanations for an opaque system by aggregating local explanations produced by a local surrogate architecture, such as LORE [14], alongside the predicted target labels from the black-box model. The framework accepts a collection of local explanations as its input and iteratively merges them to establish broader, more generalized logical rules. During each iteration, the local explanations are arranged in a priority queue based on structural similarity, and a batch of data is sampled to evaluate potential merges between candidate explanations. A merge operation is executed when a pair of highly similar explanations is extracted from the queue. This function relies on specific cut and join operators, enabling the algorithm to generalize explanations while balancing the trade-off between fidelity and model complexity.

To fuse two local explanations, Ei and Ej, the join operator is applied to non-conflicting rules, whereas the cut operator resolves conflicting rules. Consequently, join expands rule generality at the expense of local fidelity, while cut specializes the rules to preserve fidelity at the expense of generality. If the resulting merged rule satisfies predefined constraints for simplicity and predictive accuracy, the merged pair is retained, and the original expressions Ei and Ej are replaced. Ultimately, explanations exhibiting suboptimal fidelity are eliminated using the \alpha parameter, which defines a per-class trimming threshold.

Feature-based interpretability methods

Within the domain of feature-based interpretability, GA²Ms [13] is selected as an inherently transparent glass-box algorithm, while MAPLE is chosen to represent post-hoc explanatory frameworks. Both GA²Ms and MAPLE rely on underlying linear formulations, producing feature weights that quantify the relative contribution of each variable to the final prediction.

The GA²Ms algorithm builds upon Generalized Additive Models (GAM), which represent an extension of generalized linear modeling. Rather than relying on simple linear weights, a GAM assumes that the target function can be modeled as a sum of arbitrary shape functions. GA²Ms expands upon standard GAM formulations by incorporating explicit terms that capture pairwise interactions between distinct feature values:

The training pipeline begins by constructing a shallow decision tree for each input feature independently using a boosting framework, ensuring each individual tree isolates only one feature. This process is repeated across a set number of iterations, eventually producing an ensemble of trees for every variable. Next, the ensembles generated for each feature are transformed into a continuous graph by plotting the predictions of each tree. This results in a final model composed of one graph per feature. Because GA²Ms retains an additive structure, the explicit contribution of each feature to the overall prediction can be straightforwardly interpreted [13].

For inference within the GA²Ms framework, each individual shape function fi maps a feature value to a specific term contribution, acting much like a lookup table. The retrieved term contributions are summed, and the final predictions are computed by passing this total through a link function g. This additive architecture allows GA²Ms to isolate and report the precise impact of each input feature on the resulting output.

MAPLE combines classical linear regression with a tree-based interpretation of tree ensembles, utilizing a supervised neighborhood approach and feature selection to produce both global and instance-level explanations. The algorithm initially identifies the specific training points within the dataset that are most relevant to a given prediction. It then computes similarity weights for each training instance xi by determining how frequently xi and a target instance x are assigned to the same terminal leaf node across an ensemble of trees 𝐓={𝑇1,𝑇2,...,𝑇𝑘}, as formalized in Eq. 2:

The calculated weights of the training instances are subsequently embedded into a local linear model to generate both a prediction and a local explanation by solving the weighted linear regression problem outlined in:

Experimental setup

Our empirical evaluation was performed across several standard benchmark datasets, primarily retrieved from the UCI Machine Learning Repository [21]. All selected datasets represent binary classification tasks:

• Adult: The objective is to determine whether an individual's annual income exceeds 50K.

• COMPAS Two-Year: Contains recidivism risk scores predicting the likelihood of an individual reoffending within a two-year window.

• German Credit: Identifies whether a loan applicant presents a good or bad credit risk profile.

• NHANES I: Features follow-up mortality data tracking participants from the National Health and Nutrition Examination Survey Epidemiologic Follow-up Study.

• Credit Card Fraud: Contains transaction records categorized as either legitimate or fraudulent.

• Bank Marketing: Derived from a direct marketing initiative by a Portuguese banking institution, where the objective is to predict whether a client will open a term deposit account.

Experimental setup on rule-based models

To execute the rule-based evaluations, we adopt the experimental protocols established in [4]. During the data preprocessing phase, each dataset is partitioned into three independent segments: 60% of the data is allocated to train the black-box model (𝑋𝑏𝑏,𝑏𝑏), 20% is assigned to train GLocalX(𝑋𝑙𝑒,𝑌𝑙𝑒), and the remaining 20% is reserved as unseen validation data (𝑋𝑣𝑙,𝑌𝑣𝑙) to verify model performance.

GLocalX requires a trained black-box model and a local explanation framework to synthesize global rules. For this purpose, a random forest ensemble [22] serves as the target black-box model, generating predicted labels for the 20% training partition assigned to GLocalX. We use the LORE algorithm to extract individual local explanations for each instance within this partition. A conceptual overview of the required components is provided in Figure 2.

For the evaluations involving JRip, the initial 80% of the dataset is used collectively as the training partition. It is worth noting that JRip internally splits its training data into a 2/3 growing set and a 1/3 pruning set, a mechanism that closely mirrors the internal partitioning utilized by GLocalX.

Because rule-based interpretability frameworks output discrete class predictions, model performance is assessed using accuracy and fidelity. Additionally, the total number of rules in the final model is tracked as a key metric for evaluating structural complexity.

This section presents the empirical findings obtained from the rule-based and feature-based modeling experiments.

Results on rule-based models

To assess the viability of using the transparent JRip framework as a substitute for the post-hoc explanatory model GLocalX, we evaluated GLocalX across a spectrum of values for its \alpha parameter and compared the resulting rule sets with the rule set learned directly by JRip. Table 1 outlines the performance profiles of JRip alongside the various \alpha configurations of GLocalX on the adult dataset.

The empirical data indicates that on the adult dataset, GLocalX achieves its optimal trade-off between fidelity and rule base compactness at \alpha=10. Consequently, this specific configuration is selected for deeper analysis. Comparing the performance of GLocalX against JRip in Table 1 reveals that both frameworks achieve remarkably similar outcomes in terms of predictive accuracy and fidelity. The most precise rule base produced by GLocalX (at \alpha=10) yields a lower raw accuracy than JRip, though JRip produces a slightly more intricate rule structure.

Even when examining a GLocalX configuration with comparable structural complexity (\alpha=25), the core trend remains unchanged. Despite having no direct exposure to the underlying black-box model during training, JRip delivers an equally or more effective representation of its behavior. Specifically, it achieves a higher fidelity to the black-box system than GLocalX, despite the fact that GLocalX explicitly attempts to mimic the black-box outputs while JRip optimizes an entirely independent rule set.

Table 2 aggregates the experimental results across all evaluated datasets, utilizing the optimal \alpha parameter custom-tuned for each respective data source. The consolidated metrics show that both algorithms maintain highly competitive performance across rule count and predictive accuracy, with JRip consistently retaining a minor advantage.

To analyze the structural composition of these models, Table 3 contrasts the explicit rules extracted by GLocalX and JRip using the adult dataset. Although certain predictive features—such as "age", "capital-gain", and "marital-status"—appear consistently within the global rules of both GLocalX and JRip, the overall logical structures diverge substantially while maintaining nearly identical fidelity as global explanations.

This serves as a distinct manifestation of the Rashomon effect [20], which dictates that fundamentally different model configurations can capture identical predictive capacity, and that minor variations in a dataset can prompt significant structural shifts in learned symbolic models. Here, we observe this phenomenon from a novel perspective: completely distinct sets of logical rules can yield different explanations while providing the exact same level of fidelity to an underlying black-box architecture.

Furthermore, the standard interpretability-accuracy trade-off implies that highly interpretable models often experience a degradation in raw predictive accuracy. Comparing the rule structures yielded by GLocalX and JRip regarding total rule length and the volume of antecedents per rule indicates that GLocalX generally constructs shorter rules with fewer constraints than JRip.

Consequently, as a direct inference, GLocalX may offer superior cognitive clarity for human interpreters by outputting more concise and less intricate rule sets. Conversely, JRip secures superior accuracy by mapping out more complex logical pathways. It is important to emphasize that the interpretability and comprehensibility of rule bases can be analyzed from numerous conceptual dimensions and warrants deeper investigation.

Results on feature-based interpretability methods

As detailed in the preceding sections, GA²Ms and MAPLE were selected to evaluate feature-based interpretability. Here too, we aim to measure the performance of both techniques regarding their standalone predictive capacity (accuracy) alongside their alignment with the target black-box model (fidelity). We first examine the relative contribution of individual variables to the final output by plotting the feature importance hierarchies generated by both GA²Ms and MAPLE. Because the MAPLE algorithm does not natively output global feature importance as explicit weights, we utilize the average linear regression weights computed for each feature to establish its global importance ranking.

The complete metrics and graphical representations outlining the feature importance rankings for GA²Ms and MAPLE on the adult dataset are documented and illustrated in Figure 3. Comparing the outputs of MAPLE and GA²Ms confirms that variables such as "age" and "capitalgain" exert a substantial influence on the final predictions within both frameworks. However, specific features like "maritalstatus" display minimal importance inside MAPLE while acting as a primary driver within GA²Ms. This further illustrates the Rashomon effect, demonstrating that divergent feature weight allocations can be generated as valid explanations for the same underlying data distribution.

To ensure an equitable comparison of both techniques regarding fidelity, it is necessary to establish a uniform predictive mechanism for both MAPLE and GA²Ms. To accomplish this, we define the normalized feature importance weights extracted from GA²Ms and MAPLE as 𝐖𝐺∈ℝ𝑚×𝑚 and 𝐖𝑀∈ℝ𝑚×𝑚, respectively. For each explanation framework, we then apply Eq. 4 to compute the corresponding value for every individual instance in the dataset:

Within Eq. 4, the specific weights derived from MAPLE and GA²Ms are multiplied by the corresponding feature values for each sample in the dataset. To transform these continuous probability estimates into discrete binary labels, we optimize the decision threshold th using Receiver Operating Characteristic (ROC) curve analysis, as defined in Eq. 5:

where TPR and FPR represent the true positive rate and false positive rate, respectively. The resulting values are subsequently utilized to compute the Area Under the Curve (AUC) to evaluate overall accuracy and fidelity.

Using this evaluation pipeline, the accuracy and fidelity metrics for both frameworks were calculated across the various datasets, with the final values summarized in Table 4. The empirical results confirm that GA²Ms consistently outperforms MAPLE in both accuracy and fidelity across all evaluated datasets. This strongly reinforces the core hypothesis of this study: directly trained interpretable models can serve as highly effective explanations for black-box systems, even without direct access to them, because well-optimized transparent and black-box models will fundamentally correlate with one another due to learning from identical underlying data spaces.

The field of interpretable machine learning has become increasingly critical across a wide range of domain applications. The foundational principle underlying many black-box explanation methods is to approximate complex decision spaces globally or locally via an interpretable surrogate model. However, this approximation pipeline frequently sacrifices a significant portion of the original model's predictive capability, and determining whether the resulting surrogate is truly faithful to the primary black-box system remains a challenge. In this study, we demonstrated that, perhaps counterintuitively, transparent models trained entirely independently of the black-box system can achieve a level of fidelity to that black-box system that matches or exceeds the performance of post-hoc surrogates explicitly trained to mimic it.

Specifically, we evaluated GLocalX and JRip as representative post-hoc and inherently interpretable rule-based techniques, while utilizing MAPLE and GA²Ms to represent post-hoc and inherently interpretable feature-based methods. The experimental findings indicate that directly trained transparent frameworks deliver accuracy and fidelity metrics on par with advanced post-hoc explanation pipelines. However, the explicit explanations generated by these two paradigms—whether formatted as symbolic rule sets or continuous feature importances—can diverge significantly. Consequently, inherently interpretable models can be directly deployed as viable alternatives to post-hoc explanation methods. Furthermore, analyzing the structural variations between explanations yielded by rule-based versus feature-based methodologies presents an important direction for future research to optimize explanation efficiency and identify the most appropriate interpretability framework for a given dataset.

Supported by Johannes Kepler Open Access Publishing Fund and the federal state of Upper Austria.

None

None

None

Pierre Dubois, Marc Lefevre, Claire Moreau, Julien Martin & Thomas Dupont contributed to this work.

Authors and affiliations

Department of Intrinsic Interpretability and XAI Comparison, Faculty of Engineering, University of Bordeaux, Bordeaux, France
Pierre Dubois & Claire Moreau

Department of Post-Hoc Explanations and Empirical Studies, Faculty of Engineering, University of Nantes, Nantes, France
Marc Lefevre

Department of Multi-Dataset Explainability Evaluation, Faculty of Engineering, University of Strasbourg, Strasbourg, France
Julien Martin

Department of Interpretable vs. Post-Hoc Models, Faculty of Engineering, Université Paris-Saclay, Paris, France
Thomas Dupont

Corresponding author

Correspondence to Pierre Dubois

Open Access The author(s) retain copyright. This article is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. It may be shared and adapted for non-commercial purposes with appropriate attribution, an indication of changes, and distribution of adaptations under the same license. Third-party material may be subject to separate terms identified in its credit line. View the license at https://creativecommons.org/licenses/by-nc-sa/4.0/.