Post-Hoc Feature Selection Layer for Neural Networks Interpretability
This paper adapts the Feature Selection Layer (FSL) into a post-hoc interpretability tool for pre-trained neural networks on tabular data. Instead of training an embedded feature weighting layer jointly with the network, the authors freeze the pre-trained model and insert a lightweight diagonal scaling layer that learns per-feature weights via fine-tuning. The approach aims to let practitioners retrofit already deployed models to obtain feature importance scores without retraining from scratch, bridging embedded feature selection and post-hoc attribution methods.
The paper presents a practical but conceptually tenuous adaptation of the original FSL. While the method is computationally efficient and occasionally improves visual clustering metrics, its claim to reveal the features the original model considers most important is methodologically questionable, and the empirical evidence is too mixed to support strong interpretability guarantees.
The motivation for a non-invasive, post-hoc feature attribution method is clear and well-stated. The experimental design is reasonably comprehensive, spanning synthetic and real-world datasets with known ground-truth features, high-dimensional genomics data, and multiple competing post-hoc baselines including Integrated Gradients, DeepLIFT, and SHAP variants. The feature erasure analysis provides a useful sanity check by showing that removing top-ranked features degrades model performance, and source code with seeds is provided.
The central methodological flaw is the conflation of finding a sparse input projection that predicts well with frozen weights and explaining what the original model used. Post-hoc FSL trains new parameters to minimize loss; these weights can exploit patterns in the frozen layers that differ from the original model's actual feature dependencies. The regularization is also substantially weakened: the original FSL used $r(W^{\text{FSL}})=\lambda|\sum_{t=1}^{n}W_{t}^{\text{FSL}}-1|$, while post-hoc FSL replaces this with standard L1 plus ReLU, abandoning the interaction-aware sum-to-one constraint without theoretical justification.
Empirical results are uneven. On SynthA, post-hoc FSL trails embedded FSL in predictive metrics (F1 $\approx 0.916$ vs. $\approx 0.954$) and stability (Jaccard $0.759\pm0.052$ vs. $0.855$-$0.976$ for competitors, Pearson $0.770\pm0.027$ vs. $0.969$-$0.980$). The paper also admits it "fails to capture local and class-specific feature relationships" in multi-class settings, a severe limitation. Real-world microarray experiments use only 48 and 151 samples with tens of thousands of features, yielding unreliable estimates given 15-fold splits.
The evidence partially supports that post-hoc FSL can identify informative features, but comparisons to related work are inconsistent and sometimes unfavorable. On SynthA, post-hoc FSL underperforms gradient-based methods in silhouette score ($0.115$ vs. $0.181$-$0.235$) and stability, whereas on Liver it achieves the best silhouette ($0.666$). The Kruskal-Wallis/Dunn significance tests lack multiplicity correction. The paper also compares against embedded FSL, which often dominates the post-hoc variant, undermining claims that the adaptation preserves the original method's efficacy.
Code and seeds are available, which is commendable, but critical experimental details are sparse. The paper states that "All models employed different architectures" without specifying layer sizes, depths, or activation functions for each dataset. The number of post-hoc fine-tuning epochs, batch size, early stopping criteria, and the regularization strength $\lambda$ are not reported. For the Liver dataset, 15-fold cross-validation with 48 total samples means each fold draws only about 3-5 test samples, making variance estimates unreliable. These omissions would block independent reproduction.
The interpretability of complex neural networks remains a critical challenge, especially for models already deployed in high-stakes domains. To address this, we adapted the Feature Selection Layer (FSL) to optimize feature weights post-training, a variant we term post-hoc FSL. Our approach reframes the FSL as a lightweight, trainable module that integrates with already frozen pre-trained models on tabular datasets to highlight the features the original model considers most important. This post-hoc FSL learns feature relevance by fine-tuning its weights based on the original model's learned outputs. Crucially, this process is non-invasive, operating without altering the original model's architecture or its learned parameters. We conducted our experiments using both statistical and visual metrics, including accuracy, F1 score, recall, precision, weighted t-SNE and silhouette score, and also analyzed the stability of the post-hoc FSL on high-dimensional synthetic and real-world tabular datasets. We compared the post-hoc FSL feature weighting method using these metrics against the original embedded FSL and other post-hoc interpretability methods, such as Integrated Gradients, Noise Tunnel, DeepLIFT, Gradient SHAP, and Feature Ablation. Experimental results demonstrate that the post-hoc FSL feature weighting method successfully identified relevant features across the different datasets, maintaining the predictive power of the original neural network while enhancing its interpretability.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.