Structure-agnostic Causal Representation Learning
Causal representation learning typically requires knowing the causal graph upfront, yet misspecifying this structure discards predictive information and ruins OOD generalization. This paper proposes SaCRL, a kernel-based framework that jointly identifies which of three canonical causal structures governs the data and learns the corresponding invariant representation via HSIC violation metrics and SoftMin aggregation. The key insight is that correct invariances can be satisfied without sacrificing prediction, whereas wrong ones cannot, enabling automatic structure discovery during training.
SaCRL offers an elegant conceptual advance in unifying causal structure discovery with representation learning, and its theoretical framework connecting HSIC violations to structure identifiability is genuinely compelling. However, the experimental evaluation is seriously undermined by the use of test-domain oracle selection for the critical hyperparameter $\lambda$, which the DomainBed benchmark paper explicitly warns against as unrealistic. Consequently, claims of state-of-the-art performance on PACS, VLCS, and OfficeHome are unreliable, and the gap over ERM largely evaporates when the backbone is fine-tuned end-to-end.
The structural insight driving SaCRL is rigorous and well-motivated. The authors correctly observe that "under the true causal structure, the corresponding invariance is achievable by predictive representations, whereas wrong invariances incur irreducible violation," which provides a clean optimization principle for structure selection. The SoftMin relaxation over HSIC violations avoids the combinatorial explosion of discrete structure search, and the robustness experiments showing that SaCRL "acts as a soft model average instead of committing to a wrong structure" under misspecification are a genuine practical strength.
The most damaging flaw is the experimental protocol: the authors admit that they "select the invariance penalty weight $\lambda \in \{0.5,1.0,2.0,5.0\}$ per experiment by OOD accuracy, i.e., test-domain (oracle) selection in the terminology of Gulrajani and Lopez-Paz [20]." Since DomainBed's central message is that "a domain generalization algorithm should be responsible for specifying a model selection method," and oracle selection is impossible in practice, all comparisons on real-world benchmarks are biased in SaCRL's favor.\n\nThe paper also claims the three canonical structures are exhaustive under "mild conditions," but these conditions rule out cyclic or feedback relationships—a limitation highlighted by the authors' own robustness test with bidirectional $X \leftrightarrow Y$, which violates the acyclicity assumed throughout the theory. Finally, the main identification theorems rely on a "faithfulness gap" $\eta$ and "sufficient environment diversity," quantities that are not estimable from data alone.
On synthetic and semi-synthetic benchmarks where ground truth is known, SaCRL demonstrates convincing structure identification, recovering $\mathcal{G}_3$ most reliably because its invariance is unconditional. However, the semi-synthetic Bayesian networks "admit $\mathcal{G}_2$ and $\mathcal{G}_3$ but not $\mathcal{G}_1$," meaning the evaluation never tests discrimination between all three structures on real DAGs. On Colored MNIST, the authors "subsample 5000 examples per training environment and 5000 test examples to ensure balanced evaluation," a protocol that differs from prior work and lowers all accuracies, so the reported gain is not comparable to literature values. On DomainBed, the comparison set is minimal—only ERM, IRM, VREx, and CIRCE—omitting modern DG methods that might outperform the baselines under fair model selection.
The authors release code and provide substantial algorithmic detail, including random Fourier feature approximations with $D=1000$, CVaR aggregation with $\rho=0.5$, and multi-start optimization. Yet independent reproduction is severely hampered by the reliance on test-domain oracle selection for $\lambda$: without access to the test domain, one cannot reproduce the reported numbers. Additionally, the need to "run the optimization from $M \geq 10$ random initializations and select the solution with the lowest objective value," cross-fitted residuals, and EMA-conditioned encoders adds considerable engineering complexity, and the theoretical guarantees require unverifiable assumptions about the faithfulness gap and environment diversity matrix.
Causal representation learning aims to discover robust features by exploiting the causal structure underlying data generation. Existing methods require specifying the causal structure a priori, yet different structures demand fundamentally incompatible invariance constraints, and misspecification leads to representations that discard predictive information. We introduce SaCRL, a framework that jointly identifies the causal structure and learns the corresponding invariant representation without prior structural knowledge. Our approach formulates structure selection as a soft optimization over candidate invariances using HSIC-based violation metrics, with adaptive weights that automatically concentrate on the achievable structure. We provide theoretical guarantees for structure identification, including under random-feature approximation, invariance satisfaction, and out-of-distribution generalization. Empirically, SaCRL recovers the true structure on synthetic and semi-synthetic Bayesian-network benchmarks, outperforms fixed-invariance baselines on Colored MNIST, achieves state-of-the-art accuracy on three DomainBed benchmarks (PACS, VLCS, OfficeHome), and degrades gracefully under structural misspecification and limited environment diversity. Code is available at: https://github.com/ArmanBehnam/sacrl.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.