A Bregman Proximal Viewpoint on Neural Operators
Neural operators learn mappings between function spaces, yet training deep architectures remains difficult. This paper proposes viewing each operator layer as the minimizer of a Bregman-regularized functional optimization problem in Banach spaces. This unifies classical neural operators under a proximal framework and yields a new Bregman variant that adds an inverse activation term, showing improved depth scaling and accuracy on standard PDE benchmarks.
The paper provides a principled proximal framework for neural operators that is both conceptually unifying and empirically promising. Its strengths include the clean recovery of classical architectures and measurable gains in depth scaling for the proposed Fourier variant. However, the core architectural idea is adapted from a prior finite-dimensional proposal, the universal approximation theorem assumes sigmoidal activations while experiments use a SoftPlus surrogate for ReLU, and no public code release is mentioned.
The Bregman proximal interpretation is mathematically coherent. Proposition 3.4 shows that Euclidean Bregman distances recover standard operator layers with monotone activations, while Proposition 3.5 yields the novel Bregman layer via general Legendre functions. Empirically, the property that layers reduce to the identity when all weights vanish translates into measurably better depth scaling: BFNO continues improving up to 64 layers, whereas FNO and ResFNO degrade after 8--16 layers.
The main architectural novelty---a layer with an inverse activation term---was already proposed in finite dimensions by Frecon et al. (2022). Extending it to Banach function spaces is technically involved, but the practical instantiation relies on SoftPlus as a ReLU surrogate rather than the theoretically analyzed sigmoidal maps (Theorem 4.1). Moreover, the inverse term requires domain compatibility conditions (Remark 3.7), yet the implementation sidesteps these by using an activation whose inverse cancels out in composition, limiting experimental validation of the full framework.
The evidence supports the claim that BFNO improves depth scaling relative to standard FNO and ResFNO on Navier--Stokes and Burgers benchmarks. However, BFNO underperforms F-FNO on the 2D Darcy task, which the authors attribute to missing extra MLPs. Comparisons are restricted to Fourier-type operators; attention-based or DeepONet-style baselines are not evaluated empirically, leaving open whether the Bregman modification is competitive across the broader operator-learning landscape.
The experiments rely on public PDEBench datasets and follow prior FNO protocols, with layer widths and Fourier-mode counts specified. Nevertheless, no code repository URL is provided, and the implementation avoids explicit inverse activations by using SoftPlus as a near-ReLU surrogate. Exact hyperparameter schedules, initialization details for the inverse term, and random seeds are relegated to Appendix C, which is likely insufficient for independent reproduction without source code.
We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.