5 Monotone networks — Sartor et al.
5.1 Beyond thresholds: saturating activations (Sartor et al. 2025)
\(\sigma \) is right-saturating (\(\mathcal{S}^+\)) if it has a finite limit at \(+\infty \), and left-saturating (\(\mathcal{S}^-\)) if it has a finite limit at \(-\infty \). This admits unbounded activations such as ReLU on one side.
\(\mathrm{reflect}(\sigma )(x) = -\sigma (-x)\). It is an involution, preserves monotonicity, and swaps the two saturation sides.
For a right-saturating \(\sigma \) with limit \(L\), and any \(\varepsilon ,m\gt 0\), there is a gain \(\Lambda \) so that for all \(\lambda \ge \Lambda \) and \(t\ge m\), \(|\sigma (\lambda t)-L|\le \varepsilon \).
Immediate from the definition of the limit at \(+\infty \): for \(t\ge m\), \(\lambda t\to +\infty \) as \(\lambda \to \infty \), so \(\sigma (\lambda t)\) enters any \(\varepsilon \)-ball of \(L\).
If \(\sigma \) is continuous at \(b\), then for any \(\varepsilon \gt 0\) there is \(\delta \gt 0\) with \(|\sigma (t)-\sigma (b)|\le \varepsilon \) whenever \(|t-b|\le \delta \).
A restatement of continuity at \(b\) as an explicit \(\varepsilon \)–\(\delta \) bound.
Negating a layer’s input and using the reflected activation \(\mathrm{reflect}(\sigma )\) equals negating the next layer’s weights and using \(\sigma \): the weight-sign flip is absorbed by point reflection.
Expand both layer compositions and apply \(\mathrm{reflect}(\sigma )(x)=-\sigma (-x)\); the two sign flips cancel, matching the negated-weight layer applied to \(\sigma \).
For alternating monotone one-sided-saturating activations \([\sigma _1,\sigma _2,\sigma _3]=[\mathcal{S}^-,\mathcal{S}^+,\mathcal{S}^-]\), each non-constant, a monotone depth-4 MonoNet interpolates any monotone dataset to accuracy \(\varepsilon \).
theorem saturating_interpolation {d n : ℕ} (x : Fin n → (Fin d → ℝ))
(y : Fin n → ℝ) (hmono : ∀ i j, x i ≤ x j → y i ≤ y j)
(hinj : Function.Injective x) (σ₁ σ₂ σ₃ : ℝ → ℝ)
(hm₁ : Monotone σ₁) (hm₂ : Monotone σ₂) (hm₃ : Monotone σ₃)
(hs₁ : LeftSaturating σ₁) (hs₂ : RightSaturating σ₂) (hs₃ : LeftSaturating σ₃)
(hnc₁ : ∃ a b, σ₁ a < σ₁ b) (hnc₂ : ∃ a b, σ₂ a < σ₂ b)
(hnc₃ : ∃ a b, σ₃ a < σ₃ b) {ε : ℝ} (hε : 0 < ε) :
∃ N : MonoNet d, N.IsMonotone ∧ N.depth = 4 ∧
N.stack.activations = [σ₁, σ₂, σ₃] ∧ ∀ i, |N.toFun (x i) - y i| ≤ ε
Start from the threshold interpolant (theorem 21) and replace each Heaviside gate by a scaled saturating activation. With alternating sides \([\mathcal{S}^-,\mathcal{S}^+, \mathcal{S}^-]\), the gain estimate (lemma 25) makes each gate approximate its threshold arbitrarily well, and interior continuity (lemma 26) controls values near the boundary; choosing the gain large enough yields \(\varepsilon \)-accuracy. Deviation note. The paper’s exact identity \(g(x_i)=f(x_i)\) is the \(\lambda \to \infty \) idealization; the faithful finite-net statement is \(\varepsilon \)-approximate, and the non-degeneracy hypotheses are exactly \(\sigma _k\) non-constant (\(\exists a\, b,\ \sigma _k a \lt \sigma _k b\)).
For a monotone, non-constant, right-saturating \(\sigma \), any monotone dataset is \(\varepsilon \)-interpolated by a depth-4 network with all non-positive weights and the single activation \(\sigma \) on its three hidden layers — whose denotation is nonetheless monotone.
theorem nonpos_weight_universal {d n : ℕ} (x : Fin n → (Fin d → ℝ))
(y : Fin n → ℝ) (hmono : ∀ i j, x i ≤ x j → y i ≤ y j)
(hinj : Function.Injective x) (σ : ℝ → ℝ) (hmσ : Monotone σ)
(hsat : RightSaturating σ) (hnc : ∃ a b, σ a < σ b)
{ε : ℝ} (hε : 0 < ε) :
∃ N : MonoNet d, N.stack.WeightsNonpos ∧ (∀ i, N.readW i ≤ 0) ∧
N.stack.activations = [σ, σ, σ] ∧ N.depth = 4 ∧ Monotone N.toFun ∧
∀ i, |N.toFun (x i) - y i| ≤ ε
Apply theorem 28 with activations \([\mathrm{reflect}\, \sigma ,\sigma , \mathrm{reflect}\, \sigma ]=[\mathcal{S}^-,\mathcal{S}^+,\mathcal{S}^-]\), then use the Prop 3.10 sign-flip identities (proposition 27) to turn every layer into \(\sigma \) with negated weights; the final output sign flip is absorbed into the read-out, so the denotation is unchanged and stays monotone.