6 Deep constrained monotonic networks — Runje et al.
This chapter extends the Runje–Shankaranarayana (2023) constrained-monotonic architecture (their absolute-mode dense layer, split into convex and concave neuron blocks) with skip connections: gated residual blocks that stack the constrained-monotone dense layer to arbitrary depth while staying trainable, and remain sound (monotone) and universal (retain the depth-4 UAP) at every depth. The result (forthcoming) is presented in three stages: the constrained monotone dense layer itself (§6.1), the residual/deep-stack combinator and its soundness at any depth (§6.2), and the resulting universal approximation for deep monotone networks (§6.3). The embedding of non-monotone features into a partially monotone network — the original framing of this chapter — is presented as a secondary application (§6.4), extended to use the deep-residual core.
6.1 Constrained monotone dense layers
The Runje–Shankaranarayana absolute-mode dense layer — nonnegative effective weights \(|W|\), a first block of convex neurons through a monotone base activation \(\rho \) (e.g. ELU, softplus) and a second block of concave neurons through its point reflection \(\mathrm{reflect}\, \rho \), appended — is monotone.
6.2 Skip connections and deep monotone networks
A residual block combines a skip connection with a sublayer through nonnegative scalar gates: \(x \mapsto g_\alpha \, \mathrm{skip}(x) + g_\beta \, F(x)\), with \(g_\alpha ,g_\beta \ge 0\).
A ResNet — a composition of residual blocks, of any depth — in which every block has nonnegative gates and monotone skip/sublayer maps, is itself monotone.
6.3 Deep monotone universal approximation
Every continuous, coordinatewise-monotone \(f\) on the unit cube is uniformly \(\varepsilon \)-approximated by a monotone DeepMonoNet (a residual body of any depth plus a nonnegative-weight affine read-out). The witness embeds the depth-4 MonoNet of theorem 22 as a single residual block, so no depth beyond 4 is needed to retain UAP.
6.4 Partial monotonicity (secondary)
This section presents the chapter’s original application: embedding non-monotone features into a partially monotone network, extended so the monotone core may be an arbitrary-depth deep-residual network (theorem 40) rather than only the shallow depth-4 core.
6.4.1 Embedding non-monotone features
A partially monotone target \(f(u,x)\) is continuous and, for each fixed non-monotone block \(u\), monotone in the monotone block \(x\); its dependence on \(u\) is otherwise arbitrary. The architecture embeds \(u\) through an unconstrained single-hidden-layer network, clamps the embedding into \([0,1]\), concatenates it with \(x\), and feeds the result to a monotone network.
\(\mathrm{genSpanPi}\, \sigma \, d_f\) is the linear span, inside all functions \((\mathrm{Fin}\, d_f\to \mathbb {R})\to \mathbb {R}\), of the ridge units \(x\mapsto \sigma \! \left(\sum _c w_c x_c + b\right)\) — the Leshno single-hidden-layer family written with the explicit dot product, so no inner-product instance on the \(\Pi \)-type is needed.
A PartMonoNet bundles an embedding width, an unconstrained embedding \(\mathrm{emb}\colon (\mathrm{Fin}\, d_f\to \mathbb {R})\to (\mathrm{Fin}\, N\to \mathbb {R})\), and a monotone network \(\mathrm{mono}\colon \texttt{MonoNet}\, (N+d_m)\). Its denotation is \(\mathrm{toFun}(u,x)=\mathrm{mono}\big(\mathrm{append}\, (\mathrm{clamp}_{[0,1]}\circ \mathrm{emb}\, u)\; x\big)\), where \(\mathrm{clamp}_{[0,1]}\) is a fixed bounded output activation on the embedding.
6.4.2 Soundness and universal approximation
If the core MonoNet is monotone, then for every fixed \(u\) the denotation \(x\mapsto \mathrm{toFun}(u,x)\) is monotone in the (coordinatewise) \(\Pi \)-order.
theorem PartMonoNet.monotone_snd {df dm} (P : PartMonoNet df dm)
(h : P.mono.IsMonotone) (u : Fin df → ℝ) : Monotone (P.toFun u)
The clamped embedding \(\mathrm{clamp}_{[0,1]}\circ \mathrm{emb}\, u\) is a fixed vector, so appending it and varying \(x\) is monotone in the \(\Pi \)-order; compose with the monotone denotation of MonoNet. No continuity of \(\mathrm{emb}\) is required.
If \(\sigma \) densely approximates (Leshno), then for every compact \(K\subseteq \mathrm{Fin}\, d_f\to \mathbb {R}\) and every function continuous on \(K\), there is a total \(g\in \mathrm{genSpanPi}\, \sigma \, d_f\) approximating it to any accuracy on \(K\). Proved by transporting Leshno’s density across the \(\mathrm{EuclideanSpace}\, \mathbb {R}\, (\mathrm{Fin}\, d_f)\cong (\mathrm{Fin}\, d_f\to \mathbb {R})\) isometry (the Euclidean inner product equals the dot product).
Pull the target back through the isometry to a continuous function on the (compact) preimage, apply Leshno density there to get a spanning element, and push the finite ridge combination back to a total \(\Pi \)-side function; the per-generator identity is exactly the inner-product / dot product equality.
For any Leshno-admissible activation \(\sigma \) (class \(M\), not a.e. polynomial) and any jointly continuous \(f\) that is coordinatewise monotone in \(x\) on the unit cubes, and any \(\varepsilon \gt 0\), there is a PartMonoNet whose core is monotone, whose every embedding coordinate lies in \(\mathrm{genSpanPi}\, \sigma \, d_f\), and which approximates \(f\) within \(\varepsilon \) uniformly on the cube.
theorem partial_monotone_approximation {df dm : ℕ}
(σ : ℝ → ℝ) (hσ : ClassM σ) (hnp : ¬ IsAEPolynomial σ)
(f : (Fin df → ℝ) → (Fin dm → ℝ) → ℝ)
(hf : ContinuousOn (fun p => f p.1 p.2)
(Set.Icc (0 : Fin df → ℝ) 1 ×ˢ Set.Icc (0 : Fin dm → ℝ) 1))
(hmono : ∀ u ∈ Set.Icc (0 : Fin df → ℝ) 1,
∀ ⦃x y⦄, x ∈ Set.Icc (0 : Fin dm → ℝ) 1 → y ∈ Set.Icc (0 : Fin dm → ℝ) 1 →
x ≤ y → f u x ≤ f u y)
{ε : ℝ} (hε : 0 < ε) :
∃ P : PartMonoNet df dm, P.mono.IsMonotone ∧
(∀ i, (fun u => P.emb u i) ∈ genSpanPi σ df) ∧
∀ u ∈ Set.Icc (0 : Fin df → ℝ) 1, ∀ x ∈ Set.Icc (0 : Fin dm → ℝ) 1,
|P.toFun u x - f u x| ≤ ε
Fix \(\varepsilon \). By uniform continuity of \(f\) on the compact product, choose a fine tent partition of unity \(\{ \psi _i\} \) over a grid of the \(u\)-cube with support radius small enough that moving \(u\) within a cell shifts \(f\) by less than \(\varepsilon /3\) (uniformly in \(x\)). With \(g_i(x)=f(u_i,x)+C\) shifted nonnegative, the target \(F(z,x)=\big(\sum _i z_i\, g_i(x)\big)-C\) is jointly continuous and coordinatewise monotone, so theorem 22 gives a monotone network within \(\varepsilon /3\) of \(F\) on the cube. Each \(\psi _i\) is approximated within a chosen \(\eta \) by an unconstrained embedding coordinate via lemma 37, clamped into the cube; since \(F\) is Lipschitz in \(z\) this contributes another \(\varepsilon /3\), and the partition collapse (the convex combination \(\sum _i\psi _i(u)f(u_i,x)\)) is within \(\varepsilon /3\) of \(f(u,x)\). The triangle inequality over the three terms yields \(\varepsilon \). Soundness (theorem 36) makes the resulting network monotone in \(x\).
6.4.3 Box-domain variant
The unit-cube partial-monotone UAP (theorem 38) extends to any non-degenerate box \([a_F,b_F]\times [a_M,b_M]\) (i.e. \(a_F\lt b_F\) and \(a_M\lt b_M\) coordinatewise) by an affine change of variables: pull \(f\) back to the unit cube, apply theorem 38 there, then push the witness forward — the feature embedding through the affine box\(\to \)cube map on the \(u\)-side, and the monotone core through a coordinatewise affine rescaling of its trailing (monotone) block on the \(x\)-side — to recover a PartMonoNet that approximates \(f\) within \(\varepsilon \) uniformly on the box.
6.4.4 Deep-core variant
Swapping the shallow MonoNet core of PartMonoNet for a deep-residual DeepMonoNet of any depth (a DeepPartMonoNet) retains both soundness (monotone in the monotone block \(x\), for every fixed \(u\)) and partial-monotone UAP: the witness embeds the shallow core of theorem 38 via the single-block embedding of theorem 33, sharing the same embedding, so the denotation and the approximation bound transfer verbatim.