Dependencies

An ActStack is an inductively-built chain of affine layers, each followed by a one-dimensional activation.

\(\sigma \) is in class \(M\) when it is locally bounded and the closure of its set of discontinuities is Lebesgue-null.

\(\sigma \) is discriminatory for a compact \(K\) when the only signed measure \(\mu \) on \(K\) that annihilates every affine pre-composition \(x\mapsto \sigma (\left\langle w, x \right\rangle +b)\) is \(\mu =0\).

A two-layer threshold gadget whose output is the indicator \(\mathbf{1}(x\ge x_j)\) of coordinatewise domination of a fixed point \(x_j\).

\(\mathrm{genSpanPi}\, \sigma \, d_f\) is the linear span, inside all functions \((\mathrm{Fin}\, d_f\to \mathbb {R})\to \mathbb {R}\), of the ridge units \(x\mapsto \sigma \! \left(\sum _c w_c x_c + b\right)\) — the Leshno single-hidden-layer family written with the explicit dot product, so no inner-product instance on the \(\Pi \)-type is needed.

\(\mathrm{heaviside}(z)=1\) if \(z\ge 0\) and \(0\) otherwise.

A FICNN threads the input \(y\) into every layer, \(z_{i+1}=\rho _i(W^z_i z_i + W^y_i y + b_i)\), with nonnegative propagation weights \(W^z_i\ge 0\), unconstrained input skip \(W^y_i\), and convex nondecreasing activations \(\rho _i\).

\(\sigma \) is a.e. a polynomial when \(\sigma =p\) almost everywhere for some polynomial \(p\).

A MonoNet is an activation stack together with an affine read-out (weights readW and bias readBias).

A PartMonoNet bundles an embedding width, an unconstrained embedding \(\mathrm{emb}\colon (\mathrm{Fin}\, d_f\to \mathbb {R})\to (\mathrm{Fin}\, N\to \mathbb {R})\), and a monotone network \(\mathrm{mono}\colon \texttt{MonoNet}\, (N+d_m)\). Its denotation is \(\mathrm{toFun}(u,x)=\mathrm{mono}\big(\mathrm{append}\, (\mathrm{clamp}_{[0,1]}\circ \mathrm{emb}\, u)\; x\big)\), where \(\mathrm{clamp}_{[0,1]}\) is a fixed bounded output activation on the embedding.

\(\mathrm{reflect}(\sigma )(x) = -\sigma (-x)\). It is an involution, preserves monotonicity, and swaps the two saturation sides.

A residual block combines a skip connection with a sublayer through nonnegative scalar gates: \(x \mapsto g_\alpha \, \mathrm{skip}(x) + g_\beta \, F(x)\), with \(g_\alpha ,g_\beta \ge 0\).

\(\sigma \) is right-saturating (\(\mathcal{S}^+\)) if it has a finite limit at \(+\infty \), and left-saturating (\(\mathcal{S}^-\)) if it has a finite limit at \(-\infty \). This admits unbounded activations such as ReLU on one side.

A function \(\sigma \colon \mathbb {R}\to \mathbb {R}\) is sigmoidal when it is continuous with \(\sigma (t)\to 0\) as \(t\to -\infty \) and \(\sigma (t)\to 1\) as \(t\to +\infty \).

If \(\sigma \) is continuous at \(b\), then for any \(\varepsilon \gt 0\) there is \(\delta \gt 0\) with \(|\sigma (t)-\sigma (b)|\le \varepsilon \) whenever \(|t-b|\le \delta \).

If \(\sigma \) densely approximates (Leshno), then for every compact \(K\subseteq \mathrm{Fin}\, d_f\to \mathbb {R}\) and every function continuous on \(K\), there is a total \(g\in \mathrm{genSpanPi}\, \sigma \, d_f\) approximating it to any accuracy on \(K\). Proved by transporting Leshno’s density across the \(\mathrm{EuclideanSpace}\, \mathbb {R}\, (\mathrm{Fin}\, d_f)\cong (\mathrm{Fin}\, d_f\to \mathbb {R})\) isometry (the Euclidean inner product equals the dot product).

The Runje–Shankaranarayana absolute-mode dense layer — nonnegative effective weights \(|W|\), a first block of convex neurons through a monotone base activation \(\rho \) (e.g. ELU, softplus) and a second block of concave neurons through its point reflection \(\mathrm{reflect}\, \rho \), appended — is monotone.

For a right-saturating \(\sigma \) with limit \(L\), and any \(\varepsilon ,m\gt 0\), there is a gain \(\Lambda \) so that for all \(\lambda \ge \Lambda \) and \(t\ge m\), \(|\sigma (\lambda t)-L|\le \varepsilon \).

For a monotone, non-constant, right-saturating \(\sigma \), any monotone dataset is \(\varepsilon \)-interpolated by a depth-4 network with all non-positive weights and the single activation \(\sigma \) on its three hidden layers — whose denotation is nonetheless monotone.

Negating a layer’s input and using the reflected activation \(\mathrm{reflect}(\sigma )\) equals negating the next layer’s weights and using \(\sigma \): the weight-sign flip is absorbed by point reflection.

If \(\sigma \) is a.e. a polynomial, the network family spans only polynomials of bounded degree and is not dense.

Convolving an \(M\)-class \(\sigma \) with a smooth compactly-supported bump yields a \(C^\infty \) function.

Every \(f\in C(K,\mathbb {R})\) is approximated to any \(\varepsilon \gt 0\) in sup-norm by an element of the family.

The single-hidden-layer family with a continuous sigmoidal activation is dense in \(C(K,\mathbb {R})\).

Every continuous, coordinatewise-monotone \(f\) on the unit cube is uniformly \(\varepsilon \)-approximated by a monotone DeepMonoNet (a residual body of any depth plus a nonnegative-weight affine read-out). The witness embeds the depth-4 MonoNet of theorem 22 as a single residual block, so no depth beyond 4 is needed to retain UAP.

Swapping the shallow MonoNet core of PartMonoNet for a deep-residual DeepMonoNet of any depth (a DeepPartMonoNet) retains both soundness (monotone in the monotone block \(x\), for every fixed \(u\)) and partial-monotone UAP: the witness embeds the shallow core of theorem 38 via the single-block embedding of theorem 33, sharing the same embedding, so the denotation and the approximation bound transfer verbatim.

A subspace \(V\subseteq C(K,\mathbb {R})\) is dense iff every continuous linear functional vanishing on \(V\) is the zero functional.

If an \(M\)-class \(\sigma \) is not a.e. a polynomial, some smooth compactly-supported bump makes the mollification \(C^\infty \) and still not a.e. a polynomial.

Such a network denotes a convex function of its input.

Let \(f\) be convex and differentiable on \(\mathbb {R}^d\), and let \(K\) be compact. For every \(\varepsilon \gt 0\) there is a fully input-convex network, with nonnegative propagation weights and convex nondecreasing (ReLU and identity) activations, that is within \(\varepsilon \) of \(f\) uniformly on \(K\).

A non-a.e.-polynomial \(M\)-class \(\sigma \) densely approximates.

An \(M\)-class \(\sigma \) densely approximates iff it is not a.e. a polynomial.

Every continuous monotone \(f\) on the unit cube is uniformly \(\varepsilon \)-approximated by a monotone depth-4 MonoNet.

For injective points \(x\) and targets \(y\) monotone along the coordinatewise order, some monotone depth-4 MonoNet agrees with \(y\) on every \(x_i\) exactly.

A ResNet — a composition of residual blocks, of any depth — in which every block has nonnegative gates and monotone skip/sublayer maps, is itself monotone.

If \(\sigma \) is univariate-dense, then ridge functions \(x\mapsto g(\left\langle w, x \right\rangle )\) are dense in \(C(K,\mathbb {R})\) on compact \(K\).

Every continuous linear functional \(L\) on \(C(K,\mathbb {R})\) is represented by a signed regular Borel measure \(\mu \), with \(L=0\) iff \(\mu =0\).

If the core MonoNet is monotone, then for every fixed \(u\) the denotation \(x\mapsto \mathrm{toFun}(u,x)\) is monotone in the (coordinatewise) \(\Pi \)-order.

For any Leshno-admissible activation \(\sigma \) (class \(M\), not a.e. polynomial) and any jointly continuous \(f\) that is coordinatewise monotone in \(x\) on the unit cubes, and any \(\varepsilon \gt 0\), there is a PartMonoNet whose core is monotone, whose every embedding coordinate lies in \(\mathrm{genSpanPi}\, \sigma \, d_f\), and which approximates \(f\) within \(\varepsilon \) uniformly on the cube.

The unit-cube partial-monotone UAP (theorem 38) extends to any non-degenerate box \([a_F,b_F]\times [a_M,b_M]\) (i.e. \(a_F\lt b_F\) and \(a_M\lt b_M\) coordinatewise) by an affine change of variables: pull \(f\) back to the unit cube, apply theorem 38 there, then push the witness forward — the feature embedding through the affine box\(\to \)cube map on the \(u\)-side, and the monotone core through a coordinatewise affine rescaling of its trailing (monotone) block on the \(x\)-side — to recover a PartMonoNet that approximates \(f\) within \(\varepsilon \) uniformly on the box.

For alternating monotone one-sided-saturating activations \([\sigma _1,\sigma _2,\sigma _3]=[\mathcal{S}^-,\mathcal{S}^+,\mathcal{S}^-]\), each non-constant, a monotone depth-4 MonoNet interpolates any monotone dataset to accuracy \(\varepsilon \).

Every continuous sigmoidal \(\sigma \) is discriminatory.

A non-a.e.-polynomial \(M\)-class \(\sigma \) is dense on compact intervals.