2 Cybenko (1989): sigmoidal density
A function \(\sigma \colon \mathbb {R}\to \mathbb {R}\) is sigmoidal when it is continuous with \(\sigma (t)\to 0\) as \(t\to -\infty \) and \(\sigma (t)\to 1\) as \(t\to +\infty \).
\(\sigma \) is discriminatory for a compact \(K\) when the only signed measure \(\mu \) on \(K\) that annihilates every affine pre-composition \(x\mapsto \sigma (\left\langle w, x \right\rangle +b)\) is \(\mu =0\).
A subspace \(V\subseteq C(K,\mathbb {R})\) is dense iff every continuous linear functional vanishing on \(V\) is the zero functional.
Standard Hahn–Banach separation: a proper closed subspace admits a nonzero functional vanishing on it, and conversely a dense subspace forces any vanishing functional to be zero.
Every continuous linear functional \(L\) on \(C(K,\mathbb {R})\) is represented by a signed regular Borel measure \(\mu \), with \(L=0\) iff \(\mu =0\).
Assemble the Riesz–Kantorovich positive/negative decomposition of \(L\) with the positive Riesz–Markov–Kakutani theorem to obtain the representing signed measure.
Every continuous sigmoidal \(\sigma \) is discriminatory.
For a signed measure \(\mu \) annihilating all \(x\mapsto \sigma (\left\langle w, x \right\rangle +b)\), scaling the argument drives \(\sigma \) pointwise to the indicator of a half-space; dominated convergence transfers the annihilation to indicators of all half-spaces \(\{ x:\left\langle w, x \right\rangle +b\gt 0\} \). A signed measure annihilating every half-space indicator vanishes, so \(\mu =0\).
The single-hidden-layer family with a continuous sigmoidal activation is dense in \(C(K,\mathbb {R})\).
theorem universal_approximation (σ : ℝ → ℝ) (hσ : Sigmoidal σ)
{K : Set (EuclideanSpace ℝ (Fin n))} (hK : IsCompact K) :
(S σ hσ.continuous (K := K)).topologicalClosure = ⊤
By theorem 3, density reduces to: every continuous functional \(L\) vanishing on the family is zero. By theorem 4, represent \(L\) by a signed measure \(\mu \) with \(L=0\) iff \(\mu =0\). Each generator \(x\mapsto \sigma (\left\langle w, x \right\rangle +b)\) lies in the family, so \(\mu \) annihilates every affine pre-composition of \(\sigma \). By theorem 5, \(\mu =0\), hence \(L=0\), proving density.
Every \(f\in C(K,\mathbb {R})\) is approximated to any \(\varepsilon \gt 0\) in sup-norm by an element of the family.
theorem universal_approximation_eps (σ : ℝ → ℝ) (hσ : Sigmoidal σ)
{K : Set (EuclideanSpace ℝ (Fin n))} (hK : IsCompact K) [CompactSpace ↥K]
(f : C(↥K, ℝ)) {ε : ℝ} (hε : 0 < ε) :
∃ g ∈ S σ hσ.continuous (K := K), ‖f - g‖ < ε
Unfold the topological-closure statement of theorem 6 into its sup-norm \(\varepsilon \)-form; the \(\mathtt{CompactSpace}\) instance makes the norm on \(C(K,\mathbb {R})\) available.