7 Input-convex networks — Amos et al.
A FICNN threads the input \(y\) into every layer, \(z_{i+1}=\rho _i(W^z_i z_i + W^y_i y + b_i)\), with nonnegative propagation weights \(W^z_i\ge 0\), unconstrained input skip \(W^y_i\), and convex nondecreasing activations \(\rho _i\).
Such a network denotes a convex function of its input.
By induction over the layer chain: each layer maps convex coordinates to convex coordinates — a nonnegative combination of convex functions plus the affine input skip is convex, and a convex nondecreasing activation composed with a convex function is convex.
Let \(f\) be convex and differentiable on \(\mathbb {R}^d\), and let \(K\) be compact. For every \(\varepsilon \gt 0\) there is a fully input-convex network, with nonnegative propagation weights and convex nondecreasing (ReLU and identity) activations, that is within \(\varepsilon \) of \(f\) uniformly on \(K\).
Cover \(K\) by finitely many points \(x_1,\dots ,x_n\) close enough that \(f\)’s modulus of continuity is below \(\varepsilon /2\) at that scale. At each \(x_i\), the tangent plane \(y \mapsto f(x_i) + \nabla f(x_i)\cdot (y-x_i)\) minorizes \(f\) by convexity, and equals \(f\) at \(x_i\); taking the pointwise maximum of these tangent planes over \(i\) therefore stays below \(f\) everywhere and comes within \(\varepsilon \) of \(f\) on \(K\). This max-of-affine-functions is realized exactly by a nonnegative-weight ICNN with ReLU and identity activations (a max of two affine functions is \(\mathrm{ReLU}(u-v)+v\), iterated), giving the network witness.