Why σ(x) = 1/(1+e⁻ˣ): The Derivation Behind Sigmoid

Most practitioners can cite σ(x) = 1 / (1 + e⁻ˣ) from memory, but the derivation — why e, why a negative exponent, why that specific denominator structure — is typically glossed over in favour of reaching gradient descent faster. Nikhil Dasari's walkthrough reconstructs that derivation from first principles, and the reasoning has direct consequences for how engineers think about backpropagation and the vanishing-gradient problem.

Where e ≈ 2.71828 Actually Comes From

The constant e emerges from the mathematics of continuous growth. Starting from a principal at a 100% annual interest rate, compounding annually yields a growth factor of (1 + 1/1)¹ = 2. Compounding semi-annually gives (1 + 1/2)² = 2.25. Quarterly gives (1 + 1/4)⁴ ≈ 2.4414. Monthly produces (1 + 1/12)¹² ≈ 2.613. Daily reaches (1 + 1/365)³⁶⁵ ≈ 2.7146. As n → ∞ in (1 + 1/n)ⁿ, the limit is exactly e ≈ 2.71828 — the formal definition of the constant, not a curiosity.

The calculus interpretation explains why e is the natural base here. For a general exponential y = zˣ, differentiating via the limit definition yields d/dx(zˣ) = zˣ · lim_{h→0} (zʰ − 1)/h. That limit evaluates to a constant C that depends only on the base z. For z = 3, C ≈ 1.0986, so d/dx(3ˣ) ≈ 1.0986 · 3ˣ — the derivative is proportional to, but not equal to, the original function. The unique base for which C = 1 is e itself, satisfying lim_{h→0} (eʰ − 1)/h = 1 and therefore d/dx(eˣ) = eˣ exactly. This self-referential property — rate of change equals current value — is why e appears wherever smooth, continuous growth needs to be differentiated cleanly, including inside activation functions. This kind of architectural specificity in mathematical primitives separates functions that survive backpropagation cleanly from those that introduce numerical artefacts.

Constructing the Sigmoid Form Step by Step

The sigmoid's structure follows from constraints imposed by logistic regression. A linear score z = wx + b maps to (−∞, +∞), which cannot be interpreted as a probability. The requirement is a function that receives any real number and returns a value strictly inside (0, 1).

The derivation proceeds via a reciprocal argument: any number A > 1 satisfies 0 < 1/A < 1. The task reduces to constructing an expression that is always greater than 1 and varies with the input. Since eˣ > 0 for every real x, its reciprocal e⁻ˣ = 1/eˣ is also always positive. Adding 1 guarantees 1 + e⁻ˣ > 1 unconditionally. Taking the reciprocal of that sum yields 1/(1 + e⁻ˣ), bounded strictly between 0 and 1 for all real x. That expression is σ(x).

Boundary behaviour confirms the construction: as x → −∞, e⁻ˣ → +∞ so σ(x) → 0; as x → +∞, e⁻ˣ → 0 so σ(x) → 1. At x = 0, e⁰ = 1 and σ(0) = 1/(1+1) = 0.5, the symmetric midpoint.

Input–Output Behaviour at Key Values

The denominator grows large for negative inputs (suppressing output toward 0) and shrinks toward 1 for large positive inputs (pushing output toward 1). This monotonic response to sign is what makes sigmoid suitable as a binary classification boundary with a threshold of 0.5.

The Derivative and Its Role in Backpropagation

Because sigmoid appears inside neural network forward passes, its derivative must be computed during the backward pass. The derivative σ′(x) can be expressed entirely in terms of the sigmoid output itself — a form that avoids recomputing exponentials at every layer and plugs directly into the chain rule.

This closed-form self-referential derivative is also the entry point into the vanishing-gradient problem: at the saturating tails (x ≪ 0 or x ≫ 0), σ′(x) approaches 0, meaning gradient signals attenuate multiplicatively through stacked sigmoid layers. The source material positions this derivation as prerequisite knowledge before tackling vanishing and exploding gradients — a sequencing decision that reflects how the numerical behaviour of σ′(x) at extreme inputs is the mechanistic cause of training instability.

Understanding sigmoid at the level of derivation — not as a lookup formula but as a consequence of e's self-derivative property and a reciprocal-bounding construction — gives practitioners a concrete handle on why alternative activations like ReLU were adopted for deep networks and what tradeoffs sigmoid still offers in output layers and gating mechanisms. The mathematics did not change; the engineering context around it did.