How are nonlinear problems reduced to sequences of linear problems?

In numerical analysis, nonlinear problems are frequently addressed by approximating them with a sequence of simpler, linear problems. This approach is rooted in the principle of replacing a difficult problem with a "nearby" one that is easier to solve.1

Newton’s Method

The most prominent example of this reduction is Newton’s iterative method. To solve a nonlinear equation $f(x)=0$, the function's graph is approximated by a tangent line at a point near the root. The root of this linear tangent line then serves as an improved approximation for the root of the original nonlinear function. This process is repeated iteratively:2
$$x^{(k+1)}=x^{(k)}-\frac{f(x^{(k)})}{f^{\prime }(x^{(k)})}$$
where $f^{\prime }(x)$ is the derivative of the function.23

Systems of Nonlinear Equations

This concept generalizes to systems of nonlinear equations, denoted as $f(x)=0$. Here, the derivative is replaced by the Jacobian matrix, and each iteration involves solving a linear system of equations to find the next approximation.2

Nonlinear Programming

Similarly, in nonlinear programming, algorithms generate a sequence of guesses (iterates) to identify optimal values. These methods rely on information gathered at earlier points to construct linear or quadratic approximations, effectively transforming complex nonlinear optimization into a series of manageable linear steps.4