How does numerical optimization find the minimum value of a function?

In numerical analysis, optimization is the process of finding a vector of unknowns, $x$, that minimizes a real-valued objective function, $f(x)$. Because identifying a globally optimal value is often impractical, numerical algorithms typically aim to find a local optimum—the best value within a specific region of feasible solutions.12

These algorithms generally proceed by generating a sequence of guesses, known as iterates ($x^1,x^2,x^3,\dots$), with the goal of converging toward an optimal value. Each successive iterate is determined based on information gathered about the objective function and any constraints at previous points.2

The approach taken depends heavily on the problem's characteristics:
* Problem Type: Algorithms are often specialized for specific subclasses, such as large, smooth unconstrained problems or convex quadratic programming.2
* Function Properties: Problems are categorized by whether they are smooth or nonsmooth, and whether they are unconstrained or subject to equality or inequality constraints.2
* Computational Cost: Distinctions are made based on the number of variables and the computational expense required to evaluate the function.2