Start reviewing Newton's method

Numerical-optimization.md

@@ -28,7 +28,19 @@ Uniform regularization can be seen as an interpolation between Newton’s method
 
 ### Newton’s method
 
-_To be added_
+#### Review
+
+Let's say we're trying to minimize a smooth function $f$ on an affine search space with underlying vector space $V$. We can use the affine structure to take a second-order approximation of $f$ near any point $p$:
+
+$$f(p + v) \in f^{(0)}_p + f^{(1)}_p(v) + \tfrac{1}{2} f^{(2)}_p(v, v) + \mathfrak{m}^2$$
+
+Here, $v$ is a $V$-valued variable, each $f^{(k)}_p$ is a symmetric $k$-linear form on $V$, and $\mathfrak{m}$ is the ideal of smooth functions on $V$ that vanish to first order at the origin.
+
+When $f^{(2)}_p(v, v)$ is positive-definite, the second-order approximation has a unique minimum. Newton’s method is based on the hope that this minimum is near a local minimum of $f$. It works by repeatedly stepping to the minimum of the second-order approximation and then taking a new second-order approximation at that point. The minimum of the second-order approximation is nicely characterized as the place where the derivative of the second-order approximation vanishes. The *Newton step*—the vector $v \in V$ that takes us to the minimum of the second-order approximation—is therefore the solution of the following equation:
+
+$$f^{(1)}(-) + f^{(2)}(v, -) = 0.$$
+
+_To be continued_
 
 #### Uniform regularization