Integrate engine into application prototype (#15)

Port the engine prototype to Rust, integrate it into the application prototype, and use it to enforce the constraints.

### Features

To see the engine in action:

1. Add a constraint by shift-clicking to select two spheres in the outline view and then hitting the 🔗 button
2. Click a summary arrow to see the outline item for the new constraint
2. Set the constraint's Lorentz product by entering a value in the text field at the right end of the outline item
   * *The display should update as soon as you press* Enter *or focus away from the text field*

The checkbox at the left end of a constraint outline item controls whether the constraint is active. Activating a constraint triggers a solution update. (Deactivating a constraint doesn't, since the remaining active constraints are still satisfied.)

### Precision

The Julia prototype of the engine uses a generic scalar type, so you can pass in any type the linear algebra functions are implemented for. The examples use the [adjustable-precision](https://docs.julialang.org/en/v1/base/numbers/#Base.MPFR.setprecision) `BigFloat` type.

In the Rust port of the engine, the scalar type is currently fixed at `f64`. Switching to generic scalars shouldn't be too hard, but I haven't looked into [which other types](https://www.nalgebra.org/docs/user_guide/generic_programming) the linear algebra functions are implemented for.

### Testing

To confirm quantitatively that the Rust port of the engine is working, you can go to the `app-proto` folder and:

* Run some automated tests by calling `cargo test`.
* Inspect the optimization process in a few examples calling the `run-examples` script. The first example that prints is the same as the Irisawa hexlet example from the engine prototype. If you go into `engine-proto/gram-test`, launch Julia, and then

  ```
  include("irisawa-hexlet.jl")
  for (step, scaled_loss) in enumerate(history_alt.scaled_loss)
    println(rpad(step-1, 4), " | ", scaled_loss)
  end
  ```

  you should see that it prints basically the same loss history until the last few steps, when the lower default precision of the Rust engine really starts to show.

### A small engine revision

The Rust port of the engine improves on the Julia prototype in one part of the constraint-solving routine: projecting the Hessian onto the subspace where the frozen entries stay constant. The Julia prototype does this by removing the rows and columns of the Hessian that correspond to the frozen entries, finding the Newton step from the resulting "compressed" Hessian, and then adding zero entries to the Newton step in the appropriate places. The Rust port instead replaces each frozen row and column with its corresponding standard unit vector, avoiding the finicky compressing and decompressing steps.

To confirm that this version of the constraint-solving routine works the same as the original, I implemented it in Julia as `realize_gram_alt_proj`. The solutions we get from this routine match the ones we get from the original `realize_gram` to very high precision, and in the simplest examples (`sphere-in-tetrahedron.jl` and `tetrahedron-radius-ratio.jl`), the descent paths also match to very high precision. In a more complicated example (`irisawa-hexlet.jl`), the descent paths diverge about a quarter of the way into the search, even though they end up in the same place.

Co-authored-by: Aaron Fenyes <aaron.fenyes@fareycircles.ooo>
Reviewed-on: glen/dyna3#15
Co-authored-by: Vectornaut <vectornaut@nobody@nowhere.net>
Co-committed-by: Vectornaut <vectornaut@nobody@nowhere.net>

engine-proto/gram-test/Engine.jl

@@ -8,7 +8,8 @@ using Optim
 
 export
   rand_on_shell, Q, DescentHistory,
-  realize_gram_gradient, realize_gram_newton, realize_gram_optim, realize_gram
+  realize_gram_gradient, realize_gram_newton, realize_gram_optim,
+  realize_gram_alt_proj, realize_gram
 
 # === guessing ===
 
@@ -143,7 +144,7 @@ function realize_gram_gradient(
       break
     end
     
-    # find negative gradient of loss function
+    # find the negative gradient of the loss function
     neg_grad = 4*Q*L*Δ_proj
     slope = norm(neg_grad)
     dir = neg_grad / slope
@@ -232,7 +233,7 @@ function realize_gram_newton(
       break
     end
     
-    # find the negative gradient of loss function
+    # find the negative gradient of the loss function
     neg_grad = 4*Q*L*Δ_proj
     
     # find the negative Hessian of the loss function
@@ -313,6 +314,129 @@ function realize_gram_optim(
   )
 end
 
+# seek a matrix `L` for which `L'QL` matches the sparse matrix `gram` at every
+# explicit entry of `gram`. use gradient descent starting from `guess`, with an
+# alternate technique for finding the projected base step from the unprojected
+# Hessian
+function realize_gram_alt_proj(
+  gram::SparseMatrixCSC{T, <:Any},
+  guess::Matrix{T},
+  frozen = CartesianIndex[];
+  scaled_tol = 1e-30,
+  min_efficiency = 0.5,
+  backoff = 0.9,
+  reg_scale = 1.1,
+  max_descent_steps = 200,
+  max_backoff_steps = 110
+) where T <: Number
+  # start history
+  history = DescentHistory{T}()
+  
+  # find the dimension of the search space
+  dims = size(guess)
+  element_dim, construction_dim = dims
+  total_dim = element_dim * construction_dim
+  
+  # list the constrained entries of the gram matrix
+  J, K, _ = findnz(gram)
+  constrained = zip(J, K)
+  
+  # scale the tolerance
+  scale_adjustment = sqrt(T(length(constrained)))
+  tol = scale_adjustment * scaled_tol
+  
+  # convert the frozen indices to stacked format
+  frozen_stacked = [(index[2]-1)*element_dim + index[1] for index in frozen]
+  
+  # initialize search state
+  L = copy(guess)
+  Δ_proj = proj_diff(gram, L'*Q*L)
+  loss = dot(Δ_proj, Δ_proj)
+  
+  # use Newton's method with backtracking and gradient descent backup
+  for step in 1:max_descent_steps
+    # stop if the loss is tolerably low
+    if loss < tol
+      break
+    end
+    
+    # find the negative gradient of the loss function
+    neg_grad = 4*Q*L*Δ_proj
+    
+    # find the negative Hessian of the loss function
+    hess = Matrix{T}(undef, total_dim, total_dim)
+    indices = [(j, k) for k in 1:construction_dim for j in 1:element_dim]
+    for (j, k) in indices
+      basis_mat = basis_matrix(T, j, k, dims)
+      neg_dΔ = basis_mat'*Q*L + L'*Q*basis_mat
+      neg_dΔ_proj = proj_to_entries(neg_dΔ, constrained)
+      deriv_grad = 4*Q*(-basis_mat*Δ_proj + L*neg_dΔ_proj)
+      hess[:, (k-1)*element_dim + j] = reshape(deriv_grad, total_dim)
+    end
+    hess_sym = Hermitian(hess)
+    push!(history.hess, hess_sym)
+    
+    # regularize the Hessian
+    min_eigval = minimum(eigvals(hess_sym))
+    push!(history.positive, min_eigval > 0)
+    if min_eigval <= 0
+      hess -= reg_scale * min_eigval * I
+    end
+    
+    # compute the Newton step
+    neg_grad_stacked = reshape(neg_grad, total_dim)
+    for k in frozen_stacked
+      neg_grad_stacked[k] = 0
+      hess[k, :] .= 0
+      hess[:, k] .= 0
+      hess[k, k] = 1
+    end
+    base_step_stacked = Hermitian(hess) \ neg_grad_stacked
+    base_step = reshape(base_step_stacked, dims)
+    push!(history.base_step, base_step)
+    
+    # store the current position, loss, and slope
+    L_last = L
+    loss_last = loss
+    push!(history.scaled_loss, loss / scale_adjustment)
+    push!(history.neg_grad, neg_grad)
+    push!(history.slope, norm(neg_grad))
+    
+    # find a good step size using backtracking line search
+    push!(history.stepsize, 0)
+    push!(history.backoff_steps, max_backoff_steps)
+    empty!(history.last_line_L)
+    empty!(history.last_line_loss)
+    rate = one(T)
+    step_success = false
+    base_target_improvement = dot(neg_grad, base_step)
+    for backoff_steps in 0:max_backoff_steps
+      history.stepsize[end] = rate
+      L = L_last + rate * base_step
+      Δ_proj = proj_diff(gram, L'*Q*L)
+      loss = dot(Δ_proj, Δ_proj)
+      improvement = loss_last - loss
+      push!(history.last_line_L, L)
+      push!(history.last_line_loss, loss / scale_adjustment)
+      if improvement >= min_efficiency * rate * base_target_improvement
+        history.backoff_steps[end] = backoff_steps
+        step_success = true
+        break
+      end
+      rate *= backoff
+    end
+    
+    # if we've hit a wall, quit
+    if !step_success
+      return L_last, false, history
+    end
+  end
+  
+  # return the factorization and its history
+  push!(history.scaled_loss, loss / scale_adjustment)
+  L, loss < tol, history
+end
+
 # seek a matrix `L` for which `L'QL` matches the sparse matrix `gram` at every
 # explicit entry of `gram`. use gradient descent starting from `guess`
 function realize_gram(
@@ -321,7 +445,6 @@ function realize_gram(
   frozen = nothing;
   scaled_tol = 1e-30,
   min_efficiency = 0.5,
-  init_rate = 1.0,
   backoff = 0.9,
   reg_scale = 1.1,
   max_descent_steps = 200,
@@ -352,20 +475,19 @@ function realize_gram(
     unfrozen_stacked = reshape(is_unfrozen, total_dim)
   end
   
-  # initialize variables
-  grad_rate = init_rate
+  # initialize search state
   L = copy(guess)
-  
-  # use Newton's method with backtracking and gradient descent backup
   Δ_proj = proj_diff(gram, L'*Q*L)
   loss = dot(Δ_proj, Δ_proj)
+  
+  # use Newton's method with backtracking and gradient descent backup
   for step in 1:max_descent_steps
     # stop if the loss is tolerably low
     if loss < tol
       break
     end
     
-    # find the negative gradient of loss function
+    # find the negative gradient of the loss function
     neg_grad = 4*Q*L*Δ_proj
     
     # find the negative Hessian of the loss function
@@ -420,6 +542,7 @@ function realize_gram(
     empty!(history.last_line_loss)
     rate = one(T)
     step_success = false
+    base_target_improvement = dot(neg_grad, base_step)
     for backoff_steps in 0:max_backoff_steps
       history.stepsize[end] = rate
       L = L_last + rate * base_step
@@ -428,7 +551,7 @@ function realize_gram(
       improvement = loss_last - loss
       push!(history.last_line_L, L)
       push!(history.last_line_loss, loss / scale_adjustment)
-      if improvement >= min_efficiency * rate * dot(neg_grad, base_step)
+      if improvement >= min_efficiency * rate * base_target_improvement
         history.backoff_steps[end] = backoff_steps
         step_success = true
         break

engine-proto/gram-test/irisawa-hexlet.jl

@@ -74,4 +74,13 @@ if success
   for k in 5:9
     println("  ", 1 / L[4,k], " sun")
   end
-end
+end
+
+# test an alternate technique for finding the projected base step from the
+# unprojected Hessian
+L_alt, success_alt, history_alt = Engine.realize_gram_alt_proj(gram, guess, frozen)
+completed_gram_alt = L_alt'*Engine.Q*L_alt
+println("\nDifference in result using alternate projection:\n")
+display(completed_gram_alt - completed_gram)
+println("\nDifference in steps: ", size(history_alt.scaled_loss, 1) - size(history.scaled_loss, 1))
+println("Difference in loss: ", history_alt.scaled_loss[end] - history.scaled_loss[end], "\n")

engine-proto/gram-test/sphere-in-tetrahedron.jl

@@ -64,4 +64,13 @@ else
   println("\nFailed to reach target accuracy")
 end
 println("Steps: ", size(history.scaled_loss, 1))
-println("Loss: ", history.scaled_loss[end], "\n")
+println("Loss: ", history.scaled_loss[end], "\n")
+
+# test an alternate technique for finding the projected base step from the
+# unprojected Hessian
+L_alt, success_alt, history_alt = Engine.realize_gram_alt_proj(gram, guess, frozen)
+completed_gram_alt = L_alt'*Engine.Q*L_alt
+println("\nDifference in result using alternate projection:\n")
+display(completed_gram_alt - completed_gram)
+println("\nDifference in steps: ", size(history_alt.scaled_loss, 1) - size(history.scaled_loss, 1))
+println("Difference in loss: ", history_alt.scaled_loss[end] - history.scaled_loss[end], "\n")

engine-proto/gram-test/tetrahedron-radius-ratio.jl

@@ -93,4 +93,13 @@ if success
   infty = BigFloat[0, 0, 0, 0, 1]
   radius_ratio = dot(infty, Engine.Q * L[:,5]) / dot(infty, Engine.Q * L[:,6])
   println("\nCircumradius / inradius: ", radius_ratio)
-end
+end
+
+# test an alternate technique for finding the projected base step from the
+# unprojected Hessian
+L_alt, success_alt, history_alt = Engine.realize_gram_alt_proj(gram, guess, frozen)
+completed_gram_alt = L_alt'*Engine.Q*L_alt
+println("\nDifference in result using alternate projection:\n")
+display(completed_gram_alt - completed_gram)
+println("\nDifference in steps: ", size(history_alt.scaled_loss, 1) - size(history.scaled_loss, 1))
+println("Difference in loss: ", history_alt.scaled_loss[end] - history.scaled_loss[end], "\n")