Make the trust region subsolver exchangeable.

Changelog.md

@@ -5,6 +5,13 @@ All notable Changes to the Julia package `Manopt.jl` will be documented in this
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.41] - dd/mm/yyyy
+
+### Changed
+
+– `trust_regions` is now more flexible and the sub solver (Steinhaug-Toint tCG by default)
+  can now be exchanged.
+
 ## [0.4.40] – 24/10/2023
 
 ### Added

Project.toml

@@ -38,13 +38,16 @@ ColorSchemes = "3.5.0"
 ColorTypes = "0.9.1, 0.10, 0.11"
 Colors = "0.11.2, 0.12"
 DataStructures = "0.17, 0.18"
+LinearAlgebra = "1.6"
 LRUCache = "1.4"
-ManifoldDiff = "0.2, 0.3.3"
+ManifoldDiff = "0.3.8"
 Manifolds = "0.9"
 ManifoldsBase = "0.15"
+Markdown = "1.6"
 PolynomialRoots = "1"
+Random = "1.6"
 Requires = "0.5, 1"
-Statistics = "1"
+Statistics = "1.6"
 julia = "1.6"
 
 [extras]

Readme.md

@@ -74,7 +74,24 @@ To refer to a certain version or the source code in general we recommend to cite
 ```
 
 for the most recent version or a corresponding version specific DOI, see [the list of all versions](https://zenodo.org/search?page=1&size=20&q=conceptrecid:%224290905%22&sort=-version&all_versions=True).
-Note that both citations are in [BibLaTeX](https://ctan.org/pkg/biblatex) format.
+
+If you are also using [`Manifolds.jl`](https://juliamanifolds.github.io/Manifolds.jl/stable/) please consider to cite
+
+```
+@article{AxenBaranBergmannRzecki:2023,
+    AUTHOR     = {Seth D. Axen and Mateusz Baran and Ronny Bergmann and Krzysztof Rzecki},
+    DOI        = {10.1145/3618296},
+    EPRINT     = {2021.08777},
+    EPRINTTYPE = {arXiv},
+    JOURNAL    = {AMS Transactions on Mathematical Software},
+    NOTE       = {accepted for publication},
+    TITLE      = {Manifolds.jl: An Extensible {J}ulia Framework for Data Analysis on Manifolds},
+    YEAR       = {2023}
+}
+```
+
+as well.
+Note that all citations are in [BibLaTeX](https://ctan.org/pkg/biblatex) format.
 
 ## Further and Similar Packages & Links
 

docs/make.jl

@@ -108,7 +108,7 @@ tutorials_menu =
 bib = CitationBibliography(joinpath(@__DIR__, "src", "references.bib"); style=:alpha)
 makedocs(;
     format=Documenter.HTML(;
-        mathengine=MathJax3(), prettyurls=get(ENV, "CI", nothing) == "true"
+        prettyurls=false, assets=["assets/favicon.ico", "assets/citations.css"]
     ),
     modules=[
         Manopt,

docs/src/assets/citations.css

@@ -0,0 +1,19 @@
+/* Taken from https://juliadocs.org/DocumenterCitations.jl/v1.2/styling/ */
+
+.citation dl {
+  display: grid;
+  grid-template-columns: max-content auto; }
+.citation dt {
+  grid-column-start: 1; }
+.citation dd {
+  grid-column-start: 2;
+  margin-bottom: 0.75em; }
+.citation ul {
+ padding: 0 0 2.25em 0;
+ margin: 0;
+ list-style: none;}
+.citation ul li {
+ text-indent: -2.25em;
+ margin: 0.33em 0.5em 0.5em 2.25em;}
+.citation ol li {
+ padding-left:0.75em;}

docs/src/functions/gradients.md

@@ -1,13 +1,12 @@
 # [Gradients](@id GradientFunctions)
 
-For a function $f:\mathcal M→ℝ$
-the Riemannian gradient $\operatorname{grad}f(x)$ at $x∈\mathcal M$
+For a function ``f:\mathcal M→ℝ``
+the Riemannian gradient ``\operatorname{grad}f(x)`` at ``x∈\mathcal M``
 is given by the unique tangent vector fulfilling
-
-$\langle \operatorname{grad}f(x), ξ\rangle_x = D_xf[ξ],\quad
-\forall ξ ∈ T_x\mathcal M,$
-where $D_xf[ξ]$ denotes the differential of $f$ at $x$ with respect to
-the tangent direction (vector) $ξ$ or in other words the directional
+``\langle \operatorname{grad}f(x), ξ\rangle_x = D_xf[ξ],\quad
+\forall ξ ∈ T_x\mathcal M,``
+where ``D_xf[ξ]`` denotes the differential of ``f`` at ``x`` with respect to
+the tangent direction (vector) ``ξ`` or in other words the directional
 derivative.
 
 This page collects the available gradients.

docs/src/plans/index.md

@@ -17,5 +17,6 @@ Still there might be the need to set certain parameters within any of these stru
 
 ```@docs
 set_manopt_parameter!
+get_manopt_parameter
 Manopt.status_summary
 ```

docs/src/plans/objective.md

@@ -160,10 +160,10 @@ ManifoldProximalMapObjective
 get_proximal_map
 ```
 
-
 ### Hessian Objective
 
 ```@docs
+AbstractManifoldHessianObjective
 ManifoldHessianObjective
 ```
 
@@ -234,3 +234,21 @@ get_grad_inequality_constraint!
 get_grad_inequality_constraints
 get_grad_inequality_constraints!
 ```
+
+### Subproblem Objective
+
+This objective can be use when the objective of a sub problem
+solver still needs access to the (outer/main) objective.
+
+```@docs
+AbstractManifoldSubObjective
+```
+
+#### Access functions
+
+```@docs
+Manopt.get_objective_cost
+Manopt.get_objective_gradient
+Manopt.get_objective_hessian
+Manopt.get_objective_preconditioner
+```

docs/src/solvers/ChambollePock.md

@@ -3,31 +3,34 @@
 The Riemannian Chambolle–Pock is a generalization of the Chambolle–Pock algorithm [ChambollePock:2011](@citet*)
 It is also known as primal-dual hybrid gradient (PDHG) or primal-dual proximal splitting (PDPS) algorithm.
 
-In order to minimize over $p∈\mathcal M$ the cost function consisting of
+In order to minimize over ``p∈\mathcal M`` the cost function consisting of
+In order to minimize a cost function consisting of
 
 ```math
 F(p) + G(Λ(p)),
 ```
 
-where $F:\mathcal M → \overline{ℝ}$, $G:\mathcal N → \overline{ℝ}$, and
-$Λ:\mathcal M →\mathcal N$.
-If the manifolds $\mathcal M$ or $\mathcal N$ are not Hadamard, it has to be considered locally,
-i.e. on geodesically convex sets $\mathcal C \subset \mathcal M$ and $\mathcal D \subset\mathcal N$
-such that $Λ(\mathcal C) \subset \mathcal D$.
+ over ``p∈\mathcal M``
+
+where ``F:\mathcal M → \overline{ℝ}``, ``G:\mathcal N → \overline{ℝ}``, and
+``Λ:\mathcal M →\mathcal N``.
+If the manifolds ``\mathcal M`` or ``\mathcal N`` are not Hadamard, it has to be considered locally,
+i.e. on geodesically convex sets ``\mathcal C \subset \mathcal M`` and ``\mathcal D \subset\mathcal N``
+such that ``Λ(\mathcal C) \subset \mathcal D``.
 
 The algorithm is available in four variants: exact versus linearized (see `variant`)
 as well as with primal versus dual relaxation (see `relax`). For more details, see
 [BergmannHerzogSilvaLouzeiroTenbrinckVidalNunez:2021](@citet*).
 In the following we note the case of the exact, primal relaxed Riemannian Chambolle–Pock algorithm.
 
-Given base points $m∈\mathcal C$, $n=Λ(m)∈\mathcal D$,
-initial primal and dual values $p^{(0)} ∈\mathcal C$, $ξ_n^{(0)} ∈T_n^*\mathcal N$,
-and primal and dual step sizes $\sigma_0$, $\tau_0$, relaxation $\theta_0$,
-as well as acceleration $\gamma$.
+Given base points ``m∈\mathcal C``, ``n=Λ(m)∈\mathcal D``,
+initial primal and dual values ``p^{(0)} ∈\mathcal C``, ``ξ_n^{(0)} ∈T_n^*\mathcal N``,
+and primal and dual step sizes ``\sigma_0``, ``\tau_0``, relaxation ``\theta_0``,
+as well as acceleration ``\gamma``.
 
-As an initialization, perform $\bar p^{(0)} \gets p^{(0)}$.
+As an initialization, perform ``\bar p^{(0)} \gets p^{(0)}``.
 
-The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref) is fulfilled with)
+The algorithms performs the steps ``k=1,…,`` (until a [`StoppingCriterion`](@ref) is fulfilled with)
 
 1. ```math
    ξ^{(k+1)}_n = \operatorname{prox}_{\tau_k G_n^*}\Bigl(ξ_n^{(k)} + \tau_k \bigl(\log_n Λ (\bar p^{(k)})\bigr)^\flat\Bigr)
@@ -46,9 +49,9 @@ The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref
 Furthermore you can exchange the exponential map, the logarithmic map, and the parallel transport
 by a retraction, an inverse retraction, and a vector transport.
 
-Finally you can also update the base points $m$ and $n$ during the iterations.
+Finally you can also update the base points ``m`` and ``n`` during the iterations.
 This introduces a few additional vector transports. The same holds for the case
-$Λ(m^{(k)})\neq n^{(k)}$ at some point. All these cases are covered in the algorithm.
+``Λ(m^{(k)})\neq n^{(k)}`` at some point. All these cases are covered in the algorithm.
 
 ```@meta
 CurrentModule = Manopt

docs/src/solvers/adaptive-regularization-with-cubics.md

@@ -29,19 +29,17 @@ Manopt.LanczosState
 
 ## (Conjugate) Gradient Descent
 
-There are two generic functors, that implement the sub problem
+There is a generic objective, that implements the sub problem
 
 ```@docs
-AdaptiveRegularizationCubicCost
-AdaptiveRegularizationCubicGrad
+AdaptiveRagularizationWithCubicsModelObjective
 ```
 
 Since the sub problem is given on the tangent space, you have to provide
 
 ```
-g = AdaptiveRegularizationCubicCost(M, mho, σ)
-grad_g = AdaptiveRegularizationCubicGrad(M, mho, σ)
-sub_problem = DefaultProblem(TangentSpaceAt(M,p), ManifoldGradienObjective(g, grad_g))
+arc_obj = AdaptiveRagularizationWithCubicsModelObjective(mho, σ)
+sub_problem = DefaultProblem(TangentSpaceAt(M,p), arc_obj)
 ```
 
 where `mho` is the hessian objective of `f` to solve.

docs/src/solvers/cyclic_proximal_point.md

@@ -6,11 +6,11 @@ The Cyclic Proximal Point (CPP) algorithm aims to minimize
 F(x) = \sum_{i=1}^c f_i(x)
 ```
 
-assuming that the [proximal maps](@ref proximalMapFunctions) $\operatorname{prox}_{λ f_i}(x)$
+assuming that the [proximal maps](@ref proximalMapFunctions) ``\operatorname{prox}_{λ f_i}(x)``
 are given in closed form or can be computed efficiently (at least approximately).
 
 The algorithm then cycles through these proximal maps, where the type of cycle
-might differ and the proximal parameter $λ_k$ changes after each cycle $k$.
+might differ and the proximal parameter ``λ_k`` changes after each cycle ``k``.
 
 For a convergence result on
 [Hadamard manifolds](https://en.wikipedia.org/wiki/Hadamard_manifold)

docs/src/solvers/primal_dual_semismooth_Newton.md

@@ -8,13 +8,13 @@ The aim is to solve an optimization problem on a manifold with a cost function o
 F(p) + G(Λ(p)),
 ```
 
-where $F:\mathcal M → \overline{ℝ}$, $G:\mathcal N → \overline{ℝ}$, and
-$Λ:\mathcal M →\mathcal N$.
-If the manifolds $\mathcal M$ or $\mathcal N$ are not Hadamard, it has to be considered locally,
-i.e. on geodesically convex sets $\mathcal C \subset \mathcal M$ and $\mathcal D \subset\mathcal N$
-such that $Λ(\mathcal C) \subset \mathcal D$.
+where ``F:\mathcal M → \overline{ℝ}``, ``G:\mathcal N → \overline{ℝ}``, and
+``Λ:\mathcal M →\mathcal N``.
+If the manifolds ``\mathcal M`` or ``\mathcal N`` are not Hadamard, it has to be considered locally,
+i.e. on geodesically convex sets ``\mathcal C \subset \mathcal M`` and ``\mathcal D \subset\mathcal N``
+such that ``Λ(\mathcal C) \subset \mathcal D``.
 
-The algorithm comes down to applying the Riemannian semismooth Newton method to the rewritten primal-dual optimality conditions, i.e., we define the vector field $X: \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N} \rightarrow \mathcal{T} \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N}$ as
+The algorithm comes down to applying the Riemannian semismooth Newton method to the rewritten primal-dual optimality conditions, i.e., we define the vector field ``X: \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N} \rightarrow \mathcal{T} \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N}`` as
 
 ```math
 X\left(p, \xi_{n}\right):=\left(\begin{array}{c}
@@ -23,13 +23,13 @@ X\left(p, \xi_{n}\right):=\left(\begin{array}{c}
 \end{array}\right)
 ```
 
-and solve for $X(p,ξ_{n})=0$.
+and solve for ``X(p,ξ_{n})=0``.
 
-Given base points $m∈\mathcal C$, $n=Λ(m)∈\mathcal D$,
-initial primal and dual values $p^{(0)} ∈\mathcal C$, $ξ_{n}^{(0)} ∈ \mathcal T_{n}^{*}\mathcal N$,
-and primal and dual step sizes $\sigma$, $\tau$.
+Given base points ``m∈\mathcal C``, ``n=Λ(m)∈\mathcal D``,
+initial primal and dual values ``p^{(0)} ∈\mathcal C``, ``ξ_{n}^{(0)} ∈ \mathcal T_{n}^{*}\mathcal N``,
+and primal and dual step sizes ``\sigma``, ``\tau``.
 
-The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref) is reached)
+The algorithms performs the steps ``k=1,…,`` (until a [`StoppingCriterion`](@ref) is reached)
 
 1.  Choose any element
    ```math
@@ -40,7 +40,7 @@ The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref
    ```math
    V^{(k)} [(d_p^{(k)}, d_n^{(k)})] = - X(p^{(k)},ξ_n^{(k)})
    ```
-   in the vector space $\mathcal{T}_{p^{(k)}} \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N}$
+   in the vector space ``\mathcal{T}_{p^{(k)}} \mathcal{M} \times \mathcal{T}_{n}^{*} \mathcal{N}``
 3. Update
    ```math
    p^{(k+1)} := \exp_{p^{(k)}}(d_p^{(k)})
@@ -53,9 +53,9 @@ The algorithms performs the steps $k=1,…,$ (until a [`StoppingCriterion`](@ref
 Furthermore you can exchange the exponential map, the logarithmic map, and the parallel transport
 by a retraction, an inverse retraction and a vector transport.
 
-Finally you can also update the base points $m$ and $n$ during the iterations.
+Finally you can also update the base points ``m`` and ``n`` during the iterations.
 This introduces a few additional vector transports. The same holds for the case that
-$Λ(m^{(k)})\neq n^{(k)}$ at some point. All these cases are covered in the algorithm.
+``Λ(m^{(k)})\neq n^{(k)}`` at some point. All these cases are covered in the algorithm.
 
 ```@meta
 CurrentModule = Manopt