WIP: update to new JuMP nonlinear syntax #858

odow · 2023-04-21T05:44:49Z

All instances are a straight swap, dropping the NL part of macro names.
Duplicated code blocks that were added to work-around NL macros have been removed
Tests have been updated to new solution values

Questions:

~~Should the test values have changed? Macro parsing might give slightly different numerical values. Would that make a difference?~~
~~I'll comment in-line, but are the higher-order polynomial objectives correct?~~

src/core/objective.jl

src/form/acp.jl

test/docs.jl

test/opf-var.jl

test/opf.jl

odow · 2023-04-21T05:51:34Z

I also assume Juniper worked without issue, since no tests failed. Yay for a good abstraction?

codecov · 2023-04-21T06:01:18Z

Codecov Report

Merging #858 (c6c7107) into master (a8d1aad) will increase coverage by 0.02%.
The diff coverage is 96.66%.

@@            Coverage Diff             @@
##           master     #858      +/-   ##
==========================================
+ Coverage   93.87%   93.89%   +0.02%     
==========================================
  Files          43       43              
  Lines        9973     9914      -59     
==========================================
- Hits         9362     9309      -53     
+ Misses        611      605       -6

Files Changed	Coverage Δ
src/form/iv.jl	`92.57% <88.23%> (ø)`
src/core/objective.jl	`91.44% <100.00%> (+0.59%)`	⬆️
src/form/acp.jl	`93.23% <100.00%> (-0.20%)`	⬇️
src/form/acr.jl	`98.47% <100.00%> (-0.04%)`	⬇️
src/form/act.jl	`95.12% <100.00%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8d1aad...c6c7107. Read the comment docs.

src/core/objective.jl

test/opf.jl

odow · 2023-06-29T02:09:51Z

@ccoffrin there are a couple of failure here. I can reproduce the Mac 1.6 failure with

using PowerModels, Ipopt
optimizer = Ipopt.Optimizer
file = "../test/data/matpower/case5_asym.m"
pm = instantiate_model(file, ACPPowerModel, build_pf);
result = optimize_model!(pm; optimizer=optimizer)
set_start_value.(all_variables(pm.model), rand())
result = optimize_model!(pm; optimizer=optimizer)

Log is

julia> using PowerModels, Ipopt

julia> optimizer = Ipopt.Optimizer
Ipopt.Optimizer

julia> file = "../test/data/matpower/case5_asym.m"
"../test/data/matpower/case5_asym.m"

julia> pm = instantiate_model(file, ACPPowerModel, build_pf);
[info | PowerModels]: removing 1 cost terms from generator 4: [4000.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 1: [1400.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 5: [1000.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 2: [1500.0, 0.0]
[info | PowerModels]: removing 1 cost terms from generator 3: [3000.0, 0.0]

julia> result = optimize_model!(pm; optimizer=optimizer)
This is Ipopt version 3.14.4, running with linear solver MUMPS 5.4.1.

Number of nonzeros in equality constraint Jacobian...:       87
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:      198

Total number of variables............................:       20
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        0
                     variables with only upper bounds:        0
Total number of equality constraints.................:       19
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  0.0000000e+00 4.71e+00 0.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1  0.0000000e+00 7.00e-01 3.41e-03  -1.0 4.71e+00  -4.0 1.00e+00 1.00e+00h  1
   2  0.0000000e+00 6.13e-01 5.66e-03  -1.7 6.10e+16    -  1.00e+00 1.25e-01h  4
   3  0.0000000e+00 4.25e-01 1.63e-04  -1.7 1.96e+02    -  1.00e+00 1.00e+00h  1
   4r 0.0000000e+00 4.25e-01 9.99e+02  -0.7 0.00e+00    -  0.00e+00 4.77e-07R 22
   5r 0.0000000e+00 4.25e-01 5.48e+00  -0.7 2.12e+02  -4.0 1.00e+00 9.91e-04f  1
   6r 0.0000000e+00 4.25e-01 5.22e-04  -1.4 2.13e-01  -4.5 1.00e+00 1.00e+00f  1
   7r 0.0000000e+00 4.24e-01 2.28e-04  -2.1 2.12e-01  -5.0 1.00e+00 1.00e+00f  1
   8r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.80e-01    -  1.00e+00 1.00e+00s 22
   9r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.12e-01  -5.4 0.00e+00 0.00e+00R  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
  10r 0.0000000e+00 4.24e-01 5.27e-07  -2.1 2.12e-01  -5.9 1.00e+00 9.93e-01f  1
  11r 0.0000000e+00 4.24e-01 1.65e-07  -2.1 2.12e-01  -6.4 1.00e+00 9.98e-01f  1
  12r 0.0000000e+00 4.24e-01 6.89e-06  -2.1 2.12e-01  -6.9 1.00e+00 5.00e-01h  2
  13r 0.0000000e+00 4.24e-01 1.07e-05  -2.1 2.12e-01  -7.3 1.00e+00 6.25e-02h  5

... lines omitted ...

2997r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.51e-01    -  1.00e+00 5.00e-01h  2
2998r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.15e-01 -11.8 1.00e+00 5.00e-01h  2
2999r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.14e-01 -12.3 1.00e+00 1.00e+00h  1
iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
3000r 0.0000000e+00 4.24e-01 7.47e-08  -2.1 2.15e-01 -12.7 0.00e+00 4.77e-07R 22

Number of Iterations....: 3000

                                   (scaled)                 (unscaled)
Objective...............:   0.0000000000000000e+00    0.0000000000000000e+00
Dual infeasibility......:   1.8031981521322976e-04    1.8031981521322976e-04
Constraint violation....:   1.9079148135380333e-01    4.2443834851533246e-01
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   0.0000000000000000e+00    0.0000000000000000e+00
Overall NLP error.......:   1.9079148135380333e-01    4.2443834851533246e-01


Number of objective function evaluations             = 17874
Number of objective gradient evaluations             = 6
Number of equality constraint evaluations            = 18231
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 3359
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 3000
Total seconds in IPOPT                               = 4.503

EXIT: Maximum Number of Iterations Exceeded.
Dict{String, Any} with 8 entries:
  "solve_time"         => 4.50402
  "optimizer"          => "Ipopt"
  "termination_status" => ITERATION_LIMIT
  "dual_status"        => UNKNOWN_RESULT_STATUS
  "primal_status"      => UNKNOWN_RESULT_STATUS
  "objective"          => 0.0
  "solution"           => Dict{String, Any}("baseMVA"=>100.0, "gen"=>Dict{String, Any}("4"=>Dict{Str…
  "objective_lb"       => -Inf

julia> set_start_value.(all_variables(pm.model), rand());

julia> result = optimize_model!(pm; optimizer=optimizer)
[warn | InfrastructureModels]: Model already contains optimizer, cannot use optimizer specified in `optimize_model!`
This is Ipopt version 3.14.4, running with linear solver MUMPS 5.4.1.

Number of nonzeros in equality constraint Jacobian...:       87
Number of nonzeros in inequality constraint Jacobian.:        0
Number of nonzeros in Lagrangian Hessian.............:      198

Total number of variables............................:       20
                     variables with only lower bounds:        0
                variables with lower and upper bounds:        0
                     variables with only upper bounds:        0
Total number of equality constraints.................:       19
Total number of inequality constraints...............:        0
        inequality constraints with only lower bounds:        0
   inequality constraints with lower and upper bounds:        0
        inequality constraints with only upper bounds:        0

iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
   0  0.0000000e+00 4.49e+00 0.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
   1  0.0000000e+00 3.37e+00 1.26e-03  -1.0 4.49e+00  -4.0 1.00e+00 2.50e-01h  3
   2  0.0000000e+00 2.53e+00 1.10e-03  -1.0 3.37e+00  -4.5 1.00e+00 2.50e-01h  3
   3  0.0000000e+00 1.05e+00 1.53e-03  -1.0 2.53e+00  -5.0 1.00e+00 1.00e+00h  1
   4  0.0000000e+00 5.50e-03 3.19e-05  -1.0 5.20e-01  -5.4 1.00e+00 1.00e+00h  1
   5  0.0000000e+00 2.16e-07 1.10e-08  -3.8 8.15e-03  -5.9 1.00e+00 1.00e+00h  1
   6  0.0000000e+00 2.35e-14 8.37e-14  -8.6 2.03e-07  -6.4 1.00e+00 1.00e+00h  1

Number of Iterations....: 6

                                   (scaled)                 (unscaled)
Objective...............:   0.0000000000000000e+00    0.0000000000000000e+00
Dual infeasibility......:   8.3676919503834109e-14    8.3676919503834109e-14
Constraint violation....:   2.3536728122053319e-14    2.3536728122053319e-14
Variable bound violation:   0.0000000000000000e+00    0.0000000000000000e+00
Complementarity.........:   0.0000000000000000e+00    0.0000000000000000e+00
Overall NLP error.......:   8.3676919503834109e-14    8.3676919503834109e-14


Number of objective function evaluations             = 14
Number of objective gradient evaluations             = 7
Number of equality constraint evaluations            = 14
Number of inequality constraint evaluations          = 0
Number of equality constraint Jacobian evaluations   = 7
Number of inequality constraint Jacobian evaluations = 0
Number of Lagrangian Hessian evaluations             = 6
Total seconds in IPOPT                               = 0.006

EXIT: Optimal Solution Found.
Dict{String, Any} with 8 entries:
  "solve_time"         => 0.00704193
  "optimizer"          => "Ipopt"
  "termination_status" => LOCALLY_SOLVED
  "dual_status"        => FEASIBLE_POINT
  "primal_status"      => FEASIBLE_POINT
  "objective"          => 0.0
  "solution"           => Dict{String, Any}("baseMVA"=>100.0, "gen"=>Dict{String, Any}("4"=>Dict{Str…
  "objective_lb"       => -Inf

So it looks like Ipopt just gets caught in a local optima that is infeasible and it can't figure out a search direction to escape. But starting from a random point works well.

Is the asymmetric case known to be problematic?

test/runtests.jl

test/docs.jl

test/model.jl

test/opf-obj.jl

odow · 2023-09-05T01:19:00Z

Failure is jump-dev/JuMP.jl#3487

* All instances are a straight swap, dropping the NL part of macro names. * Duplicated code blocks that were added to work-around NL macros have been removed * Tests have been updated to new solution values Questions: * Should the test values have changed? Macro parsing might give slightly different numerical values. Would that make a difference? * I'll comment in-line, but are the higher-order polynomial objectives correct?

test/runtests.jl

odow · 2023-09-05T03:48:46Z

test/opf-var.jl

-            @test result["termination_status"] == OPTIMAL || result["termination_status"] == ALMOST_OPTIMAL
-            @test isapprox(result["objective"], 7505.33; atol = 1e0)
-            #@test isapprox(result["objective"], 7637.95; atol = 1e0)
+            @test result["termination_status"] in (OPTIMAL, ALMOST_OPTIMAL, ITERATION_LIMIT)


This example hits an iteration limit, but it still finds the objective. Guess just a tolerance issue.

odow · 2023-09-15T03:40:59Z

Will update once @ccoffrin makes some larger unrelated changes.

odow · 2024-01-02T03:23:01Z

@ccoffrin should I update this PR, or do you want to release v0.20 first?

ccoffrin · 2024-01-02T03:47:01Z

I'll release v0.20 first, then planning v0.21 to be the update to the new NL interface.

ccoffrin · 2024-01-02T17:37:16Z

For your situational awareness, I am thinking v0.21 will be a minimalist update that moves PowerModels over to the new NL interface with smallest number of changes and minor code simplifications within single functions (basically what this PR is). Then in v0.22 I will explore redesigning broader architectural choices in PowerModels to really take advantage of the fact that NL expressions are first class.

ccoffrin · 2024-01-04T00:56:57Z

closing in favor of #901