Solution file format

[jkiviluo]

This should probably be a discussion topic rather than an issue, but I'd like to know if there are other solution file formats available or planned?

For my part, I'd like to read the solution to MathProg in order to use the code that writes out results, but the solution file format in MathProg is different:

s bas 17169 14089 f f 1.06684e+11
i 1 b 1.06684e+11 0
i 2 s 947.477 1579.87
i 3 s 864.792 1579.87
i 4 s 814.937 1579.87
...

[jajhall]

We only have the raw (computer-readable) and pretty (human-readable) formats at present, but it's not too much trouble to write out in a different format.

I can guess what some of the above refers to, but where is this MathProg format defined?

[jkiviluo]

I have not found a reference documentation, but the GLPK / GNU Mathprog source code has a file that does the output. I'm attaching it here.

This would give a pathway to start using HiGHS with GLPK / Mathprog model files that typically use GLPSOL as the solver. I believe there are quite a lot of those models lying around.

/* prsol.c (write basic solution in printable format) */

/***********************************************************************
*  This code is part of GLPK (GNU Linear Programming Kit).
*  Copyright (C) 2009-2016 Free Software Foundation, Inc.
*  Written by Andrew Makhorin <[email protected]>.
*
*  GLPK is free software: you can redistribute it and/or modify it
*  under the terms of the GNU General Public License as published by
*  the Free Software Foundation, either version 3 of the License, or
*  (at your option) any later version.
*
*  GLPK is distributed in the hope that it will be useful, but WITHOUT
*  ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
*  or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public
*  License for more details.
*
*  You should have received a copy of the GNU General Public License
*  along with GLPK. If not, see <http://www.gnu.org/licenses/>.
***********************************************************************/

#include "env.h"
#include "prob.h"

#define xfprintf glp_format

int glp_print_sol(glp_prob *P, const char *fname)
{     /* write basic solution in printable format */
      glp_file *fp;
      GLPROW *row;
      GLPCOL *col;
      int i, j, t, ae_ind, re_ind, ret;
      double ae_max, re_max;
      xprintf("Writing basic solution to '%s'...\n", fname);
      fp = glp_open(fname, "w");
      if (fp == NULL)
      {  xprintf("Unable to create '%s' - %s\n", fname, get_err_msg());
         ret = 1;
         goto done;
      }
      xfprintf(fp, "%-12s%s\n", "Problem:",
         P->name == NULL ? "" : P->name);
      xfprintf(fp, "%-12s%d\n", "Rows:", P->m);
      xfprintf(fp, "%-12s%d\n", "Columns:", P->n);
      xfprintf(fp, "%-12s%d\n", "Non-zeros:", P->nnz);
      t = glp_get_status(P);
      xfprintf(fp, "%-12s%s\n", "Status:",
         t == GLP_OPT    ? "OPTIMAL" :
         t == GLP_FEAS   ? "FEASIBLE" :
         t == GLP_INFEAS ? "INFEASIBLE (INTERMEDIATE)" :
         t == GLP_NOFEAS ? "INFEASIBLE (FINAL)" :
         t == GLP_UNBND  ? "UNBOUNDED" :
         t == GLP_UNDEF  ? "UNDEFINED" : "???");
      xfprintf(fp, "%-12s%s%s%.10g (%s)\n", "Objective:",
         P->obj == NULL ? "" : P->obj,
         P->obj == NULL ? "" : " = ", P->obj_val,
         P->dir == GLP_MIN ? "MINimum" :
         P->dir == GLP_MAX ? "MAXimum" : "???");
      xfprintf(fp, "\n");
      xfprintf(fp, "   No.   Row name   St   Activity     Lower bound  "
         " Upper bound    Marginal\n");
      xfprintf(fp, "------ ------------ -- ------------- ------------- "
         "------------- -------------\n");
      for (i = 1; i <= P->m; i++)
      {  row = P->row[i];
         xfprintf(fp, "%6d ", i);
         if (row->name == NULL || strlen(row->name) <= 12)
            xfprintf(fp, "%-12s ", row->name == NULL ? "" : row->name);
         else
            xfprintf(fp, "%s\n%20s", row->name, "");
         xfprintf(fp, "%s ",
            row->stat == GLP_BS ? "B " :
            row->stat == GLP_NL ? "NL" :
            row->stat == GLP_NU ? "NU" :
            row->stat == GLP_NF ? "NF" :
            row->stat == GLP_NS ? "NS" : "??");
         xfprintf(fp, "%13.6g ",
            fabs(row->prim) <= 1e-9 ? 0.0 : row->prim);
         if (row->type == GLP_LO || row->type == GLP_DB ||
             row->type == GLP_FX)
            xfprintf(fp, "%13.6g ", row->lb);
         else
            xfprintf(fp, "%13s ", "");
         if (row->type == GLP_UP || row->type == GLP_DB)
            xfprintf(fp, "%13.6g ", row->ub);
         else
            xfprintf(fp, "%13s ", row->type == GLP_FX ? "=" : "");
         if (row->stat != GLP_BS)
         {  if (fabs(row->dual) <= 1e-9)
               xfprintf(fp, "%13s", "< eps");
            else
               xfprintf(fp, "%13.6g ", row->dual);
         }
         xfprintf(fp, "\n");
      }
      xfprintf(fp, "\n");
      xfprintf(fp, "   No. Column name  St   Activity     Lower bound  "
         " Upper bound    Marginal\n");
      xfprintf(fp, "------ ------------ -- ------------- ------------- "
         "------------- -------------\n");
      for (j = 1; j <= P->n; j++)
      {  col = P->col[j];
         xfprintf(fp, "%6d ", j);
         if (col->name == NULL || strlen(col->name) <= 12)
            xfprintf(fp, "%-12s ", col->name == NULL ? "" : col->name);
         else
            xfprintf(fp, "%s\n%20s", col->name, "");
         xfprintf(fp, "%s ",
            col->stat == GLP_BS ? "B " :
            col->stat == GLP_NL ? "NL" :
            col->stat == GLP_NU ? "NU" :
            col->stat == GLP_NF ? "NF" :
            col->stat == GLP_NS ? "NS" : "??");
         xfprintf(fp, "%13.6g ",
            fabs(col->prim) <= 1e-9 ? 0.0 : col->prim);
         if (col->type == GLP_LO || col->type == GLP_DB ||
             col->type == GLP_FX)
            xfprintf(fp, "%13.6g ", col->lb);
         else
            xfprintf(fp, "%13s ", "");
         if (col->type == GLP_UP || col->type == GLP_DB)
            xfprintf(fp, "%13.6g ", col->ub);
         else
            xfprintf(fp, "%13s ", col->type == GLP_FX ? "=" : "");
         if (col->stat != GLP_BS)
         {  if (fabs(col->dual) <= 1e-9)
               xfprintf(fp, "%13s", "< eps");
            else
               xfprintf(fp, "%13.6g ", col->dual);
         }
         xfprintf(fp, "\n");
      }
      xfprintf(fp, "\n");
      xfprintf(fp, "Karush-Kuhn-Tucker optimality conditions:\n");
      xfprintf(fp, "\n");
      glp_check_kkt(P, GLP_SOL, GLP_KKT_PE, &ae_max, &ae_ind, &re_max,
         &re_ind);
      xfprintf(fp, "KKT.PE: max.abs.err = %.2e on row %d\n",
         ae_max, ae_ind);
      xfprintf(fp, "        max.rel.err = %.2e on row %d\n",
         re_max, re_ind);
      xfprintf(fp, "%8s%s\n", "",
         re_max <= 1e-9 ? "High quality" :
         re_max <= 1e-6 ? "Medium quality" :
         re_max <= 1e-3 ? "Low quality" : "PRIMAL SOLUTION IS WRONG");
      xfprintf(fp, "\n");
      glp_check_kkt(P, GLP_SOL, GLP_KKT_PB, &ae_max, &ae_ind, &re_max,
         &re_ind);
      xfprintf(fp, "KKT.PB: max.abs.err = %.2e on %s %d\n",
            ae_max, ae_ind <= P->m ? "row" : "column",
            ae_ind <= P->m ? ae_ind : ae_ind - P->m);
      xfprintf(fp, "        max.rel.err = %.2e on %s %d\n",
            re_max, re_ind <= P->m ? "row" : "column",
            re_ind <= P->m ? re_ind : re_ind - P->m);
      xfprintf(fp, "%8s%s\n", "",
         re_max <= 1e-9 ? "High quality" :
         re_max <= 1e-6 ? "Medium quality" :
         re_max <= 1e-3 ? "Low quality" : "PRIMAL SOLUTION IS INFEASIBL"
            "E");
      xfprintf(fp, "\n");
      glp_check_kkt(P, GLP_SOL, GLP_KKT_DE, &ae_max, &ae_ind, &re_max,
         &re_ind);
      xfprintf(fp, "KKT.DE: max.abs.err = %.2e on column %d\n",
         ae_max, ae_ind == 0 ? 0 : ae_ind - P->m);
      xfprintf(fp, "        max.rel.err = %.2e on column %d\n",
         re_max, re_ind == 0 ? 0 : re_ind - P->m);
      xfprintf(fp, "%8s%s\n", "",
         re_max <= 1e-9 ? "High quality" :
         re_max <= 1e-6 ? "Medium quality" :
         re_max <= 1e-3 ? "Low quality" : "DUAL SOLUTION IS WRONG");
      xfprintf(fp, "\n");
      glp_check_kkt(P, GLP_SOL, GLP_KKT_DB, &ae_max, &ae_ind, &re_max,
         &re_ind);
      xfprintf(fp, "KKT.DB: max.abs.err = %.2e on %s %d\n",
            ae_max, ae_ind <= P->m ? "row" : "column",
            ae_ind <= P->m ? ae_ind : ae_ind - P->m);
      xfprintf(fp, "        max.rel.err = %.2e on %s %d\n",
            re_max, re_ind <= P->m ? "row" : "column",
            re_ind <= P->m ? re_ind : re_ind - P->m);
      xfprintf(fp, "%8s%s\n", "",
         re_max <= 1e-9 ? "High quality" :
         re_max <= 1e-6 ? "Medium quality" :
         re_max <= 1e-3 ? "Low quality" : "DUAL SOLUTION IS INFEASIBLE")
            ;
      xfprintf(fp, "\n");
      xfprintf(fp, "End of output\n");
#if 0 /* FIXME */
      xfflush(fp);
#endif
      if (glp_ioerr(fp))
      {  xprintf("Write error on '%s' - %s\n", fname, get_err_msg());
         ret = 1;
         goto done;
      }
      ret = 0;
done: if (fp != NULL) glp_close(fp);
      return ret;
}

/* eof */

[jajhall]

OK I've read through this and can produce the corresponding output from HiGHS

[jajhall]

Thanks. This helps me work out the format of the solution file, but it sounds as if this system involves an input file format (see [http://manpages.ubuntu.com/manpages/trusty/man1/glpsol.1.html]) that we can't read

[jkiviluo]

Thanks for considering this. I'm not sure I quite understand, since I thought HiGHS would just need to write the solution file in the same format as GLPSOL. My workflow is as follows:

Read Mathprog model with GLPSOL and write out a MPS file:
glpsol -m myModel.mod --wmps myModel.mps

Then run highs and write to a solution file:
highs myModel.mps --write_model_file myModelSolved.sol

And finally run glpsol again to process the results:
glpsol -m myModel.mod -r myModelSolved.sol

Then I will get out all the results that myModel.mod is formatted to output. In my case myModel.mod is an energy systems model with 2000+ lines of code. I've done similar workflow with Coin CLP before - it outputs almost the same format as GLPSOL/Mathprog and I was able to make it work.

[mingodad]

I would be interested in work a way to integrate glpsol with HIGHS directly without intermediary files like I did with LPSOLVE here https://github.com/mingodad/GLPK/blob/09dbcba20cc2a07b20029775380696115f5d2163/examples/glpsol.c#L1013 also I'm interested in know how much performance improvement my changes to GMPL can bring to your model and data.

[jkiviluo]

Do you mean that you would integrate HiGHS as an alternative solver in GLPK? That would of course be great!

[mingodad]

Yes that's it !

[jajhall]

Thanks. I thought you were aiming to eliminate the need to call glpsol, and just have HiGHS read the Mathprog model. The only other thing we'd have to do is communicate to HiGHS what format of solution file is required as, by default, the '.sol' extension yields a native solution file format being used by other people.

[mingodad]

You can post this suggestion/request to the GLPK mailing lists http://mail.gnu.org/mailman/listinfo/bug-glpk and/or http://mail.gnu.org/mailman/listinfo/help-glpk .

[mingodad]

I just implemented OBJSENSE for freemps on my fork of GLPK the binaries are available here https://github.com/mingodad/GLPK/actions/runs/2324392030 built from github actions.

[mingodad]

Trying to test the last HiGHS with glpsol whit this model from GLPK/examples/hashi.mod it seems that the solution output format generated by HiGHS is not in a format accepted by glpsol, I've tried with write_solution_style 1 and 2, see attached files.

I'm using binaries from this github actions build https://github.com/mingodad/GLPK/actions/runs/2339697539 .

Commands executed:

/usr/bin/time  ./glpsol-linux-amd64 -m hashi.mod  -w hashi-glpk.sol
/usr/bin/time  ./glpsol-linux-amd64 -m hashi.mod  --read hashi-glpk.sol
/usr/bin/time  ./glpsol-linux-amd64 -m hashi.mod  --check --wfreemps hashi.mps
/usr/bin/time  highs --options_file=hashi-highs.set hashi.mps
/usr/bin/time  ./glpsol-linux-amd64 -m hashi.mod  --read hashi-highs2.sol

hashi-highs.set:

write_solution_to_file=true
solution_file=hashi-highs2.sol
write_solution_style=2

./glpsol-linux-amd64 -m hashi.mod  --read hashi-highs1.sol
GLPSOL: GLPK LP/MIP Solver, v4.65-ex, glp_double size 8
Parameter(s) specified in the command line:
 -m hashi.mod --read hashi-highs1.sol
Reading model section from hashi.mod...
168 lines were read
168 lines were read
Generating edge_sel...
Generating satisfy_vertex_demand...
Generating no_crossing1...
Generating no_crossing2...
Generating no_crossing3...
Generating no_crossing4...
Generating flow_conservation...
Generating connectivity_vub1...
Generating connectivity_vub2...
Generating cost...
Model has been successfully generated
Reading MIP solution from 'hashi-highs1.sol'...
hashi-highs1.sol:1: error: line designator missing or invalid
Unable to read problem solution

./glpsol-linux-amd64 -m hashi.mod  --read hashi-highs2.sol
GLPSOL: GLPK LP/MIP Solver, v4.65-ex, glp_double size 8
Parameter(s) specified in the command line:
 -m hashi.mod --read hashi-highs2.sol
Reading model section from hashi.mod...
168 lines were read
168 lines were read
Generating edge_sel...
Generating satisfy_vertex_demand...
Generating no_crossing1...
Generating no_crossing2...
Generating no_crossing3...
Generating no_crossing4...
Generating flow_conservation...
Generating connectivity_vub1...
Generating connectivity_vub2...
Generating cost...
Model has been successfully generated
Reading MIP solution from 'hashi-highs2.sol'...
hashi-highs2.sol:1: error: line designator missing or invalid
Unable to read problem solution

hashi-glpsol-highs.zip

[jajhall]

I'm using binaries from this github actions build https://github.com/mingodad/GLPK/actions/runs/2339697539 .

If these are built using master, then write_solution_style = 2 will correspond to a format that is deprecated and will become write_solution_style = -1 . The glpsol output is only in the mathprog-sol branch (using write_solution_style = 2) that is currently the subject of PR #863

Nonetheless, I'll have a look at your test case on the mathprog-sol branch

[jajhall]

OK I now see the difference between
glpsol -m hashi.mod -w hashi-glpk.sol
and
glpsol -m hashi.mod -o hashi-glpk.sol
I was following the latter on the basis of @jkiviluo 's comments above, using the quoted glpk source (and equivalent for MIP) - even though the output was nothing like the following computer-readable output of @jkiviluo 's original query

s bas 17169 14089 f f 1.06684e+11
i 1 b 1.06684e+11 0
i 2 s 947.477 1579.87
i 3 s 864.792 1579.87
i 4 s 814.937 1579.87

No matter, this computer-readable output is soon generated by looking at glp_write_sol in wrsol.c rather than glp_print_sol in prsol.c.

[jajhall]

HiGHS will now write out the solution in glpsol format if the following options are set - either within code that calls HiGHS, or on the command line if they are included in an options file foo.set and there is the command line option --options_file=foo.set

write_solution_to_file=true
solution_file=foo.bar
write_solution_style=2

It works if HiGHS reads an MPS file, but not (properly) if it reads a .lp file, since your LP file reader doesn't retain constraint names (yet).

The only thing that I can't match is where/if the objective is written out as one of the model rows. It appears that glpsol does this when working from a .mod file, but not when working from an MPS file. It's also unclear where the objective is written out: before or after the constraints. This seems to depend on where the objective is defined in the .mod file.

I've tested on @jkiviluo 's for_HiGHS_testing.zip, but would welcome other experiments being done

The new code is currently the subject of #863

Views? bugs?

Thanks

[jkiviluo]

I've been trying to compile HiGHS all this morning, but I'm having lot of issues (I suppose because I'm on Windows). CMake file for HiGHS seems to be setup only for GCC compilers and I was first trying to use Visual Studio compilers. I got that sorted out, but a warning would have been nice (I really don't know much about this, but maybe the CMakeLists.txt could give a warning).

Now I'm getting a complaint about not having 'OSI' package and I'm not getting any good clues from the internet. Does either of you run Windows and would have binaries for the mathprog-sol branch?

[jajhall]

Sorry about this. No-one on the HiGHS team works in Windows, so we struggle with Windows build issues. Other than the release binaries provided by the JuMP people, we get individual builds to work, but lack a big picture view. I'll pass on your message about warnings to our software engineer.

If you're getting a CMake build failure due to not having OSI, then I think that -DFAST_BUILD=on will ignore OSI, but maybe that only works with CMake on Linux/MacOS.

I'm about to code up a work-around for the "cost row reporting" issue.

[jkiviluo]

This morning was nicer and I was able to compile with MSYS2 (a Cygwin based package manager for building Windows native binaries - should have started with that in the first place instead of trying to patch up everything myself).

I also got my initial workflow working. Really big thanks!

[jkiviluo]

I had to copy zlib1.dll to Highs folder to make it work - I should probably set some compiler flag that I have no clue about.

[jajhall]

Good to hear. The ability to read compressed files is relatively new to HiGHS, so it sounds as if this isn't caught in the CMake

The fix for handling the cost row more generally is almost complete

[jajhall]

HiGHS (mathprog-sol branch) is now writing out in the raw format, but there is still an issue relating to empty cost rows, and it's a product of glpsol, not HiGHS

Running

• glpsol -m hashi.mod --check --wfreemps hashi.mps
• glpsol -m hashi.mod -w hashi-glpk-mod.sol
• glpsol hashi.mps -w hashi-glpk-mps.sol

hashi-glpk-mod.sol has one more row than hashi-glpk-mps.sol, since glpsol interprets the MPS file's empty cost row as an empty free row and removes it.

Hence

• glpsol -m hashi.mod --read hashi-glpk-mod.sol succeeds
• glpsol -m hashi.mod --read hashi-glpk-mps.sol fails

Since HiGHS is reading hashi.mps and I aped the behaviour of glpsol on MPS files,

• glpsol -m hashi.mod --read hashi-highs.sol fails

When glpsol is run from a '.mod' file, if it always reports an empty cost row in a .sol file, there is a "fix"

When HiGHS is run from an MPS file, it can also report an empty cost row in a .sol file.

Of course this makes HiGHS inconsistent with glpsol run from an .mps file, but this is due to the glpsol inconsistency.

I should observe, in passing that HiGHS solves hashi.mps in 0.5s, whereas glpsol takes 52s, so there's a major performance gain to be had!

[jkiviluo]

For this modified simple_max (I should have renamed it to simple_min), HiGHS puts the rows in bit different order than glpsol. I then thought it is the same cost row problem you were saying earlier - maybe it's not. Above is HiGHS output and below is GLPSOL output. Row i3 from glpsol is row i1 in HiGHS. This somehow throws the GLPSOL solution file reading off track and it starts to pick up rows in funny ways. Including the objective function value (it prints out 15 instead of -10 when doing that display Objective.val).

c Problem:    simple_max
c Rows:       3
c Columns:    2
c Non-zeros:  6
c Status:     OPTIMAL
c Objective:  Objective = -10 (MINimum)
c
s bas 3 2 f f -10
i 1 b -10 0
i 2 u 15 -0.4
i 3 u 20 -0.2
j 1 b 5 0
j 2 b 5 0
e o f

[jkiviluo]

Looking at that with thinking turned on, glpsol seems to assume that the first i row should hold the objective function values. But as you say above, maybe it depends on the location of the objective function in the model code.

[jkiviluo]

Indeed, when I move the objective equation in model code to be the last, then the results based on the HiGHS solution file are printed correctly and results from glpsol solution file incorrectly (using the old solution file before moving the objective function).

[jajhall]

Thanks. I'd also deduced that the position of the objective in the .mod file was important, but that the information is lost when the MPS or .LP file is written (I think). This is why the version of HiGHS that I'm checking has an option to control where the objective is written in a glpsol solution file, since we can't expect all users to move the objective as you have.

[jajhall]

So, based on experiments with simple_max:

• in an MPS file the objective appears in the ROWS section in the position corresponding to where it was defined in the .mod file. [I've tried all three possible positions.] When HiGHS is run from the MPS file, it can remember this and reproduce the required position of the objective in the .sol file
• in an LP file the objective appears first (of course), so the information about its position in the .mod file is lost. When HiGHS is run from the LP file, the glpsol_cost_row_location option can be used to define where the cost row should appear in the .sol file

[mingodad]

I've been looking at HiGHS C api to try add a direct bind to it from glpsol and I think that having a way to build the model step by step would facilitate the work for me and for others that could try create bindings for other purposes too.
Would be nice to have an api function that build the model and preallocate memory as needed with info provided beforehand with something like:

HighsInt Highs_buildModel(void* highs, const HighsInt num_col,
                         const HighsInt num_row, const HighsInt num_nz,
                         const HighsInt q_num_nz, const HighsInt a_format,
                         const HighsInt q_format, const HighsInt sense);

So this way we could pass the data step by step without HiGHS needing to allocate/reallocate constantly because it would already know how big the model will be.

As it is right now if the data we have doesn't match the format expected by HiGHS we need duplicate the memory allocation to discard after populating HiGHS with data, this is a problem for models with big data.

Example:
As it's now suppose we have a model with 6GB of data that doesn't match HiGHS format, to pass that data to HiGHS we would need to allocate 6GB of memory to copy our data to a format accepted by HiGHS as input, then HiGHS would again allocate 6GB of data to copy the data we pass to it, then we'll delete the temporary data we have allocated.

With a function like the one I proposed above, we'll tell HiGHS to preallocate the model data and then we could ask with other api functions the pointer to each data segment and populate it directly without need temporary storage (of course users can mess with direct memory access but that's always the case anyway). So HiGHS allocate 6GB and we just copy our data to it.

[jkiviluo]

@mingodad If you get this working, please send a note in this discussion.

[jajhall]

There is a way to build a model step-by-step, and it's what you'd have to do in order to avoid having a full copy of the model both locally and in HiGHS. You call Highs_addRows periodically, after calling Highs_addCols or Highs_addVars to add the variables that have been "discovered" whilst parsing the .mod file since the last call to Highs_addRows. This is how JuMP (does) and Pyomo (will) build models in HiGHS. The overhead of repeatedly resizing the internal HighsModel instance isn't large, as resize (typically) doubles the space reserved if it exceeds the current capacity. Indeed, I'd guess that the cost of parsing the .mod file twice - as I guess Glpsol is doing if it has the dimensions of the model but not the values - is likely to be much more.

What you propose would be better called Highs_ReserveModel and will just ensure that the right amount of space is reserved for each of the vectors in the internal HighsModel instance. Since plenty more space is allocated when solving a model, it doesn't seem to be worth having Highs_ReserveModel due to the confusion that it'll cause some users.

[mingodad]

I don't agree with you here because I can see a big win in simplification an performance benefit for the C API users-

[jajhall]

Then maybe I don't understand how you're looking to build your model after the call to reserve space in HiGHS. What "simplification" do you mean. For me you're adding a method call that's useful only for users who know their final model dimensions, but don't have all the values to pass the model in one call.

If you don't have the whole model stored locally, then you'll have to pass it in chunks of rows as you parse the .mod file, no?

What I'm suggesting is that you have HiGHS resize as you do this, with log_2(n) capacity doublings for the internal vectors. That way users are blind to internal space allocation.

If the HiGHS process I'm suggesting was meaningfully inefficient, the JuMP guys would have pointed it out, as they've helped us eliminate other inefficiencies.

Where's the big gain with what you propose?

[odow]

The cost of loading is almost never the bottleneck for a few reasons:

1. if you're going to pass in 6GB of data, then that's a really large number of non-zeros, and HiGHS probably can't solve it or will take a really long time.
2. during the solve process multiple copies of the problem get created anyway, so you need much more memory than the 6GB to actually solve the problem

[jajhall]

@odow is right that loading the problem is (obviously) always going to be relatively cheap compare with solving it, so long as the cost of loading is linear in model size - as it (now) is with HiGHS

If the problem is really easy, then HiGHS might be able to solve it, particularly if presolve reduces it dramatically in size. 🙂

The number of internal copies is (now) minimal: roughly two copies of the presolved model in addition to the original

[jajhall]

My work on creating glpsol .sol files is now in HiGHS master. By default, and when possible, HiGHS writes out the solution as if it were generated by running glpsol from a .mod file. This is possible if HiGHS runs from an MPS file generated by glpsol, but not if it's run from an LP file generated by glpsol since the position of the cost row derived from the .mod file is lost. Hence using MPS as the intermediate file format is strongly advised!

By setting the HiGHS option glpsol_cost_row_location, the location (or absence) of the cost row in the glpsol .sol file can be controlled, allowing HiGHS to run on an MPS or LP file as if glpsol were run on that file.

[mingodad]

I'm still getting the same error when trying to solve the hashi model:

./glpsol-linux-amd64 -m hashi.mod  --read hashi-highs2.sol
GLPSOL: GLPK LP/MIP Solver, v4.65-ex, glp_double size 8
Parameter(s) specified in the command line:
 -m hashi.mod --read hashi-highs2.sol
Reading model section from hashi.mod...
168 lines were read
168 lines were read
Generating edge_sel...
Generating satisfy_vertex_demand...
Generating no_crossing1...
Generating no_crossing2...
Generating no_crossing3...
Generating no_crossing4...
Generating flow_conservation...
Generating connectivity_vub1...
Generating connectivity_vub2...
Generating cost...
Model has been successfully generated
Reading MIP solution from 'hashi-highs2.sol'...
hashi-highs2.sol:8: error: number of rows mismatch
Unable to read problem solution

[jajhall]

I wasn't using hashi.mod for testing, sorry, because I thought qfit.mod was equivalent (and Glpsol takes ages to solve hashi)

I get the behaviour you observe, and it's due to a Glpsol quirk that I'd not anticipated. Compare the following

• qfit.mod - pure equation-solving with no "minimize"/"maximize" statement
• hashi.mod - feasibility problem with a "minimize cost: 0" statement

Both put an "N" row into the MPS file

• qfit.mps has N R0000000
• hashi.mps has N cost

Glpsol has a cost row in the .sol file for hashi.mod, but not qfit.mod

HiGHS doesn't have the information from the MPS file on whether to include a cost row, or not.

Unless HiGHS interprets a cost row name of R0000000 as being "artificial" and omits it. Of course, this doesn't work if someone has the line

"minimize R0000000: 0"

in the .mod file for a feasibility problem!

The latter seems unlikely, so I'll try interpreting R0000000 as being "artificial" and omit it with a warning

[jajhall]

Modifications have been merged in to master

[jkiviluo]

I've seen HiGHS totally freeze after getting a solve. Seems to happen only sometimes but more frequently with larger models. I was using a version I compiled Sunday morning - I will do further tests with the latest version. Here is it what it looks like:

[jajhall]

I've built HiGHS master in VScode on Windows, and there was an expensive assert - so only in debug - that meant that HiGHS too ages to load a larger MPS file. I've cleared that and pushed to master. However, there's no sign of stalling after the solve

[jkiviluo]

In principle sharing would be ok, but in practice it's quite involved, since there is lot of machinery behind the model (it's really a model generator). However, HiGHS also stalled with transp.mod. I just had to run it around 10 times before it happened. I'm attaching transp.txt (change to transp.mps) for convenience.

transp.txt

My colleague will try to compile HiGHS with MSVC - any tips what did you do to get make work with VScode?

[jajhall]

OK, if it stalled with transport.mod then that's OK. I note that you had to try multiple times. The only thought I had just now is whether I close the solution file - but I can't see why that would be a non-deterministic problem.

With VScode, it seemed to find the CMake stuff and just "worked", after a couple of years of not being able to run HiGHS in any debugger. I'm still not able to run step-by-step, but at least I see where exceptions occur without using print statements! Sorry, coding on Windows is a closed book to me

[jkiviluo]

My colleague was able to compile with MSVC and HiGHS seems to be stable with that. So, it seems it has something to do with compiler and/or its settings. I have very little experience with compiling so it can very well be that I did something stupid.

We will report if there are further issues.

[jajhall]

That's good to hear. Although we would want HiGHS to run correctly with every architecture and compiler, it's very hard with some Windows variants.

[mingodad]

In other programs that behave like it's on an endless loop I usually execute it with GDB and then when it's frozen CTRL-C to get the gdb prompt and look around.

[jajhall]

I'm using VScode on Windows. I don't know how to use GDB

[jajhall]

I've created the branch fix-881 with some addiitonal logging information when writing out a Glpsol solution file.. For a small model you should see

Solving the original LP from the solution after postsolve
Model status : Optimal
Simplex iterations: 87
Objective value : 2.2549496316e+05
HiGHS run time : 0.01
Writing the solution to a file
Writing Glpsol solution file for model "adlittle"
Written Glpsol solution line for row 1
Written Glpsol solution line for row 2
Written Glpsol solution line for row 4
Written Glpsol solution line for row 8
Written Glpsol solution line for row 16
Written Glpsol solution line for row 32
Written Glpsol solution line for col 1
Written Glpsol solution line for col 2
Written Glpsol solution line for col 4
Written Glpsol solution line for col 8
Written Glpsol solution line for col 16
Written Glpsol solution line for col 32
Written Glpsol solution line for col 64
Written Glpsol solution file for model "adlittle"
HiGHS run ended

[jkiviluo]

BTW, it did get all they way to the end of this before freezing.

Further testing with the new version indicates that it really is stable now. Thanks a lot!

[jajhall]

That's good, thanks. I'll still check whether the '.sol' file gets closed.

Enjoy the performance!