Add support for Benchmarking Derivative-Free Optimization Algorithms

[rgaiacs]

This was requested by @deisegema.

The reference for this is J. Moré and S. Wild website and article.

[rgaiacs]

As requested in the article, we need the dimensions of the problems. @abelsiqueira and I will use a external file to pass that information.
For CUTEst, this can be obtained from https://github.com/lpoo/cute-problem-chooser.

[abelsiqueira]

3 years late, but I think I'll work in this now.
It's slightly more complicated, because the data profile needs the function value at all iterations.

[abelsiqueira]

It looks like I'll have to completely change a few things to make it work. I'm thiking

• We have F_{s,p,k}, the function value at iteration k for problem p and solver s. It seems to me that the easier agnostic way to store this is: one folder for each solver, one file for each solver, one line for each iteration.
• Since data and perf have fundamentally different parsing, the Prof class will have to change. They are similar only in the plotting, basically (Almost all plotting, in reality).
• I'm thinking in changing the fact that the backends are Profilers. So there would be two Profiler: Data and Perf, which do their own parsing and scaleing (which is how we've called the computation of the performance function).
• I'm too used to Julia, so I'm not thinking OOP. The backends would derive from the same class? Is there a reason for this?

[rgaiacs]

one folder for each solver, one file for each solver, one line for each iteration.

Do you mean "one folder for each solver, one file for each problem, one line for each iteration."?

Since data and perf have fundamentally different parsing, the Prof class will have to change.

There is a few ways that you can do it. My suggestion is to remove this line and add two new classes to https://github.com/ufpr-opt/perprof-py/blob/master/perprof/prof.py#L36. Each class is for one profiler. You will end up with something like

class Pdata(object):
    ...

class XProfiling(Pdata):
    def __init__(self):
        self.data = ...
        Pdata.__init__(self)

class YProfiling(Pdata):
    def __init__(self):
        self.data = ...
        Pdata.__init__(self)

You will need to change things at the https://github.com/ufpr-opt/perprof-py/blob/master/perprof/main.py.

I'm thinking in changing the fact that the backends are Profilers. So there would be two Profiler: Data and Perf, which do their own parsing and scaleing (which is how we've called the computation of the performance function).

If other methods are going to be different, change it at Pdata to return NotImplementedError.

I'm too used to Julia, so I'm not thinking OOP. The backends would derive from the same class? Is there a reason for this?

See my response above. You know how to contact me for chat. :-)

[abelsiqueira]

Do you mean "one folder for each solver, one file for each problem, one line for each iteration."?

Yes

Thanks for the input. I'm doing something like that. I hope to post something today with the basic differences.

[abelsiqueira]

#181 handles the big picture for data profile, and other things.
The finer points to do

• Change --tau, which means the maximum ratio value, for something more generic. xmax?.
• Add argument for rtol in data profile.
• Solver names other than folder names. Pass as comma separated or read from file.