Different .osil directories for multithreading

[Ph0non]

Hi and thanks for the help (https://groups.google.com/forum/#!topic/julia-opt/KzkWeVY2woI)

If I use multithreading to optimize more problems in less time I get sometimes an error that results.osrl is empty. It seems that every worker use the same Pkg.dir("CoinOptService", ".osil") and overwrite the used file for another worker.

It would be nice if there is some way to use subdirectories for every worker.

[tkelman]

Yeah, the default behavior is not at all suitable for solving multiple problems simultaneously, sorry about that. I should make a note about that on the readme (or if you want to suggest an edit in a PR, go right ahead). Can you try setting the osil, osrl, and osol keyword arguments to OsilSolver to independent file names for each problem? Pretty sure that should work. (You'll need to manually clean them up afterwards, at the moment I'm leaving those files around for debugging purposes.)

I'm not sure what feasibility reformulations you might try without knowing much about the structure of your problem. It looked like you were doing a lot of database access and other things to populate the problem data. The .osil file should be a self-contained way of reproducing the formulation including all data though, might be good to share that with the Couenne mailing list (if you can) as a problem it was having trouble with vs Bonmin was able to solve it. Usually I'd expect it to be the other way around, Bonmin is a heuristic and only gives local optima when the continuous part of your MINLP is non-convex, vs Couenne does more sophisticated (and computationally expensive) relaxations to try to get a global solution.

I'm also curious about what compilation troubles you had, if they're anything I could improve on. If you're on OSX, sorry I didn't finish packaging binaries of Couenne, CppAD, and OS via Homebrew yet. That would make things a little easier and I should get back to finishing that.

[Ph0non]

About the problem:

I try to calculate so called conservative nuclide vectors. A nuclide vector (NV) is a theoretical distributions of radioactive nuclides in a population of given samples. This distribution needn't to be representive. This distribution must fulfill the requirement, that the specific activity of the NV over the specific activity of a given sample must be greater equal 1 to be conservative.

The boundaries for this requirement are calcaluted here:

@addNLConstraint(m, 1 ≤ C[j,1,l+h] * a[j,k,l+h] * ∑{f_red[i,k] * x[i], i=1:length(Xvar)} /
 ∑{eff_arr_red[i,1] * x[i], i=1:length(Xvar)} ≤ g )

C -> The sum of the so called Co60-equivalent (each nuclide have a different sensibility to be detected)
j -> index over the relevant nuclides (mean over all samples a minimum of 0.01%)
l+h -> the year for which the NV should be valid (decay correction), h is 0 or 1, the NV need to be conservative for the beginning and end of a year
C[1,:,y] -> the clearance paths (y=1 -> FMA, y=2 -> FMB)
a -> a factor with the amount of the relevant nuclides and the clearance values
f_red -> 1./clearance values reduced to the relevant nuclides
eff_arr_red -> efficiency to detect the nuclides (also used in C)
Xvar -> the variables of the model
g -> some upper bound to get the least overestimate

The objective (maximize the sum of Co60 and Cs137) is choosen to get best measurability in practice.

compilation problems: It was not clear which packages OS needed (i use the errors to get to know which packages i also need. At least the path to coinutils was not set and I didn't found the variable to set it. So i copy the files from /usr/include/coin to the folders OS-2.9.1/Bcp/src/include and another I already forgotten.

I also get the following error in some case:

OSSolverService: OsiBranchingObject.cpp:629: virtual OsiBranchingObject* 
OsiSimpleInteger::createBranch(OsiSolverInterface*, const OsiBranchingInformation*, int)
 const: Assertion 'fabs(value-nearest)>info->integerTolerance_' failed.

The corresponding osol and osil files:
https://gist.github.com/Ph0non/1ecf6326cb409821b945

I can't reproduce the :Infeasible status for a feasible problem in Bonmin again. Couenne works, but not as intended. I increase the upper bound with each iteration and :Infeasible status but with some low upper bound couenne produce the status :Error. The results file say it all, but shouldn't be it an :Infeasible instead of :Error?
https://gist.github.com/Ph0non/25a1233ec157fa463d7e

[tkelman]

At least the path to coinutils was not set and I didn't found the variable to set it. So i copy the files from /usr/include/coin to the folders OS-2.9.1/Bcp/src/include and another I already forgotten.

You shouldn't need to install OS system-wide here. Most likely what happened was it couldn't find the compiled libraries from Cbc. What platform are you on, are you using the latest version of Cbc, and have you done Pkg.build("Cbc")? Are you using a system-wide installation of Cbc? If so, which version of Cbc, and did you compile it yourself or get it from a package manager (which distribution?)? If you installed Cbc some time ago and it was not using the latest version, you could clear out Pkg.dir("Cbc","deps","usr") and re-run Pkg.build("Cbc"); Pkg.build("CoinOptServices") and I believe that would fix any problems finding coinutils. I don't think I'll be able to support system versions of Cbc or OS from package managers though since the version numbers will most likely not be new enough.

I have not seen COUENNE INITIALIZE PROBLEM before. I'd have to look into Couenne's source code to see what kind of conditions can cause that. It's likely not a conventional infeasibility-detection exit condition, and is triggering some different code path where the solver is not as sure what's wrong.

I notice your osil file has quite a few occurrences of coef="0.0", I guess I'm not explicitly checking the coefficient value for zeros at any point. I should perhaps remove those. Also since all of the nonlinear constraints appear to be "affine / affine", I think you should be able to reformulate this problem as a linear fractional problem then you wouldn't need Bonmin/Couenne, a regular MILP solver like Cbc would work.

[tkelman]

Your variables are all nonnegative, and the denominator of every nonlinear constraint looks to be the same expression

          <sum>
            <variable idx="0" coef="1.0"/>
            <variable idx="1" coef="0.341399997"/>
            <variable idx="2" coef="0.0"/>
            <variable idx="3" coef="0.0"/>
            <variable idx="4" coef="0.0"/>
            <variable idx="5" coef="0.0"/>
            <variable idx="6" coef="0.0"/>
          </sum>

so you should be able to multiply both sides of a ≤ (b / c) ≤ d by c, giving the linear constraint a * c ≤ b ≤ d * c, since a and d are constants, b and c are affine expressions and c is nonnegative.

[Ph0non]

I installed JuMP which came with Cbc, or not? I didn't install somethin systemwide before. All was JuMP und Ipopt through julia.
Now Cbc should be installed systemwide after CoinOptService could find it (and find it after install through apt)

I use aptosid (debian sid) x64 with apt and sid sources + additional aptosid sources (bug fixes + patches).

coef="0.0" is a result of the detector efficiency (Co60 equivalent). Most relevant nuclides are non measureable (no gamma emitter) for the used detector.

In your example idx=0 is Co60 (Co60 equivalent 1) and idx=1 is Cs137 (Co60 equivalent of 0.3414). This is also the reson to maximize the sum of both.

[tkelman]

Cbc does not get installed by default, but it is listed as a dependency of this package. So when you did Pkg.add("CoinOptServices"), then Cbc should have automatically been installed. Does Pkg.test("Cbc") work?

The only place I set coef is here https://github.com/tkelman/CoinOptServices.jl/blob/92a37a13a30e19b811ab295cce5365defe630f37/src/translations.jl#L211-L232 where I am definitely not checking the value at the moment. The best I could do I think is return <number value="0"> instead of <variable idx="$N" coef="0.0"/>, but maybe that's better to leave to the solver to figure out, or expression-level preprocessing that JuMP currently doesn't do much (any?) of.

[Ph0non]

When I build CoinOptService I get the message, that Cbc is build. Pkg.test("Cbc") returns a few errors. If I would know how to redirect REPL to a file I would show you.

Is this such a big deal if tere are a few 0.0 coeff in a sum in the denominator(which would definitely not be zero)?

./julia/julia -e 'Pkg.test("Cbc")' > ./Cbctest
https://gist.github.com/Ph0non/b975edbb6d97c461b858
(after line 908)

Is there an upper limit for constraints?
numberofConstraints="1023"

[tkelman]

It looks like the only problem was at [qcqpmodel] Test quad constraints (discrete) ? That's an issue I've seen with Bonmin, it can depend on the version of Bonmin that was built. It also looks like a print-level patch that I have here was not applied for some reason. Have you done Pkg.update() recently? What do Pkg.installed("Cbc") and Pkg.installed("CoinOptServices") say?

Is this such a big deal if tere are a few 0.0 coeff in a sum in the denominator(which would definitely not be zero)?

No, not really. Just a bit more I/O and processing for the solver to do, I don't know if or when it will be smart enough to remove those terms. You could avoid creating them in the first place by looping over only indices where the coefficients are nonzero in your problem formulation.

Is there an upper limit for constraints? numberofConstraints="1023"

I highly doubt it... that's either a coincidence or a serious bug somewhere.

Regardless, higher priority for you should be trying a MILP formulation. If you don't need to deal with experimental MINLP features, you shouldn't have to. You have an affine expression with all nonnegative coefficients and all nonnegative variables in the denominator, so it's a valid transformation to multiply that out.

[Ph0non]

Pkg.installed("Cbc")
v"0.1.6"
Pkg.installed("CoinOptServices")
v"0.0.4"

Thanks for the suggestion. Is MILP the same as MIP? (http://jump.readthedocs.org/en/latest/installation.html#getting-solvers)

I modified the problem but get a few strange results from JuMP itself with a double sided constraint. But this is not for this issue, which is already miles away from the original problem. ;)

Conclusion: CoinOptService.jl works nearly fine for problems in the real world :)
Very good work.

P.S.: Cbc works VERY fast.

[tkelman]

Hm, with those versions you should have the latest builds of everything so I'm not sure why the print level patch wouldn't get applied, or the qcqpmodel test would have any problems...

Thanks for the suggestion. Is MILP the same as MIP? (http://jump.readthedocs.org/en/latest/installation.html#getting-solvers)

Yeah, see also the table at http://www.juliaopt.org/ - I think the one from JuMP's docs hasn't been updated to account for MINLP capability, which was implemented only a few months ago for KNITRO.jl. JuMP has had mixed-integer-linear capabilities since the beginning (or well before I started using it anyway), the nonlinear stuff is newer. And the table hasn't been updated for this package since it's still kind of experimental and not formally a part of JuliaOpt, though it's designed to work together with JuliaOpt packages.

I modified the problem but get a few strange results from JuMP itself with a double sided constraint.

Oh right, you will need to split that up into 2 separate inequality constraints, since range constraints in JuMP need to have constants on either side I think.

which is already miles away from the original problem.

I could probably add a note in the readme about solving multiple problems simultaneously to close this issue. I don't quite want to pick a random filename by default and delete it after solving the problem, since looking at the generated osil files is really useful (and I don't need access to your data sources to run the same optimization problem). Maybe I'll eventually do this as default behavior but add a debug flag for leaving the files around.

P.S.: Cbc works VERY fast.

Your problem is pretty small, so I'm not surprised.

[tkelman]

At least the path to coinutils was not set and I didn't found the variable to set it.

Ah, I see this happening. I think this can happen if you don't have pkg-config installed.