Skip to main content

Results

Four competitors submitted their solvers, one of which we could not run on our machines despite many efforts. Of the remaining three, one solver obtained feasible solutions to only 4 out of the 14 instances (7 training and 7 testing). For this solver, I set the objective value and running times for the infeasible instances to a number higher than the maximum of the other two solvers, just to see if it would at least be competitive. I will refer to this solver as the “infeasibleSolver”.
 
As for the benchmark solvers, they are there just to provide a feasible solution, they were not run to optimize the problems, as that required tuning their parameters, which I had no access to.
 
The analysis of the three remaining solvers was done in two ways:
 
1.      Equal weights for the training and test instances:
a.      The AD normality test for both objective values and running times was insignificant for only one solver, though another had a p-value of 0.053; thus, I adopted nonparametric tests.
b.      The F-test was significant for the objective value, indicating that at least one solver is different than the other two.
c.      Pair-wise comparisons, based on the objective value, using the MW test knocked out the infeasibleSolver, yet failed to distinguish between the other two. The 95.4% CI for η1 - η2 was (-375216, 605393) and the p-value was 0.6625.
d.      Turning to the running time as a secondary objective to break the tie, the MW test was significant with the 95.4% CI being (-1964.0, -288.9) and a p-value of 0.0044.
 
2.      Weighted values for training and test instances (0.7 for testing and 0.3 for training); as stated in the rules of the competition:
a.      The AD normality test for both objective values and running times was significant for all solvers; thus, only nonparametric tests are possible.
b.      The F-test was significant for the objective value, indicating that at least one solver is different than the other two.
c.      Pair-wise comparisons, based on the objective value, using the MW test knocked out the infeasibleSolver, yet failed to distinguish between the other two. The 95.4% CI for η1 - η2 was
(-207625, 268860) and a p-value of 0.7304.
d.      Turning to the running time as a secondary objective to break the tie, the MW test was significant with the 95.4% CI being (-1306.8, -298.4) and a p-value of 0.0038.
 
Conclusion:
Based on the competition rules, the solver presented by Artelys is the winner as it obtains solutions that are not significantly worse than the best, but does so in a significantly less time.
The second place winner is Dr. Emanuele Tresoldi as his solver obtains solutions that are insignificantly better than the others, yet it does so with a significantly more time.
 
The data is in the attached file. 
AttachmentSize
Results summary.xlsx26.61 KB