Skip to main content

Rules

Challenge rules

The challenge runs over two stages: development and testing. 

The development stage (1st-April-2016 to 17th-June-2016):

Participants are to build their own solvers (a model and a search algorithm) for the total waste water network design and operation optimization problem. Participants should keep in mind that the evaluation of their solver is based on both a set of training instances, and a set of test instances. The evaluation criteria is described at the bottom of this page.

To aid in building a solver, the following resources are available in the Resources page:

  • A problem description.
  • Training instance data along with a feasible solution for each instance.
  • A file format description of both the input instance data files, and the output solution data files. 
  • A mathematical model. Again, it is not mandatory to use the model provided, it is only there for assistance. 
  • A verification tool to assess the feasibility and the quality of the solutions the participants reach.
  • A leaderboard that shows the best solution quality obtained for each training instance. Participants may upload their solution files of any instance through their account, we will verify the solution(s) and update the leaderboard's page once a week.

 

The testing stage (20th-June-2016 to 16th-September-2016):

Once the development stage is over, the participants are to submit their solvers for evaluation. This is done based on both the training set and a new, previously unseen, test set. We will run all submissions on the same machines for a maximum of 3 hours of CPU time, after which the solver should print the best solution it finds to a file matching the format used during the training phase. File format descriptions are found in the Resources page.

All experiments are performed on a single node of a cluster containing machines equipped with an Intel Xeon E3-1220 processor clocked at 3.10 GHz and 8 GB RAM. Each solver is allowed to utilize up to 4 cores for parallel processing. 

Important: upon submission, a participant must specify the requirements needed to run his, or her, solver such as: operating system, external libraries, external solvers (commercial or not), etc.

 

Evaluation criterion:

Evaluation is based on two measures: solution quality (primary) and running time (secondary), obtained for both the training set and test set. Each solver is ranked from 1 to k, k being the number of participants, on each measure. The average and variance of the ranks, for each measure, are then calculated for each solver across all instances with a 0.3 weight assigned to a training instances, and a 0.7 weight assigned to test instances. 

The "best" solver is determined as follows:

  • Find the solver with the best average rank for the solution quality measure. 
  • Discard any other solver that has an average rank that is statistically worse than the best, based on the primary measure.
  • If one solver remains, a winner is identified. It is then removed from the pile of solvers and the process is repeated to identify the second and third places.
  • If in any round more than one solver remains, we consider the secondary measure (running time) in the same fashion; that is, the solver with the best average rank for the running time, among the remaining solvers, is used to discard the others. Ideally, one solver will remain, if not, then we have multiple winners at that spot (fist, second, or third).

 

Award split in case of ties:

  • If two or more solvers occupy the same rank, the budget allocated to that rank and all subsequent ranks will be summed up and divided equally among the owners of these solvers. 
  • It is possible that solvers occupying the second or third positions are awarded nothing if three, or more, solvers occupy the first position; or two, or more, solvers occupy the second position, respectively.