How the SAT Competition 2003 will proceed?

SAT Competition 2003

February - May, 2003

This is subject to change. Last modification: December 4, 2002.

A new SAT competition is going to start on February 2003 and continue until May 2003. The results of the competition will be given at the SAT 2003 conference (Italy). That competition follows the SAT Competition 2002 started on March 10, 2002 and continued until (and during) SAT 2002 Symposium (May 6-9, 2002, Cincinatti, Ohio, USA).

Because a competition needs strict winners, according to the competition rules, this page describe how we will find them. Of course, this is only related to the rules we have defined. This page describes this rules and how we'll choose the "best" SAT solvers (3 awards will be given) for this competition, and what benchmarks will be the hardest (according to our rules). For further details, please take a look at the technical report concerning the SAT 2002 competition.

Competition calendar

Month Week Competition phases
February 3 to 9 0
February 10 to 16 1 Submission of Benchmarks/Solvers (Feb,14th)
February 17 to 23 2 Compliance testing
February/March 24 to 2 3 Buggy solvers returned (Feb, 28th)
March 3 to 9 4 Buggy solvers resubmitted (March, 7th)
March 10 to 16 5 First stage starts (March, 14th)
March 17 to 23 6 first stage
March 24 to 30 7 first stage
March/April 31 to 6 8 first stage
April 7 to 13 9 first stage
April 14 to 20 10 Second stage starts (April, 14th)
April 21 to 27 11 second stage
April/May 28 to 4 12 second stage
May 5 to 11 13 Awards revealed (during conference)

As shown on the above calendar, the competition will be 13 weeks long, and separated in 3 rounds:

Details of each round

First round (red/white, weeks 1-4): before the actual competition starts, solvers and benchmarks will be checked for compliance (essentially input/output). For each submitter, an account will be opened on a Linux machine over the web, where compilation may be checked. Please do notice that before week 5, no submitted solvers will be tested on any submitted (new) benchmark.

Second round, first stage (green, weeks 5-9): this is the end of the interactive part of the competition. We will run each solver on each applicable benchmark (i.e., we will not run incomplete solvers on the benchmarks known to be unsatisfiable). If a solver does not answer within time limit (to be determined when we know the number of submissions), it is interrupted. Randomized solvers will be started several times (at least 3 times) and the median time will be taken into account ("time limit" is charged for the unsuccessful tries). Time limit will be fixed after week 1, when the number of submission will be known (this number will be made public).

Third round, second stage (green, weeks 5-9): for this last stage of the competition we select top N solvers (determined from the first stage by the same method as will be used for determining the winners after the second stage, considering only the best variant of a given solver) in each of the 6 categories and run them on the M smallest unsolved benchmarks (again, separately in each category), smaller benchmarks come earlier. We suppose N= M =4 now, but Laurent Simon may change it right after the deadline if the number of benchmarks and/or solvers and/or our computers will be far from what we predict. Clearly, at this stage the time limit will be substantially larger.

Results (yellow, week 13): awards will be given during the SAT2003 conference. We will also try to give a first analysis of results (see below for awards).

Who are the winners?

There will be 8 prizes. We will grant 6 winners for the solvers and 2 for the benchmarks:

What are the awards?

A beautiful certificate, unless some people are willing to provide prizes to motivate the competition (then please contact the organizers).

What are the differences with the previous (2002) competition?

According to some important bias in the SAT 2002 competition (we identifed them in our SAT 2002 competition report), we made the following decisions:

Important dates

[Top] [SAT 2002 competition] [SAT 2003] [SAT-Ex] [SATLIB] [SAT Live!]

Last modification: 3 December 2002.