Metric & Limits

---

To evaluate planners, we used the agile metric used in the last IPC, i.e. a planner is regarded better w.r.t. that metric if it finds any solution to the problem faster.

• The score of a planner on a solved task is 1 if the task was solved within 1 second and 0 if the task was not solved within the resource limits.
• If the task was solved in t seconds (1 ≤ t ≤ T) with a time limit of T seconds then its score is min{1, 1 - log(t)/log(T)}. The score of a planner is the sum of its scores for all tasks, but normalized so that each domain contributes equally to the overall score.

The time and memory limits were as follows:

• 8 GB memory limit
• 30 min time limit (using 1 CPU core)

Rules regarding wrong outputs:

• If an invalid plan is returned, all planning problems in the domain are counted as unsolved.
• If that happens in more than one domain, the participating planner is disqualified in the respective track.

All these rules were known to the planners before planner submission.