Datasets

We created essentially two sets of data:

All produced plans (including invalid ones) and
all log files of the planners for the competition (with all detailed results) and

Note that the plans can also be used for other purposes, such as for research in plan verification, or correcting invalid plans.

Plans

In addition to the logs, we also offer the plans that were generated by the planners in the IPC as a resource. This can for example be used for research in plan verification. In addition to plans from the IPC planners, we have also gathered plans from runs of other planners (mostly the PANDA planner). We have removed all duplicate plans, i.e. when the same plan was generated by multiple planners. The current version of the plans is available on github (this repository will soon be replaced by another). We will update the repository from time-to-time as new plans become available.

In total we provide:

10,961 valid and 1,406 invalid plans from total-order domains, and
1,211 valid and 138 invalid plans from partial-order domains.

Log Files

For both tracks of the IPC, you can download a full dataset and a reduced dataset. Note that all experiments were executed 10 times with different seeds. Per run, we defined a base seed. For every run, the actually used seed was the base seed plus the instance number. The full dataset contains the log files for all planners on every instance. Since this is a lot of data we also created a reduced data set.

Problem Instance Renaming

In the logs provided below, every instance is referred to with a numerical code. For technical reasons, the instance numbers are separated between the partial-order and the total-order tracks and partially overlap! After the experiments were run, the structure of the domain repository was changed to allow for the usage of LAB. As such, the file names of the instances in the log files may not be correct anymore. The following two files contain a mapping of instance numbers to the instance files. The paths refer to the github repository.

Instance name mapping of partial order track (40 kB)
Instance name mapping of total order track (111 kB)

Full Data Set

Every log contains the full output of the planner and the output of the subsequent verification of the found plan. In some cases, the verification took longer than the maximum time-limit of the job on our compute cluster (primaliry very long plans in the Towers domain). All such plans were valid. The planners are identified by a number which is shown in the table of planner above. The file names of the full logs comprise several information. The most notable one is the last number before the long sequence of dashs: it is the instance number (see above).

partial order track (15.8 MB as zip)
total order track (10 zip files, each 20 to 40 MB)

Reduced Data Set

In addition, we offer a reduced dataset. This dataset contains one line per run of a planner in which it found a solution (not necessarily a correct one). Each line contains 6 entries:

A string describing the planner. The number contained in this string corresponds to the number of the planner.
The instance number (see above).
The domain number. This number has no deeper meaning, it is just a unique number given to all instances of one domain.
Runtime in seconds. This already includes the reduction of 0.12 seconds given to all planners due to singularity startup time.
Peak memory usage in kilobytes. This already includes the reduction by 20000 kb given to all planners due to the memory used by singularity.
Length of the found plans in number of actions.

The reduced data sets can be found here:

partial order track (131 KB)
total order track (1 MB)

Detailed Logs per Planner

The tables below show the status information on all competitors of the 2020 IPC, including those who dropped out before its official start. (Note that this page was used in the early stages of the IPC so that competitors could see whether their planner had issues or not, so they could fix them before the actual competition started. The page was thus continuously updated, but is now static and won't change anymore.)

For each competitor we show whether the planner currently compiles and whether it solves a given set of sample problems. The logs of the compilation and every run are available for download by clicking on the blue links.

The test instances can be downloaded by clicking at the columns headers.

Each of the six sample problems is divided into two sections. The one on the top shows whether the planner is able to find a plan (green) or not (red). The second section below shows whether the generated plan is valid. Green indicates that the plan is valid, red that it is not, and blue that no plan was found in the first place. If the planner produced only a primitive plan and did not provide a decomposition witness, we will indicate with cyan that the primitive plan is executable. Yellow indicates instances that the planner does not support some feature of the respective problem instance (either due to recursion or partial order).

For the last column, we let each planner run for 10 seconds on a larger set of benchmark instances.

As a summary per domain we report:

SOL - the planner has solved the problem with a valid plan,
INVAL - the planner has solved the problem, but the plan failed verification, and
TLE - the planner exceeded the time limit of 10 seconds.

Note that INVAL here also includes cases where the planner produced only a primitive plan which was valid.

Competitor	Compile Status	rover01	satellite01	um-translog01	transport01	p-rover01	p-transport01	10 seconds run (SOL,INVAL,TLE)
Lilotane recursion: yes partial order: no	Compile Log 495.34 sec 519912 kbyte	Planner Log 0.45 sec 24312 kbyte	Planner Log 0.36 sec 24732 kbyte	Planner Log 0.50 sec 24552 kbyte	Planner Log 0.38 sec 24832 kbyte	not supported	not supported	Miconic: 7 / 0 / 0 Zenotravel: 5 / 0 / 0 UM-Translog: 22 / 0 / 0 Satellite: 25 / 0 / 0 Woodworking: 11 / 0 / 0 SmartPhone: 7 / 0 / 0 rover: 19 / 0 / 1 transport: 28 / 0 / 2 Planner Log
Lilotane recursion: yes partial order: no	Compile Log 495.34 sec 519912 kbyte	Verify Log 0.00 sec 3292 kbyte	Verify Log 0.00 sec 2492 kbyte	Verify Log 0.01 sec 4664 kbyte	Verify Log 0.01 sec 2700 kbyte	no plan	no plan
HPDL-planner recursion: yes partial order: yes	Compile Log 210.51 sec 435944 kbyte	Planner Log 0.67 sec 24228 kbyte	Planner Log 0.68 sec 24084 kbyte	Planner Log 0.76 sec 24136 kbyte	Planner Log 0.91 sec 24068 kbyte	Planner Log 0.76 sec 24476 kbyte	Planner Log 0.66 sec 24220 kbyte	Miconic: 7 / 0 / 0 Zenotravel: 5 / 0 / 0 UM-Translog: 22 / 0 / 0 Satellite: 25 / 0 / 0 Woodworking: 3 / 0 / 8 SmartPhone: 2 / 0 / 5 rover: 13 / 0 / 7 transport: 1 / 0 / 29 Planner Log
HPDL-planner recursion: yes partial order: yes	Compile Log 210.51 sec 435944 kbyte	Verify Log 0.00 sec 3372 kbyte	Verify Log 0.00 sec 2500 kbyte	Verify Log 0.01 sec 4668 kbyte	Verify Log 0.00 sec 2704 kbyte	Verify Log 0.01 sec 3368 kbyte	Verify Log 0.00 sec 2704 kbyte
HyperTensioN recursion: yes partial order: no	Compile Log 31.81 sec 186360 kbyte	Planner Log 0.37 sec 24704 kbyte	Planner Log 0.36 sec 24784 kbyte	Planner Log 0.38 sec 25244 kbyte	Planner Log 0.41 sec 24884 kbyte	not supported	not supported	Miconic: 7 / 0 / 0 Zenotravel: 5 / 0 / 0 UM-Translog: 22 / 0 / 0 Satellite: 25 / 0 / 0 Woodworking: 6 / 0 / 5 SmartPhone: 7 / 0 / 0 rover: 19 / 0 / 1 transport: 30 / 0 / 0 Planner Log
HyperTensioN recursion: yes partial order: no	Compile Log 31.81 sec 186360 kbyte	Verify Log 0.01 sec 3932 kbyte	Verify Log 0.00 sec 2492 kbyte	Verify Log 0.01 sec 4668 kbyte	Verify Log 0.00 sec 2704 kbyte	no plan	no plan
PDDL4J recursion: yes partial order: no	Compile Log 183.17 sec 711248 kbyte	Planner Log 2.27 sec 99224 kbyte	Planner Log 2.18 sec 94976 kbyte	Planner Log 2.22 sec 107300 kbyte	Planner Log 2.01 sec 96372 kbyte	not supported	not supported	Miconic: 7 / 0 / 0 Zenotravel: 5 / 0 / 0 UM-Translog: 22 / 0 / 0 Satellite: 25 / 0 / 0 Woodworking: 6 / 0 / 5 SmartPhone: 0 / 0 / 7 rover: 20 / 0 / 0 transport: 28 / 0 / 2 Planner Log
PDDL4J recursion: yes partial order: no	Compile Log 183.17 sec 711248 kbyte	Verify Log 0.00 sec 3212 kbyte	Verify Log 0.00 sec 2492 kbyte	Verify Log 0.02 sec 4668 kbyte	Verify Log 0.00 sec 2708 kbyte	no plan	no plan
PDDL4J recursion: yes partial order: yes	Compile Log 219.01 sec 705092 kbyte	Planner Log 2.13 sec 100244 kbyte	Planner Log 2.20 sec 93680 kbyte	Planner Log 2.23 sec 99680 kbyte	Planner Log 2.43 sec 94628 kbyte	Planner Log 2.22 sec 91740 kbyte	Planner Log 2.22 sec 94964 kbyte	Miconic: 7 / 0 / 0 Zenotravel: 5 / 0 / 0 UM-Translog: 22 / 0 / 0 Satellite: 25 / 0 / 0 Woodworking: 5 / 0 / 6 SmartPhone: 0 / 0 / 7 rover: 20 / 0 / 0 transport: 26 / 0 / 4 Planner Log
PDDL4J recursion: yes partial order: yes	Compile Log 219.01 sec 705092 kbyte	Verify Log 0.00 sec 3216 kbyte	Verify Log 0.00 sec 2496 kbyte	Verify Log 0.01 sec 4668 kbyte	Verify Log 0.00 sec 2704 kbyte	Verify Log 0.00 sec 3240 kbyte	Verify Log 0.00 sec 2700 kbyte
pyHiPOP recursion: yes partial order: yes	Compile Log 102.70 sec 416480 kbyte	Planner Log 5.51 sec 69188 kbyte	Planner Log 2.53 sec 38240 kbyte	Planner Log 57.21 sec 1539936 kbyte	Planner Log 2.15 sec 44764 kbyte	Planner Log 60.13 sec 24968 kbyte	Planner Log 2.53 sec 47664 kbyte	Miconic: 2 / 0 / 5 Zenotravel: 4 / 0 / 1 UM-Translog: 17 / 0 / 5 Satellite: 11 / 0 / 14 Woodworking: 2 / 0 / 9 SmartPhone: 4 / 0 / 3 rover: 4 / 0 / 16 transport: 1 / 0 / 29 Planner Log
pyHiPOP recursion: yes partial order: yes	Compile Log 102.70 sec 416480 kbyte	Verify Log 0.01 sec 3936 kbyte	Verify Log 0.00 sec 2492 kbyte	Verify Log 0.02 sec 4668 kbyte	Verify Log 0.00 sec 2704 kbyte	no plan	Verify Log 0.00 sec 2700 kbyte

Competitor	test01	test02	test03	test04	test05	test06	test07
	empty method	forall	forall2	no abstracts	constants in domain	synonymes	arguments
Lilotane recursion: yes partial order: no	Planner Log 0.46 sec 24684 kbyte	Planner Log 0.37 sec 25156 kbyte	Planner Log 0.37 sec 24612 kbyte	Planner Log 0.34 sec 24740 kbyte	Planner Log 0.37 sec 24880 kbyte	Planner Log 0.33 sec 24548 kbyte	Planner Log 0.38 sec 24524 kbyte
Lilotane recursion: yes partial order: no	Verify Log 0.89 sec 2332 kbyte	Verify Log 0.05 sec 2344 kbyte	Verify Log 0.01 sec 2356 kbyte	Verify Log 0.00 sec 2328 kbyte	Verify Log 0.00 sec 2340 kbyte	Verify Log 0.00 sec 2480 kbyte	Verify Log 0.00 sec 2344 kbyte
HPDL-planner recursion: yes partial order: yes	Planner Log 0.78 sec 24836 kbyte	Planner Log 0.63 sec 24284 kbyte	Planner Log 0.60 sec 24248 kbyte	Planner Log 0.61 sec 24236 kbyte	Planner Log 0.63 sec 24556 kbyte	Planner Log 0.59 sec 23868 kbyte	Planner Log 0.63 sec 23996 kbyte
HPDL-planner recursion: yes partial order: yes	Verify Log 0.17 sec 2312 kbyte	Verify Log 0.01 sec 2344 kbyte	Verify Log 0.00 sec 2356 kbyte	Verify Log 0.00 sec 2328 kbyte	Verify Log 0.00 sec 2344 kbyte	Verify Log 0.00 sec 2484 kbyte	Verify Log 0.00 sec 2348 kbyte
HyperTensioN recursion: yes partial order: no	Planner Log 0.37 sec 24496 kbyte	Planner Log 0.34 sec 24196 kbyte	Planner Log 0.36 sec 24288 kbyte	Planner Log 0.35 sec 24584 kbyte	Planner Log 0.38 sec 24672 kbyte	Planner Log 0.36 sec 24964 kbyte	Planner Log 0.37 sec 25188 kbyte
HyperTensioN recursion: yes partial order: no	Verify Log 0.00 sec 2336 kbyte	Verify Log 0.00 sec 2344 kbyte	Verify Log 0.00 sec 2352 kbyte	Verify Log 0.00 sec 2332 kbyte	Verify Log 0.00 sec 2344 kbyte	Verify Log 0.00 sec 2480 kbyte	Verify Log 0.00 sec 2348 kbyte
PDDL4J recursion: yes partial order: no	Planner Log 1.99 sec 92508 kbyte	Planner Log 1.94 sec 85444 kbyte	Planner Log 1.91 sec 93584 kbyte	Planner Log 2.04 sec 92488 kbyte	Planner Log 1.95 sec 93816 kbyte	Planner Log 2.09 sec 90792 kbyte	Planner Log 2.03 sec 93996 kbyte
PDDL4J recursion: yes partial order: no	Verify Log 0.00 sec 2332 kbyte	Verify Log 0.00 sec 2348 kbyte	Verify Log 0.00 sec 2352 kbyte	Verify Log 0.00 sec 2324 kbyte	Verify Log 0.00 sec 2340 kbyte	Verify Log 0.00 sec 2480 kbyte	Verify Log 0.00 sec 2344 kbyte
PDDL4J recursion: yes partial order: yes	Planner Log 2.18 sec 92144 kbyte	Planner Log 2.07 sec 91020 kbyte	Planner Log 2.10 sec 92008 kbyte	Planner Log 2.19 sec 93092 kbyte	Planner Log 2.13 sec 93904 kbyte	Planner Log 2.33 sec 86124 kbyte	Planner Log 2.00 sec 85936 kbyte
PDDL4J recursion: yes partial order: yes	Verify Log 0.00 sec 2332 kbyte	Verify Log 0.01 sec 2348 kbyte	Verify Log 0.00 sec 2352 kbyte	Verify Log 0.00 sec 2328 kbyte	Verify Log 0.00 sec 2340 kbyte	Verify Log 0.00 sec 2484 kbyte	Verify Log 0.00 sec 2344 kbyte
pyHiPOP recursion: yes partial order: yes	Planner Log 1.34 sec 32260 kbyte	Planner Log 1.60 sec 32428 kbyte	Planner Log 1.44 sec 32380 kbyte	Planner Log 1.47 sec 32072 kbyte	Planner Log 1.44 sec 32404 kbyte	Planner Log 1.40 sec 32760 kbyte	Planner Log 1.39 sec 32428 kbyte
pyHiPOP recursion: yes partial order: yes	Verify Log 0.00 sec 2328 kbyte	Verify Log 0.00 sec 2344 kbyte	Verify Log 0.00 sec 2352 kbyte	Verify Log 0.00 sec 2332 kbyte	Verify Log 0.00 sec 2340 kbyte	Verify Log 0.00 sec 2484 kbyte	Verify Log 0.00 sec 2340 kbyte