Datasets


We created essentially two sets of data:

Note that the plans can also be used for other purposes, such as for research in plan verification, or correcting invalid plans.

Plans

In addition to the logs, we also offer the plans that were generated by the planners in the IPC as a resource. This can for example be used for research in plan verification. In addition to plans from the IPC planners, we have also gathered plans from runs of other planners (mostly the PANDA planner). We have removed all duplicate plans, i.e. when the same plan was generated by multiple planners. The current version of the plans is available on github (this repository will soon be replaced by another). We will update the repository from time-to-time as new plans become available.

In total we provide:

Log Files

For both tracks of the IPC, you can download a full dataset and a reduced dataset. Note that all experiments were executed 10 times with different seeds. Per run, we defined a base seed. For every run, the actually used seed was the base seed plus the instance number. The full dataset contains the log files for all planners on every instance. Since this is a lot of data we also created a reduced data set.

Problem Instance Renaming

In the logs provided below, every instance is referred to with a numerical code. For technical reasons, the instance numbers are separated between the partial-order and the total-order tracks and partially overlap! After the experiments were run, the structure of the domain repository was changed to allow for the usage of LAB. As such, the file names of the instances in the log files may not be correct anymore. The following two files contain a mapping of instance numbers to the instance files. The paths refer to the github repository.

Full Data Set

Every log contains the full output of the planner and the output of the subsequent verification of the found plan. In some cases, the verification took longer than the maximum time-limit of the job on our compute cluster (primaliry very long plans in the Towers domain). All such plans were valid. The planners are identified by a number which is shown in the table of planner above. The file names of the full logs comprise several information. The most notable one is the last number before the long sequence of dashs: it is the instance number (see above).

Reduced Data Set

In addition, we offer a reduced dataset. This dataset contains one line per run of a planner in which it found a solution (not necessarily a correct one). Each line contains 6 entries:

The reduced data sets can be found here:

Detailed Logs per Planner

The tables below show the status information on all competitors of the 2020 IPC, including those who dropped out before its official start. (Note that this page was used in the early stages of the IPC so that competitors could see whether their planner had issues or not, so they could fix them before the actual competition started. The page was thus continuously updated, but is now static and won't change anymore.)

For each competitor we show whether the planner currently compiles and whether it solves a given set of sample problems. The logs of the compilation and every run are available for download by clicking on the blue links.

The test instances can be downloaded by clicking at the columns headers.

Each of the six sample problems is divided into two sections. The one on the top shows whether the planner is able to find a plan (green) or not (red). The second section below shows whether the generated plan is valid. Green indicates that the plan is valid, red that it is not, and blue that no plan was found in the first place. If the planner produced only a primitive plan and did not provide a decomposition witness, we will indicate with cyan that the primitive plan is executable. Yellow indicates instances that the planner does not support some feature of the respective problem instance (either due to recursion or partial order).

For the last column, we let each planner run for 10 seconds on a larger set of benchmark instances.

As a summary per domain we report:

Note that INVAL here also includes cases where the planner produced only a primitive plan which was valid.

CompetitorCompile Status rover01 satellite01 um-translog01 transport01 p-rover01 p-transport01 10 seconds run (SOL,INVAL,TLE)
Lilotane
recursion: yes
partial order: no
Compile Log
495.34 sec
519912 kbyte
Planner Log
0.45 sec
24312 kbyte
Planner Log
0.36 sec
24732 kbyte
Planner Log
0.50 sec
24552 kbyte
Planner Log
0.38 sec
24832 kbyte
not supported not supported Miconic:
7 / 0 / 0
Zenotravel:
5 / 0 / 0
UM-Translog:
22 / 0 / 0
Satellite:
25 / 0 / 0
Woodworking:
11 / 0 / 0
SmartPhone:
7 / 0 / 0
rover:
19 / 0 / 1
transport:
28 / 0 / 2
Planner Log
Verify Log
0.00 sec
3292 kbyte
Verify Log
0.00 sec
2492 kbyte
Verify Log
0.01 sec
4664 kbyte
Verify Log
0.01 sec
2700 kbyte
no plan no plan
HPDL-planner
recursion: yes
partial order: yes
Compile Log
210.51 sec
435944 kbyte
Planner Log
0.67 sec
24228 kbyte
Planner Log
0.68 sec
24084 kbyte
Planner Log
0.76 sec
24136 kbyte
Planner Log
0.91 sec
24068 kbyte
Planner Log
0.76 sec
24476 kbyte
Planner Log
0.66 sec
24220 kbyte
Miconic:
7 / 0 / 0
Zenotravel:
5 / 0 / 0
UM-Translog:
22 / 0 / 0
Satellite:
25 / 0 / 0
Woodworking:
3 / 0 / 8
SmartPhone:
2 / 0 / 5
rover:
13 / 0 / 7
transport:
1 / 0 / 29
Planner Log
Verify Log
0.00 sec
3372 kbyte
Verify Log
0.00 sec
2500 kbyte
Verify Log
0.01 sec
4668 kbyte
Verify Log
0.00 sec
2704 kbyte
Verify Log
0.01 sec
3368 kbyte
Verify Log
0.00 sec
2704 kbyte
HyperTensioN
recursion: yes
partial order: no
Compile Log
31.81 sec
186360 kbyte
Planner Log
0.37 sec
24704 kbyte
Planner Log
0.36 sec
24784 kbyte
Planner Log
0.38 sec
25244 kbyte
Planner Log
0.41 sec
24884 kbyte
not supported not supported Miconic:
7 / 0 / 0
Zenotravel:
5 / 0 / 0
UM-Translog:
22 / 0 / 0
Satellite:
25 / 0 / 0
Woodworking:
6 / 0 / 5
SmartPhone:
7 / 0 / 0
rover:
19 / 0 / 1
transport:
30 / 0 / 0
Planner Log
Verify Log
0.01 sec
3932 kbyte
Verify Log
0.00 sec
2492 kbyte
Verify Log
0.01 sec
4668 kbyte
Verify Log
0.00 sec
2704 kbyte
no plan no plan
PDDL4J
recursion: yes
partial order: no
Compile Log
183.17 sec
711248 kbyte
Planner Log
2.27 sec
99224 kbyte
Planner Log
2.18 sec
94976 kbyte
Planner Log
2.22 sec
107300 kbyte
Planner Log
2.01 sec
96372 kbyte
not supported not supported Miconic:
7 / 0 / 0
Zenotravel:
5 / 0 / 0
UM-Translog:
22 / 0 / 0
Satellite:
25 / 0 / 0
Woodworking:
6 / 0 / 5
SmartPhone:
0 / 0 / 7
rover:
20 / 0 / 0
transport:
28 / 0 / 2
Planner Log
Verify Log
0.00 sec
3212 kbyte
Verify Log
0.00 sec
2492 kbyte
Verify Log
0.02 sec
4668 kbyte
Verify Log
0.00 sec
2708 kbyte
no plan no plan
PDDL4J
recursion: yes
partial order: yes
Compile Log
219.01 sec
705092 kbyte
Planner Log
2.13 sec
100244 kbyte
Planner Log
2.20 sec
93680 kbyte
Planner Log
2.23 sec
99680 kbyte
Planner Log
2.43 sec
94628 kbyte
Planner Log
2.22 sec
91740 kbyte
Planner Log
2.22 sec
94964 kbyte
Miconic:
7 / 0 / 0
Zenotravel:
5 / 0 / 0
UM-Translog:
22 / 0 / 0
Satellite:
25 / 0 / 0
Woodworking:
5 / 0 / 6
SmartPhone:
0 / 0 / 7
rover:
20 / 0 / 0
transport:
26 / 0 / 4
Planner Log
Verify Log
0.00 sec
3216 kbyte
Verify Log
0.00 sec
2496 kbyte
Verify Log
0.01 sec
4668 kbyte
Verify Log
0.00 sec
2704 kbyte
Verify Log
0.00 sec
3240 kbyte
Verify Log
0.00 sec
2700 kbyte
pyHiPOP
recursion: yes
partial order: yes
Compile Log
102.70 sec
416480 kbyte
Planner Log
5.51 sec
69188 kbyte
Planner Log
2.53 sec
38240 kbyte
Planner Log
57.21 sec
1539936 kbyte
Planner Log
2.15 sec
44764 kbyte
Planner Log
60.13 sec
24968 kbyte
Planner Log
2.53 sec
47664 kbyte
Miconic:
2 / 0 / 5
Zenotravel:
4 / 0 / 1
UM-Translog:
17 / 0 / 5
Satellite:
11 / 0 / 14
Woodworking:
2 / 0 / 9
SmartPhone:
4 / 0 / 3
rover:
4 / 0 / 16
transport:
1 / 0 / 29
Planner Log
Verify Log
0.01 sec
3936 kbyte
Verify Log
0.00 sec
2492 kbyte
Verify Log
0.02 sec
4668 kbyte
Verify Log
0.00 sec
2704 kbyte
no plan Verify Log
0.00 sec
2700 kbyte





Competitor test01 test02 test03 test04 test05 test06 test07
empty method forall forall2 no abstracts constants in domain synonymes arguments
Lilotane
recursion: yes
partial order: no
Planner Log
0.46 sec
24684 kbyte
Planner Log
0.37 sec
25156 kbyte
Planner Log
0.37 sec
24612 kbyte
Planner Log
0.34 sec
24740 kbyte
Planner Log
0.37 sec
24880 kbyte
Planner Log
0.33 sec
24548 kbyte
Planner Log
0.38 sec
24524 kbyte
Verify Log
0.89 sec
2332 kbyte
Verify Log
0.05 sec
2344 kbyte
Verify Log
0.01 sec
2356 kbyte
Verify Log
0.00 sec
2328 kbyte
Verify Log
0.00 sec
2340 kbyte
Verify Log
0.00 sec
2480 kbyte
Verify Log
0.00 sec
2344 kbyte
HPDL-planner
recursion: yes
partial order: yes
Planner Log
0.78 sec
24836 kbyte
Planner Log
0.63 sec
24284 kbyte
Planner Log
0.60 sec
24248 kbyte
Planner Log
0.61 sec
24236 kbyte
Planner Log
0.63 sec
24556 kbyte
Planner Log
0.59 sec
23868 kbyte
Planner Log
0.63 sec
23996 kbyte
Verify Log
0.17 sec
2312 kbyte
Verify Log
0.01 sec
2344 kbyte
Verify Log
0.00 sec
2356 kbyte
Verify Log
0.00 sec
2328 kbyte
Verify Log
0.00 sec
2344 kbyte
Verify Log
0.00 sec
2484 kbyte
Verify Log
0.00 sec
2348 kbyte
HyperTensioN
recursion: yes
partial order: no
Planner Log
0.37 sec
24496 kbyte
Planner Log
0.34 sec
24196 kbyte
Planner Log
0.36 sec
24288 kbyte
Planner Log
0.35 sec
24584 kbyte
Planner Log
0.38 sec
24672 kbyte
Planner Log
0.36 sec
24964 kbyte
Planner Log
0.37 sec
25188 kbyte
Verify Log
0.00 sec
2336 kbyte
Verify Log
0.00 sec
2344 kbyte
Verify Log
0.00 sec
2352 kbyte
Verify Log
0.00 sec
2332 kbyte
Verify Log
0.00 sec
2344 kbyte
Verify Log
0.00 sec
2480 kbyte
Verify Log
0.00 sec
2348 kbyte
PDDL4J
recursion: yes
partial order: no
Planner Log
1.99 sec
92508 kbyte
Planner Log
1.94 sec
85444 kbyte
Planner Log
1.91 sec
93584 kbyte
Planner Log
2.04 sec
92488 kbyte
Planner Log
1.95 sec
93816 kbyte
Planner Log
2.09 sec
90792 kbyte
Planner Log
2.03 sec
93996 kbyte
Verify Log
0.00 sec
2332 kbyte
Verify Log
0.00 sec
2348 kbyte
Verify Log
0.00 sec
2352 kbyte
Verify Log
0.00 sec
2324 kbyte
Verify Log
0.00 sec
2340 kbyte
Verify Log
0.00 sec
2480 kbyte
Verify Log
0.00 sec
2344 kbyte
PDDL4J
recursion: yes
partial order: yes
Planner Log
2.18 sec
92144 kbyte
Planner Log
2.07 sec
91020 kbyte
Planner Log
2.10 sec
92008 kbyte
Planner Log
2.19 sec
93092 kbyte
Planner Log
2.13 sec
93904 kbyte
Planner Log
2.33 sec
86124 kbyte
Planner Log
2.00 sec
85936 kbyte
Verify Log
0.00 sec
2332 kbyte
Verify Log
0.01 sec
2348 kbyte
Verify Log
0.00 sec
2352 kbyte
Verify Log
0.00 sec
2328 kbyte
Verify Log
0.00 sec
2340 kbyte
Verify Log
0.00 sec
2484 kbyte
Verify Log
0.00 sec
2344 kbyte
pyHiPOP
recursion: yes
partial order: yes
Planner Log
1.34 sec
32260 kbyte
Planner Log
1.60 sec
32428 kbyte
Planner Log
1.44 sec
32380 kbyte
Planner Log
1.47 sec
32072 kbyte
Planner Log
1.44 sec
32404 kbyte
Planner Log
1.40 sec
32760 kbyte
Planner Log
1.39 sec
32428 kbyte
Verify Log
0.00 sec
2328 kbyte
Verify Log
0.00 sec
2344 kbyte
Verify Log
0.00 sec
2352 kbyte
Verify Log
0.00 sec
2332 kbyte
Verify Log
0.00 sec
2340 kbyte
Verify Log
0.00 sec
2484 kbyte
Verify Log
0.00 sec
2340 kbyte