Evaluation of MetaDL, Error Prone and Spot Bugs

How to produce data:

• Download Defects4J, Error Prone and Spot Bugs

./scripts/download_defects4j.sh
./scripts/download_static_checkers.sh

• Copy over MetaDL

cp -r /path/to/metadl ./static-checkers

• Run the performance and analyisis quality:

./scripts/time_checkers_on_defects4j.sh

• The results are produced in /study/output-fixed
• Analyse the results:

python3  python/CompareMetaDL.py  study/output-fixed/mdl_parsed.json \
         study/output-fixed/ep_parsed.json study/output-fixed/sb_parsed.json

Data archive

MetaDL version used: 50ac5e08a5ef7f514a44dcf8065166abe2f3c248

Performance data for a source-only run of MetaDL

• data/oopsla2020/study-mdl-so/
• used for reporting on the performance of MetaDL-SO

Analysis quality data, with a full run of MetaDL

• data/oopsla2020/study-all
• used for reporting on relative preicison and recall of our analyses

Performance data for MetaDL(full), SpotBugs and ErrorProne

• data/oopsla2020/study-1
• data/oopsla2020/study-2
• data/oopsla2020/study-3
• used for report on the performance of MetaDL(full), SpotBugs and ErrorProne
• 3 runs to have some confidence in the stability of results