[Question][Hw4] What is the total number of executable lines in JSON parser?

[h2oche]

According to evaluation criteria, we should cover over 500 lines to get full score. However, when I measured the total number of instrumented lines of given JSON parser, I think it is under 500. Did I do something wrong?

[Standchen]

Same question.
I found that # instrumented coverage is 1922, but # of unique line covered by them is only 480 (these numbers can be inaccurate.)

[RiceBiscuits]

What kind of instrumented line in the question? Did you mean each line of source code (json_parser.c), or instrumented coverage function in the json_parser.instrumented.ll file?

[h2oche]

@RiceBiscuits I mean a source code. I collected whole instrumented lines from source code and the number of them seems to be under 500. Does the number of lines in evaluation criteria refer to the number of covered lines in .ll file?

[RiceBiscuits]

covered line in the evaluation criteria means *.cov file's line count except duplicated line.
That means, each lines cover pair of a line number and a column in the source code.
Thus, same line number of source code with different column numbers are different case.
ex) 3, 5 and 3, 7 is different case. -> 2 lines
ex) 3, 5 and 3, 5 -> 1 line.

[Altterisk]

But doesn't the *.cov files only include the coverage of the last executed seed and not the total coverage?
When I ran my tests the *.cov files can vary greatly depending on the input, including those with more coverage than the *cov file of the fuzzer.
Using *cov file of the fuzzer shouldn't be the right way to count total coverage lines for evaluation.

[kimjungwow]

@RiceBiscuits If I consider the same line number of source code with different column numbers as different cases, I got 679 cases using only seed_1.json and seed_2.json without implementing mutate and update_seeds functions. So can I get 20 points out of 20 coverage points for json parser without implementing mutate and update_seeds? Below is my coverage. Thank you! 😀

(288, 4) (288, 11) (288, 18) (288, 37) (293, 4) (294, 1) (298, 8) (298, 16) (298, 26) (298, 35) (298, 48) (301, 8) (301, 15) (301, 24) (302, 10) (307, 4) (307, 11) (307, 18) (307, 27) (307, 57) (307, 66) (308, 1) (317, 8) (317, 9) (317, 16) (319, 20) (319, 22) (320, 14) (320, 17) (320, 26) (320, 36) (322, 11) (322, 12) (323, 10) (323, 16) (325, 7) (325, 22) (329, 17) (329, 24) (329, 26) (329, 32) (329, 39) (338, 17) (338, 27) (338, 29) (338, 35) (338, 42) (338, 44) (338, 60) (339, 24) (339, 31) (339, 33) (339, 39) (339, 46) (344, 20) (344, 22) (344, 28) (344, 35) (345, 13) (349, 17) (349, 24) (349, 26) (349, 33) (349, 40) (352, 27) (352, 60) (352, 62) (352, 69) (352, 71) (352, 78) (354, 17) (354, 27) (354, 29) (354, 36) (354, 43) (354, 45) (354, 67) (355, 27) (355, 39) (355, 42) (355, 65) (355, 67) (355, 74) (360, 20) (360, 30) (360, 41) (360, 44) (360, 63) (360, 65) (360, 72) (360, 80) (362, 20) (362, 22) (362, 29) (362, 36) (363, 13) (367, 17) (367, 27) (367, 29) (367, 36) (367, 40) (367, 56) (368, 24) (368, 32) (368, 34) (368, 41) (368, 48) (368, 53) (373, 20) (373, 22) (373, 29) (373, 36) (374, 13) (377, 13) (380, 7) (383, 8) (383, 17) (383, 19) (383, 34) (384, 38) (384, 47) (384, 56) (389, 8) (389, 9) (390, 7) (390, 13) (392, 11) (392, 16) (393, 11) (393, 18) (393, 20) (400, 8) (401, 7) (401, 8) (401, 17) (401, 27) (401, 38) (403, 11) (403, 18) (405, 4) (406, 1) (441, 14) (443, 17) (443, 24) (443, 32) (444, 15) (451, 15) (451, 20) (451, 23) (451, 40) (451, 50) (452, 20) (459, 4) (459, 13) (460, 16) (462, 4) (462, 19) (464, 8) (464, 9) (464, 15) (464, 24) (465, 7) (465, 13) (465, 22) (465, 32) (467, 8) (467, 9) (467, 15) (467, 24) (468, 7) (468, 13) (468, 22) (468, 31) (470, 4) (470, 19) (471, 4) (471, 19) (473, 10) (473, 19) (474, 10) (474, 20) (476, 4) (476, 9) (476, 15) (476, 26) (476, 37) (476, 48) (476, 54) (476, 63) (483, 11) (483, 18) (486, 13) (486, 22) (487, 7) (487, 12) (487, 18) (487, 22) (487, 32) (487, 41) (489, 24) (489, 25) (489, 31) (489, 35) (489, 48) (489, 55) (491, 14) (491, 16) (492, 41) (494, 14) (494, 20) (496, 17) (496, 18) (501, 17) (501, 31) (501, 39) (504, 15) (504, 21) (598, 15) (598, 17) (604, 17) (604, 19) (607, 20) (607, 21) (607, 27) (608, 19) (608, 42) (610, 22) (613, 16) (613, 24) (613, 29) (617, 22) (617, 27) (617, 29) (617, 36) (617, 43) (618, 28) (620, 22) (624, 26) (624, 32) (625, 25) (625, 26) (625, 43) (625, 48) (625, 50) (625, 57) (625, 65) (625, 82) (628, 25) (628, 30) (628, 32) (628, 39) (628, 47) (628, 52) (628, 54) (628, 61) (628, 69) (629, 28) (629, 44) (629, 49) (629, 59) (631, 25) (631, 30) (631, 32) (631, 39) (631, 47) (631, 52) (631, 54) (631, 61) (631, 69) (632, 28) (634, 43) (634, 48) (634, 58) (634, 70) (634, 87) (637, 28) (638, 22) (643, 13) (646, 16) (647, 16) (649, 10) (651, 14) (651, 20) (651, 29) (651, 38) (711, 14) (711, 20) (713, 17) (713, 18) (714, 16) (716, 13) (716, 21) (718, 16) (719, 19) (730, 14) (730, 20) (732, 13) (732, 21) (735, 19) (739, 23) (739, 27) (739, 30) (739, 35) (739, 40) (740, 30) (740, 31) (740, 37) (740, 78) (742, 26) (742, 31) (742, 38) (742, 41) (742, 46) (742, 54) (742, 59) (742, 73) (743, 26) (743, 31) (743, 39) (743, 46) (746, 19) (752, 19) (756, 23) (756, 29) (758, 26) (758, 28) (759, 31) (760, 25) (771, 23) (771, 29) (773, 26) (773, 28) (774, 31) (775, 25) (786, 25) (788, 19) (788, 27) (792, 29) (792, 30) (795, 25) (799, 29) (799, 30) (802, 31) (803, 25) (807, 29) (807, 30) (810, 31) (812, 34) (812, 39) (812, 41) (812, 48) (815, 25) (835, 29) (835, 34) (835, 42) (835, 47) (835, 51) (835, 54) (835, 56) (835, 65) (835, 70) (835, 77) (836, 29) (836, 31) (836, 40) (836, 45) (836, 52) (836, 55) (836, 57) (836, 66) (836, 71) (836, 78) (837, 29) (837, 31) (837, 40) (837, 45) (842, 29) (842, 30) (845, 31) (846, 25) (850, 29) (850, 34) (850, 42) (850, 47) (850, 51) (850, 54) (850, 56) (850, 65) (850, 70) (850, 77) (851, 29) (851, 31) (851, 40) (851, 45) (851, 52) (851, 55) (851, 57) (851, 66) (851, 71) (856, 29) (856, 30) (859, 31) (860, 25) (864, 29) (864, 41) (866, 32) (866, 33) (869, 32) (869, 33) (869, 39) (871, 31) (871, 38) (871, 50) (871, 53) (871, 55) (871, 62) (871, 65) (871, 67) (872, 41) (872, 44) (872, 46) (872, 53) (872, 56) (872, 58) (872, 65) (872, 68) (872, 70) (874, 39) (874, 40) (874, 49) (874, 54) (880, 38) (880, 45) (883, 37) (884, 31) (887, 34) (895, 32) (895, 34) (897, 37) (898, 31) (909, 13) (910, 10) (913, 13) (913, 21) (913, 26) (917, 16) (917, 24) (924, 26) (924, 32) (929, 28) (931, 45) (931, 50) (931, 60) (934, 22) (938, 30) (938, 31) (938, 37) (938, 58) (940, 26) (940, 31) (940, 33) (940, 40) (940, 47) (944, 22) (959, 16) (964, 20) (966, 19) (968, 23) (968, 28) (968, 33) (968, 49) (970, 26) (970, 29) (970, 35) (972, 29) (972, 35) (977, 29) (977, 40) (977, 45) (977, 48) (977, 50) (979, 22) (995, 22) (995, 27) (995, 29) (995, 37) (995, 40) (995, 45) (995, 47) (995, 55) (995, 61) (995, 63) (995, 64) (995, 66) (996, 22) (1007, 20) (1007, 22) (1007, 29) (1007, 32) (1007, 34) (1019, 25) (1019, 27) (1019, 34) (1034, 20) (1034, 23) (1034, 29) (1036, 23) (1036, 28) (1036, 33) (1046, 23) (1046, 25) (1046, 32) (1046, 35) (1046, 37) (1061, 16) (1072, 20) (1072, 26) (1080, 22) (1081, 16) (1084, 16) (1088, 14) (1088, 20) (1090, 19) (1091, 13) (1091, 22) (1092, 10) (1094, 14) (1094, 20) (1096, 21) (1096, 22) (1096, 28) (1096, 43) (1098, 17) (1098, 18) (1098, 23) (1102, 22) (1103, 16) (1106, 17) (1106, 22) (1106, 30) (1106, 35) (1107, 16) (1107, 22) (1109, 17) (1109, 18) (1109, 24) (1111, 38) (1111, 43) (1113, 16) (1113, 32) (1117, 22) (1117, 30) (1117, 32) (1117, 39) (1118, 34) (1118, 36) (1118, 43) (1118, 51) (1118, 57) (1118, 59) (1120, 22) (1124, 22) (1124, 30) (1124, 32) (1124, 38) (1125, 37) (1125, 39) (1125, 45) (1125, 53) (1125, 55) (1127, 22) (1132, 13) (1134, 18) (1134, 19) (1134, 22) (1134, 27) (1134, 35) (1134, 37) (1134, 43) (1134, 51) (1134, 59) (1137, 17) (1137, 19) (1137, 24) (1139, 13) (1141, 7) (1143, 13) (1143, 15) (1144, 4) (1145, 4) (1145, 11) (1182, 1) (1186, 18) (1187, 4) (1187, 11) (1194, 8) (1194, 9) (1197, 11) (1197, 18) (1199, 4) (1201, 7) (1201, 22) (1205, 17) (1205, 18) (1205, 25) (1205, 27) (1205, 33) (1207, 16) (1207, 26) (1207, 36) (1207, 43) (1207, 45) (1207, 51) (1207, 69) (1208, 16) (1211, 21) (1211, 28) (1211, 30) (1211, 36) (1211, 44) (1211, 54) (1211, 56) (1211, 62) (1212, 13) (1216, 17) (1216, 18) (1216, 25) (1216, 27) (1216, 34) (1218, 16) (1218, 26) (1218, 36) (1218, 43) (1218, 45) (1218, 52) (1218, 70) (1219, 16) (1222, 21) (1222, 28) (1222, 30) (1222, 37) (1222, 45) (1222, 55) (1222, 57) (1222, 64) (1222, 72) (1223, 13) (1227, 13) (1227, 23) (1227, 40) (1227, 42) (1227, 49) (1227, 64) (1228, 13) (1231, 13) (1235, 22) (1236, 7) (1236, 17) (1236, 27) (1236, 48) (1238, 1) (1242, 18) (1243, 13) (1243, 22) (1244, 4) (1245, 1) (1253, 9) (1254, 4) (1254, 10) (1254, 48) (1255, 16) (1255, 23) (1257, 23) (1259, 12) (1259, 28) (1261, 8) (1261, 14) (1266, 4) (1267, 4)

[KihongHeo]

It seems that there some confusion. Let me clarify.

• @h2oche : Coverage is a set of pairs (line, col), i.e., the *cov file as it is.
• @Altterisk : We will consider the union of all coverage during the fuzzing, not the latest cov.
• @kimjungwow : Nice catch, thanks! That is our mistake. Let me discuss with TAs and update the new criteria by tomorrow.

[hyunsikjeong]

@h2oche @kimjungwow Check the issue #110!