qemurr: create

qemu: start documenting unmerged replay reverse debugging while testing it

Author: cirosantilli

README.adoc

@@ -1665,7 +1665,13 @@ And then search for a line of type:
 
 === GDB step debug early boot
 
-TODO: why can't we break at early startup stuff such as:
+Break at the very first instruction executed by QEMU:
+
+....
+./rungdb -C
+....
+
+TODO why can't we break at early startup stuff such as:
 
 ....
 ./rungdb extract_kernel
@@ -6096,11 +6102,17 @@ Only tested in x86.
 
 PCI driver for our minimal `pci_min.c` QEMU fork device:
 
+....
+./run -- -device lkmc_pci_min
+....
+
+then:
+
 ....
 insmod /pci_min.ko
 ....
 
-Source:
+Sources:
 
 * Kernel module: link:kernel_module/pci_min.c[].
 * QEMU device: https://github.com/cirosantilli/qemu/blob/lkmc/hw/misc/lkmc_pci_min.c
@@ -6124,11 +6136,7 @@ What happened:
 
 Kernel messages and printks from inside QEMU are shown all together, to see that more clearly, run in <<graphic-mode>> instead.
 
-Works because we add to our default QEMU CLI:
-
-....
--device lkmc_pci_min
-....
+We don't enable the device by default because it does not work for vanilla QEMU, which we often want to test with this repository.
 
 Probe already does a MMIO write, which generates an IRQ and tests everything.
 
@@ -7022,6 +7030,12 @@ This awesome feature allows you to examine a single run as many times as you wou
 ./run -F '/rand_check.out;/poweroff.out;' -R
 ....
 
+A convenient shortcut to do both at once to test the feature is:
+
+....
+./qemurr -F '/rand_check.out;/poweroff.out;'
+....
+
 By comparing the terminal output of both runs, we can see that they are the exact same, including things which normally differ across runs:
 
 * timestamps of dmesg output
@@ -7046,8 +7060,7 @@ EXT4-fs (sda): re-mounted. Opts: block_validity,barrier,user_xattr
 TODO replay with network gets stuck:
 
 ....
-./run -F '/sbin/ifup -a;wget -S google.com;/poweroff.out;' -r
-./run -F '/sbin/ifup -a;wget -S google.com;/poweroff.out;' -R
+./qemurr -F '/sbin/ifup -a;wget -S google.com;/poweroff.out;'
 ....
 
 after the message:
@@ -7058,14 +7071,15 @@ adding dns 10.0.2.3
 
 There is explicit network support on the QEMU patches, but either it is buggy or we are not using the correct magic options.
 
+Solved on unmerged c42634d8e3428cfa60672c3ba89cabefc720cde9 from https://github.com/ispras/qemu/tree/rr-180725
+
 TODO `arm` and `aarch64` only seem to work with initrd since I cannot plug a working IDE disk device? See also: https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg05245.html
 
 Then, when I tried with <<initrd>> and no disk:
 
 ....
 ./build -aA -i
-./run -aA -F '/rand_check.out;/poweroff.out;' -i -r
-./run -aA -F '/rand_check.out;/poweroff.out;' -i -R
+./qemurr -aA -F '/rand_check.out;/poweroff.out;' -i
 ....
 
 QEMU crashes with:
@@ -7076,6 +7090,37 @@ ERROR:replay/replay-time.c:49:replay_read_clock: assertion failed: (replay_file
 
 I had the same error previously on x86-64, but it was fixed: https://bugs.launchpad.net/qemu/+bug/1762179 so maybe the forgot to fix it for `aarch64`?
 
+Solved on unmerged c42634d8e3428cfa60672c3ba89cabefc720cde9 from https://github.com/ispras/qemu/tree/rr-180725
+
+===== QEMU reverse debugging
+
+TODO get working.
+
+QEMU replays support checkpointing, and this allows for a simplistic "reverse debugging" implementation proposed at https://lists.gnu.org/archive/html/qemu-devel/2018-06/msg00478.html on the unmerged link:https://github.com/ispras/qemu/tree/rr-180725[]:
+
+....
+./run -F '/rand_check.out;/poweroff.out;' -r
+./run -F '/rand_check.out;/poweroff.out;' -R -d
+....
+
+On another shell:
+
+....
+./rungdb start_kernel
+....
+
+In GDB:
+
+....
+n
+n
+n
+n
+reverse-continue
+....
+
+and we are back at `start_kernel`
+
 ==== QEMU trace multicore
 
 TODO: is there any way to distinguish which instruction runs on each core? Doing:

br2/default

@@ -28,6 +28,7 @@ BR2_PACKAGE_FILE=y
 BR2_PACKAGE_OVERRIDE_FILE="../buildroot_override"
 BR2_PACKAGE_PCIUTILS=y
 # For qemu-ga on guest. TODO: do something with it, and document it.
+# Maybe: https://superuser.com/questions/930588/how-to-pass-commands-noninteractively-to-running-qemu-from-the-guest-qmp-via-te
 BR2_PACKAGE_QEMU=y
 BR2_PACKAGE_STRACE=y
 BR2_ROOTFS_OVERLAY="../rootfs_overlay"

qemurr

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -eu
+./run -r "$@"
+./run -R "$@"

run

@@ -270,7 +270,7 @@ else
     common_mkdir
     qemu_executable="qemu-system-${arch}"
   else
-    qemu_executable="${buildroot_out_dir}/host/usr/bin/qemu-system-${arch}"
+    qemu_executable="${common_qemu_variant_dir}/${arch}-softmmu/qemu-system-${arch}"
   fi
   extra_flags="${extra_flags_qemu} ${extra_flags}"
   qemu_common="\
@@ -303,12 +303,14 @@ ${vnc}"
         driveif=none
         rrid=',id=img-direct'
         root='root=/dev/sda'
+        snapshot=
       else
         driveif=virtio
         root='root=/dev/vda'
         rrid=
+        snapshot=,snapshot
       fi
-      extra_flags="${extra_flags} -drive 'file=${common_images_dir}/rootfs.ext2.qcow2,format=qcow2,if=${driveif},snapshot${rrid}' \\
+      extra_flags="${extra_flags} -drive 'file=${common_images_dir}/rootfs.ext2.qcow2,format=qcow2,if=${driveif}${snapshot}${rrid}' \\
 "
       if [ -n "$rr" ]; then
         extra_flags="${extra_flags} \\
@@ -323,25 +325,22 @@ ${vnc}"
     extra_flags="${extra_flags} \
 -object filter-replay,id=replay,netdev=net0 \\
 -icount 'shift=7,rr=${rr},rrfile=${common_qemu_rrfile}' \\
+"
+    virtio_gpu_pci=
+  else
+    virtio_gpu_pci="-device virtio-gpu-pci \\
 "
   fi
   case "$arch" in
     x86_64)
       if "$kgdb"; then
         extra_append="${extra_append} kgdboc=ttyS0,115200"
       fi
-      if "$prebuilt"; then
-        custom_devices=
-      else
-        custom_devices="-device lkmc_pci_min \\
-"
-      fi
       cmd="\
 ${qemu_common} \
 -M pc \\
 -append '${root} nopat ${extra_append}' \\
 -device edu \\
-${custom_devices} \\
 ${extra_flags} \
 "
     ;;
@@ -354,7 +353,7 @@ ${qemu_common} \
 -M virt \\
 -append '${root} ${extra_append}' \\
 -cpu cortex-a15 \\
--device virtio-gpu-pci \\
+${virtio_gpu_pci} \
 ${extra_flags} \
 "
     ;;
@@ -367,7 +366,7 @@ ${qemu_common} \
 -M virt \\
 -append '${root} ${extra_append}' \\
 -cpu cortex-a57 \\
--device virtio-gpu-pci \\
+${virtio_gpu_pci} \
 -kernel '${common_images_dir}/Image' \\
 ${extra_flags} \
 "
@@ -376,7 +375,7 @@ ${extra_flags} \
       if ! "$ramfs"; then
         root='root=/dev/hda'
         extra_flags="${extra_flags} \
--drive 'file=${common_images_dir}/rootfs.ext2.qcow2,format=qcow2,snapshot' \\
+-drive 'file=${common_images_dir}/rootfs.ext2.qcow2,format=qcow2${snapshot}' \\
 "
       fi
       cmd="\

rungdb

@@ -7,7 +7,8 @@ before=
 lx_symbols="-ex 'lx-symbols ../kernel_module-1.0/' \\
 "
 kgdb=false
-while getopts A:a:b:gkL:n:X OPT; do
+docontinue=true
+while getopts A:a:b:CgkL:n:X OPT; do
   case "$OPT" in
     A)
       after="$OPTARG"
@@ -18,6 +19,10 @@ while getopts A:a:b:gkL:n:X OPT; do
     b)
       before="$OPTARG"
       ;;
+    C)
+      # No Continue.
+      docontinue=false
+      ;;
     g)
       gem5=true
       ;;
@@ -84,7 +89,12 @@ ${gdb} \
 -ex 'file vmlinux' \\
 -ex 'target remote localhost:${common_gdb_port}' \\
 ${brk} \
--ex 'continue' \\
+"
+fi
+if "$docontinue"; then
+  echo asdf
+  cmd="${cmd} \
+-ex continue \\
 ${lx_symbols} \
 "
 fi