Dev 0.6 (#4)

* Fix config argument bug

* Fix packaging error that was only allowing lapidary to be run from the
directory itself. Now can be installed via pip.

* Cleaning up the checkpointing engine as I start working on keyframes.

- Adding some documentation (docstrings)
- Converting random print statements into log messages using the logger
modules.
- Also moved the LICENSE to the proper place in the directory hierarchy
so that it shows up on github.

* Fix bug in Spec17 plugin that wasn't allowing for customizable spec
locations.

- Spec2017Bench now overrides the SPEC17_ROOT variable in the makefile,
allowing spec to now build properly.
- Added a bit more documentation.

* Fixed the bug with early-termination in GDB

- It wasn't actually an issue with the signal handler like I thought. It
was actually that I wasn't catching a GDB error from parsing the "show
language" command. It is now fixed.
- Some other minor tweaks.

* Legacy bug cleanup.

Before we can merge 0.6, we need to do the following things:
- Get single simulate and parallel simulate working again (currently I
need to test after rebuilding gem5).
- Get keyframes fully implemented, which will dramatically increase
usability.

* Fixed single simulate

- Parallel still terminates early for some reason

* Parallel simulate is working again!

- Need to clean up the stdout/stderr interweaving.
- Should also clean up some of the logging infra

* Fixed stdout/stderr redirection

Tips and tricks:
- stdout/stderr should be in text mode (at least for python2.7, I think)
- need to seek to 0 to print
- can't use io.StringIO because python expects fileno() to be
implemented on stdout/stderr

* A bit of clean-up

* Add some keyframes scaffolding.

Author: iangneal

.gitignore

@@ -142,7 +142,9 @@ lapidary-code.code-workspace
 .vscode/ipch/681ea811cfbf718c/get_fs_base.ipch
 .vscode/ipch/681ea811cfbf718c/mmap_address.bin
 
-# Proprietary
+# Proprietary/testing related
 spec-cpu2017
 workspace_lapidary_spec17/
 *.bak
+log.txt
+simulation_results

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "gistsnip"]
 	path = gistsnip
 	url = https://github.com/loov/gistsnip.git
+[submodule "gem5_master"]
+	path = gem5_master
+	url = https://gem5.googlesource.com/public/gem5

.lapidary.yaml

@@ -1,4 +1,4 @@
-gem5_path: ./custom_gem5
+gem5_path: ./gem5_master
 gem5_features:
   syscall_trace: true
 
@@ -7,5 +7,5 @@ libc_path: ./libc/glibc/build/install
 gem5_flag_config_plugin: ./test/test_configurations.py
 
 spec2017_config:
-  spec2017_src_path: ./spec-cpu2017
+  spec2017_src_path: ../cooldown/spec-cpu2017
   workspace_path: ./workspace_lapidary_spec17

lapidary/LICENSE LICENSElapidary/LICENSE renamed to LICENSE

@@ -12,22 +12,40 @@
 from time import sleep
 
 from lapidary.utils import *
-from lapidary.checkpoint.Checkpoints import GDBCheckpoint
+from lapidary.checkpoint.Checkpoints import *
 
 class GDBCheckpointConverter:
 
-    def __init__(self, gdb_checkpoint):
+    def __init__(self, gdb_checkpoint, keyframe=None, diffs=None):
         assert isinstance(gdb_checkpoint, GDBCheckpoint)
         assert gdb_checkpoint.is_valid_checkpoint()
+        assert keyframe is None or isinstance(keyframe, Gem5Checkpoint)
         self.gdb_checkpoint = gdb_checkpoint
         self.mappings = self.gdb_checkpoint.get_mappings()
+        self.keyframe = keyframe
+        self.diff_frames = diffs
 
     @staticmethod
     def compress_memory_image(file_path):
+        '''
+            gem5 can accept either compressed or uncompressed pmem files 
+            (compressed with gzip), as long as they have the same file name.
+
+            The only downside is that simulations take longer to start.
+        '''
         subprocess.call(['gzip', '-f', str(file_path)])
         gzip_path = Path(str(file_path) + '.gz')
         gzip_path.rename(file_path)
 
+
+    def _create_diff_file(self, src_pmem_fp, new_pmem_fp):
+        if self.keyframe is None:
+            return new_pmem_file
+        
+        from diff_match_patch import diff_match_patch
+        dmp = diff_match_patch()
+
+
     def create_pmem_file(self):
 
         with self.gdb_checkpoint.get_pmem_file_handle() as pmem_raw,\

gem5_master

@@ -60,6 +60,14 @@ def __str__(self):
 
 
 class Gem5Checkpoint(GDBCheckpoint):
-
     def is_valid_checkpoint(self):
         return super().is_valid_checkpoint(self) and self.pmem_file.exists()
+
+class Gem5DiffCheckpoint(GDBCheckpoint):
+    
+    def __init__(self, checkpoint_directory):
+        super().__init__(self, checkpoint_directory)
+        self.pmem_diff_file = self.pmem_file + '.diff'
+    
+    def is_valid_checkpoint(self):
+        return super().is_valid_checkpoint(self) and self.pmem_diff_file.exists()