Feature/change detection (#43)

* multiple file uploads supported when authenticated
* change detection is working
* working through the bugs
* partial fix for bulk automation failing
* fixed issue with restarting automated job
* duplicate issue is resolved, again
* added scheduled automation monitoring to context menus
* fixing bugs
* wrap db insert in transaction. 
* Converted mhtml to html to improve viewability of artifacts
* mhtml to html on the fly added
* fixing render issues, added hash to attachment data structure
* remove unused file
* remove more dead code
* fixed deep save issue when using the context menu
* working on improving badge updates
* add timestamp to debug statement
* timestamp the last time the scheduled automation ran

Author: yooper

README.md

@@ -4,84 +4,132 @@
 
 # Your Rapport – Collect, Reflect, and Collaborate.
 
-Your Rapport is an open source "Chrome Extension" digital archiving and intelligence tool that collects online conversations using intelligent screenshot automation 
-and makes the content fully searchable for professionals, amateurs, and archivists who need to preserve, analyze, or audit digital 
-dialogue across platforms, Your Rapport bridges visual capture with text-based search, turning ephemeral interactions 
-into permanent, searchable records. Your Rapport is an Open Source tool that implements the best practices for 
-protecting your privacy and documenting online content. You can easily import, export, or print screenshots from 
-your collection. Your Rapport is free to use, but has a couple pro features you will need to pay for, eventually. 
-This enables us to continue development and support for this product. 
-
-Please consider supporting this project with a [pro license](https://buy.stripe.com/4gM5kDbRcgWW8d7gLedAk00).
-
-Check out the wiki for more in depth information. 
+Your Rapport is an open source digital archiving platform that runs entirely within your browser. It uses plugins written 
+in json to provide additional functionality. The functionality aids in adding flexibility without introducing lots of 
+custom code or complexity. This product is meant to simplify digital archiving by providing tools that can automate most
+of the collection process. 
 
 ### Getting Started 
 After installing the Chrome extension from https://chromewebstore.google.com/detail/your-rapport/clkaalonjdkliiaadkgodlfbiipidjmn, 
 "Your Rapport" will automatically be ready to collect.
 
-There are several options for collecting a screenshot of a web page:
- * `Mouse - Right Click` and select the **Autoscroll Collect** menu option with the "Your Rapport" logo
- * `Alt+S` collects a single screenshot and deep copy which is an mhtml file
+After installing, click the "Your Rapport" pin in your extension tab and select the action you want performed or,  
+
+There are several options and actions available when archiving data:
+ * `Alt+S` collects a single screenshot and deep copy which is a mhtml file and a html version of the web page.
  * `Alt+A` autoscroll and collect multiple screenshots, or stop the autoscroll.
  * `Alt+X` opens up the dashboard where you can search, print, share, or delete your collection. 
- * `Alt+Q` quick scan the web page for selectors and show the counts on the extension pin. 
- * Click the "Your Rapport" pin in your extension tab and select the action you want performed.  
-
-Your Rapport is an open source commercial tool for the following reasons:
- * Transparency in how software works and where the data goes is an important security and privacy concern to all of us
- * A commercial tool is the only viable way to support developing a standard set of open source tools, useful for doing online research
- * Keeps the infrastructure costs lower by not having additional overhead with privatized Software as a Service approach
- * Implement best practices based on community feedback
- * The target price for the Pro license can be set to $3 a month support us [here](https://buy.stripe.com/4gM5kDbRcgWW8d7gLedAk00) 
-
-🌟 Key Features
-
-📸 Smart Screenshot Capture
-Automatically takes screenshots of chats, posts, comments, and threads across the web.
+ * `Alt+Q` quick scan opens a sub-panel that displays data extracted insights from the live web page. 
+ * `Mouse - Right Click` gives several options depending upon what you click on within a web page
 
-🤖 Automated Bulk Screenshot Captures
-Your Rapport lets you provide a list of URLs to scrape. This saves you time by being able to scrape large amounts of
-data, while you work on other tasks. You can also right click on a link to add the URL to the automation queue for 
-future scraping.
 
-🔍 Searchable Text Extraction
-Uses advanced algorithms to extract text from screenshots, making every captured conversation searchable by keyword, 
-username, or phrase.
+Consider supporting this project for $3 a month, available through stripe [pro license](https://buy.stripe.com/4gM5kDbRcgWW8d7gLedAk00).
 
-📗 Quick Scan 
-Shows how many of your keyword selectors are contained on the web page within the Extension Pin. Making it
-easy to determine if the current page has any pertinent information
+[Change Log](https://github.com/yooper/your-rapport/wiki/Change-Log)
 
+[Wiki Docs](https://github.com/yooper/your-rapport/wiki)
 
-📚 Discovery Plugins
-Are small json scripts that allow you to link your selectors to specific websites for enrichment. Example, if you find a 
-phone number and have installed the Discovery Plugin from the Package Management screen, you will now be able to streamline
-searching against several of the data brokers. 
+Check out the wiki for more in depth information. Or review the source code in the github repo. 
 
-🌱 Companion Extensions
-Are chrome extensions that will provide improvements to your workflow. For example, "[Who Am I](https://chromewebstore.google.com/detail/who-am-i/gdnhlhadhgnhaenfcphpeakdghkccfoo)"
- a Chrome extension, provides username enumeration across a couple thousand websites. Who Am I has a button in its
-UI that will trigger the "Your Rapport" extension to scrape the selected social website with the click of a button. 
-
-💬 Platform-Agnostic
-Works across social media platforms, messaging apps (via web interfaces), forums, and more.
-
-🔐 Private & Secure
-Your captured content stays on your machine utilizing your browser's security and privacy protections.
-
-🔗 Share Your Rapport Collection 
-The data exported from Your Rapport is in a Non-Proprietary format known as JSON.
-Your collection can be downloaded as "json" file(s) which lets you easily share them with
-others using email or other file sharing apps. The screenshots can be individually downloaded, too. 
+# Key Features
+* Mostly Free
+* Archive thousands of web pages, images, articles, recipes, or other content from the web. 
+* Search using text, tags, selectors or dates to filter and dive into your digital archives.
+* Source code is Open Source
 
-🖨️ Printable Report(s)
-There is a lightweight print function that lets you download a PDF with the screenshot and metadata.
 
+# Technical and Product Roadmap 
+
+Below are the stages used to process collected information, commonly referred to as an ETL pipeline. This workflow is 
+designed to run entirely inside a Chrome extension. Discovery Plugins connect events to actions. For example, you might 
+save a web page, an MHTML file, an image, or a snippet of JavaScript and then trigger a scan by a remote service to 
+extract people, places, or names from the text, or run OCR to pull text from an image. The extension collects the 
+content through your browser and stores it locally. When you choose to send data to a remote service, you can also store
+the response locally so it becomes part of your archive. Below is a breakdown of each stage involved in processing 
+online content within the Chrome extension.
+
+* [Collecting](Collecting-Web-Content)
+  * Digital Media Content
+    * [Deep Save](Deep-Save)
+    * [Auto Collect](Auto-Collect)
+    * [Audio Collect](Audio-Collect) (Not Supported, yet)
+    * [Video Collect](Video-Collect) (Not Supported, yet)
+  * [Media Types](supported-collection-types)
+    * [Text Types](Text-Types)
+    * [Image Types](Image-Types)
+    * [Audio Types](Audio-Types) (Not Supported, yet)
+    * [Video Types](Video-Types) (Not Supported, yet)
+  * [Formats](Formats)
+    * [MHTML](MHTML)
+    * [HTML](HTML)
+    * [JSON](JSON)
+    * [Base64](Base64)
+  * [Automations](Setting-Up-And-Running-Automations)
+    * [Bulk Url Input](Your-Rapport-Bulk-Collection)
+    * [Scheduled Automations](Scheduled-Automations)
+* [Persistence](Persistence)
+  * [Indexeddb](IndexedDB) (In app database)
+  * [Importing](import-a-collection)
+    * [Uploading Dialog](Uploading-Dialog)
+    * [Auto Sync](Auto-Sync) (Not Available, yet)
+  * [Exporting](exporting-and-sharing-your-collection)
+    * [Sync to Local Disk](Sync-to-Local-Vault) (Pro Feature)
+    * [Sync to Remote Source](Sync-to-Remote-Source) (Available through Discovery Plugins)
+    * [Export to File](Export-to-File)
+  * [Change Data Capture](Change-Data-Capture)
+    * [Change Detection](Change-Detection)
+    * [Audit Logging](Audit-Logging) (Not Available, yet)
+* [Indexing](Indexing)
+  * [Search](Working-With-Your-Rapport-Collections-in-the-Search-Dashboard)
+    * [Free Text](Free-Text-Search)
+    * [Tags](Your-Rapport-Tag-Management)
+    * [Selectors](Your-Rapport-Selectors)
+    * [Domains](Domains)
+    * [Boolean Operators](Boolean-Operators) (Not Available, yet)
+  * [Schemas](Schemas)
+    * [Rapport](Rapport-Schema)
+    * [DiscoveryPlugin](DiscoveryPlugin-Schema)
+    * [Artifact](Artifact-Schema)
+* [Analysis](Analysis)
+  * [Merging Screenshots](Merging-Screenshots)
+  * [Quick Scan](Quick-Scan) 
+  * [SQL](SQL) (Not Available, yet)
+  * [Web Application Integrations](Web-Application-Integrations)
+  * [Chrome Extension Integrations](Chrome-Extension-Integrations)
+  * [Data Viewer](Data-Viewer)
+  * [Data Integrity Hash](Data-Integrity-Hash)
+* [Workflow](Workflow)
+  * [Api Keys](Api-Key-Management)
+  * [Discovery Plugins](discovery-plugins-tutorial)
+    * [Events](Discovery-Plugin-Events)
+      * [Create](Discovery-Plugin-Create)
+      * [Update](Discovery-Plugin-Update)
+      * [Delete](Discovery-Plugin-Delete)
+    * [Actions](Discovery-Plugin-Actions)
+      * [Create Tab](Create-Tab)
+      * [Submit Form](Submit-Form)
+      * [Foreground Runner](Foreground-Runner)
+      * [Background Runner](Background-Runner)
+  * [Case Management](Case-Management) (Not Available, yet)
+    * [Create](Case-Create)
+    * [Update](Case-Update)
+    * [Delete](Case-Delete)
+  * [Reporting](Reporting)
+    * [Basic Reports](Basic-Reports)
+    * [Dashboard Reports](Dashboard-Reports) (Not Available, yet)
+  * [Simplified Export](Simplified-Export-For-AI-Tools) (For AI Tools)
+
+
+Your Rapport is an Open Source tool that implements the best practices for 
+protecting your privacy and archiving online content. You can easily import, export, or print screenshots from 
+your collection. Your Rapport is free to use, but has a couple pro features you will need to pay for, such as local sync
+or advanced change detection algorithms when collecting data. This enables us to continue development and support for this product. 
 
-🧠 Best Data Integrity Practices, so far:
-* Signed Hash for verifying each screenshot's integrity
-* Multiple timestamps are collected
-* Multiple attributes about the computer that took the screenshot are captured
-* Source code is Open Source
+Your Rapport is an open source commercial tool for the following reasons:
+ * Transparency in how software works and where the data goes is an important security and privacy concern to all of us
+ * A commercial tool is the only viable way to support developing a standard set of open source tools, useful for doing online research or archiving
+ * Keeps the infrastructure costs lower by not having additional overhead with privatized Software as a Service approach
+ * Implement best practices based on community feedback
+ * Provide the data in multiple formats and make it easy to transfer to different locations
+ * The target price for the Pro license can be set to $3 a month support us [here](https://buy.stripe.com/4gM5kDbRcgWW8d7gLedAk00)