Feature: Extensions can add new devices (#1887)

Author: k9ert

docs/extensions.md

@@ -0,0 +1,11 @@
+
+# Callback methods
+
+Callback methods will get called from within Specter Desktop Core for various reasons. You can implement them in your service-class in order to react on those occasions.
+
+Checkout the `cryptoadvance.specter.services.callbacks` file for all the specific callbacks.
+
+Some important one is the `after_serverpy_init_app` which passes a `Scheduler` class which can be used to setup regular tasks. A list of currently implemented callback-methods along with their descriptions are available in [`/src/cryptoadvance/specter/services/callbacks.py`](https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/services/callbacks.py).
+
+
+

docs/extensions/callbacks.md

@@ -0,0 +1,55 @@
+# Data storage
+As an extension developer, you have the choice to completely manage your own persistence. Don't hesitate to completely do your own thing. However, if your requirements are not that complicated, you can rely/use one of two options: You either have data which need encryption (via the passwords of the users) or you don't have that requirement.
+
+## ServiceEncryptedStorage
+Some `Services` will require user secrets (e.g. API key and secret). Each Specter `User` will have their own on-disk encrypted `ServiceEncryptedStorage` with filename `<username>_services.json`. Note that the user's secrets for all `Services` will be stored in this one file.
+
+This is built upon the `GenericDataManager` class which supports optional encrypted fields. In this case all fields are encrypted. The `GenericDataManager` encryption can only be unlocked by each `User`'s individual `user_secret` that itself is stored encrypted on-disk; it is decrypted to memory when the `User` logs in.
+
+For this reason `Services` cannot be activated unless the user is signing in with a password-protected account (the default no-password `admin` account will not work).
+
+_Note: During development if the Flask server is restarted or auto-reloads, the user's decrypted `user_secret` will no longer be in memory. The Flask context will still consider the user logged in after restart, but code that relies on having access to the `ServiceEncryptedStorage` will throw an error and/or prompt the user to log in again._
+
+It is up to each `Service` implementation to decide what data is stored; the `ServiceEncryptedStorage` simply takes arbitrary json in and delivers it back out.
+
+This is also where `Service`-wide configuration or other information should be stored, _**even if it is not secret**_ (see above intro about not polluting other existing data stores).
+
+## ServiceEncryptedStorageManager
+Because the `ServiceEncryptedStorage` is specific to each individual user, this manager provides convenient access to automatically retrieve the `current_user` from the Flask context and provide the correct user's `ServiceEncryptedStorage`. It is implemented as a `Singleton` which can be retrieved simply by importing the class and calling `get_instance()`.
+
+This simplifies code to just asking for:
+```python
+from .service_encrypted_storage import ServiceEncryptedStorageManager
+
+ServiceEncryptedStorageManager.get_instance().get_current_user_service_data(service_id=some_service_id)
+```
+
+As a further convenience, the `Service` base class itself encapsulates `Service`-aware access to this per-`User` encrypted storage:
+```python
+@classmethod
+def get_current_user_service_data(cls) -> dict:
+    return ServiceEncryptedStorageManager.get_instance().get_current_user_service_data(service_id=cls.id)
+```
+
+Whenever possible, external code should not directly access these `Service`-related support classes but rather should ask for them through the `Service` class.
+
+## `ServiceUnencryptedStorage`
+A disadvantage of the `ServiceEncryptedStorage` is, that the user needs to be freshly logged in in order to be able to decrypt the secrets. If you want to avoid that login but your extension should still store data on disk, you can use the `ServiceUnencryptedStorage`.
+
+In parallel with the `ServiceEncryptedStorageManager` there is also a `ServiceUnencryptedStorageManager` which is used exactly the same way.
+
+## `ServiceAnnotationsStorage`
+Annotations are any address specific or transaction specific data from a `Service` that we might want to present to the user (not yet implemented). Example: a `Service` that integrates with a onchain store would have product/order data associated with a utxo. That additional data could be imported by the `Service` and stored as an annotation. This annotation data could then be displayed to the user when viewing the details for that particular address or tx.
+
+Annotations are stored on a per-wallet and per-`Service` basis as _unencrypted_ on-disk data (filename: `<wallet_alias>_<service>.json`).
+
+_Note: current `Service` implementations have not yet needed this feature so displaying annotations is not yet implemented._
+
+## Data Storage Class Diagram
+
+Unfortunately, the two unencrypted classes are derived from the encrypted one rather than having it the other way around or having abstract classes. This makes the diagram maybe a bit confusing.
+
+[![](https://mermaid.ink/img/pako:eNqVVMFuwjAM_ZUqJzaVw66IIU0D7bRd0G6VItO4LFvqoCRlqhj_PpeWASLduhyiqH7v-dlOsxO5VSgmIjfg_VzD2kGZUcKr3Z-Q0Ol8DgGegWCNLpl-jcfJEt1W57ig3NWbgGoZrONoS-oJXjBfCaPcPxI-ENkAQVvyF7RHS4VeVw5WBpea1gaDpZbZZ6eT_9VyzMK18_8oZeIuE8l4fMsn4tOQRvZm7FPra24XZgLjK4--38BFTe1-uCORAe3acLOk1KSDlCOPpkgTxSBZWKPQpUlniUcnP7C-f7GENycmQYnSFvLdc7zQBk-hn1r4OxrlTxFjQY3ORKSHbUfcn3vuqTFm_MIyt4h3pX1zraTCA_0sX0b9ya5varxPB7DUKk0-wfC1HSh_PeJdFB7_Mc6sTKf--Hk2G96769lHhJrlW7xc1bJp538qGpQjJnVqhUhFia4ErfiROwhlIrxhiZmY8FFhAZUJmciogVYbHj8ulOb8YlKA8ZgKqIJd1pSLSXAVHkHdW9mh9t9YNMxZ)](https://mermaid-js.github.io/mermaid-live-editor/edit#pako:eNqVVMFuwjAM_ZUqJzaVw66IIU0D7bRd0G6VItO4LFvqoCRlqhj_PpeWASLduhyiqH7v-dlOsxO5VSgmIjfg_VzD2kGZUcKr3Z-Q0Ol8DgGegWCNLpl-jcfJEt1W57ig3NWbgGoZrONoS-oJXjBfCaPcPxI-ENkAQVvyF7RHS4VeVw5WBpea1gaDpZbZZ6eT_9VyzMK18_8oZeIuE8l4fMsn4tOQRvZm7FPra24XZgLjK4--38BFTe1-uCORAe3acLOk1KSDlCOPpkgTxSBZWKPQpUlniUcnP7C-f7GENycmQYnSFvLdc7zQBk-hn1r4OxrlTxFjQY3ORKSHbUfcn3vuqTFm_MIyt4h3pX1zraTCA_0sX0b9ya5varxPB7DUKk0-wfC1HSh_PeJdFB7_Mc6sTKf--Hk2G96769lHhJrlW7xc1bJp538qGpQjJnVqhUhFia4ErfiROwhlIrxhiZmY8FFhAZUJmciogVYbHj8ulOb8YlKA8ZgKqIJd1pSLSXAVHkHdW9mh9t9YNMxZ)
+
+## Implementation Notes
+Efforts has been taken to provide `Service` data storage that is separate from existing data stores in order to keep those areas clean and simple. Where touchpoints are unavoidable, they are kept to the absolute bare minimum (e.g. `User.services` list in `users.json`, `Address.service_id` field).

docs/extensions/data-storage.md

@@ -0,0 +1,41 @@
+# Adding Devices
+
+Devices are the main building blocks for singlesig and multisig wallets. Different Hardwarewallets are represented as devices as well as the Bitcoin Core Hotwallet or the Electrum Wallet which might hold private keys and export xpubs into Specter Desktop.
+
+To Create your own Device, you have to specify the modules containing subclasses of `Device` in `service.py`:
+
+```
+class DiceService(Service):
+    # [...]
+    devices = ["mynym.specterext.myextensionid.devices.mydevice"]
+```
+
+You don't have to place the device in that devices-subdirectory but that's recommended. Here is an example with some explanations:
+
+```python
+# [...]
+from cryptoadvance.specter.device import Device
+
+class MyDevice(Device):
+    # the device_type is a string representation of this class which will be used in the
+    # json-file of a device of that type. Simply use the class-name lowercase
+    # and make sure it's unique
+    device_type = "mydevice"
+    # Will be shown when adding a new device and as a searchterm
+    name = "Electrum"
+    # The Icon. Use a b/w.svg
+    icon = "electrum/img/devices/electrum_icon.svg"
+    # optional, You might want to have a more specific template for creating a new device
+    template = "electrum/device/new_device_keys_electrum.jinja"
+
+    # If your device is a classic Hardwarewallets, it might have one of these features:
+    sd_card_support = True
+    qr_code_support = True
+
+    # auto, off or on
+    # seedsigner uses on. By default it's auto.
+    qr_code_animate = "off"
+
+```
+
+For sure there might be various methods to overwrite. Please have a look into the `Device` class.

docs/extensions/devices.md

@@ -0,0 +1,148 @@
+# Frontend aspects
+
+## controller.py
+
+You can have your own frontend with a blueprint (flask blueprints are explained [here](https://realpython.com/flask-blueprint/)). If you only have one, it needs to have a `/` route in order to be linkable from the `choose your plugin` page. 
+If you create your extension with a blueprint, it'll also create a controller for you which, simplified, looks like this:
+```python
+rubberduck_endpoint = ScratchpadService.blueprint
+
+def ext() -> ScratchpadService:
+    ''' convenience for getting the extension-object'''
+    return app.specter.ext["rubberduck"]
+
+def specter() -> Specter:
+    ''' convenience for getting the specter-object'''
+    return app.specter
+
+
+@rubberduck.route("/")
+@login_required
+@user_secret_decrypted_required
+def index():
+    return render_template(
+        "rubberduck/index.jinja",
+    )
+[...]
+```
+
+You can also have more than one blueprint. Define them like this in your service class:
+```python
+    blueprint_modules = { 
+        "default" : "mynym.specterext.rubberduck.controller",
+        "ui" : "mynym.specterext.rubberduck.controller_ui"
+    }
+```
+
+You have to have a default blueprint which has the above mentioned index page.
+In your controller, the endpoint needs to be specified like this:
+
+```python
+ui = RubberduckService.blueprints["ui"]
+```
+
+## Templates and Static Resources
+
+The minimal url routes for `Service` selection and management. As usualy in Flask, `templates` and `static` resources are in their respective subfolders. Please note that there is an additional directory with the id of the extension which looks redundant at first. This is due to the way blueprints are loading templates and ensures that there are no naming collisions. Maybe at a later stage, this can be used to let plugins override other plugin's templates.
+
+## Modifying non-extension pages
+
+You might have an extension which wants to inject e.g. JavaScript code into each and every page of Specter Desktop. You can do that via overwriting one of the `inject_in_basejinja_*` methods in your service-class:
+```python
+    @classmethod
+    def inject_in_basejinja_head(cls):
+        ''' e.g. rendering some snippet '''
+        return render_template("devhelp/html_inject_in_basejinja_head.jinja")
+
+    @classmethod
+    def inject_in_basejinja_body_top(cls):
+        ''' or directly returning text '''
+        return "<script>console.log('Hello from body top')"
+
+    @classmethod
+    def inject_in_basejinja_body_bottom(cls):
+        return "something here"
+```
+
+For this to work, the extension needs to be activated for the user, though.
+
+### Extending dialogs
+You can extend the settings dialog or the wallet-dialog with your own templates. To do that, create a callback method in your service like:
+
+```python
+from cryptoadvance.specter.services import callbacks
+# [...]
+    def callback_add_settingstabs(self):
+        ''' Extending the settings tab with an own tab called "myexttitle" '''
+        return [{"title": "myexttitle", "endpoint":"myext_something"}]
+
+    def callback_add_wallettabs(self):
+        ''' Extending the wallets tab with an own tab called "mywalletdetails" '''
+        return [{"title": "mywalletdetails", "endpoint":"myext_mywalletdetails"}]
+```
+
+In this case, this would add a tab called "myexttitle" and you're now supposed to provide an endpoint in your controller which might be called `myext_something` e.g. like this:
+
+```python
+@myext_endpoint.route("/settings_something", methods=["GET"])
+def myext_something():
+    return render_template(
+        "myext/some_settingspage.jinja",
+        ext_settingstabs = app.specter.service_manager.execute_ext_callbacks(
+            callbacks.add_settingstabs
+        )
+    )
+```
+
+If you want to have an additional wallet tab, you would specify something like:
+
+```python
+@myext_endpoint.route("/wallet/<wallet_alias>/mywalletdetails", methods=["GET"])
+def myext_mywalletdetails(wallet_alias):
+    wallet: Wallet = app.specter.wallet_manager.get_by_alias(wallet_alias)
+    return render_template(
+        "myext/mywalletdetails.jinja",
+        wallet_alias=wallet_alias,
+        wallet=wallet,
+        specter=app.specter,
+        ext_wallettabs = app.specter.service_manager.execute_ext_callbacks(
+            callbacks.add_wallettabs
+        )
+    )
+```
+
+The `some_settingspage.jinja` should probably look exactly like all the other setting pages and you would do this like this:
+
+```jinja
+{% extends "base.jinja" %}
+{% block main %}
+	<form action="?" method="POST" onsubmit="showPacman()">
+		<input type="hidden" class="csrf-token" name="csrf_token" value="{{ csrf_token() }}"/>
+		<h1 id="title" class="settings-title">Settings</h1>
+		{% from 'settings/components/settings_menu.jinja' import settings_menu %}
+		{{ settings_menu('myext_something', current_user, setting_exts) }}
+		<div class="card" style="margin: 20px auto;">
+			<h1>{{ _("Something something") }} </h1>
+		</div>
+	</form>
+{% endblock %}
+```
+
+![](./images/extensions/add_settingstabs.png)
+
+
+A reasonable `mywalletdetails.jinja` would look like this:
+
+```jinja
+{% extends "wallet/components/wallet_tab.jinja" %}
+{% set tab = 'my details' %}
+{% block content %}
+	<br>
+	<div class="center card" style="width: 610px; padding-top: 25px;">
+        Some content here for the wallet {{ wallet_alias }}
+	</div>
+{% endblock %}
+```
+
+![](./images/extensions/add_wallettabs.png)
+