Added blueprints area, plus changes to the cloud-resilience guidance

Author: andyblundell

README.md

@@ -40,7 +40,7 @@ The framework is a companion to:
 
 The framework consists of:
 
-* [Engineering principles](principles.md)
+* [Engineering principles](principles.md) and [blueprints](blueprints.md)
 * [Engineering quality review tool](insights/review.md)
 * [Communities of practice guidelines](communities/communities-of-practice.md) and active communities:
   * [Product Development Test Automation Working Group](communities/pd-test-automation-working-group.md)

blueprints.md

@@ -0,0 +1,5 @@
+# Engineering blueprints
+
+This is a list of blueprint solutions to common problems which are referenced within this quality framework:
+
+- [Cross-account backups on AWS](blueprints/backups-aws.md)

blueprints/backups-aws.md

@@ -0,0 +1,10 @@
+# Cross-account backups on AWS
+
+## Context
+
+- These notes are part of a broader set of [blueprints](../blueprints.md)
+- This blueprint relates to [service reliability](../practices/service-reliability.md) and specifically to use of [cloud services](../practices/cloud-services.md)
+
+## TBC
+
+... TBC ...

practices/cloud-services.md

@@ -21,18 +21,28 @@
 - Prefer serverless platform as a service (PaaS) over infrastructure as a service (IaaS) (see [outsource bottom up](../patterns/outsource-bottom-up.md)).
 - Where not serverless use ephemeral and immutable infrastructure.
 - Engage your cloud supplier early on in the development process. They have various tools and processes to help you (e.g. [AWS Well-Architected Review](https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&wa-lens-whitepapers.sort-order=desc)).
-- Understand cloud supplier SLAs.
-- Make systems self-healing.
-  - Prefer technologies which are resilient by default.
-  - Favour global-scoped (e.g. [CloudFront](https://aws.amazon.com/cloudfront/) or [Front Door](https://azure.microsoft.com/en-gb/pricing/details/frontdoor/)) or region-scoped services (e.g. [S3](https://aws.amazon.com/s3/), [Lambda](https://aws.amazon.com/lambda/), [Azure Functions](https://azure.microsoft.com/en-gb/services/functions/)) to availability-zone (AZ) scoped (e.g. [VMs](https://azure.microsoft.com/en-gb/services/virtual-machines/), [RDS DBs](https://aws.amazon.com/rds/)) or single-instance services (e.g. [EC2 instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)).
+- Make systems self-healing and resilient:
+  - Be aware that terms such as "region" have different meanings across different cloud vendors
+    - For example, it is not valid to compare the number of UK "regions" in AWS and Azure
+    - High levels of resilience can be achieved using UK-based cloud services for providers such as AWS and Azure, if the full scope & resilience of the clouds is used
+  - Also sometimes conflated in terms of resilience are cross-account and cross-region:
+    - As a minimum, all systems should have a tamper-proof cross-account backup to protect against account compromise, e.g. ransomware atttack: see [blueprint for AWS-based systems](../blueprints/backups-aws.md)
+    - You may wish to additionally consider cross region backups to protect against region failure
+  - Be aware of the resilience of any systems on which your system depends - for example, in a region-failure scenario, a standby for your system in a second region won't help if your system relies on another system which only runs in the single region which has failed
+  - Be aware of the difference between the resilience of cloud and your system's resilience in cloud
+    - Understand the SLAs of the cloud services you use.
+    - Every cloud service you use introduces more dependencies and more opportunities for service issues ...
+    - ... but, bespoke engineering to avoid using cloud vendor services introduces additional complexity and opportunities for reliability issues
+    - ... and, the risks are typically far greater for bespoke engineering, therefore: favour cloud services over bespoke engineering
+  - Prefer technologies which are resilient by default: favour global-scoped (e.g. [CloudFront](https://aws.amazon.com/cloudfront/) or [Front Door](https://azure.microsoft.com/en-gb/pricing/details/frontdoor/)) or region-scoped services (e.g. [S3](https://aws.amazon.com/s3/), [Lambda](https://aws.amazon.com/lambda/), [Azure Functions](https://azure.microsoft.com/en-gb/services/functions/)) to availability-zone (AZ) scoped (e.g. [VMs](https://azure.microsoft.com/en-gb/services/virtual-machines/), [RDS DBs](https://aws.amazon.com/rds/)) or single-instance services (e.g. [EC2 instance storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html)).
   - For AZ-scoped services, use redundancy to create required resilience (e.g. [AWS Auto Scaling Groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) or [Azure Scale/Availability Sets](https://docs.microsoft.com/en-us/azure/virtual-machines/availability)), and:
     - For stateless components use active-active configurations across AZs (e.g. running stateless containers across multiple AZs using [AWS Elastic Kubernetes Service](https://aws.amazon.com/eks/))
     - For stateful components, e.g. databases, consider use of active-active configurations across AZs (e.g. [Aurora Multi-Master](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html)), but be aware of the added complexity conflict resolution for asynchronous replication can bring and potential performance impact where synchronous replication is chosen.
   - Consider use of multiple regions (e.g. for AWS eu-west-1 [Dublin] as well as eu-west-2 [London]) as a way to improve availability, though ensure data sovereignty implications are understood and accepted (see below).
   - Understand failover (e.g. [RDS failover](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html#:~:text=Failover%20times%20are%20typically%2060%E2%80%93120%20seconds.)) and failed instance replacement times and engineer to accommodate these.
-- Be aware of data sovereignty implications of using any systems hosted outside the UK.
-  - Make sure your information governance lead is aware and included in decision making.
-  - Consider SaaS tools the team uses as well as the systems we build.
+  - Be aware of data sovereignty implications of using any systems hosted outside the UK.
+    - Make sure your information governance lead is aware and included in decision making.
+    - Consider SaaS tools the team uses as well as the systems we build.
 - Services should scale automatically up and down.
   - If possible, drive scaling based on metrics which matter to users (e.g. response time), but balance this with the benefits of choosing leading indicators (e.g. CPU usage) to avoid slow scaling from impacting user experience.
   - Understand how rapidly demand can spike and ensure scaling can meet these requirements. Balance scaling needs with the desire to avoid over provisioning and use [pre-warming](https://petrutandrei.wordpress.com/2016/03/18/pre-warming-the-load-balancer-in-aws/) of judiciously where required. Discuss this with the cloud provider well before go live they can assist with pre-warming processes ([AWS](https://aws.amazon.com/premiumsupport/programs/iem/)).