Migrate architecture-pillars

This commit migrates the content of https://github.com/LBHackney-IT/architecture-pillars at commit LBHackney-IT/architecture-pillars-microsite@d1d9a2a.

Author: spikeheap

docs/architecture-pillars/adr.md

@@ -0,0 +1,49 @@
+---
+id: adr
+title: Architecture Decision Records
+---
+### Context
+
+Previously technical decisions were captured as part of spike documentation that was kept in project specific google drive. They were not open for other projects to review, adopt and adapt. Often, new developers were not aware of decisions as they were not aware of where to look for documentation. 
+
+Hence, we agreed to create Architecture Decision Records (ADRs) and add them to a single github repo [https://github.com/LBHackney-IT/lbh-adrs] to ensure that we have enough documentation, that all decisions are kept in the same location that is easy to find and to document how and why a decision was reached within a codebase. 
+
+This will also achieve governance and uniformity around all projects. Also ADRs help to give context around the decisions that were taken so that we can revisit them. Other benefits that we have identified when using ADRs are:
+- Improves onboarding for new developers
+- Improves agility when handing over project ownership between external team to internal or vice versa
+- Improves alignment across the teams regarding best practices
+
+### What is an Architectural Decision Record?
+
+An architecture decision record (ADR) is a document that captures an important architectural decision made along with its context and consequences of adopting the decision.
+
+### When to write an ADR?
+An ADR should be written whenever a decision of significant impact is made; it is up to each project team to align on what defines a significant impact. They can be:
+
+- Backfilling a decision which was made previously.
+- Proposing large changes to a solution/spike.
+- Proposing no/small changes for a spike
+- Proposing changes that differ from the overall agreed standard across our current ecosystem.
+
+### How to start using ADRs
+
+*Decision identification:*
+- How urgent and how important is Architecture Decision?
+- Design methods and practices can assist with decision identification and decision making.
+- Ideally maintain a decision to-do list which aligns with the service to-do list.
+
+*Decision making:*
+-Group decision making via Community of practices or project team workshops to validate the findings can help in decision making.
+- Better informed decision via ADRs which are available openly and people can collaborate on it.
+
+*Decision enactment and enforcement:*
+- ADRs are used in software design; hence they have to be communicated to, and accepted by, various stakeholders of the services that fund, develop, consume and operate it.
+- Architecturally evident coding styles and code reviews that focus on architectural concerns and decisions are two related practices.
+- ADRs also have to be (re-)considered when modernizing a software system in software evolution.
+
+*Decision sharing (optional):*
+- Many ADRs recurring across various project development..
+- Experiences from other projects and reusable components could enforce knowledge management strategy and contribute towards our emerging Community of practices meetups such as Data and Architecture etc.
+- Dependency matrix evaluation.
+
+

docs/architecture-pillars/api_compliance.md

@@ -0,0 +1,31 @@
+---
+id: api_compliance
+title: API Compliance Checklist
+--- 
+
+### Context
+Every API deployed to development, staging and production environments must be compliant with the set of standards listed in this document. An API should not be promoted from one environment to another if it does not satisfy all requirements listed.
+
+The set of compliance items that form this checklist are put in place to ensure that any API developed does not duplicate effort, is built in a reusable way, follows security best practices, is consistent with other APIs and follows all development standards defined.
+
+The APIs compliance checklist will be used as part of future Service Standard Assessments and ongoing check-ins to ensure any identified issues are tackled early on and no technical debt is accumulated.
+
+### Checklist
+1. The API has corresponding SwaggerHub documentation for all of the API endpoints it exposes.
+2. The API has completed the [API specification assessment process](https://playbook.hackney.gov.uk/api-specifications/assessment_process/)
+3. The API has been developed in Hackney’s preferred tech tech stack, unless otherwise agreed and as per standards defined in our [API playbook](https://playbook.hackney.gov.uk/API-Playbook/).
+4. The API has been developed following the TDD approach and has end-to-end tets in place 
+    - End-to-end tests guide for DynamoDB
+    - End-to-end tests guide for PostgreSQL
+5. The API has monitoring and logging tools enabled, as per defined standards.
+    - X-Ray is enabled for request tracing
+    - Canaries are created for availability monitoring
+    - CloudWatch is used for application logging.
+6. The API has vulnerability scanning enabled via SonarCloud.
+    - The API should also have no pending findings to review in SonarCloud.
+7. The API has the following infrastructure compliance checks in place:
+    - Terraform-compliance for the AWS resources provisioned as part of the same API deployment
+    - Serverless safeguards to ensure that the API is using an authorizer for security
+8. In Production, the API has been tested as per the ‘Production testing checklist’
+9. In Production, the API’s infrastructure is built as per [Production deployment - Live service infrastructure requirements](https://docs.google.com/document/d/1UrT6u4j8AlyPf-aD_E4c30uH27MJgIJoVxYR9kKGzFw/edit)
+

docs/architecture-pillars/api_spec_asessment_process.md

@@ -0,0 +1,62 @@
+---
+id: api_spec_asessment_process
+title: API Specifications Assessment Process
+---
+
+## Purpose
+The purpose of this process is to provide a clear,open and  consistent method of providing new and amended API specifications, evaluating them and getting them published in a way that is easy for a wider audience to access.
+
+Our API specifications have become a fundamental part of our API development process; all new APIs begin from a set of design specifications.  As the number of APIs we are developing continues to grow, we have begun to identify areas of inconsistency that we want to be able to improve on including:
+
+- Where the specifications are stored - specs tend to be stored in different locations by different projects making them difficult to track down when needed which ends up re-inventing the wheel.  If they cannot be found then we lose their value.
+- The review and approval process - once a specification has been approved, if there needs to be changes to the specifications, there needs to be a consistent approach for getting this done or we risk implementing changes that could present any of a number of risks to consumers of the API including developing against outdated specifications and systems not working as expected.
+
+Our API specifications are different from the API documentation in Swagger whose main purpose is to describe the various endpoints of the APIs.  The specifications add more light into the design process and attempt to capture the decisions behind approaches or changes to an API along with the user and data needs.  This further contributes to building our APIs in a more consistent way as departures from the standard set of tools and methodologies can be clearly documented here.
+
+We believe in the value of collaboration and having colleagues contribute to the design process allows for shared learning and potentially improves the quality of the output of these reusable APIs.  Having this process in place allows us to inject a diverse set of knowledge into the design of the API which would lead to positive and consistent outcomes for our development.
+
+## Vision
+
+- A single, centralised repository holding all of our API design specifications.
+- A way to easily and consistently access published API specifications in a way that is familiar to people.
+- A way to standardise the management of changes to specifications.
+- A way to link the API specifications to the API catalog going forward.
+
+## Developer Needs
+- As a developer I want a way to easily find API specifications so that I can better understand and use these APIs effectively as well as stay up to date on any changes that may affect my product.
+- As a developer I want to be able to publish specifications in a way that colleagues, stakeholders and interested parties can easily access and provide feedback on.
+
+## The Assessment Process
+Below is a diagram that provides some illustration of the process for assessing/evaluating our API specifications.  Each step in the process is expanded upon a bit further:
+
+Click image to open in a new tab. 
+[![](./docs-images/api_spec_assessment_process.png)](./docs-images/api_spec_assessment_process.png)
+
+** Draft Specification ** - The individual or project will draft the API specification as part of their internal design process, ensuring that the specification meets the needs of the identified users and follows our development standards as outlined in the API Playbook.  Once the draft is completed it should be exported to an MD (markdown) file and added to the API specifications repository [repository url to be added].
+
+** Raise the PR ** - Once the specification is ready for review, a pull request is raised to facilitate review and merge into the main branch.  If this is a new document, a link to it should be added to the repository’s navigation component.  A pull request can be reviewed either collaboratively at our tech meetups or offline by the development manager with a retrospective review if the pull request brings significant change.
+
+** Agreement Stage ** - If the specification is agreed then the pull request is approved and merged to the main branch.  Once merged it will trigger an update to the specifications web page, publishing any changes.  If, however, the specification is not agreed, any comments or concerns raised are added to the pull request and it is returned to the individual or project proposing the specification.  The individual or project will then review and address the requested changes, either by making improvements or by providing justification for maintaining the original design.
+
+** Subsequent Discoveries or Reviews ** - It is expected that API specifications will continue to evolve as further discoveries are made.  This process will allow for specifications to be iterated and improved.  The changes will be made on a new branch, separate from the main branch.  Once the changes have been completed a pull request will be raised and the review process will be triggered again.
+
+## Versioning
+While we iterate and improve on our API specifications, there needs to be a way to refer to previously agreed versions of the specifications.  For this we will use the features of Github to track changes to our designs.  With this in place, consumers of our API will be able to see how the APIs have evolved and address any changes that may impact their use.
+
+## Data Meetup Feedback
+
+https://ideaflip.com/b/wa4zzqf97nke/
+
+## Decision
+
+This process was proposed at our technical architecture meetup meeting on Tuesday 20th July, 2021.  The process was reviewed and agreed with the following items to consider:
+- If we will be able to fully monitor who and where each API is being used across the system(s) so that we can update users; for instance where there are changes in specification.
+- If the use of APIs will be added to a fuller system diagram.
+- We currently have Swagger docs generated by the APIs themselves as well as manually added to Swagger Hub.  Having a third source of documentation might be a bit much.  We will need to look at why we have Swagger documents generated in multiple locations.
+- We need to check that we do not add any new security risks by publishing more detailed information about our designs.
+- We will need to clearly distinguish between target specifications and actual; the idea being these specification documents will be more of a target and the - --Swagger docs will represent the actual.
+- We need to ensure that there is a standardised and recorded testing process for changes to the API.
+
+## Consequences
+
+Additional effort will be required to convert the document from a Google Doc (or original format) to the Markdown (MD) file format.  This could potentially contribute to documents not being in sync.

docs/architecture-pillars/auto_scaling.md

@@ -0,0 +1,53 @@
+---
+id: auto_scaling
+title: Auto-scaling for AWS resources used in Software Engineering
+---
+
+### Context
+
+To achieve cost optimization, all AWS services provisioned as part of the software delivery lifecycle, where capacity is pre-configured at the point of provisioning, must have an associated auto-scaling policy to ensure that we are only paying for what we use. 
+
+Cloud resources auto-scaling is the process of automatically scaling up or down based on traffic patterns and usage.
+
+Enabling auto-scaling will reduce the possibility of over-provisioning due to incorrect predictions of what capacity will be required.  
+
+This document outlines the common AWS resources used as part of delivering software that will require an auto-scaling policy to be applied. 
+
+**Note:** EC2 has been excluded from the list as we strive to use serverless technologies when delivering digital services. EC2 is only used for our bastion hosts, which are already provisioned with minimal capacity.
+
+### RDS
+
+AWS RDS currently supports only [storage autoscaling](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIOPS.StorageTypes.html#USER_PIOPS.Autoscaling) as an automated way to scale database instances. 
+
+Auto-scaling applies to an RDS database instance when the following factors are true: 
+- Free available space is less than 10 percent of the allocated storage.
+- The low-storage condition lasts at least five minutes.
+- At least six hours have passed since the last storage modification, or storage optimization has completed on the instance, whichever is longer.
+
+To provide cost optimization, RDS database instances should be provisioned with a smaller allocated storage space and an auto-scaling option enabled. The provisioned storage capacity **should not** be based on predictions for future service needs - it should instead reflect the foreseeable future data storage needs and scale up automatically only if required.
+
+
+**Considerations:**
+- Always set a reasonable maximum and minimum storage capacity to be used for auto-scaling to avoid scenarios where data storage needs have grown exponentially for a reason different to genuine service needs. 
+    - Example for such scenario is large logs stored in Postgres due to failing DMS task resulting in database storage used increasing continuously. In such situations, alarms should be in place to prompt investigation.
+- If you are not sure how much storage space will be required for a database, please start with the default amount suggested by AWS for the given database instance type and enable auto-scaling.
+
+### DynamoDB
+You can use the [AWS Application Auto Scaling](https://docs.aws.amazon.com/autoscaling/application/userguide/what-is-application-auto-scaling.html) service to set up automated scaling policies for DynamoDB.
+
+DynamoDB autoscaling can be applied to both -  a table and a Global Secondary Index (GSI).
+
+At Hackney, DynamoDB tables uses provisioned capacity mode, which means that we are billed based on read and write capacity units that a table is provisioned with. 
+
+To achieve cost optimization, all DynamoDB tables must have a scaling policy associated that will increase or decrease the RCUs and WCUs based on the traffic patterns.
+
+A guide to enabling auto-scaling for DynamoDB tables using Terraform can be found [here](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/appautoscaling_policy).
+
+### AWS Lambda
+Hackney does not use provisioned concurrency for the majority of the Lambda functions used. This means that Lambda handles concurrency and scaling automatically, as described in the [official documentation](https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html). For this reason, there is no need to apply auto-scaling as this is already handled by default.
+
+For any Lambda functions using provisioned concurrency, an auto-scaling policy must be applied, AWS Application Auto Scaling service can be used to automate the scaling process as described [here](https://docs.aws.amazon.com/autoscaling/application/userguide/services-that-can-integrate-lambda.html).
+
+
+
+