diff --git a/.checkov.yaml b/.checkov.yaml new file mode 100644 index 0000000..b99a071 --- /dev/null +++ b/.checkov.yaml @@ -0,0 +1,6 @@ +# +# +# +skip-check: +- CKV_OPENAPI_3 # Schema don't need cleartext checks. +- CKV_OPENAPI_4 # Schema don't need security definitions. diff --git a/.github/ISSUE_TEMPLATE/config.yaml b/.github/ISSUE_TEMPLATE/config.yaml new file mode 100644 index 0000000..5294d08 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yaml @@ -0,0 +1,50 @@ +name: Bug Report +description: File a bug report +title: "[Bug]: " +labels: ["bug", "triage"] +body: + - type: textarea + id: what-happened + attributes: + label: I Expect + placeholder: Tell us what you see! + value: |- + + ## When (optional) + + - + - + + ## I Expect + + - + - + + ## Instead + + - + + ## Notes + + - + validations: + required: true + - type: textarea + id: logs + attributes: + label: Relevant log output + description: >- + Please copy and paste any relevant log output. + This will be automatically formatted into code, + so no need for backticks. + render: shell + - type: checkboxes + id: terms + attributes: + label: Code of Conduct + description: > + By submitting this issue, + you agree to follow our [Code of Conduct](https://example.com) + options: + - label: I agree to follow this project's Code of Conduct + required: true diff --git a/.github/ISSUE_TEMPLATE/issue-simple.md b/.github/ISSUE_TEMPLATE/issue-simple.md new file mode 100644 index 0000000..cc764d5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/issue-simple.md @@ -0,0 +1,29 @@ +--- +name: Minimal issue template +about: Minimal Issue Template +title: '[bug]: ' +labels: '' +assignees: '' +--- + +## When I (optional) + +1. xx +1. xx +1. xx + +## I expect + +- [ ] yy +- [ ] zz + +## Instead + +- +- +- + +## Notes + +Attach sanitized logs, screenshots, outputs. +CC folks diff --git a/.github/POSTMORTEM.md b/.github/POSTMORTEM.md new file mode 100644 index 0000000..b398461 --- /dev/null +++ b/.github/POSTMORTEM.md @@ -0,0 +1,142 @@ +--- +# This is a template for a postmortem reports inspired by +# the teamdigitale's one published on medium.com. +# For the original version, see the references section. +title: Fake Postmortem - Cloud connectivity incident +date: 2018-05-23 +summary: >- + Fake Postmortem inspired by the following: The Digital Team's websites were unreachable for 28 hours due to a cloud provider + outage. +authors: +- name: Mario Rossi +- name: Franco Bianchi +references: +- https://medium.com/team-per-la-trasformazione-digitale/document-postmortem-technology-italian-government-public-administration-99639a0a7877 +- https://abseil.io/resources/swe-book/html/ch02.html#blameless_postmortem_culture +glossary: {} +keywords: [] +... +--- +# Postmortem - Template for a postmortem report + +## Summary + +**Impact**: + +The following services cannot be reached: + +- Dashboard Team +- Three-Year ICT Plan +- Designers Italia +- Developers Italia +- Docs Italia +- Forum Italia + +**Duration**: +28 hours + +**Cause**: +OpenStack network outage - cloud provider _Cloud SPC Lotto 1_ + +## Context + +The Digital Team's websites are based mainly on static HTML generated by the source content of the repositories on GitHub. The HTML code is published via a web server (nginx) and exposed according to HTTPS protocol. Forum Italia (http://forum.italia.it) is the only exception to this deployment model, and is managed separately via Docker containers. At any given time, one or more web servers can be deployed on the cloud provider's (Cloud SPC Lotto 1) OpenStack virtual machines, using the API provided by the platform. + +Cloud resources (virtual machines and volume data) are allocated towards services according to the Agency for Digital Italy's Cloud SPC contract. + +## Impact and damage assessment + +On 19/05/2018, the following services became unreachable due to an internal connectivity issue of the Cloud Service Provider "Cloud SPC": + +- Dashboard Team +- Three-Year ICT Plan +- Designers Italia +- Developers Italia +- Docs Italia +- Forum Italia + +## Causes and Contributing Factors + +According to a postmortem document released by the supplier on 2018-06-07, the interruption of connectivity experienced by the 31 users (tenants) of the SPC Cloud service was triggered by a planned update of the OpenStack platform carried out on the night of Thursday 2018-05-17. + +### Detection + +The problem was detected the following morning (2018-05-18), thanks to reports from users who were no longer able to access the services provided on the Cloud SPC platform. + +### Causes + +The document states that a restart of the control nodes of the OpenStack platform (nodes that handle OpenStack's management services: neutron, glance, cinder, etc.) caused “an anomaly” in the network infrastructure, blocking the traffic on several computing nodes (nodes where virtual instances are executed), and causing virtual machines belonging to 31 users to become unreachable. +The postmortem document also explains how a bug in the playbook (update script) would have blocked network activities by modifying the permissions of the file `/var/run/neutron/lock/neutron-iptables`, as indicated in the platform's official documentation. + +Again, according to the supplier, restarting the nodes was necessary for the application of security updates for Meltdown and Spectre (CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754). + +The unavailability of the Cloud SPC infrastructure was undoubtedly the root cause of the problem, but the lack of an application-level protection mechanism for the Digital Team's services prolonged their unavailability. +Indeed, due to the fact that the possibility of the entire cloud provider becoming unreachable had not been taken into account during the design phase of the services, it was not possible to respond adequately to this event. +Despite the SPC Cloud provider's failover mechanisms, the web services were not protected from generalized outages capable of undermining the entire infrastructure of the only Cloud provider at our disposal. + +## Actions taken + +WRITEME: A list of action items taken to mitigate/fix the problem + +- * Action 1 + * Owner +- * Action 2 + * Owner +... + +## Preventive actions + +WRITEME: A list of action items to prevent this from happening again + + + +## Lessons learned + +### What went wrong + +The Cloud SPC platform cannot currently distribute virtual machines through data centers or different regions (OpenStack region). +It would have been useful to be able to distribute virtual resources through independent infrastructures, even infrastructures provided by the same supplier. + +### What should have been done + +In hindsight, the Public Administration should have access to multiple cloud providers, so as to ensure the resilience of its services even when the main cloud provider is interrupted. + +### Where we got lucky + +WRITEME: What things went right that could have gone wrong + +### What should we do differently next time + +The most important lesson we learned from this experience is the need to continue investing in the development of a cross-platform, multi-supplier Cloud model. +This model would guarantee the reliability of Public Administration services even when the main cloud provider becomes affected by problems that make it unreachable for a long period of time. + +## Timeline + +A timeline of the event, from discovery through investigation to resolution. +All times are in CEST. + +### 2018-05-17 + +22.30 CEST: The SPC MaaS alert service sends alerts through email indicating that several nodes can no longer be reached. + +### 2018-05-19 + +6:50 CEST: The aforementioned services, available at the IP address 91.206.129.249, can no longer be reached + +### 2018-05-19 + +08:00 CEST: The problem is detected and reported to the supplier + +09:30 CEST: The machines are determined to be accessible through OpenStack's administration interface (API and GUI) and internal connectivity reveals no issue. Virtual machines can communicate through the tenant's private network, but do not connect to the Internet. + +15:56 CEST: The Digital Team sends the supplier and CONSIP a help request via email + +18:00 CEST: The supplier communicates that they have identified the problem, which turns out to be the same problem experienced by the DAF project, and commence work on a manual workaround + +19:00 CEST: The supplier informs us that a fix has been produced and that it will be applied to the virtual machines belonging to the 31 public administrations (tenants) involved. + +### 2018-05-20 + +11:10 CEST: The supplier restores connectivity to the VMs of the AgID tenant + +11:30 CEST: The Digital Team reboots the web services and the sites are again reachable diff --git a/.github/PULL_REQUEST_TEMPLATE/pr-simple.md b/.github/PULL_REQUEST_TEMPLATE/pr-simple.md new file mode 100644 index 0000000..a2f5db1 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/pr-simple.md @@ -0,0 +1,13 @@ +## This PR + +- [ ] +- [ ] +- [ ] + +## It's done + +- Rationale of the implementation + +## Checks + +- [ ] This PR conforms the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml new file mode 100644 index 0000000..714da34 --- /dev/null +++ b/.github/workflows/lint.yml @@ -0,0 +1,90 @@ +# Run linting action with some custom setup. + +name: Lint + +on: + push: + branches: + - main + pull_request: + branches: + - main + + # Allows you to run this workflow manually from the Actions tab + workflow_dispatch: + +permissions: read-all +jobs: + lint: + # The type of runner that the job will run on + runs-on: ubuntu-latest + steps: + # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Super-Linter + uses: super-linter/super-linter/slim@v7.1.0 + env: + # Either disable MULTI_STATUS or pass the GITHUB_TOKEN. + MULTI_STATUS: false + # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + VALIDATE_MARKDOWN: false + VALIDATE_MARKDOWN_PRETTIER: false + # Disabled for conflicts with the isort version used in pre-commit + # you can re-enable it if you align your local isort with + # the one in the super-linter image. + VALIDATE_PYTHON_ISORT: false + VALIDATE_XML: false + # + # YAML validation is delegated to pre-commit. + # + VALIDATE_YAML: false + VALIDATE_YAML_PRETTIER: false + VALIDATE_OPENAPI: false + VALIDATE_NATURAL_LANGUAGE: false + # + # JS/CSS delegated to eslint. + # + VALIDATE_CSS: false + VALIDATE_JAVASCRIPT_ES: false + VALIDATE_JAVASCRIPT_STANDARD: false + VALIDATE_JSON_PRETTIER: false + VALIDATE_JSON: false + # + # TS/TSX delegated to eslint. + # + VALIDATE_TSX: false + VALIDATE_TYPESCRIPT_ES: false + # JAVASCRIPT_ES_CONFIG_FILE: eslint.config.mjs + VALIDATE_TYPESCRIPT_STANDARD: false + # TYPESCRIPT_ES_CONFIG_FILE: eslint.config.mjs + VALIDATE_JSCPD: false # Disable copy-paste detection. + + pre-commit: + # The type of runner that the job will run on + runs-on: ubuntu-latest + container: python:3.9 + steps: + - uses: actions/checkout@v4 + + - name: Run commit hooks. + run: | + pip3 --no-cache-dir install pre-commit + git --version + pwd + ls -la + id + git config --global --add safe.directory "$PWD" + pre-commit install + pre-commit run -a + + # Store (expiring) logs on failure. + # Retrieve artifacts via `gh run download`. + - uses: actions/upload-artifact@v4 + if: failure() + with: + name: pre-commit.log + path: /github/home/.cache/pre-commit/pre-commit.log + retention-days: 5 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..a902160 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,28 @@ +# +# Run pre-commit hooks. You can run them without installing +# the hook with +# +# $ pre-commit run --all-files +# +# See https://pre-commit.com for more information +# See https://pre-commit.com/hooks.html for more hooks +repos: +- repo: https://github.com/pre-commit/pre-commit-hooks + rev: v6.0.0 + hooks: + # Manage spaces. + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-added-large-files + - id: check-symlinks + # Check file syntax/format + - id: check-xml + - id: check-json + - id: check-yaml + args: [--allow-multiple-documents] + # Security checks. + - id: detect-private-key + - id: detect-aws-credentials + args: + # See https://github.com/pre-commit/pre-commit-hooks/issues/174 + - --allow-missing-credentials diff --git a/.yamllint b/.yamllint new file mode 100644 index 0000000..132b71f --- /dev/null +++ b/.yamllint @@ -0,0 +1,26 @@ +# +# yamllint configuration files. It disables some checks to ease the integration +# with other yaml tools (eg. pre-commit autoformatter, ...) +# +extends: default + +rules: + document-end: disable + document-start: disable + truthy: disable + brackets: disable + line-length: + max: 90 + indentation: + indent-sequences: consistent + +# Specify the paths to be processed +ignore: | + pnpm-lock.yaml + node_modules + +# Override rules for specific paths +overrides: + - path: "apps/example/public/schemas/*.yaml" + rules: + line-length: disable diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..0815977 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,114 @@ +# CONTRIBUTING + +To contribute to this repository, please follow the guidelines below. + +## pre-commit + +Pre-commit checks your files before committing. It can lint, format or do +other checks on them. + +Once you install it via + + pip3 install pre-commit --user + +You can run it directly via + + pre-commit run --all-files + +Or install it as a pre-commit hook + + pre-commit install + +## Making a PR + +Contributing to a repository is done via pull requests (PR). +It is important to keep the code base clean and consistent in time, +in order to make it maintanable +and reduce unuseful deployments (see [CI](#ci)). + +A correct development process, with code reviews, is part of a correct +shift-left strategy. + +Following this procedure will help you to make a clean PR. +Each PR should be associated with an issue and a branch; +if the PR already exists, you can just start working from it. + +1. If there's no issue for your PR, create one where you describe the expected behavior and the current behavior; +1. If you are not a member of the organization, fork the repository and fetch from both your fork and the origin + + GH=ioggstream # use your github username + git clone -o teamdigitale https://github.com/teamdigitale/dati-semantic-schema-editor-build + cd dati-semantic-schema-editor-build + git remote add origin git@github.com:${GH}/dati-semantic-schema-editor-build.git + +1. Create a branch for your PR fetching from the main branch, using your username and issue-number as branch name. + Before checkout, make sure you have the latest version of the `teamdigitale/main` branch. + + ISSUE=123 # use the issue number + BRANCH=${GH}-${ISSUE} + git fetch --all + git checkout -b ${BRANCH} teamdigitale/main + + If the PR already exists, you can continue to work on it, always fetching the latest version + and ensuring that your working copy is up to date. Otherwise, you risk to work waste time + resolving conflicts. + + git fetch --all # Always download latest changes + git checkout teamdigitale/${BRANCH} + +1. Make your changes (this includes [pre-commit checks](#pre-commit)) and review them when adding. + This is an important and overlooked step, especially when + you are working alone or on a large PR. Moreover this allows you to split your changes in multiple commits + or to discard some of changes that you still want to temporarily keep in your working directory. + + git add -p + +1. You can now commit them. If your PR fixes the issue, + the commit message should start with `Fix: #ISSUE` where `ISSUE` is the issue number. + Otherwise, a reference to the issue can be added in the commit message body. + + git add . + git commit -m "Fix: #$ISSUE. Brief description of the changes." + + If the PR does not fix the issue, you can always reference it + in the commit messages. + + git commit -m "Brief description of the changes. See #ISSUE." + +1. Now you can push the branch and create the PR. + If your branch is published on your fork, you can create the PR directly + from github. + + git push origin ${BRANCH} + + When opening the PR from the web interface, please indicate: + + - if the PR is a draft one, prefixing it with the `WIP:` string + or using the **draft PR** functionality of github; + - the target branch, e.g. `teamdigitale/main`; + - what has been done, including the fixed issues (e.g. `Fix: #123`); + - when useful, describe the solution. + + If the PR is not ready to merge, you can still: + + - notify your colleagues tagging them (e.g. `CC: @ioggstream`); + - ask for a review if you have the associated permissions + (e.g. "Add reviewers" on github); + - proof-read it from the code-hosting platform WebUI, tag colleagues + or [suggest changes](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/incorporating-feedback-in-your-pull-request) + + This project requires that PRs are rebased before being merged, + in order to ensure a clear history. + Further information on rebasing and merging is available on + the [Linux kernel website](https://docs.kernel.org/maintainer/rebasing-and-merging.html) + and on [Atlassian](https://www.atlassian.com/git/tutorials/merging-vs-rebasing). + +1. Once the PR is merged, you can delete your local and remote branches, + and fetch the latest version from the upstream repository. + The code-hosting platform can be configured to automatically remove + remote branches automatically after merge. + +## CI + +Each PR is tested by a CI workflow that runs on GitHub Actions. +The final step might include a deployment to PyPI or to an OCI image registry. diff --git a/README.md b/README.md index 3153e20..35eacb1 100644 --- a/README.md +++ b/README.md @@ -1 +1,16 @@ -# dati-semantic-schema-editor-build \ No newline at end of file +# dati-semantic-schema-editor-build + +This project creates a build of the [Italia OpenAPI Schema Editor](https://github.com/teamdigitale/dati-semantic-schema-editor) +to be embedded on the https://schema.gov.it website. + +See also: + +- https://github.com/teamdigitale/dati-semantic-schema-editor +- https://github.com/teamdigitale/dati-semantic-kubernetes + +## Contributing + +Please, see [CONTRIBUTING.md](CONTRIBUTING.md) for more details on: + +- using [pre-commit](CONTRIBUTING.md#pre-commit); +- following the git flow and making good [pull requests](CONTRIBUTING.md#making-a-pr).