-
Notifications
You must be signed in to change notification settings - Fork 8
Deployment Guidelines
So you have managed to deploy your apps and/or services to an OpenRiskNet Virtual Environment (VE), probably by means of creating some templates that allow simple deployment and undeployment. So are you done?
Certainly not! OpenRiskNet needs services to be of production quality, so that third parties will find them easy to deploy, and will satisfy strict security requirements. Think of this as needing to have your services at a state that a security concious pharmaceutical company will be willing to us.
Here are some guidelines that should be handled to get to this state.
We are proposing different levels of compliance for ORN applications:
- Manually Installable: Materials and instructions exist (most likely here) for deploying the application. The process is manual but other users should be able to follow the instructions.
- Good Hygiene: pods do not run as root user, using a project-specific user for deployment, have readiness and liveness probes, have defined Container resource requests and limits and applications have enabled authentication where appropriate.
- Console Installable: Applications are deployable through the web console using either the Template Service Broker or the Ansible Service Broker (note: in future 'Operators' will likely supersede these).
We recommend that all applications aim to get to at least level 2.
A document describing the current status of ORN partner applications can be found here.
Many containers still run as the root user. This is bad as it introduces potential security risks. Better to run as a specific non-privileged user. This is still not ideal as there is potential 'leak through' of user processes between containers.
Best is to allow your container to run as an arbitrarily assigned user. This is the default in OpenShift and means that your container has to be able to run as any user ID and you do not know in advance what that user ID will be.
Sometimes this is not practical, or even possible, so its OK to fall back to run as a dedicated non-privileged user, but this requires the settings in your project to be changed to allow this. Avoid running as the root user unless that is absolutely necessary, and that should hardly ever really be needed.
AIM: your containers can be run without the need to modify the default security settings.
The VE is likely to have an admin user (or one can be created using the system:admin user).
It is tempting to use this user because you can do anything.
The problem with this user is that you can do anything!
But please avoid the OpenShift admin user.
If you are logged in as a standard user (see here for details) then you initially have very few privileges and you cannot access existing projects.
You can however create a new project, and you will then be an admin for that project (note: this assumes the default
configuration - the ability to create new projects can be disabled for normal users). Having created the new project
you can do pretty well whatever you want within that project and that is usually a good approach. If you need access to
other projects (e.g. to be able to access a database or SSO in the openrisknet-infra) then you either need to request
access from an admin user in that project, or a cluster-admin user, or ask them to perform the necessary operations
on your behalf.
In some cases it might be more appropriate to perform operations as a less privileged user. You can either remove the
admin role from your user or create an additional user for that purpose.
Incidentally, a low-privilege developer user may already be available on the VE.
If so you can use developer to create the project and deploy the application.
Instead of adding the required privilege to your new user ask yourself whether you can avoid the privilege.
More information is present in the next section.
OpenShift users create projects and deploy applications but actions
performed by the launched Pods take place using a Service Account (SA) whose
actions are constrained by a Security Context Constraint (SCC). If
you do not specify a service account in your application a built-in (default)
account and SCC is used. This account has limited privileges,
especially with regard to allowing containers to run that need to be run as root.
Resist the temptation, easy though it is, to adjust the capabilities of the
default service account. Instead create (using the admin account) your own account,
which is typically done project-by-project. You can then adjust the privilege
of your project-specific service account without disturbing the system-wide default.
An OpenShift blog that describes Service Accounts and the related topic of Security Context Constraints (SCCs), which control the actions that a Pod can perform and what it has the ability to access, can be found in the article Understanding Service Accounts and SCCs.
With a service account created in your project you define the Service Account alongside the container definition in your application template. Ideally your template should be parameterised so that a lot of significant information can be provided externally. A typical template definition might look like this (edited for readability): -
parameters:
- name: APP_SA
value: my-service-account
objects:
- kind: DeploymentConfig
apiVersion: v1
spec:
template:
spec:
serviceAccountName: ${APP_SA}
The expectation nowadays is that HTTPS should be used for all traffic and that all certificates should be signed by a trusted CA. Use of plain HTTP or self-signed certificates is frowned on.
The ACME Controller tool that is expected to be deployed to any ORN VE makes this very simple to achieve. All that is needed is to add this annotation to your route and Let's Encrypt certificates will be generated and automatically renewed for your routes.
metadata:
annotations:
kubernetes.io/tls-acme: "true"
As a guide its best to set this value of this annotation to false while you
are setting things up and then switch it to true when you are finished as
Let's Encrypt has fairly strict quotas on the number of certificates that
can be generated and its easy to exceed this when testing.
Let users know that your application is available for use. On the current ORN production site this involved adding a link to your app (the public routes) in this landing page.
To do this edit the index.html in this GitHub repo.
Committing a change to this repo will result in the page automatically being redeployed a few minutes
later.
Also, add your application to the list of services on the main ORN website.
Make your services discoverable by the ORN Service Registry by adding annotations to your Service defintions that the registry will discover when your services start or stop.
This is described in Annotating-your-service-to-make-it-discoverable.
Make sure your pods have suitable probes (health checks). OpenShift uses these to determine whether to attach any designated Service to your Pod and also to determine whether the application behind the service is healthy.
They are extremely valuable and our FAQ on the topic OpenShift Container Probes provides some some helpful guidance.
Use of a liveness probe is beneficial but remember that OpenShift will restart your Pod if this probe fails.
OpenShift accommodates the definition of Resource requests and limits for your Pods providing OpenShift with guidance on its needs for the CPU and Memory. A good description of these can be found in this blog post.
When defined, OpenShift will use the values to schedule the Pod onto a node with sufficient available resources and ensure that the defined limits are not exceeded when the Pod is running (for the benefit of other Pods on the same node)
Refer to the Compute Resources section of the OpenShift Quotas and Limit Ranges guide for more details.
A CPU Request (measured in millicores) defines the minimum amount of CPU your application Pod needs and OpenShift wil ensure that you're put on a node where you can get it. A Limit is used to place an upper-limit on your Pod's central-processor use.
A Memory Request ensures that you will have the memory you need (as OpenShift will place you on a node that can satisfy your request). A Limit prevents you from using more memory than specified.
While OpenShift can gracefully throttle the CPU for your Pod, preventing it from
using more than your defined limit, it cannot do the same for memory. If you
don't have enough CPU then you are essentially forced to slow down. But, if you
need more memory little can be done to avoid potential OutOfMemory
exceptions in your application (that you're unlikely to be able to handle anyway)
and so your Pod will be automatically killed.
Consequently if you try and use more CPU than your limit then OpenShift prevents you and you can continue to run. But, if you try and allocate more memory than your limit your Pod will be killed (and restarted).
A template for a Pod that needs at least half a core with up to 2 cores (in bursts) and that uses 128MB of memory but no more that 256MB would contain the following resource definition: -
parameters:
- name: CPU_REQUEST
value: 500m
- name: CPU_LIMIT
value: 2000m
- name: MEM_REQUEST
value: 128Mi
- name: MEM_LIMIT
value: 256Mi
objects:
- kind: DeploymentConfig
apiVersion: v1
spec:
template:
spec:
containers:
- image: me/my-image:latest
resources:
requests:
cpu: ${CPU_REQUEST}
memory: ${MEM_REQUEST}
limits:
cpu: ${CPU_LIMIT}
memory: ${MEM_LIMIT}
For the benefit of the entire VE you should strive to profile your application so that you understand its processing an memory needs in typical stead state and under maximum load. This is generally good advice but the benefit of doing so means that you can define sensible ane VE-friendly resources for the Pods in your application. Benefiting you and everyone in the VE.
Add your application as a client in the OpenRiskNet Realm in Keycloak and setup your application to require
authenticated users. This ensures that users get a good Single Sign On experience and we can effectively monitor how
much the different ORN applications are being used.
See Adding-Authentication for guidelines on how to do this.
An ORN VE provides a number of infrastructure components. If your application provides these themselves consider
switching to using these so that you can avoid needing to manage them yourself.
The current infrastructure components are:
- PostgreSQL database
- RabbitMQ message queue
- Keycloak for SSO (Red Hat SSO)
If you see you are providing something that could be better handled as an infrastructure component (e.g. a differnt type of database) then work with us to make this happen.
Managers of other VEs will want to deploy your application. Make this easy by adding it to the OpenShift Service Catalog (not to be confused with the OpenRiskNet Registry) or a project that has been specifically setup to house project templates.
If you are using a Template to deploy you are probably half way there already and can use the Template Service Broker. Further, the VE may have a project that is specifically designed to deploy applications from templates. If so then our recipe Serving applications from the OpenShift Template Service Broker (TSB) provides you with a helpful guide to serving your application from it.
Templates and the TSB are a quick-and-easy way to deploy simple applications (those that require little in terms of existing services) but they are limited in that removal of the application requires manual effort (typically by removal of the project the application was deployed to).
For better full lifecycle control of applications you will need to invest time creating an Ansible Playbook Bundle or, beyond OpenShift 3.11, an Operator. Both of which are currently beyond the scope of this guide.
For further reading visit the following: -
- Ansible Playbook Bundles with the Ansible Service Broker.
- The Operator Framework. Which is a technology preview in OpenShift 3.11.
TODO - Do we need to cover APBs?
You should aim to have your application automatically re-deployed when it is updated. There are several approaches, but the 2 most common may be:
- Pod containers update whenever a new Docker image is pushed to Docker Hub.
- Pod containers are rebuilt and deployed when updated code is pushed to GitHub.
OpenShift can automatically re-deploy your Pod container(s) based on various triggers. A trigger can be a change to an application's deployment configuration or the change in the external image. Our knowledge base article Automatic Redeployment with ImageStreams is a handy reference guide.
This approach is useful if you have a pre-existing test and deployment Framework that manages the publication of your container images.
TODO - Write Me.