Add a .travis.yml file

[bhgrant8]

Detailed Description

As referenced in #3 we do not have a functioning .travis.yml file to complete deployment of packages successfully to AWS Services. This file will be necessary for our CI/CD pipeline

Context

By templating this file, it will lead to individual projects being successful in a deploy, and having their services run in the aws cloud environment

• will have to add in the different parts to travis
• confirm any variables which need to be set/which can be simplified
• whether to use travis deployers (https://docs.travis-ci.com/user/deployment)
• what we need to script (or could pull from last year)
• what benefits do we gain with github releases?
• test a deploy

Possible Implementation

This is rough sketch of what we need, but implementation still needs fleshing out

Your Environment

• Version (https://github.com/hackoregon/2019-backend-cookiecutter-django/releases): 0.4.2 as of writing
• Operating System and version: n/a
• Link to your project: n/a

[nam20485]

@BrianHGrant I am going to start looking at getting some of this going. Once I have a basic outline of the implementation we should discuss to make sure that the direction I am going in matches your expectations.

[nam20485]

I have a basic structure in place for a multi-stage build that contains each of the steps you have outlined in your slide: https://github.com/hackoregon/2019-sandbox-backend/blob/travis/.travis.yml

(I am building until it works correctly inside of my template-generated repo. Once it functions there, we can replace the generic bits with template parameters and move it into the 2019-backend-cookiecutter-django repo so that it is generated correctly for users of the cookiecutter template)

Currently,

1. The Deploy GitHub Release stage is completing successfully.
2. The Deploy PyPI Package stage is not working (I am working on this one now)
3. The Build Image stage fails when it gets to the pip install hackoregon_sandbox step in the Docker image creation. This is expected at this point as we have not successfully uploaded our package to pypi)

The rest of the stages (test, deploy to ECR/ECS) are not yet tested but are set up to function using the old scripting that has been known to function correctly in the past, so will probably just require some wiring up in to this.

I can continue to work on the individual pieces, but the main question I have at this point is we need to decide on the workflow for deploying a release.

Currently the workflow is based on git tagging. That is:

When a commit is made only the build and test stages run, as the other stages involve releasing things and are not needed unless we intend to create an explicit release. This essentially emulates our build process from last season.

Once a tag is created and pushed, the other stages run, in addition to build and test. A GitHub release is made, the PyPI package is created and uploaded to PyPI, (build and test run), and then the image is deployed to ECR/ECS.

Please take a look and provide feedback on the workflow I have outlined, as well as in general and if any of this matches what you had in mind. :) I haven't used travis' stage or deploy provider functionality before so I am totally open to suggestions or other directions.

I would like to get people's feedback on the structure prior to going to much further. Please add any people to the discussion you think should or would provide helpful feedback.

@znmeb
@DingoEatingFuzz
@BrianHGrant

[bhgrant8]

Thanks @nam20485 great work putting the general pieces together. I have not used deployers/stages but seems to be the correct approach and follows the general path I was thinking about.

I've been going back and forth on when and how much to run tests. Since I've setup the production build to install via pypi, we can't currently build and test prior to the deploy, so we end up deploying code to pypi prior to running tests in the integration environment.

If developers ran adequate tests prior to pushing, then maybe ok? Otherwise I see options:

1. we would need to add a build and test stage with debug build,
2. an additional build version (dev/prod/?) that may configure a generally production build, but does a pip install either within container from local directory (without the -e flag) or from github repo?

[nam20485]

OK I have a lot of this working now. Here is the latest version for reference: https://github.com/hackoregon/2019-sandbox-backend/blob/travis/.travis.yml

All the stages with the exception of the last one, deploy to ECR/ECS, now seem to be functioning correctly.

Here is an example from a successful run: https://travis-ci.org/hackoregon/2019-sandbox-backend

The deploy to ECR/ECS fails as expected, but the docker image build and test succeeds.

Are we OK with the workflow I have implemented, that is, official releases are deployed only on a git tag command? Normal commits to any branch only build the image and test it.

Since I've setup the production build to install via pypi, we can't currently build and test prior to the deploy, so we end up deploying code to pypi prior to running tests in the integration environment.

You raise an important issue here. Let me think about it and get back to you, but I do see a couple options, like you mentioned.

[DingoEatingFuzz]

I'm just catching up on all of this, so please excuse me if I overlooked anything, but my thoughts are as follows:

1. Heavy +1 to using build stages with Travis. The after_success trick we've been using is an unnecessary hack of the past, so thank you @nam20485 for exploring this new approach.
2. Also a fan of git tagging builds. Note that this isn't something we did in the past, we only tag docker images before pushing to ECR. Ideally we tag in git and share that tag with the docker build so we can trace things back to code.
3. Running automated tests before pushing to PyPi is an absolute must. It's industry standard to vet code before publishing. Publishing then running tests is an unacceptable alternative.
4. How are you currently tagging the sandbox project? Is that done by hand and then pushed? It would be nice to have tooling around this just to remove the possibility of fat-fingering tags and whatnot.

[nam20485]

Not at all, I appreciate the feedback. And it looks like you have picked up on the salient points.

Heavy +1 to using build stages with Travis. The after_success trick we've been using is an unnecessary hack of the past, so thank you @nam20485 for exploring this new approach.

At first I just didn't see any other way to accomplish what Brian wanted besides using stages. But now that I have a lot of it put together and have been working on it quite a bit I really like how it has turned out. Much less of a hack than the shoehorning that was happening in the after_success approach.

Also a fan of git tagging builds. Note that this isn't something we did in the past, we only tag docker images before pushing to ECR. Ideally we tag in git and share that tag with the docker build so we can trace things back to code.

In this approach the code is tagged via git and then that tag is applied not only to the Docker image, but to all of the rest of the build release products, i.e. the GitHub Release and the PyPI package. The entire process is traceable back to the specific code that was built and deployed. The GitHub release preserves the source so we can easily inspect and roll-back to the previous version as well.

Running automated tests before pushing to PyPi is an absolute must. It's industry standard to vet code before publishing. Publishing then running tests is an unacceptable alternative.

I completely agree. I wouldn't consider an approach without it to be a viable solution. The latest iteration that I am exploring includes a step at the beginning to build and test the production image prior to moving forward with any of the other stages of the build.

How are you currently tagging the sandbox project? Is that done by hand and then pushed? It would be nice to have tooling around this just to remove the possibility of fat-fingering tags and whatnot.

Current work-flow is that when a a release is desired, a git tag 0.1.26 && git push --tags command is manually performed. Its a simple one-step command that sets the version tag and kicks off the rest of the build, but it does require some human intervention. However, I have discovered a simple python tool called bumpversion that combines automatically incrementing the version number with setting and pushing the tag, simply by issuing a command like bumpversion. I believe that will assuage your concern.