Blue Stingray CI/CD Devops Process – AWS

Blue Stingray implements a continuous integration and deployment strategy, utilizing a number of 3rd party solutions and tooling, to increase code quality, application stability, and team efficiency.

Goals

The Blue Stingray team is focused on the following goals while developing a devops strategy for any project:

Supports three layers of application environments for Development, Staging/QA, and Production.
Provides seamless integration between the code repository and the devops pipeline, to reduce friction, increase efficiency of the development team.
Allows for short feedback cycles. Developers should have the ability to continuously deliver and iterate on product development as often as multiple times a day, if needed, without incurring overhead costs for server management and deployments. This will increase overall ROI for the stakeholders, allowing them to see development progress in a Staging/QA environment quicker.
Integrates an error monitoring and reporting tool through each major stage in the devops pipeline to reduce risk to the production application and increase the transparency of potential issues to the development team.
Allows for teams and developers to work concurrently, reducing bottlenecks in the development process.

Development Principles

To have a successful CI/CD process in place, each member of the development team has a set of guiding principles or strategies that they follow when working on projects:

Modular and bite sized. Each feature, bug, or improvement to the application should be a small and easy to understand chunk. This supports the reducing feedback cycles and reducing risk to code merging.
Automate. Every piece of the process that can be automated should be automated. This is for the sake of efficiency, but even more importantly to reduce the risk of human error.
Transparency. All work done should be visible and transparent to the team. This is typically done by commit often. Working in smaller chunks of code allows this to happen more easily. Also, working with good tooling such allows for all work and workflows to be accessible by any member of the team.
Iteration and flexibility. Development work is not one and done. Each piece of the project is being updated, improved, maintained, and refactored over time. Developers approach projects with this mentality understanding that development must be flexible to change. This flexibility of work approved every aspect of the development and devops process.

Tooling

Blue Stingray uses the following tooling on custom development projects. We have worked with a large range of solutions based on our clients specific needs and restrictions. In this section, multiple solutions will be listed for each type of tool:

Code repositories (AWS CodeCommit, Github, Bitbucket). Provides a central repository for project development artifacts and assets.
Infrastructure as code via containers (Docker). Containers allow for strict configuration of infrastructure and dependencies that can be deployed on top of any existing infrastructure. This will provide stability, reduce the risk of conflicts between environments.
CI/CD pipeline (AWS CodePipeline). The core framework for defining and automating steps of the development process.
Testing framework. We’ve worked with dozens of testing frameworks. They are dependent on the technology being used and the requirements of the project.
Application monitoring (AWS CloudWatch). Allows for the development team to be notified about potential risks to the application or project based on the application code.
Server monitoring (AWS CloudWatch). Allows for the development team to be notified about potential risks to the application or project based on the cloud servers.

Standard Workflow

Each project will differ slightly based on the requirements of the project. This section provides an example of a standard CI/CD workflow for a custom development project at Blue Stingray.

Setup

Each project at Blue Stingray will have the following setup occur:

Code repositories considered with a skeleton project which can be built and run by any developer on the team via a local container.
Each code repository will be integrated with the CI/CD pipeline with a default set of pipeline steps.
The project will contain skeleton test suites for each type of test category. Those will typically start with:
- Unit tests.
- Integration tests.
- Functional/UI tests.

CI/CD Standard Steps

Development performed by the Blue Stingray team locally. Each developer runs a copy of the project on their local machine via docker. This involves the developer performing feature development and developing automated tests for the specific piece that they are working with.
Features pushed into a central code repository.
The pipeline recognizes change: deploys the test plan against code changes.
- On test failure: notifies developers.
- On test success: continues to next pipeline stage.
At least one developer on the team manually reviews the code looking for potential issues or improvements.
- On review failure: notifies original developer and requests changes.
- On review failure: moves it to the next pipeline stage.
Application build is deployed to a Staging/QA environment, awaits acceptance validation from stakeholders.
- On validation rejection: notifies developers.
- On validation acceptance: continues to next pipeline stage.
Application build is deployed to the Production environment.

Build and Test Processes

Each project uses different technologies, so the specific build and test tooling changes per project.

The testing plan will involve a combination of automated and manual testing. The Blue Stingray team will include automated test cases in the application code for each feature developed. The developers who write the feature code write the automated tests along side it. This is more efficient and less error prone, avoiding miscommunication issues, for application development than using a separate QA team that is not directly familiar with the application code. The application framework provides tooling for creating automated test cases. These automated tests will be ran and confirmed at stages 3, 4, and 5 in the pipeline process and code moves through the pipeline stages.

For example, if Blue Stingray if development a custom development Laravel application on AWS, then a typical “build and test” stage in the pipeline (see #3 in the standard steps above), then the following automated processes will be run:

Dependencies installed into a containerized environment based on a consistent image definition.
Code is linted for issues or improvements:
- Syntax and style lint.
- Performance lint.
- Security lint.
SQL database is configured and migrated with latest schema/changes.
Each test suite is run:
- Backend PHPUnit unit performed.
- Backend PHPUnit integration performed.
- Frontend Mocha unit performed.
- Laravel Dusk UI performed.
- k6 load testing performed.

In addition to automated testing, both Blue Stingray and the clients of Blue Stingray participate in manual acceptance testing of applications and requirements throughout the development process.