Notice: Our URL has changed! Please update your bookmarks. Dismiss

Elements of an OpenShift CI/CD Pipeline

This document’s objective is to explain some foundational concepts related to building a CI/CD pipeline and how they relate to OpenShift. This is intended to be an introductory guide to what is in fact a rather vast topic. More in-depth and implementation-oriented guidelines are linked from this document when appropriate.

Immutable infrastructure

The general concept of immutable infrastructure is that there is only one way to introduce change into a system, and that way is to destroy and recreate that system. The main benefit of this approach is that changes are introduced in a very controlled way, minimizing drifts and misconfiguration. In addition, having a single method for introducing change allows more focus on automation and optimization. Infrastructure as code is a common approach used to achieve immutable infrastructure (this topic is out of scope for this document).

Systems and infrastructure can be rapidly destroyed and recreated. Therefore, immutability should not be interpreted as rigidity, but rather as a very prescriptive rule about how change can be introduced. For the lifetime of a given instance of a system, it is actually immutable.

The concept of immutable infrastructure can be applied to container-based solutions and deployment pipelines with the idea of progressively locking the infrastructure:

ImmutableInfrastructure

Pieces of the infrastructure are locked down in different phases and each infrastructure piece becomes immutable after being locked down.

The Code Build phase locks the business logic. This is about creating a deployable artifact starting from our code. We have always done this step.
The Image Build phase locks the binary execution environment and creates a container image. After the container image is created, the business logic can be executed anywhere with no drift (for binaries and other runtime dependencies).
The Topology Definition phase locks the topology of the solution. Assuming the deployment involves a solution comprised of several components, this phase creates and locks the topology that describes which components must be deployed and their dependencies. The declarative API is leveraged to create a manifest of the compute, networking, and storage resources that are needed.

Notice that we have already locked down most of the infrastructure and we haven’t deployed anything yet.

The last phase performs the deployment to an environment. During deployment, environment-dependent properties are injected, including public properties as well as confidential credentials.

To comply with the principle of immutable infrastructure after deployment, the only way to introduce new change in the environment is to repeat the previous processes. This is true regardless of the nature of the change that we want to introduce, which might include:

  • Code release

  • Runtime updates

  • Security Patches

  • Configuration changes

In traditional environments, the last three items are usually the responsibility of the operations team, and the developer team is not necessarily involved when they are performed. With immutable infrastructure and containers, deployment processes need to be radically rethought and need to involve the developer teams.

The phases of the progressing locking process that we have described can implemented in a CI/CD pipeline. The following sections discuss the types of pipelines and the types of steps that are part of a pipeline.

Pipeline Stages and Pipeline Flow

A pipeline stage is a logically grouped set of tasks intended to achieve a specific function within a pipeline (e.g. Build the App, Deploy the App, Test the App, Promote the App). The pipeline succeeds when all stages have completed without failure. Typically, stages run serially (one after the other) and in a consistent order, but some may run in parallel. We refer to the movement from one stage to the next as triggering.

Note
We dive deeper into these functions in Types of Pipeline Stages.

The ultimate goal for a successful pipeline is that it is able to run all the way through automatically (automatic triggering), taking the workload all the way into a production state without any intervention my humans. This level of continuation allows for development teams to release small amounts of code quickly and with low risk.

TruePipeline

To achieve a pipeline with this level of capabilities requires a high level of investment on the part of the development and operations teams to build proper testing and validation into the automated pipeline. This ensures quality and compliance of the code before deploying it into production. For this reason, many pipelines initially include manual triggers — stops or pauses after certain stages, requiring manual intervention to run tests, review code, or receive sign-off before approving the pipeline to continue on to a higher stage.

PointAndClick

There is a continuum between strictly manually triggered pipelines and automatically triggered pipelines. Most organizations may begin with full manual triggering between stages, but should look to remove as many of those manual triggers as feasible in order to reduce bottlenecks in the system.

Types of Pipeline Stages

Typically, there are four steps in a pipeline:

Step Type Description

Build

Creates or builds artifacts used by other steps.

Promotion

Deploys artifacts created in a build step to a specified environment.

Validation

Validates actions performed in a previous step, typically build or deploy. If the validation step fails, the pipeline process might be aborted.

Compliance

These type of steps are sometimes needed to make the pipeline compliant with enterprise release management standards.

Note
Because the primary task of a pipeline is to promote software through various Software Development Lifecycle (SDLC) environments to production, an MVP (Minimum Viable Product) pipeline may only require build and promotion steps, with validation and compliance checking being performed by humans.

The recommended approach is to create an MVP pipeline, and only add validation and compliance steps after the MVP pipeline is verified.

MVPPipeline

Build Stage

Build steps include everything needed to build a container image. Although version control is always in effect for container images, it might also be required for other deployment artifacts (for example, jar files) for compliance reasons. The pipeline owner must work with the release policies owner to ensure compliance requirements are met.

There are three ways to build a docker image:

Build type Description

External Docker build

Performs a standard Dockerfile build external to the OpenShift cluster. In this case, OpenShift is independent of the build.

Internal Docker build

Performs a standard Dockerfile build executed as a pod inside of OpenShift.

S2I build

Performs a source to image build that includes intermediate artifacts and docker image and that is executed as one process inside a pod.

Each of these build types produces the desired output, but has different advantages and disadvantages. Because builds are spiky types of workload, Internal Docker Builds are well-suited for container platforms in which capacity is given back to the pool at the end of the build. Running builds inside of OpenShift can increase overall environment efficiency. However, running builds internally in OpenShift creates garbage that needs to be removed (see this sections of the doc on garbage collection.

The main advantage of S2I builds is that they don’t require learning Dockerfile syntax. Building good docker images is an acquired skill and it may not be worth the cost of training all the developers in an organization. Another advantage is that the build steps in a S2I build are not run with root privileges. Another advantage is that the build steps in S2I builds run as the user defined in the base image. This gives an organization the ability to control the permission scope of the build, while still allowing the developer full ability to self serve.

On the other hand, because S2I builds are not part of the Kubernetes API, S2I builds are not portable across Kubernetes-based platforms. In addition, although it is possible to customize the S2I process, learning to customize S2I builds can be as time consuming and complex as learning how to properly build docker images.

More information for these build types can be found in our OpenShift Builds deep dive.

Binary Builds

If a build process already exists and the objective is to customize it to deploy to OpenShift, the binary build is a good option. The binary build uses artifacts produced in a previous step as input to an OpenShift build (either docker or S2I).

Check out our Binary Build tutorial to see this in action.

BinaryBuild

Jenkins Pipeline Build Strategy

A recent addition to the set of available OpenShift build types is the Jenkins pipeline build strategy. A Jenkins pipeline build is a build strategy that includes a mechanism to trigger a Jenkins build from within OpenShift via an API call. The build logic resides in Jenkins.

Promotion Stage

Promotion steps are used to deploy solutions (comprised of multiple components) to SDLC environments. In OpenShift, an environment is a project residing in a cluster. Assuming there is an existing OpenShift template that represents the solution, the deployment is performed with the following command:

oc process <template> ... | oc apply -f -

This command will instantiate the template to the target environment using the environment dependent template parameters (not to be confused with the environment dependent properties) and applies it to OpenShift.

The apply command determines if there are differences between the current state and the new, desired state, and if so, applies the differences as patches to the existing API objects. Because existing objects are not destroyed, the related applications do not incur outages.

If triggers are active at this point, a new deployment commences. In general, automatic triggers for deployments should be carefully considered. In fact if we have a pipeline orchestrator ( such as Jenkins) the logic to trigger a rollout should reside in the orchestrator.

Check out our guide on OpenShift Templates for more details on these concepts.

Rollout Strategies

The following rollout strategies are possible:

Rollout strategy Managed by Description

Recreate

OpenShift

Destroys and recreates pods, causing outages.

Rolling

OpenShift

Gradually creates new pods and destroys old ones in such a way to avoid outages.

Blue/Green

Pipeline

Maintains two sets of pods (old and new) and splits traffic between the two sets, gradually moving more and more traffic to the new pods.

Canary

Pipeline

Sends a small amount of traffic to the new pods. Once the new version is validated, works as blue/green.

A/B (*)

Pipeline + Service Mesh

Sends traffic to one version or the other based on some content in the request.

Shadow

Pipeline + Service Mesh

Sends traffic to both old and new pods, traffic to new pods is managed in a way to avoid side effects. Metrics are collected, analyzed, and compared with old traffic.

(*) Note: this definition of A/B testing differs from the one that you can find in the OpenShift documentation.

Deployment strategy characteristics

Rollout strategy Zero downtime Real traffic testing Targeted user Infrastructure cost Complexity

Recreate

no

no

no

low

low

Rolling

yes

no

no

low

low

Blue/Green

yes

no

no

high

medium

Canary

yes

no

yes

low

medium

A/B

yes

yes

yes

low

high

Shadow

yes

yes

no

high

high

Validation Stage

Validation steps tend to be organization specific and, in part, reflect the development methodology adopted by that organization.

Validation steps can be classified as environment independent and environment dependent. To catch errors as quickly as possible, environment independent steps should be executed before the first deployment. Conversely, environment dependent steps can only be executed in an environment after the solution has been deployed.

Examples of environment independent and environment dependent validation steps are provided in the following tables.

Environment independent validation steps:

Validation step Common tool/technology implementations

Code static analysis

SonarQube

Code static security scanning

Veracode

Unit test

JUnit

Image security scanning

CloudForms (with atomic scan), Twistlock, BlackDuck, Aquasec.

Image security scanning is a new step introduced by container technology. It ensures the quality of images in terms of security vulnerabilities (CVEs)
This process has two sides: on one side we need to scan the images to determine if there are known vulnerabilities in it, on the other side we need to prevent images that don’t pass the quality check from running as containers. This enforcement resides outside of the pipeline and can be configured in OpenShift via the image admission control.

An example image scanning pipeline step can be found here.

Environment dependent validation steps:

Validation step Common tool / technology implementations

Integration test

Various tools and techniques. BDD seems to be emerging for web and mobile apps.

Canary test

Pipeline orchestrated and tested with integration testing tools.

Chaos test

Gremlin, Fabric8 Chaos Monkey.

Behavioral Driven Development (BDD) testing is a valuable option and should be used for testing web and mobile applications.
BDD test cases are created using a pseudo natural language (Gherkin), allowing them to be quickly and easily created by non-technical users. The test cases are translated into machine instructions at test runtime (a popular one is Cucumber).
The general architecture of BDD tests is illustrated in the following diagram:

BDD

The above architecture can be fully deployed and contained in OpenShift. Multiple combinations of browser/operating system and mobile device/operating system tests can be executed in parallel. The advent of Windows containers will provide additional opportunities for testing applications running on Internet Explorer and Edge.

Compliance Stage

Compliance steps may be needed when pipelines must be compliant with an organization’s release process and change management.

Examples of compliance steps are provided in the following table:

Step Description

Manual approval

Suspends pipeline execution until manual approval to continue is provided .

Container image signing

Cryptographically signs images

Deployment manifest upload

Uploads the manifest of what is deployed to a centralized change management tool.

Container image signing provides a method for making an assertion cryptographically verifiable. When an image is signed (a fact to which any meaning can be assigned, for example, the image is approved for execution ), anyone can verify that assertion. Again having assertions is only one piece of the process, the other piece, which resides outside of the pipeline, is having a way to verify and enforce them. Here you can find an example of a pipeline with an container image signing step.

Google is currently working on a more general purpose assertion framework for deployment pipelines (see Grapheas and Kritis).

Ownership of the Pipeline

It is important to identify who in the organization owns the release process, tools, and pipelines. For traditional ITIL-based organizations, there is typically a release management team that owns the release process. That team might also own the tools involved in supporting the pipelines, and, in some cases, own the actual pipelines.

Ownership should be clearly defined for all pipeline-related aspects:

  • Release policies and compliance rules

  • Templates and other general reusable pipeline artifacts

  • Tools needed to support the pipeline

  • Individual pipelines

The current trend for agile organizations is to have developer teams own the pipelines for their specific applications. In this case, the development teams are responsible for creating and updating the pipelines, as well as ensuring pipeline compliance with enterprise-issued release management standards.

Because pipeline processes can be described as code, developers must learn the pipeline Domain Specific Language (DSL), but otherwise have a short learning curve.

More traditional organizations may not be comfortable with this approach because they want to have a clear separation of duty between development activities and deployment tasks.

Each organization is different and will have to determine their own ownership model for pipelines. For more on the organizational impact of adopting container technologies, see our guide, OpenShift and the Organization.

Metrics and Dashboards

A wealth of metrics can be collected from CI/CD pipelines, and much of the value of a pipeline is lost if metrics are not collected and made available in dashboards. DevOps teams utilize metrics for fast feedback.

Useful metrics might include: number of deployment errors and total pipeline execution time.

Although there are not many tools currently available for providing pipeline metrics, one example worth examining is Hygieia from Capital One.

A pipeline dashboard is illustrated in the following screen capture:

hygiea screenshot

Conclusions

This guide has described various deployment pipeline concepts in the context of OpenShift Container Platform and OCI containers. Immutable infrastructure and container platform automation don’t change the strategy or goals of continuous delivery but they do affect the operation of the pipeline. Definitions of pipeline phases, types, and ownership can be adjusted to leverage OpenShift behavior. Build, promotion, validation, and compliance steps can all benefit from OpenShift features designed to support CICD. And pipeline metrics and dashboards can integrate with Kubernetes to provide useful feedback on deployment automation performance. We recommend using the OpenShift automation and integration possibilities described here to enhance your deployment pipelines to provide rapid, frictionless, higher quality software releases.