Date
Tags devops

Part 1

This is part one of a one part series that covers generalities of CI/CD. Note that this is not a recipe for doing something specific with something like Jenkins.. this is an opinionated and high level overview of basic principles.

If you're most interested in ..., you can leave this part and move directly to part 2.

Project / Repo Structure

  1. Keep directory structure as flattened as you reasonably can. This is a place where CI implementation code can easily get coupled to project code. For example, while project refactors could certainly break individual tests, they should not break our ability to execute tests.

  2. Get clarity on repo role and purpose. Does it cover a single libraries or tool? Does it bundle diverse libraries/tool so they can be shipped together (Dockerfiles, helm charts)? Repos should be monolingual if possible for simplicity when added static-analysis procedures, etc.

  3. Separate config from code if you can, leaving at most just enough config-template for running unittests. If you must bundle the config with your project code, use encryption for secrets with commit-hooks to enforce it.

  4. For simplicity, don't use submodules unless you really must (looking at you, golang).

  5. Avoid junking up the repo with large files. If you need these in your build, versions should be pinned explicitly, and pulled and pushed from something like S3, Artifactory, etc. Make appropriate use of something like git lfs from the beginning if you really must throw big files (jars and stuff) in version control.

CI Pipeline Stages

  • Pull: Wherein one or more project repositories are cloned.

  • Inject: Wherein any project configuration is retrieved if necessary, possibly decrypted or rendered. Configuration might be in a variety of external places to start with, say in another repo (clone it), or in consul or zookeeper (curl it).

  • Static Analysis: Wherein project code and config structure is analyzed syntactically. The "static" part means that code is NOT executed.. this is crucial so that we don't have to care whether the code has side-effects.

  • Build: Wherein project code is built. This is possibly a no-op for non-compiled languages, but the analog would be setup with Ruby's rvm or python's virtualenv. Builds should be happening from scratch in all cases, not caching rvm or virtualenv state between builds.

  • Test: Wherein testing happens, possibly lots of testing that includes Units, Integration, E2E. All build stages EXCEPT possibly testing should follow a "fail fast" execution paradigm.

  • Tag: Wherein push happens, for both metadata and artifacts. This includes merging, simple branch tagging, or artifact deployment. This might be a push to ECR, S3, Dockerhub, Github, Nexus, Artifactory, etc. I call it "tagging" to highlight the fact that even if you aren't doing something as simple as pushing git tags, you need be to pushing a versioned artifact.

  • Deploy: Wherein servers are updated, hopefully in some kind of cool blue/green setup or with autoscaling groups and rolling deployments.

CI Pipelines as Code

CI Pipelines as Code

Part 1 of 1 in "Intro to CI/CD"