GOTCHAS USING TERRAFORM IN A SECURE DELIVERY

PIPELINE
by Anton Babenko
 2

AGENDA
▸ Key concepts in Terraform

▸ How CI/CD pipeline can look like

▸ How Terraform can be used there

▸ FTP (Frequent Terraform Problems)

 3

ANTON BABENKO
Terraform AWS fanatic

Organise {HashiСorp User Group, AWS User Group,

DevOpsDays} Oslo

github.com/terraform-aws-modules

github.com/antonbabenko

@antonbabenko

linkedin.com/in/antonbabenko

anton@antonbabenko.com
 4

Write, plan, and create infrastructure as code

www.terraform.io
 5

image from cloudcraft.co

CI/CD PIPELINE 9

WHY CI/CD PIPELINE IS IMPORTANT?

It gives you fair idea about bottlenecks that likely to occur and increase
confidence when it comes to avoiding these bottlenecks.
CI/CD PIPELINE 10

image from www.i-visionblog.com

GETTING FROM 0 TO 1 11

READ => WRITE => TERRAFORM APPLY

WAY AHEAD… 12

TERRAFORM MODULES
Reuse code & «not invented here»

▸ registry.terraform.io - 300+ modules

▸ github.com/terraform-aws-modules
WAY AHEAD…

TERRAFORM WORKSPACE
Just use terraform_remote_state instead… EOF
 14

CI/CD TOOLS
▸ CircleCI/TravisCI

▸ Plugins for Concourse/Drone/Jenkins/Ansible

 15

ATLANTIS

A unified workflow for collaborating on

Terraform through GitHub and GitLab

www.runatlantis.io
 16

▸ GUI for plan, apply, locking, state rollbacks

▸ Sentinel - policy as code

▸ Private modules registry

▸ Enterprise pricing
 17

PRE-SUMMARY
Terraform code

CI/CD options
CI/CD GOTCHAS #1 18

GENERAL
▸ Remote state only

▸ Error handling:

▸ retry X times

▸ terraform plan -parallelism=1

▸ terraform plan -detailed-exitcode

CI/CD GOTCHAS #2 19

TERRAFORM DEPENDENCIES & PLUGINS

▸ terraform init

▸ .terraform/*

▸ vendor, fork (modules, providers)

▸ private repositories
CI/CD GOTCHAS #3 20

SECRETS
▸ Part of CI system, environment variables

▸ Stored in state file

▸ Use provider specific features:

▸ PGP (aws_iam_login_profile, aws_iam_access_key)

▸ RDS (iam_database_authentication_enabled) or change outside

CI/CD GOTCHAS #3 21
CI/CD GOTCHAS #3 22
ACCESS CONTROL 23

THE PRINCIPLE OF LEAST PRIVILEGE REQUIRES THAT EVERY PROCESS MUST

BE ABLE TO ACCESS ONLY THE INFORMATION AND RESOURCES THAT ARE
NECESSARY FOR ITS LEGITIMATE PURPOSE.

Principle of least privilege

ACCESS CONTROL 24

GRANT ONLY THE PERMISSIONS REQUIRED TO

PERFORM A TASK.

AWS IAM best practices

ACCESS CONTROL IN AWS 25
 26

SECURE DELIVERY PIPELINE

▸ Access control

▸ «Do what you really want»

▸ PR approval system - https://github.com/capitalone/checks-out

▸ Depends on how you structure your code

DEEP(ER) DIVE 27

TERRAFORM DESIGN PATTERNS

▸ Resource modules (eg, terraform-aws-modules)

▸ Infrastructure modules

▸ Composition
DESIGN PATTERNS 28

TERRAFORM RESOURCE MODULES (EG, TERRAFORM-AWS-MODULES)

▸ Create resources (obviously)

▸ No relations to other modules (usually)

▸ Very flexible

▸ Versioning
DESIGN PATTERNS 29

TERRAFORM INFRASTRUCTURE MODULES

▸ Use specific version of resource modules

▸ Company-wide standards (eg, tags and names)

▸ May use code generators (jsonnet, cookiecutter, etc)

▸ Versioning
DESIGN PATTERNS 30

TERRAFORM COMPOSITION
▸ Use specific version of infrastructure or resource modules

▸ Provide all the values for region, environment, module, etc

▸ Terragrunt is awesome
TYPES OF DELIVERY PIPELINES 31

BASIC PIPELINE
▸ git checkout -b new-feature

▸ git commit

▸ git push origin new-feature

▸ Open a pull-request

▸ …

▸ PR is approved, new-feature is merged to master (protected branch)

▸ terraform apply in master branch

TYPES OF DELIVERY PIPELINES 32

BASIC PIPELINE (SAMPLE)

TYPES OF DELIVERY PIPELINES 33

PIPELINE FOR MODULES

▸ Generate missing configurations (jsonnet, for eg)

▸ Run automated tests (kitchen-terraform, for eg)

▸ Publish summary as a comment

▸ Notify downstream users - https://github.com/justwatchcom/github-releases-

notifier 

TOOLS 34

RELATED TOOLS
▸ github.com/antonbabenko/pre-commit-terraform

▸ github.com/wata727/tflint

▸ github.com/segmentio/terraform-docs

▸ github.com/kamatama41/tfenv

▸ github.com/gruntwork-io/terragrunt
 36

PIPELINES ARE GOOD, BUT NOT ENOUGH

▸ Refactoring

▸ Upgrades

▸ Rollbacks

▸ Force unlock
FREQUENT TERRAFORM PROBLEMS (FTP) 37

reddit.com/r/Terraform
FTP 38

REFACTORING
▸ terraform state mv

▸ Upgrade versions of modules

▸ Run terraform commands recursively

github.com/antonbabenko/terrible
FTP 39

TERRAFORM UPGRADE
▸ Enabled versioning on state bucket, right!?

▸ Use tfenv to manage Terraform version

FTP 40

TERRAFORM LOCKS
▸ terraform force-unlock

▸ Remember Atlantis
 41

DEMO
https://github.com/antonbabenko/terraform-deployment-pipeline-talk
 42

FINAL SUMMARY
▸ Use CI/CD for automated tasks (check, plan, apply, destroy, merge, promotion)

▸ Use minimal roles + MFA

KEEP IT SIMPLE, STUPID 43

No workspaces and no
advanced arguments in CLI
QUESTIONS?