Using Terraform Cloud to Manage GitHub Repositories

Terraform Cloud is a hosted service that helps manages the coordination of terraform builds when working with remote teams. It has the option to run locally, or within the service. Runs are queued in order, thereby reducing the chance of conflict or overwrites across a team. Other useful features are the ability to role back, team based permissions, and change audit trail.

What this Page Covers

Prerequisites

  • GitHub organisation admin rights

Set Up Terraform Cloud

Next you’ll be asked to link up a version control provider. This is essential as the code needs to come from somewhere!

Link to GitHub

Login to GitHub as organisation admin, go to the organisation settings, Developer Settings section and click on OAuth Apps. Fill in the details:
* the OAuth app name (Terraform Cloud)
* the Homepage URL (https://app.terraform.io)
* a callback URL, a dummy value is fine as we will get this from Terraform Cloud in step two.

Then register the app. Copy the Client ID and the Client Secret to use in step two.

Step Two Add the VCS Provider to Terraform Cloud

Go back to Terraform Cloud and add VCS Provider. Fill in the copied:
* Client ID
* Client Secret
as directed and click on “Create VCS Provider”. Now copy the Callback URL as we will need to change the dummy value from step one.

Step Three Update the GitHub OAuth App

Go back to GitHub and locate the OAuth app created in step one. In the “Authorization Callback URL” field, replace the dummy value with the Callback URL we copied in the previous stage and click “Update Application”

Step Four Request Access for Terraform Cloud

Go back to the Terraform Cloud VCS Provider page, and click on “Connect Organization”. Follow the redirect pages and click on the green “authorize” button. Assuming you are the GitHub admin, check your emails and approve the request, or contact an admin and ask them to do so. More detailed instructions from TC are here.

Terraform Cloud now has the ability to use GitHub as a source for executions. Next we will set up a GitHub repository for managing all our repositories, teams and members. PRs in this repo will be planned, and on merge, the changes will be applied from TC.

Creating A Terraform Cloud Managed Repo

  • manually create the management repo
  • create terraform files and add to GitHub
  • create TC and GitHub tokens
  • import the management repo to TC
  • switch to TC executions

Manually Create a Repo

We first need to manually create our starting repo and then import to Terraform Cloud. Go to GitHub and create a repo called github-management, make sure to select “Initialise this repository with a readme” option, “add .gitignore Terraform) and “add a license MIT”. Now clone the repo locally:

git clone git@github.com:<org-name>/github-management.git

Create Terraform Files and Add to GitHub

To run terraform we will need to add the GitHub provider, a TC backend and a repository.tf file for the repo import. In the provider.tf change the organisation to your GitHub one and in the backend.tf change the organisation to your TC one.

provider.tf
# Configure the GitHub Provider
provider "github" {
version = "~> 2.2"
organization = "<github-org-name>"
# Export the GITHUB_TOKEN as an environment variable
}
backend.tf
# Add the terraform cloud backend for running locally
terraform {
backend "remote" {
hostname = "app.terraform.io"
organization = "<tc-org-name>"

workspaces {
name = "github-management"
}
}
}
repository.tf
# The terraform resource for the repository
resource "github_repository" "github-management" {
name = "github-management"
description = "Terraform based repository to manage all our GutHub repositories"

private = false
has_issues = true
has_wiki = false
allow_merge_commit = false
allow_squash_merge = true
allow_rebase_merge = true
auto_init = false
license_template = "mit"
topics = ["config", "terraform"]
}

Add the files to GitHub, note that we do not need to add the backend.tf file as TC creates its own. If you want teams to test the plan locally, add the file. For the setup, we push straight to master because TC will sync with master when connecting the workspace.

git add provider.tf repository.tf
git commit -m "Adding provider and repo files"
git push origin master

Create GitHub and Terraform Cloud Tokens

Tokens allow access to the relevant GitHub and TC APIs. We need these tokens locally for the initial terraform import.

For GitHub: go to your profile (top right) >>Settings>>Developer Settings>>Personal Access Tokens and create a token called terraform_cloud with:

  • all repo rights
  • admin:org read and write
  • all user rights
  • delete_repo (for archiving)

Copy the token, we will need this for running the terraform import and TC will need for calling GitHub APIs

For TC: click on your account icon at the top right and go to User Settings and then Tokens. Generate a new token called tf_remote and copy. In your home directory, create a file called .terraformrc, edit the contents like this:

credentials "app.terraform.io" {
token = "<paste-token-here>"
}

Import the management repo to TC

terraform initInitializing the backend...Successfully configured the backend "remote"! Terraform will automaticallyuse this backend unless the backend configuration changes.

Go to terraform cloud and you will see a workspace called github-management

Showing the initial setup of a TC workspace

At this point nothing has really happened, we are just preparing. Now import the repository like this:

export GITHUB_TOKEN=<token-from-above>
terraform import github_repository.github_management github-management
github_repository.github_management: Importing from ID "github-management"...
github_repository.github_management: Import prepared!Prepared github_repository for importgithub_repository.github_management: Refreshing state... [id=github-management]Import successful!The resources that were imported are shown above. These resources are now inyour Terraform state and will henceforth be managed by Terraform.Releasing state lock. This may take a few moments...

If you get an error here, go to Settings/General and change the execution mode to local (remote should be ok, but I have had issues in the past). Now switch back to TC and you will see the state has updated:

Workspace state view after repo import

Switch to Terraform Cloud Executions

Link the TC Workspace to GitHub

There are a few things in the workspace we need to update in the TC console.

  • go to Variables and add and Environment Variable called GITHUB_TOKEN with the github token we copied above, and check the “Sensitive” box to make non-readable
  • go to Settings/Version Control, then select the VCS we linked above, choose the repository <org-name>/github-management and update VCS Settings.

In the workspaces summary page the Repo column will have updated like this

Workspace view after GitHub repo configured

Click on the github-management link to go to this page

Showing the workspace home page after GitHub configuration

The first run needs manually triggering, so click on “Queue plan”. Look at the plan before approving:

Plan successful with unexpected resource destroy

Oh oh, we have a problem here. The license template is forcing a rebuild of my repo! Let’s check the state

Showing the license template import failure

Looks like the the license template did not import correctly. The problem can be fixed by removing license_template variable from the github_repository resource. I filed an issue for this.

Update the file and push to GitHub and queue again. This time it all looks good.

Plan: 0 to add, 1 to change, 0 to destroy.

Click on “Confirm and Apply”, write a message and everything should be good like this

Successful run confirmation

Switch to Terraform Cloud Auto Run

git checkout -b feature/branch-protection# Set up baseline configs for the repo
resource "github_branch_protection" "team_baseline_config" {
repository = github_repository.github_management.name
branch = "master"

required_status_checks {
# require up to date before merging
strict = true
contexts = ["atlas/mononoke/github-management", ]
}
required_pull_request_reviews {
dismiss_stale_reviews = true
require_code_owner_reviews = false
}
}
git push -u origin fix/branch-protection

Then go to GitHub and create the PR. TC will pick this up and run the plan. After a few minutes you should see something like this in GitHub at the bottom of the PR:

TC posts back to GitHub with check status

Click on “Squash and merge” and TC will run plan and then apply. Go the the relevant TC workspace and this is what you should get:

Boom, all applied!

Now the initial repo is set up, you can set up users, teams and more repos. In the long run, it’s far more convenient and transparent to configure GitHub this way. When you leave a project, you can even create your own PR to remove yourself!

Improvements and Gotchas

  • Avoid switching between remote and local runs for TC. Stick with remote.
  • TC can run on branches if you really want to, best avoided, running on branches leads to the dark side.
  • A single repo can have multiple workspaces configured — in workspace General Settings, set the Terraform Working Directory.
  • Use a bot user GitHub token for running TC, to avoid dependency on individuals who may leave.
  • Create a repo and workspace to configure TC workspaces.
  • Use Terraform Registry modules for GitHub.

Links

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store