Using Terraform Cloud to Manage GitHub Repositories

8 min readDec 8, 2019

Terraform Cloud is a hosted service that helps manages the coordination of terraform builds when working with remote teams. It has the option to run locally, or within the service. Runs are queued in order, thereby reducing the chance of conflict or overwrites across a team. Other useful features are the ability to role back, team based permissions, and change audit trail.

What this Page Covers

This page covers setting up a Terraform Cloud workspace to manage your GitHub repositories.

Prerequisites

a GitHub organisation
GitHub organisation admin rights

Set Up Terraform Cloud

The setup involves some flipping between GitHub and Terraform Cloud. Head over to TC and signup for an account. Once the account is created, you’ll get the option of creating a new organisation or join an existing team. Select create a new organisation. By default the creator, will be placed in the owners group, with full permissions. Go ahead and create an organisation with a unique name and the owners email address.

Next you’ll be asked to link up a version control provider. This is essential as the code needs to come from somewhere!

Link to GitHub

Step One Create an OAuth App

Login to GitHub as organisation admin, go to the organisation settings, Developer Settings section and click on OAuth Apps. Fill in the details:
* the OAuth app name (Terraform Cloud)
* the Homepage URL (https://app.terraform.io)
* a callback URL, a dummy value is fine as we will get this from Terraform Cloud in step two.

Then register the app. Copy the Client ID and the Client Secret to use in step two.

Step Two Add the VCS Provider to Terraform Cloud

Go back to Terraform Cloud and add VCS Provider. Fill in the copied:
* Client ID
* Client Secret
as directed and click on “Create VCS Provider”. Now copy the Callback URL as we will need to change the dummy value from step one.

Step Three Update the GitHub OAuth App

Go back to GitHub and locate the OAuth app created in step one. In the “Authorization Callback URL” field, replace the dummy value with the Callback URL we copied in the previous stage and click “Update Application”

Step Four Request Access for Terraform Cloud

Go back to the Terraform Cloud VCS Provider page, and click on “Connect Organization”. Follow the redirect pages and click on the green “authorize” button. Assuming you are the GitHub admin, check your emails and approve the request, or contact an admin and ask them to do so. More detailed instructions from TC are here.

Terraform Cloud now has the ability to use GitHub as a source for executions. Next we will set up a GitHub repository for managing all our repositories, teams and members. PRs in this repo will be planned, and on merge, the changes will be applied from TC.

Creating A Terraform Cloud Managed Repo

In this section we will create a repo a repo that is managed by Terraform Cloud. We need to:

manually create the management repo
create terraform files and add to GitHub
create TC and GitHub tokens
import the management repo to TC
switch to TC executions

Manually Create a Repo

We first need to manually create our starting repo and then import to Terraform Cloud. Go to GitHub and create a repo called github-management, make sure to select “Initialise this repository with a readme” option, “add .gitignore Terraform) and “add a license MIT”. Now clone the repo locally:

git clone git@github.com:<org-name>/github-management.git

Create Terraform Files and Add to GitHub

To run terraform we will need to add the GitHub provider, a TC backend and a repository.tf file for the repo import. In the provider.tf change the organisation to your GitHub one and in the backend.tf change the organisation to your TC one.

provider.tf
# Configure the GitHub Provider
provider "github" {
  version      = "~> 2.2"
  organization = "<github-org-name>"
  # Export the GITHUB_TOKEN as an environment variable
}backend.tf
# Add the terraform cloud backend for running locally
terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "<tc-org-name>"

    workspaces {
      name = "github-management"
    }
  }
}repository.tf
# The terraform resource for the repository
resource "github_repository" "github-management" {
  name        = "github-management"
  description = "Terraform based repository to manage all our GutHub repositories"

  private            = false
  has_issues         = true
  has_wiki           = false
  allow_merge_commit = false
  allow_squash_merge = true
  allow_rebase_merge = true
  auto_init          = false
  license_template   = "mit"
  topics             = ["config", "terraform"]
}

Add the files to GitHub, note that we do not need to add the backend.tf file as TC creates its own. If you want teams to test the plan locally, add the file. For the setup, we push straight to master because TC will sync with master when connecting the workspace.

git add provider.tf repository.tf
git commit -m "Adding provider and repo files"
git push origin master

Create GitHub and Terraform Cloud Tokens

Tokens allow access to the relevant GitHub and TC APIs. We need these tokens locally for the initial terraform import.

For GitHub: go to your profile (top right) >>Settings>>Developer Settings>>Personal Access Tokens and create a token called terraform_cloud with:

all repo rights
admin:org read and write
all user rights
delete_repo (for archiving)

Copy the token, we will need this for running the terraform import and TC will need for calling GitHub APIs

For TC: click on your account icon at the top right and go to User Settings and then Tokens. Generate a new token called tf_remote and copy. In your home directory, create a file called .terraformrc, edit the contents like this:

credentials "app.terraform.io" {
   token = "<paste-token-here>"
}

Import the management repo to TC

Now we need to initialise terraform so run

terraform initInitializing the backend...Successfully configured the backend "remote"! Terraform will automaticallyuse this backend unless the backend configuration changes.

Go to terraform cloud and you will see a workspace called github-management

Showing the initial setup of a TC workspace

At this point nothing has really happened, we are just preparing. Now import the repository like this:

export GITHUB_TOKEN=<token-from-above>
terraform import github_repository.github_management github-management
github_repository.github_management: Importing from ID "github-management"...github_repository.github_management: Import prepared!Prepared github_repository for importgithub_repository.github_management: Refreshing state... [id=github-management]Import successful!The resources that were imported are shown above. These resources are now inyour Terraform state and will henceforth be managed by Terraform.Releasing state lock. This may take a few moments...

If you get an error here, go to Settings/General and change the execution mode to local (remote should be ok, but I have had issues in the past). Now switch back to TC and you will see the state has updated:

Switch to Terraform Cloud Executions

Now all the preparation is done, we can switch to running executions in TC, so that a PR update will run plan, and a merge will run plan and apply without intervention.

Link the TC Workspace to GitHub

There are a few things in the workspace we need to update in the TC console.

go to Variables and add and Environment Variable called GITHUB_TOKEN with the github token we copied above, and check the “Sensitive” box to make non-readable
go to Settings/Version Control, then select the VCS we linked above, choose the repository <org-name>/github-management and update VCS Settings.

In the workspaces summary page the Repo column will have updated like this

Workspace view after GitHub repo configured

Click on the github-management link to go to this page

Showing the workspace home page after GitHub configuration

The first run needs manually triggering, so click on “Queue plan”. Look at the plan before approving:

Plan successful with unexpected resource destroy

Oh oh, we have a problem here. The license template is forcing a rebuild of my repo! Let’s check the state

Showing the license template import failure

Looks like the the license template did not import correctly. The problem can be fixed by removing license_template variable from the github_repository resource. I filed an issue for this.

Update the file and push to GitHub and queue again. This time it all looks good.

Plan: 0 to add, 1 to change, 0 to destroy.

Click on “Confirm and Apply”, write a message and everything should be good like this

Switch to Terraform Cloud Auto Run

The final step is to change the workspace to auto apply on merge, after all without this, where is our automation? A merge to master should represent the state we want, or why merge? In TC, go the Settings/General and cahnge the “Apply Method” to Auto apply. Then we need to be sure the master branch is protected and that PRs are validated by TC. Create a branch, add the following lines to repository.tf and then commit to a branch

git checkout -b feature/branch-protection# Set up baseline configs for the repo
resource "github_branch_protection" "team_baseline_config" {
  repository     = github_repository.github_management.name
  branch         = "master"

  required_status_checks {
    # require up to date before merging
    strict = true
    contexts = ["atlas/mononoke/github-management", ]
  }
  required_pull_request_reviews {
    dismiss_stale_reviews      = true
    require_code_owner_reviews = false
  }
}git push -u origin fix/branch-protection

Then go to GitHub and create the PR. TC will pick this up and run the plan. After a few minutes you should see something like this in GitHub at the bottom of the PR:

TC posts back to GitHub with check status

Click on “Squash and merge” and TC will run plan and then apply. Go the the relevant TC workspace and this is what you should get:

Now the initial repo is set up, you can set up users, teams and more repos. In the long run, it’s far more convenient and transparent to configure GitHub this way. When you leave a project, you can even create your own PR to remove yourself!

Improvements and Gotchas

Make sure you enable 2FA for both GitHub and TC, and download the recovery codes.
Avoid switching between remote and local runs for TC. Stick with remote.
TC can run on branches if you really want to, best avoided, running on branches leads to the dark side.
A single repo can have multiple workspaces configured — in workspace General Settings, set the Terraform Working Directory.
Use a bot user GitHub token for running TC, to avoid dependency on individuals who may leave.
Create a repo and workspace to configure TC workspaces.
Use Terraform Registry modules for GitHub.

Links

Terraform Cloud
Terraform Provider for GitHub
Terraform Provider for TC
Terraform Registry