Scalable and Organized Terraform Project Structure

Due to the lack of official guidelines for structuring larger Terraform projects, teams often face challenges like waiting for state locks and slow apply times. This post presents a scalable Terraform project structure designed to address these challenges.

• Starting Simple
• Challenges of a Simple Structure
• Solution: Split State by Service and Environment
• Managing Cross-State Dependencies
• Avoiding Code Duplication
• Conclusion

Starting Simple

When starting a new Terraform project, a common approach is to organize the project by environment, such as development, staging, and production, with each environment maintaining its own Terraform state. In this setup, each environment's Terraform files either define resources directly or call modules to provision the required infrastructure. While this structure works perfectly well for simple projects, it can become a bottleneck as the projects grow or teams scale up.

├── modules
│   ├── x
│   ├── y
│   └── z
├── development
│   ├── main.tf
│   └── terraform.tf
├── staging
│   ├── main.tf
│   └── terraform.tf
└── production
    ├── main.tf
    └── terraform.tf

Challenges of a Simple Structure

This simple project structure introduces several key challenges:

• State locking: Multiple teammates cannot work in parallel due to Terraform state locking, creating bottlenecks.
• Slow plan and apply times: As the number of Terraform resources grows, the duration of terraform operations increases, slowing down development and amplifying locking challenges.
• Provider upgrades: Upgrading Terraform providers becomes daunting because all breaking changes must be addressed simultaneously.
• State Reconstruction: If something happens to the state file, reconstructing it (e.g., using terraform import) becomes significantly harder.

Solution: Split State by Service and Environment

Explore the example source code here.

To address these issues, consider splitting the project (and thus the Terraform state) into smaller, more manageable pieces. Divide the project by services, and within each service, organize by environments (optionally adding further subdivisions like regions).

This approach provides several benefits:

• Teammates can work in parallel on different services without conflicting locks.
• Smaller state files mean quicker applies and thus improved productivity.
• Terraform provider upgrades can be performed gradually impacting fewer resources

├── modules
│   ├── x
│   ├── y
│   └── z
└── services
    ├── x
    │   ├── development
    │   ├── staging
    │   └── production
    ├── y
    │   ├── development
    │   ├── staging
    │   └── production
    └── z
        ├── development
        ├── staging
        └── production
            ├── europe-west1
            └── us-west1

Managing Cross-State Dependencies

A common challenge with the proposed project structure is managing dependencies between services or states. For example, one service might rely on outputs from another service’s state. Instead of passing outputs directly, use indirection techniques like:

• Use DNS instead of IPs.
• Fetch secrets directly from a secret manager.
• Utilize Terraform data sources to reference shared resources.

For instance, consider two services that need to reference a common Postgres service:

resource "google_sql_database_instance" "postgres" {
  name = "my-shared-postgres"
  ...
}

data "google_sql_database_instance" "postgres" {
  name = "my-shared-postgres"
}

data "google_sql_database_instance" "postgres" {
  name = "my-shared-postgres"
}

Avoiding Code Duplication

Another challenge is avoiding code duplication across environments. To tackle this, keep Terraform files in the services folder simple. They should primarily call modules and pass minimal variables, such as the environment name. Differentiations between environments should be handled within the module.

For example, consider a module that provisions a Postgres database with different resource specifications depending on the environment. The module abstracts environment-specific details, reducing duplication, and improving maintainability and readability (from the module, it's immediately clear how the resources will differ between the environments).

locals {
  tier = {
    development = "db-f1-micro"
    staging     = "db-custom-2-7680"
    production  = "db-custom-4-15360"
  }
}

resource "google_sql_database_instance" "postgres" {
  name = "my-shared-postgres"

  settings {
    tier = locals.tier[var.env]
  }
}

module "shared_postgres" {
  source = "../../../modules/shared_postgres"
  env    = "development"
}

module "shared_postgres" {
  source = "../../../modules/shared_postgres"
  env    = "production"
}

Conclusion

While there is no official guideline, proposed Terraform project structure can scale to support hundreds of services without compromising team productivity.

However, there are additional challenges to consider:

• Each state must download the necessary providers, often redundantly. This could be optimized using a shared local cache.
• Changes to one service may require re-applying dependent services. Automating this process and resolving circular dependencies can be difficult. Fortunately, most changes occur in the application layer, where other tools (GitOps) can be helpful.

Need help with your Terraform project or want to upgrade your team's skills?
Let's connect