Terraform: Updating State Using “Moved” Block

This post illustrates how you can rename existing resources or restructure the Terraform codebase without destroying and recreating the resources using moved block introduced in Terraform 1.1. It also explains some limitations using this new construct.

PROBLEM: MODIFYING EXISTING RESOURCE NAME

Using a simple resource block below as an example…

resource "random_pet" "current" {}

On apply, one resource is created and the state now tracks that resource.

$ terraform state list
random_pet.current

When changing the resource name from current to new

resource "random_pet" "new" {}

The generated plan indicates the resource will be destroyed and recreated, which may cause potential data loss (ex: if the resource is a database) or unintended ripple effects (because the resource ID has changed).

In this case, congratulations! We have become pet killers.

# random_pet.current will be destroyed
# (because random_pet.current is not in configuration)
- resource "random_pet" "current" {
    - id        = "willing-fowl" -> null
    - length    = 2 -> null
    - separator = "-" -> null
  }

# random_pet.new will be created
+ resource "random_pet" "new" {
    + id        = (known after apply)
    + length    = 2
    + separator = "-"
  }

Plan: 1 to add, 0 to change, 1 to destroy.

Solving With Terraform Earlier Than 1.1

Before Terraform 1.1, the only way to change the resource name without recreating the resource is to use terraform state mv.

$ terraform state mv random_pet.current random_pet.new
Move "random_pet.current" to "random_pet.new"
Successfully moved 1 object(s).

We can also verify that the resource name has changed in the state.

$ terraform state list
random_pet.new

Solving With Terraform 1.1

Terraform 1.1 introduced moved block where the state change process can be done directly in Terraform source code without using terraform state mv.

To pull this off, we changed the resource name from current to new. Then, a moved block is added to facilitate the state change without affecting the provisioned resource.

resource "random_pet" "new" {}

moved {
  from = random_pet.current
  to   = random_pet.new
}

Now, we can verify the generated plan to ensure the provisioned resource does not get destroyed and recreated before applying the move.

# random_pet.current has moved to random_pet.new
resource "random_pet" "new" {
  id = "willing-fowl"
  # (2 unchanged attributes hidden)
}

Plan: 0 to add, 0 to change, 0 to destroy.

The state should have the updated resource name.

$ terraform state list
random_pet.new

So, what if the moved block is left in the codebase, and we do another plan or apply?

$ terraform apply
random_pet.new: Refreshing state... [id=willing-fowl]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

As you can see, leaving the moved block in the codebase does not affect the state or the resource, and Terraform is smart enough to ignore it. In another word, once the “move” has occurred, we can safely remove the moved block.

PROBLEM: RESTRUCTURING CODEBASE USING “FOR_EACH”

Let’s try something a little more complicated.

Let’s assume we have 3 random pets where we call the resource block 3 times. Perhaps, you inherited the old codebase from pre v0.13 days where for_each construct was not introduced yet.

resource "random_pet" "a" {}
resource "random_pet" "b" {}
resource "random_pet" "c" {}

Here’s how the state looks like.

$ terraform state list
random_pet.a
random_pet.b
random_pet.c

Let’s assume we want to refactor the existing codebase to leverage the for_each construct.

resource "random_pet" "main" {
  for_each = toset(["a", "b", "c"])
}

Solving With Terraform Earlier Than 1.1

With older version of Terraform, we have to perform the following terraform state mv 3 times.

$ terraform state mv random_pet.a 'random_pet.main["a"]' 
Move "random_pet.a" to "random_pet.main[\"a\"]"
Successfully moved 1 object(s).

$ terraform state mv random_pet.b 'random_pet.main["b"]' 
Move "random_pet.b" to "random_pet.main[\"b\"]"
Successfully moved 1 object(s).

$ terraform state mv random_pet.c 'random_pet.main["c"]' 
Move "random_pet.c" to "random_pet.main[\"c\"]"
Successfully moved 1 object(s).
$ terraform state list 
random_pet.main["a"]
random_pet.main["b"]
random_pet.main["c"]

Solving With Terraform 1.1

To pull this off with Terraform 1.1, we call the moved block 3 times.

resource "random_pet" "main" {
  for_each = toset(["a", "b", "c"])
}

moved {
  from = random_pet.a
  to   = random_pet.main["a"]
}

moved {
  from = random_pet.b
  to   = random_pet.main["b"]
}

moved {
  from = random_pet.c
  to   = random_pet.main["c"]
}

In the generated plan, we want to verify that there are no changes made to the provisioned resources before applying changing the state.

# random_pet.a has moved to random_pet.main["a"]
resource "random_pet" "main" {
  id = "poetic-collie"
  # (2 unchanged attributes hidden)
}

# random_pet.b has moved to random_pet.main["b"]
resource "random_pet" "main" {
  id = "glad-lobster"
  # (2 unchanged attributes hidden)
}

# random_pet.c has moved to random_pet.main["c"]
resource "random_pet" "main" {
  id = "suited-fly"
  # (2 unchanged attributes hidden)
}

Plan: 0 to add, 0 to change, 0 to destroy.

As you can see, we end up with the same outcome.

$ terraform state list
random_pet.main["a"]
random_pet.main["b"]
random_pet.main["c"]

The upside is we can still stick with our terraform apply to adjust the state without using terraform state mv.

LIMITATIONS

It’s worth pointing out that it’s all sunshine and rainbows here.

The moved block solves just part of the state manipulation problems, and you may still need to use terraform state [action] for other use cases (at least with Terraform 1.1.x).

Limitation #1: Removing Resources(s) from State

In rare cases, you may want Terraform to stop tracking a resource in its state file. Perhaps, you run into a race condition problem when attempting to destroy the resources, which prevents Terraform from completing the process successfully. As a result, all resources are fully destroyed (via “eventual consistency”) but unfortunately, a few non-existent resources are still being tracked in the Terraform’s state file.

In this case, you can’t remove the resource from the state by assigning an empty string…

moved {
  from = random_pet.current
  to   = ""
}
$ terraform apply
╷
│ Error: Invalid expression
│
│   on main.tf line 21, in moved:
│   21:   to   = ""
│
│ A single static variable reference is required: only attribute access and indexing with constant keys. No calculations, function calls, template expressions, etc are
│ allowed here.

… or to null

moved {
  from = random_pet.current
  to   = null
}
$ terraform apply
╷
│ Error: Invalid address
│
│   on main.tf line 21, in moved:
│   21:   to   = null
│
│ Resource specification must include a resource type and name.
╵

The only way to pull this off is to use terraform state rm.

$ terraform state rm random_pet.current
Removed random_pet.current
Successfully removed 1 resource instance(s).

Limitation #2: Moving Resource(s) Between State Files

The moved block allows us to rename resource or restructure our codebase all within the same state file. However, it is not possible to move resources from one state file to another state file.

For example, let’s assume we have one Terraform workspace that manages the following 3 resources.

# workspace A

resource "google_folder" "a" {
  parent       = "folders/123456789012"
  display_name = "a"
}

resource "google_folder" "b" {
  parent       = "folders/123456789012"
  display_name = "b"
}

resource "google_folder" "c" {
  parent       = "folders/123456789012"
  display_name = "c"
}

… and we want to move resource c to another workspace.

# Workspace A

resource "google_folder" "a" {
  parent       = "folders/123456789012"
  display_name = "a"
}

resource "google_folder" "b" {
  parent       = "folders/123456789012"
  display_name = "b"
}
# Workspace B

resource "google_folder" "c" {
  parent       = "folders/123456789012"
  display_name = "c"
}

The only way to pull this off is to identify the resource ID for c in workspace A first…

# Workspace A

$ terraform state show google_folder.c
# google_folder.c:
resource "google_folder" "c" {
  create_time     = "2022-04-30T15:00:29.114Z"
  display_name    = "c"
  folder_id       = "999999999999"
  id              = "folders/999999999999"
  lifecycle_state = "ACTIVE"
  name            = "folders/999999999999"
  parent          = "folders/123456789012"
}

Then, remove that resource from workspace A’s state.

# Workspace A

$ terraform state rm google_folder.c
Removed google_folder.c
Successfully removed 1 resource instance(s).

Finally, import the resource in workspace B’s state.

# Workspace B

$ terraform import google_folder.c folders/999999999999
google_folder.c: Importing from ID "folders/999999999999"...
google_folder.c: Import prepared!
  Prepared google_folder for import
google_folder.c: Refreshing state... [id=folders/999999999999]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Wildcard Subdomains in /etc/hosts

This post illustrates how you use a DNS forwarder to manage wildcard subdomains so that you don’t have to explicitly list each subdomain in /etc/host file.

PROBLEM

When trying to map multiple subdomains (ex: a.localhost, b.localhost, c.localhost, d.localhost) to the same IP, it is not possible to do the following in /etc/hosts:

# /etc/hosts

1.2.3.4 *.localhost

Rather, each subdomain has to be explicitly defined:

# /etc/hosts

1.2.3.4 a.localhost b.localhost c.localhost d.localhost

It requires you to babysit and manage these wildcard subdomains over time, but you do have a good job security.

SOLUTION

Configuration

Install a DNS forwarder using Homebrew.

brew install dnsmasq

Create a configuration to map the wildcard subdomains to the same IP.

sudo bash -c \
  'echo "address=/localhost/1.2.3.4" > /usr/local/etc/dnsmasq.d/localhost.conf'

Restart the service.

sudo brew services restart dnsmasq

Create /etc/resolver directory.

sudo mkdir -p /etc/resolver

Create a custom DNS resolver where the file name is the domain name.

sudo bash -c \
  'echo "nameserver 127.0.0.1" > /etc/resolver/localhost'

Verification

Flush the DNS cache first.

sudo killall -HUP mDNSResponder

Verify that ping command on each subdomain resolves to the correct IP.

$ ping -c 1 a.localhost
PING a.localhost (1.2.3.4): 56 data bytes

$ ping -c 1 b.localhost
PING b.localhost (1.2.3.4): 56 data bytes

$ ping -c 1 a.b.c.localhost
PING a.b.c.localhost (1.2.3.4): 56 data bytes

Enabling Python VirtualEnv in JupyterLab

This post illustrates how you can enable Python virtualenv in GCP JupyterLab so that you can organize your .ipynb files to use different virtual environments to keep track of Python package dependencies.

PROBLEM

  • You are using GCP JupyterLab.
  • You want to adhere to the Python development best practices by not polluting the global environment with your Python packages so that you can generate a cleaner “pip freeze” in the future.
  • You want each Notebook file (.ipynb) to have its own environment so that you can run them with different package versions.
  • You configured a Python virtual environment, but “pip install” from the Notebook file still installs the packages in the global environment.
  • You are fed up.

SOLUTION

Configuring Virtual Environments

In the JupyterLab Notebook’s terminal, create an empty directory to organize all virtual environments.

mkdir virtualenv

Ensure ipykernel is installed. This is used to create new IPython kernels.

python3 -m pip install ipykernel

For each new virtual environment, run the following commands to perform these steps:

  • Get into the base virtual environment directory.
  • Define a new virtual environment name. Replace [NEW_ENV_NAME] with a new name.
  • Create new Python virtual environment.
    • The –system-site-packages option ensures you can still use the “data-sciencey” packages that come pre-installed with the GCP JupyterLab Notebook within your new virtual environment.
  • Jump into the newly created virtual environment.
  • Create a new IPython kernel.
  • Exit from virtual environment.
cd virtualenv
VENV=[NEW_ENV_NAME] # Update this!
python3 -m venv $VENV --system-site-packages
source $VENV/bin/activate
python -m ipykernel install --user --name=$VENV
deactivate

Configuring Notebook File

Create a new Notebook file (.ipynb).

In Select Kernel dialog, select the kernel that you created. In this example, there are 2 new virtual environments (“smurfs” and “thundercats”).

Selecting a kernel in JupyterLab

To perform a simple test, install a new package.

%pip install pandas==1.3.0

IMPORTANT: You need to use IPython’s Magics (literally speaking) to ensure the packages are installed in the virtual environment.

  • %pip = This uses the pip package manager within the current kernel. Magic!
  • ! pip = This uses the pip package manager from the underlying OS. No magic!

From the menu bar, select Kernel -> Restart Kernel and Clear All Outputs… . Always restart the kernel when new packages are installed with %pip.

Kernel -> Restart Kernel and Clear All Outputs in JupyterLab

Inspect the package version. This should show the version you have just installed.

import pandas
print(pandas.__version__)

To verify this actually works, create another Notebook file pointing to another kernel. In this file, install the same package but with different version.

Testing Python Virtualenv in JupyterLab