Terragrunt: “plan-all” while Passing Outputs between Modules

PROBLEM

Terragrunt has a feature that allows one module to pass outputs to another module.

For example, if “project-prod” module wants to consume “subfolders” output from “folder” module, it can be done like this in “project-prod” module’s terragrunt.hcl:-

include {
    path = find_in_parent_folders()
}

dependency "folder" {
    config_path = "../folder"
}

inputs = {
    env_folders = dependency.folder.outputs.subfolders
}

The challenge is when running commands such as plan-all, it will fail with the following error:-

Cannot process module Module [...] because one of its 
dependencies, [...], finished with an error: /my/path/folder/terragrunt.hcl 
is a dependency of /my/path/project-prod/terragrunt.hcl 
but detected no outputs. Either the target module has not 
been applied yet, or the module has no outputs. If this 
is expected, set the skip_outputs flag to true on the 
dependency block.

SOLUTION

This error occurs because the generated plan for “folder” module has not been applied yet (ie: the infrastructure does not exist), hence there are no outputs to pass to “project-prod” module to satisfy plan-all.

To fix this, mock outputs can be supplied:-

include {
    path = find_in_parent_folders()
}

dependency "folder" {
    config_path = "../folder"

    mock_outputs = {
        subfolders = {
            "dev" = {
                "id" = "temp-folder-id"
            }
            "prod" = {
                "id" = "temp-folder-id"
            }
            "uat" = {
                "id" = "temp-folder-id"
            }
        }
    }
}

inputs = {
    env_folders = dependency.folder.outputs.subfolders
}

Finally, when running apply-all, it will use the runtime outputs instead of provided mock outputs to build the rest of the infrastructure.

GCP + Terraform: “google: could not find default credentials” Error

PROBLEM

When running any Terraform commands (init, plan, etc) from a different server, the following error is thrown:-

Error: google: could not find default credentials. 
See https://developers.google.com/accounts/docs/application-default-credentials 
for more information.

  on  line 0:
  (source code not available)

SOLUTION

One recommended way is to set up a service account by following the instruction from the above link.

Another way, for developement purpose, is to install Google Cloud SDK and run the following gcloud command, which will generate an Application Default Credentials (ADC) JSON file based on your user account and store it in a location where the SDK can find it automatically:-

gcloud auth application-default login

Terraform: “Error acquiring the state lock” Error

PROBLEM

When running terraform plan, the following error is thrown:-

Acquiring state lock. This may take a few moments...

Error: Error locking state: Error acquiring the state lock: writing "gs://my/bucket/terraform.tfstate/default.tflock" failed: googleapi: Error 412: Precondition Failed, conditionNotMet
Lock Info:
  ID:        1234567890
  Path:      gs://my/bucket/folder/terraform.tfstate/default.tflock
  Operation: migration destination state
  Who:       mike@machine
  Version:   0.12.12
  Created:   2019-10-30 12:44:36.410366 +0000 UTC
  Info:      


Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

SOLUTION

One way is to disable locking by passing -lock=false flag.

However, if you are sure the lock isn’t properly released, to perform a force unlock, run this command:

terraform force-unlock [LOCK_ID]

In this case…

terraform force-unlock 1234567890

Webpack: Managing Tree Shaking

PROBLEM

Sometimes, Webpack’s tree-shaking may accidentally eliminate imported code from import statements.

For example, we may have a root JS file that imports a CSS file:-

import React from 'react';
import ReactDOM from 'react-dom';
import '../css/index.css';

...

… and for some reason, the CSS file will never appear in the final bundle even if the Webpack config contains proper rules to handle CSS files.

SOLUTION

To prevent Webpack from removing any “unreferenced” code (ex: global CSS files, JS polyfills, etc), list these side effects under package.json, for example:-

{
  "name": "front-end-stack",
  "sideEffects": [
    "*.css"
  ]
}

Azure: Deploying WAR File to Tomcat

PROBLEM

Typically, when using ZipDeploy to push a WAR file (ex: my-app.war) to an Azure instance, we need to:-

  • Rename my-app.war to ROOT.war.
  • Place ROOT.war under webapps/.
  • Zip up webapps/.
  • Use ZipDeploy to push the zip file to an Azure instance.

This zip file will automatically be unzipped under site/wwwroot:-

D:\home\site\wwwroot
└── webapps
    └── ROOT.war

Tomcat detects ROOT.war and will try to unpack the WAR file under ROOT/:-

D:\home\site\wwwroot
└── webapps
    ├── ROOT
    │   ├── META-INF
    │   │   └── ...
    │   └── WEB-INF
    │       └── ...
    └── ROOT.war

The problem is sometimes Tomcat is unable to fully unpack ROOT.war because some files may be locked by some running processes. As a result, the web page fails to load and shows 404 error.

SOLUTION

A better and more stable solution is to use WarDeploy to push WAR file to an Azure instance because:-

  • It simplifies the deployment process because we don’t need to create a zip file with webapps/ containing a WAR file named ROOT.war.
  • Instead of relying on Tomcat to unpack the WAR file, WarDeploy will do the unpacking step elsewhere before copying to site/wwwroot/webapps/.

To use WarDeploy, you can use curl command or PowerShell script to do so.

Here’s an example of the PowerShell script:-

Param(
    [string]$filePath,
    [string]$apiUrl,
    [string]$username,
    [string]$password
)

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username, $password)))

Invoke-RestMethod -Uri $apiUrl -Headers @{ Authorization = ("Basic {0}" -f $base64AuthInfo) } -Method POST -InFile $filePath -ContentType "multipart/form-data"

If the PowerShell script above is called deploy.ps1, to push my-app.war to my-app Azure instance:-

deploy.ps1 -filePath D:\my-app.war -apiUrl https://my-app.scm.azurewebsites.net/api/wardeploy -username [USER] -password [PASSWORD]

Depending on the size of the WAR file, you may get “The operation has timed out” error after 100 seconds:-

Invoke-RestMethod : The operation has timed out
At D:\agent\_work\2ea6e947a\my-app\deploy.ps1:18 char:1
+ Invoke-RestMethod -Uri $apiUrl -Headers @{Authorization=("Basic {0}"  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

This will most likely to occur if the WAR file is too big (80MB+).

To fix this, increase the time out duration by adding -TimeoutSec option to Invoke-RestMethod statement:-

Invoke-RestMethod [OTHER_OPTIONS...] -TimeoutSec 300

Feign + Eureka: UnknownHostException when Attempting to Invoke a Service

PROBLEM

When attempting use Feign to invoke a service through Eureka, the following exception occurs:-

MY-HOST-NAME executing GET http://donkey-kong-service/throw/barrels/10
feign.RetryableException: MY-HOST-NAME executing GET http://donkey-kong-service/throw/barrels/10
	at feign.FeignException.errorExecuting(FeignException.java:84)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:113)
	...
Caused by: java.net.UnknownHostException: MY-HOST-NAME
		at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)

SOLUTION

Go to the donkey-kong-service app and add the following line to the application.properties:-

eureka.instance.prefer-ip-address=true

Now, when Feign tries to invoke the service, the IP address will be used instead of the OS’ reported hostname.

Spring Boot: Restarting App using Dev Tools with IntelliJ IDEA

Spring Boot provides spring-boot-devtools module that allows the app to “smartly” restart whenever the files on the classpath have changed.

Because the rarely changed classes (ex: 3rd party JARs) are separated out into a different classloader from the app’s actively developed classes’ classloader, it allows Spring Boot to quickly restart the app compared to “cold start”.

DEPENDENCY

First, add the following dependency:-

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-devtools</artifactId>
	<scope>runtime</scope>
</dependency>

CONFIGURING INTELLIJ IDEA

In IntelliJ IDEA:-

  • Click SHIFT twice to bring up the “Search History” dialog.
  • Select “Actions” tab.
  • Type “Registry” in the search box.
  • Select “Registry…”.

In the “Registry” dialog:-

  • Find “compiler.automake.allow.when.app.running” key.
  • Check the checkbox.
  • Close the dialog.

In IntelliJ IDEA “Preferences” dialog:-

  • Go to “Build, Execution, Deployment” » “Compiler”.
  • Check “Build project automatically”.
  • Close the dialog.

Finally, instead of running Maven goals to run the Spring Boot app, select the Application class (annotated with @SpringBootApplication) and run it from IntelliJ IDEA.

Anytime the app’s class files have changed, IntelliJ IDEA will compile the app, which will then trigger Spring Boot Dev Tools to restart the app.