Table of Contents

There are various ways for a Cloud Run instance to connect to a Cloud SQL instance. Regardless of the solutions, if both resources reside in the same GCP organization, connecting using a Cloud SQL’s private IP with SSL/TLS encryption is highly recommended to prevent network traffic from going out to the internet.

It is possible to have a Cloud SQL instance with both private and public IPs enabled, and there are legitimate reasons for doing so. For example, it allows the developers to connect to the Cloud SQL instance outside the GCP environment. However, utmost care is needed to ensure the private IP is always used when connecting from the Cloud Run instance.

This article evaluates the pros and cons of each solution and provides some recommendations.

Cloud SQL Configuration

For consistency’s sake, SSL/TLS encryption is enabled on the Cloud SQL instance using client certificates.

A client certificate is also created.

Note: Some solutions, such as Cloud SQL Proxy and Cloud SQL Language Connectors, do not need the client certificate. So, these steps can be safely skipped. From the documentation:

With Cloud SQL Auth Proxy and Cloud SQL Connectors, client and server identities are also automatically verified regardless of the SSL mode setting.

Options

Option A: Cloud SQL Language Connectors

This solution uses Cloud SQL Language Connectors, a programming language library maintained by Google.

The following configurations are NOT required:

• SSL certificates in the Cloud SQL instance
• Cloud SQL volume mounting in the Cloud Run instance.

Python App

import os

from google.cloud.sql.connector import Connector, IPTypes

# ... simplified for brevity

connector = Connector(IPTypes.PRIVATE)  # use Private IP

connector.connect(
  os.getenv('DB_INSTANCE_CONNECTION_NAME'),  # project:region:instance
  'pymysql',
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  db=os.getenv('DB_NAME'),
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage client certificates.

Cons

• Only supports Python, Java, Go, and Node.js.

Option B: Managed Cloud SQL Proxy in Cloud Run

This solution uses a managed Cloud SQL Proxy that is configurable directly in the Cloud Run instance. Once configured, a Cloud SQL volume mount is created in the Cloud Run instance.

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  unix_socket=f'/cloudsql/{os.getenv("DB_INSTANCE_CONNECTION_NAME")}',  # /cloudsql/project:region:instance
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage SSL certificates.

Cons

• It doesn’t work if the Cloud SQL instance has both public IP and private IP enabled. There’s no way to specify which IP to use, and Cloud SQL Proxy automatically uses the public IP if it exists.
  • Note: If you know how to pull this off, please drop a comment below.
• Additional configurations must be considered when running in production, which introduces many points of failure.
• It may exceed the default Cloud SQL Admin API quota limit.

Option C: Manually Configure Cloud SQL Proxy

This solution manually configures Cloud SQL Proxy via Dockerfile instead of relying on the managed Cloud SQL volume mount from Cloud Run.

Dockerfile

FROM python:3.12-alpine

RUN apk update && apk add curl

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8080

RUN curl -o cloud-sql-proxy https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.12.0/cloud-sql-proxy.linux.amd64
RUN chmod +x cloud-sql-proxy

CMD ["sh", "-c", "./cloud-sql-proxy --private-ip --port 3306 ${DB_INSTANCE_CONNECTION_NAME} & sleep 5 && python app.py"]

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  host=os.getenv('DB_HOST'),  # 127.0.0.1
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage client certificates.
• Unlike the managed Cloud SQL Proxy volume mount solution, this solution will work even if the Cloud SQL instance has both private and public IPs.

Cons

• Need to manage the Cloud SQL Proxy installation and configuration in the Dockerfile.
• All the cons described in the managed Cloud SQL Proxy solution above still apply here.

Option D: Use Cloud SQL’s Private IP and Client Certificate

Instead of relying on external tools/libraries, the application code connects directly to the Cloud SQL instance using the private IP where the client certificate is also provided.

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  host=os.getenv('DB_HOST'), # Cloud SQL's Private IP
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  ssl={
    'ca': os.getenv('DB_SSL_SERVER_CA'),     # /app/certs/ca/server-ca.pem
    'cert': os.getenv('DB_SSL_CLIENT_CERT'), # /app/certs/cert/client-cert.pem
    'key': os.getenv('DB_SSL_CLIENT_KEY'),   # /app/certs/key/client-key.pem
    'check_hostname': False,
  }
)

Note: The hostname checking must be disabled because the database host (Cloud SQL’s Private IP) will never
match the common name specified in the certificate. If the hostname checking is not disabled, it will produce the following error:

Error connecting to database: (2003, “Can’t connect to MySQL server on ’10.x.x.x’ ([SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: IP address mismatch, certificate is not valid for ’10.x.x.x’. (_ssl.c:1000))”)

Pros

• No additional wrappers like Cloud SQL Proxy or Cloud SQL Language Connectors are required.

Cons

• Additional configurations and complexity on the IaC and application code are needed to manage and ingest the client certificate securely.

Recommendations

Best Solution – Option A

Use Cloud SQL Language Connectors if your app uses one of the supported programming languages (Python, Java, Go, or Node.js).

It eliminates many unnecessary configurations on the IaC code and application code. In other words, there’s no need to do the following:

• Create a client certificate for Cloud SQL.
• Store server certificate, client certificate, and client key in Secret Manager securely.
• Mount files containing the server certificate, client certificate, and client key as volumes in the Cloud Run instance.
• Pass certificate information to the application code via environment variables.

None of the above is needed.

Less codebase = Less chance of errors.

Distance Next Best Solution – Option D

Connect directly using the Cloud SQL instance’s Private IP and client certificate.

It eliminates the need to rely on additional cloud configuration or external library (i.e., fewer points of failure).

However, it introduces additional configurations in IaC and application code to maintain and ingest the certificate described above.

Least Favorite Solution – Option B & C

While both Cloud SQL Proxy solutions (volume mount in the Cloud Run instance or installing it directly via Dockerfile) don’t require a client certificate, they come with additional considerations and complexities when running in production.

Furthermore, the volume mount solution won’t work if the Cloud SQL instance has both private and public IPs.

The post Cloud Run: Connecting to Cloud SQL with Private IP and SSL appeared first on My Shitty Code.

Table of Contents

This tutorial shows how to improve the performance of the remote pipeline that builds a Docker image using docker build, which takes 15 minutes to 20 seconds.

Note: Although Google’s Cloud Build is used here, this solution can be applied in GitHub Actions, Azure Pipeline, or any pipeline-driven Docker builds.

The Challenge with Docker Build

Given the following Google’s Cloud Build script:

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    args: ["build", "-t", "${_NAME}", "."]

images:
  - ${_NAME}

substitutions:
  _LOCATION: us-central1
  _REPO: shared
  _IMAGE: chatbot
  _NAME: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true

When kicking off the build…

gcloud builds submit --config=cloudbuild.yml --project=[PROJECT ID]

… Docker builds the image successfully and pushes it to the Artifact Registry.

When rerunning the same build multiple times, it takes similar time to run:

In the example above, building the image with no changes to the Dockerfile takes an average of 15 minutes.

Solution

When running the build locally, Docker takes advantage of the layer caching by storing the data in the file system. This results in faster subsequent builds.

However, when using a remote pipeline, the agent assigned to the pipeline job is ephemeral. Hence, Docker must always rebuild all layers, resulting in a long build time. To fix this annoyance, we must instruct Docker to cache the layers elsewhere.

Introducing Buildx

Docker has a relatively unheard CLI command called Buildx that allows the layers to be cached remotely. Buildx has been the default build client since Docker Engine 23.0 and Docker Desktop 4.19.

To investigate, run the following commands locally to verify:

$ docker --version
Docker version 25.0.3, build 4debf41

$ docker build --help
Usage:  docker buildx build [OPTIONS] PATH | URL | -

// TRUNCATED

Step 1: Enable experimental mode

At the time of writing, Google’s Cloud Build uses a Docker version older than v23.0.

Hence, an environment variable must be set to enable the experimental mode when using an older Docker version.

# cloudbuild.yml

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled

Step 2: Specify build driver

A different Docker driver must be explicitly specified to use the layer caching feature in the older Docker version.

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx create
        --driver docker-container
        --use

Step 3: Configure build caches

Now, configure Docker to fetch and store the caches.

# cloudbuild.yml

steps:
  # ...

  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx build
        --cache-from ${_NAME}:cache
        --cache-to type=registry,ref=${_NAME}:cache,mode=max
        -t ${_NAME} .
        --push

In this example, the cache image is stored in the same image repository in Artifact Registry but with a different tag, cache.

To ensure all layers’ build information is cached, mode=max is specified, too.

Step 4: Putting everything together

The modified build file looks like this:

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx create
        --driver docker-container
        --use

  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx build
        --cache-from ${_NAME}:cache
        --cache-to type=registry,ref=${_NAME}:cache,mode=max
        -t ${_NAME} .
        --push

substitutions:
  _LOCATION: us-central1
  _REPO: shared
  _IMAGE: chatbot
  _NAME: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled

Step 5: Analyze build results

Now, submit the build multiple times to examine the results.

The build logs show that the first run takes about 12 minutes because the cache has yet to exist. Once the cache exists, the subsequent build takes an average of 20 seconds.

The reason for the gap between 4 PM and 7 PM was that nobody had the time to sit for 12 minutes for the build to finish.

Step 6: Verify image repository

The images stored in the Artifact Registry look like this:

As you can see, the final image (with the latest tag) and the cache image (with the cache tag) are successfully stored in the Artifact Registry. The 20-second build is exclusively spent pulling a few gigabytes of cache image from the Artifact Registry.

The post Supercharge Docker Build Pipeline by 97% appeared first on My Shitty Code.

PROBLEM

You want to volume mount a FUSE-mounted directory to a container, for example:

When attempting to run the container…

docker run -it --rm -v /my-kfc-bucket:/home busybox

… an error occurred:

docker: Error response from daemon: error while creating
mount source path '/my-kfc-bucket': mkdir /my-kfc-bucket: 
file exists.

SOLUTION

Unmount the existing FUSE-mounted directory.

sudo umount /my-kfc-bucket

Mount it back with the following option. Because this command with -o allow_other must be executed with sudo privilege, you will need to change the root ownership to yourself (via –uid and –gid) so that you can easily read/write within the directory.

sudo gcsfuse \
  -o allow_other \
  --uid $(id -u) \
  --gid $(id -g) \
  gcs-bucket /my-kfc-bucket  

If it is successful, the output should look like this:

Start gcsfuse/0.40.0 (Go version go1.17.6) for app "" using mount point: /my-kfc-bucket
Opening GCS connection...
Mounting file system "gcs-bucket"...
File system has been successfully mounted.

Rerun the docker container.

docker run -it --rm -v /my-kfc-bucket:/home busybox

Now, you can read/write the GCS bucket’s data from the container. In this example, the GCS bucket’s data is located in /home.

The post GCSFuse + Docker: “Error while creating mount source path ‘/a’: mkdir /a: file exists.” appeared first on My Shitty Code.

You want to push a new image to Google Container Registry (GCR) or pull an existing image from GCR.

SOLUTION

Pushing a New Image to GCR

Prepare your Dockerfile.

FROM alpine:3.7

# some content...

Create an image and tag it with a path pointing to GCR within a project.

There are several variations of GCR’s hostname (ex: gcr.io, us.gcr.io, eu.gcr.io, etc) depending on the data center’s location.

The GCR path has the following format: [HOSTNAME]/[PROJECT-ID]/[IMAGE].

docker build -t gcr.io/shitty-project/shitty-repo .

Log into GCP.

gcloud auth login

Register gcloud as a Docker credential helper.

gcloud auth configure-docker

Push the image to GCR.

docker push gcr.io/shitty-project/shitty-repo

View pushed image.

gcloud container images list-tags gcr.io/shitty-project/shitty-repo

DIGEST        TAGS    TIMESTAMP
78b36c0b456d  latest  2019-03-07T16:19:53

The repository and image can also be viewed in GCP Console.

Pulling an Existing Image from GCR

Proceed with the authentication process if it is not done already.

gcloud auth login
gcloud auth configure-docker

Pull the image from GCR.

docker pull gcr.io/shitty-project/shitty-repo

The post GCP + Container Registry: Pushing/Pulling Images appeared first on My Shitty Code.

When running the Docker container interactively (ex: docker run –rm -it myimage), you want to run a startup script every time.

SOLUTION

For Ubuntu, Debian and Centos images, write the startup script to /root/.bashrc:

# UBUNTU
FROM ubuntu:latest
RUN echo "echo 'Welcome!'" >> /root/.bashrc
WORKDIR /home

# DEBIAN
FROM debian:latest
RUN echo "echo 'Welcome!'" >> /root/.bashrc
WORKDIR /home

# CENTOS
FROM centos:latest
RUN echo "echo 'Welcome!'" >> /root/.bashrc
WORKDIR /home

For Alpine image, it’s a little different because it uses Ash shell. Besides writing the startup script to /root/.profile, you also need to set that path to an environment variable called ENV:

FROM alpine:latest
ENV ENV=/root/.profile
RUN echo "echo 'Welcome!'" > $ENV
WORKDIR /home

The post Docker: Executing Startup Script When Running Container Interactively appeared first on My Shitty Code.