Table of Contents

There are various ways for a Cloud Run instance to connect to a Cloud SQL instance. Regardless of the solutions, if both resources reside in the same GCP organization, connecting using a Cloud SQL’s private IP with SSL/TLS encryption is highly recommended to prevent network traffic from going out to the internet.

It is possible to have a Cloud SQL instance with both private and public IPs enabled, and there are legitimate reasons for doing so. For example, it allows the developers to connect to the Cloud SQL instance outside the GCP environment. However, utmost care is needed to ensure the private IP is always used when connecting from the Cloud Run instance.

This article evaluates the pros and cons of each solution and provides some recommendations.

Cloud SQL Configuration

For consistency’s sake, SSL/TLS encryption is enabled on the Cloud SQL instance using client certificates.

A client certificate is also created.

Note: Some solutions, such as Cloud SQL Proxy and Cloud SQL Language Connectors, do not need the client certificate. So, these steps can be safely skipped. From the documentation:

With Cloud SQL Auth Proxy and Cloud SQL Connectors, client and server identities are also automatically verified regardless of the SSL mode setting.

Options

Option A: Cloud SQL Language Connectors

This solution uses Cloud SQL Language Connectors, a programming language library maintained by Google.

The following configurations are NOT required:

• SSL certificates in the Cloud SQL instance
• Cloud SQL volume mounting in the Cloud Run instance.

Python App

import os

from google.cloud.sql.connector import Connector, IPTypes

# ... simplified for brevity

connector = Connector(IPTypes.PRIVATE)  # use Private IP

connector.connect(
  os.getenv('DB_INSTANCE_CONNECTION_NAME'),  # project:region:instance
  'pymysql',
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  db=os.getenv('DB_NAME'),
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage client certificates.

Cons

• Only supports Python, Java, Go, and Node.js.

Option B: Managed Cloud SQL Proxy in Cloud Run

This solution uses a managed Cloud SQL Proxy that is configurable directly in the Cloud Run instance. Once configured, a Cloud SQL volume mount is created in the Cloud Run instance.

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  unix_socket=f'/cloudsql/{os.getenv("DB_INSTANCE_CONNECTION_NAME")}',  # /cloudsql/project:region:instance
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage SSL certificates.

Cons

• It doesn’t work if the Cloud SQL instance has both public IP and private IP enabled. There’s no way to specify which IP to use, and Cloud SQL Proxy automatically uses the public IP if it exists.
  • Note: If you know how to pull this off, please drop a comment below.
• Additional configurations must be considered when running in production, which introduces many points of failure.
• It may exceed the default Cloud SQL Admin API quota limit.

Option C: Manually Configure Cloud SQL Proxy

This solution manually configures Cloud SQL Proxy via Dockerfile instead of relying on the managed Cloud SQL volume mount from Cloud Run.

Dockerfile

FROM python:3.12-alpine

RUN apk update && apk add curl

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8080

RUN curl -o cloud-sql-proxy https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.12.0/cloud-sql-proxy.linux.amd64
RUN chmod +x cloud-sql-proxy

CMD ["sh", "-c", "./cloud-sql-proxy --private-ip --port 3306 ${DB_INSTANCE_CONNECTION_NAME} & sleep 5 && python app.py"]

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  host=os.getenv('DB_HOST'),  # 127.0.0.1
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
)

Pros

• Very concise configuration in both IaC and application code since there’s no need to manage client certificates.
• Unlike the managed Cloud SQL Proxy volume mount solution, this solution will work even if the Cloud SQL instance has both private and public IPs.

Cons

• Need to manage the Cloud SQL Proxy installation and configuration in the Dockerfile.
• All the cons described in the managed Cloud SQL Proxy solution above still apply here.

Option D: Use Cloud SQL’s Private IP and Client Certificate

Instead of relying on external tools/libraries, the application code connects directly to the Cloud SQL instance using the private IP where the client certificate is also provided.

Python App

import os

import pymysql

# ... simplified for brevity

pymysql.connect(
  host=os.getenv('DB_HOST'), # Cloud SQL's Private IP
  database=os.getenv('DB_NAME'),
  user=os.getenv('DB_USER'),
  password=os.getenv('DB_PASS'),
  ssl={
    'ca': os.getenv('DB_SSL_SERVER_CA'),     # /app/certs/ca/server-ca.pem
    'cert': os.getenv('DB_SSL_CLIENT_CERT'), # /app/certs/cert/client-cert.pem
    'key': os.getenv('DB_SSL_CLIENT_KEY'),   # /app/certs/key/client-key.pem
    'check_hostname': False,
  }
)

Note: The hostname checking must be disabled because the database host (Cloud SQL’s Private IP) will never
match the common name specified in the certificate. If the hostname checking is not disabled, it will produce the following error:

Error connecting to database: (2003, “Can’t connect to MySQL server on ’10.x.x.x’ ([SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: IP address mismatch, certificate is not valid for ’10.x.x.x’. (_ssl.c:1000))”)

Pros

• No additional wrappers like Cloud SQL Proxy or Cloud SQL Language Connectors are required.

Cons

• Additional configurations and complexity on the IaC and application code are needed to manage and ingest the client certificate securely.

Recommendations

Best Solution – Option A

Use Cloud SQL Language Connectors if your app uses one of the supported programming languages (Python, Java, Go, or Node.js).

It eliminates many unnecessary configurations on the IaC code and application code. In other words, there’s no need to do the following:

• Create a client certificate for Cloud SQL.
• Store server certificate, client certificate, and client key in Secret Manager securely.
• Mount files containing the server certificate, client certificate, and client key as volumes in the Cloud Run instance.
• Pass certificate information to the application code via environment variables.

None of the above is needed.

Less codebase = Less chance of errors.

Distance Next Best Solution – Option D

Connect directly using the Cloud SQL instance’s Private IP and client certificate.

It eliminates the need to rely on additional cloud configuration or external library (i.e., fewer points of failure).

However, it introduces additional configurations in IaC and application code to maintain and ingest the certificate described above.

Least Favorite Solution – Option B & C

While both Cloud SQL Proxy solutions (volume mount in the Cloud Run instance or installing it directly via Dockerfile) don’t require a client certificate, they come with additional considerations and complexities when running in production.

Furthermore, the volume mount solution won’t work if the Cloud SQL instance has both private and public IPs.

The post Cloud Run: Connecting to Cloud SQL with Private IP and SSL appeared first on My Shitty Code.

Table of Contents

This tutorial shows how to improve the performance of the remote pipeline that builds a Docker image using docker build, which takes 15 minutes to 20 seconds.

Note: Although Google’s Cloud Build is used here, this solution can be applied in GitHub Actions, Azure Pipeline, or any pipeline-driven Docker builds.

The Challenge with Docker Build

Given the following Google’s Cloud Build script:

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    args: ["build", "-t", "${_NAME}", "."]

images:
  - ${_NAME}

substitutions:
  _LOCATION: us-central1
  _REPO: shared
  _IMAGE: chatbot
  _NAME: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true

When kicking off the build…

gcloud builds submit --config=cloudbuild.yml --project=[PROJECT ID]

… Docker builds the image successfully and pushes it to the Artifact Registry.

When rerunning the same build multiple times, it takes similar time to run:

In the example above, building the image with no changes to the Dockerfile takes an average of 15 minutes.

Solution

When running the build locally, Docker takes advantage of the layer caching by storing the data in the file system. This results in faster subsequent builds.

However, when using a remote pipeline, the agent assigned to the pipeline job is ephemeral. Hence, Docker must always rebuild all layers, resulting in a long build time. To fix this annoyance, we must instruct Docker to cache the layers elsewhere.

Introducing Buildx

Docker has a relatively unheard CLI command called Buildx that allows the layers to be cached remotely. Buildx has been the default build client since Docker Engine 23.0 and Docker Desktop 4.19.

To investigate, run the following commands locally to verify:

$ docker --version
Docker version 25.0.3, build 4debf41

$ docker build --help
Usage:  docker buildx build [OPTIONS] PATH | URL | -

// TRUNCATED

Step 1: Enable experimental mode

At the time of writing, Google’s Cloud Build uses a Docker version older than v23.0.

Hence, an environment variable must be set to enable the experimental mode when using an older Docker version.

# cloudbuild.yml

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled

Step 2: Specify build driver

A different Docker driver must be explicitly specified to use the layer caching feature in the older Docker version.

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx create
        --driver docker-container
        --use

Step 3: Configure build caches

Now, configure Docker to fetch and store the caches.

# cloudbuild.yml

steps:
  # ...

  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx build
        --cache-from ${_NAME}:cache
        --cache-to type=registry,ref=${_NAME}:cache,mode=max
        -t ${_NAME} .
        --push

In this example, the cache image is stored in the same image repository in Artifact Registry but with a different tag, cache.

To ensure all layers’ build information is cached, mode=max is specified, too.

Step 4: Putting everything together

The modified build file looks like this:

# cloudbuild.yml

steps:
  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx create
        --driver docker-container
        --use

  - name: gcr.io/cloud-builders/docker
    entrypoint: bash
    args:
      - -c
      - >-
        docker buildx build
        --cache-from ${_NAME}:cache
        --cache-to type=registry,ref=${_NAME}:cache,mode=max
        -t ${_NAME} .
        --push

substitutions:
  _LOCATION: us-central1
  _REPO: shared
  _IMAGE: chatbot
  _NAME: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}

options:
  logging: CLOUD_LOGGING_ONLY
  machineType: E2_HIGHCPU_32
  dynamicSubstitutions: true
  env:
    - DOCKER_CLI_EXPERIMENTAL=enabled

Step 5: Analyze build results

Now, submit the build multiple times to examine the results.

The build logs show that the first run takes about 12 minutes because the cache has yet to exist. Once the cache exists, the subsequent build takes an average of 20 seconds.

The reason for the gap between 4 PM and 7 PM was that nobody had the time to sit for 12 minutes for the build to finish.

Step 6: Verify image repository

The images stored in the Artifact Registry look like this:

As you can see, the final image (with the latest tag) and the cache image (with the cache tag) are successfully stored in the Artifact Registry. The 20-second build is exclusively spent pulling a few gigabytes of cache image from the Artifact Registry.

The post Supercharge Docker Build Pipeline by 97% appeared first on My Shitty Code.

Table of Contents

This post shows how you can use Rclone to back up your data from a Synology NAS to a storage bucket in GCP.

Why Backing Up Synology NAS

Backing up data is like wearing a seat belt when driving, where nothing bad happens 99% of the time. However, when the 1% strikes unexpectedly one day, the “future” you will be grateful that you do not lose any precious data, such as childhood photos, important documents, etc.

When you own several machines at home, it makes sense to centralize the shared data in a local NAS, such as Synology NAS. You may also configure Time Machine to seamlessly back up each Mac to Synology NAS every weekend. It is equally important to back up the data in Synology NAS to the cloud. Your car’s seat belt might be faulty, but you still have an airbag. If a thief breaks into your house, steals your shiny NAS, and burns your house down, you still have your data in the cloud.

The moral of the story is always to back up your data, preferably using the 3-2-1 rule.

Why Rclone

The most confusing part of the Synology NAS backup process is there are several ways to do this. The three popular solutions are Cloud Sync, Hyper Backup, and Rclone. Each solution allows us to push data to various cloud providers. This Reddit post nicely sums it all up. If I can summarize the most significant disadvantage of each solution in a few words, it will be:

• Cloud Sync silently ignores file names with unsupported characters.
• Hyper Backup landlocks you to Synology solution because you can’t browse the backed-up files in the cloud.
• Rclone doesn’t have a pretty GUI.

Rclone seems more palatable among these solutions because I know all my data will be backed up. When disaster strikes next time, I also don’t have to deal with the proprietary format during my data recovery process. Who knows if Synology will still be around 20 years from now?

Why GCP

In the past, I configured my Synology NAS to perform a seamless backup to CrashPlan using this Docker solution. It has worked flawlessly for many years. The main reasons I want to move the backups to my personal GCP org are:

• I have free GCP credits. Even without the GCP credits, based on the GCP Pricing Calculator, it is still cheaper to back up my data to a regional Archive storage bucket than CrashPlan.
• I can customize the number of backup versions and retention policies.
• It is faster to recover all my data if needed.
• I want to secure the data and GCP resources myself.

You can swap out GCP with a cloud provider of your choice. For example, you can use Rclone to back up all your photos to Amazon Photos for free if you have a Prime membership and your photo formats are supported. My RAW images (supported) also have corresponding XMP files (not supported), and I want to back up other file formats too.

Configuration

This diagram below helps you to visualize the solution.

Setting up GCP Resources

The best way to secure the data backups is to create a separate GCP project where only a specific service account has sufficient permission to write to a storage bucket. At a high level:

• Create a GCP project.
• Create a storage bucket.
  • Enable object versioning and limit them to N versions (to prevent rising storage costs).
• Create a service account.
• Grant service account with Storage Object Admin (roles/storage.objectAdmin) at the project level.
• Create a service account key.
• Download the service account key file in JSON format to the local machine first. In this post, this file is named service-account.json.
• BONUS: If you are paranoid, configure a VPC Service Control perimeter and put that project within the perimeter. This ensures only the whitelisted IPs can access these project resources. This way, even if the service account key is compromised, adversaries cannot access the protected resources unless they also spoof the IPs.

Setting Up Rclone in Synology NAS

Ways to Install Rclone

There are multiple ways to install Rclone in Synology NAS:

• Install Rclone directly in Synology NAS.
• Install Rclone using Docker (rclone/rclone).
• Install Rclone with GUI using Docker (romancin/rclonebrowser).

In this post, I use the second solution (without GUI) because simpler implementation means less chance of the tool breaking in the future. Besides, this is a “set-and-forget” process, and I don’t need to use the GUI most of the time.

What’s in /docker/appdata/rclone

/docker/appdata should already exist when the Docker package is installed in Synology NAS. You only need to create a child folder named rclone (or any name you like). This folder contains three files:

• excludes.txt = File patterns to be excluded from the backup process.
• rclone.conf = Rclone configuration.
• service-account.json = The service account key file that you downloaded to your local machine from GCP.

Here’s my example of excludes.txt, and customize it to your needs:

@eaDir/**
.**

rclone.conf can be generated using rclone config command. However, that must be done outside Synology NAS (i.e., on your local machine). That said, most of this command’s interactive steps are unnecessary since you created the storage bucket earlier. All you need is exactly this and nothing else:

[gcp]
type = google cloud storage
service_account_file = /config/rclone/service-account.json

Downloading Docker Image

Go to DSM > Docker to download rclone/rclone image.

Creating Docker Container (with –dry-run)

This step ensures the Rclone is configured properly before any files are copied to the storage bucket to prevent incurring unnecessary costs.

First, create a new container.

Use rclone/rclone image.

No changes to the network settings.

Enter the container name, and go to Advanced Settings.

Under Execution Command, enter the following command:

sync /data gcp:[BUCKET_NAME] --exclude-from /config/rclone/excludes.txt --gcs-bucket-policy-only --verbose --dry-run

Replace [BUCKET_NAME] with your own bucket name.

IMPORTANT: Synology does not allow us to edit the execution command after creating a container!

No changes to the port settings.

Define the volume mounts:

• /config/rclone = contains the Rclone configuration files.
• /data = contains data to be backed up.
  • Because you can’t perform a volume mount on root /, each root folder needs to be volume-mounted to a subfolder within /data that you wish to back up.

TIP: Consider mounting a data folder with the least amount of data when doing the dry run test. You can add/remove folders on an existing container later.

Review the summary.

Once the container is created, click on the toggle to run it.

Review the container log. You can drop a few files into the mounted folder and rerun the container. The files should show up in the log.

Creating Docker Container (without –dry-run)

Now that you have a working container to perform the dry run test, it’s time to back up the data to the cloud.

As mentioned earlier, it is not possible to edit the existing container’s execution command. The existing container contains the –dry-run flag on the execution command. The simplest way is to export the container setting, which will be used to create a new container without the –dry-run flag.

Export just the container settings to a local machine.

Open the JSON file, locate and remove –dry-run from cmd. Save the file.

{
   "CapAdd" : [],
   "CapDrop" : [],
   "cmd" : "sync /data gcp:[BUCKET_NAME] --exclude-from /config/rclone/excludes.txt --gcs-bucket-policy-only --verbose --dry-run",
   "cpu_priority" : 0,
   "enable_publish_all_ports" : false,
   ...
}

Return to the Docker container window to import this JSON file to create a new container.

Upload the modified JSON file and provide a new container name.

Now, there are 2 Rclone containers: one with the –dry-run flag and another without it. It is up to you whether to retain or delete the container with the –dry-run flag.

Scheduling Docker Container to Run

To ensure the Rclone runs to back up the files to the cloud, create a task scheduler. In this example, a triggered task is created so that Rclone can perform the backups on boot-up.

Ensure to run the task as root.

Specify the following script to execute. This script ensures the Docker service is running first before starting the container to perform the backup.

while (synopkg status Docker | jq -r '.status' | grep -vq "running"); do
  sleep 10
done

docker container start rclone

That’s it! Every time the Synology NAS boots up, the Rclone container will run to perform any necessary data backup to the cloud.

The post Rclone: Backing Up Synology NAS to GCP appeared first on My Shitty Code.

Table of Contents

This post shows you how to fix the dreaded NO_MATCHING_ACCESS_LEVEL error when dealing with VPC Service Control in GCP.

Background

Let’s assume you want to protect a project and its resources by putting it within a VPC SC perimeter. You must also ensure your trusted device(s) can access the protected project resources.

In this case, an access level is created to capture the IP address (or CIDR ranges) that requires access to the protected resources.

Once an access level is created, a VPC SC perimeter is also created with the ingress policy configured to allow any identities from the whitelisted IPs access to the protected project resources.

Problem

One easy way to test the VPC SC perimeter is to interact with a GCS bucket from the command line. If the perimeter is configured properly, you should see a list of files. Otherwise, you will get a VPC SC error like below.

> gsutil ls gs://my-shitty-bucket
AccessDeniedException: 403 Request is prohibited by organization's policy. 
vpcServiceControlsUniqueIdentifier: qQgF9j6AzA413Mfinr2fC5c67-TQoJRuNFKfCzAfXXXXXXXX

When looking up the VPC SC UID for more details, it states there is no matching access level, even though the VPC SC perimeter is configured with an access level containing a matching IP address.

Solution

The most likely reason for this error is you have accidentally created a scoped access policy (see the red warning below) without an unscoped access policy at the org level.

To verify this, go to Security > VPC Service Controls > MANAGE POLICIES.

In this example, the “main” access policy is scoped to a project (covered by blue color).

There are 2 ways to fix this.

Option A: Keep Existing Scoped Access Policy, Create Dummy Unscoped Access Policy

If you share this GCP org with other team members and you want to get it working without impacting other GCP resources, the easiest way is to create another access policy that is unscoped (i.e., no project or folder specified) at the org level. This unscoped access policy does not have any VPC SC perimeters configured.

In this example, an unscoped access policy named “default” is created.

Once the changes are propagated, you can execute the same gsutil command successfully.

>  gsutil ls gs://my-shitty-bucket
gs://my-shitty-bucket/my-shitty-file.jpg

Option B (Preferred): Delete Existing Scoped Access Policy, Create Unscoped Access Policy with VPC SC Perimeter

Unfortunately, an existing scoped access policy cannot be unscoped. So, if you accidentally create a scoped access policy, you can delete it and create an unscoped access policy instead, which requires you to reconfigure the VPC SC perimeter. The upside is you won’t confuse yourself with multiple access policies if you truly don’t need the scoped access policy.

In this example, an unscoped access policy named “default” is created, and it also has the VPC SC perimeter configured with the same settings.

Once the changes are propagated, the same gsutil command will also execute successfully.

The post VPC SC: 2 Ways to Fix NO_MATCHING_ACCESS_LEVEL Error appeared first on My Shitty Code.

Table of Contents

What is text-bison

Google’s PaLM 2 for text (text-bison) is recently GA’d. This foundation model is optimized for natural language tasks and it comes in 4 sizes (gecko, otter, bison, and unicorn). To date, unicorn doesn’t exist yet, no pun intended… unless one decides to stick an ice cream cone on a pony’s forehead at a county fair right now. If you know how to use OpenAI’s ChatGPT or Google Bard, you already know how to use text-bison.

Getting Started

The fastest way to explore text-bison is to leverage Generative AI Studio, select the model to use, and get going with prompt design with absolutely no code. The pricing is listed here. For text-bison, you can either use freeform or structured form.

Freeform Form

The freeform form is basically a giant text area for you to write anything you want. In this example, I want to put AI to good use… by analyzing one of Bruno Mars’ lyrics.

By providing concise instructions and the lyric snippet, the AI model accurately describes what Bruno’s trying to tell us, how he feels, and ways for him to improve his personal relationship.

Structured Form

The structured form leverages the few-shot prompt technique to assist the model to provide a more specific response. In this case, you provide some context and a few examples of how the model should respond. Then, you test it by providing an input for the model to generate an output.

Export Code

In either freeform or structured form, once you are pleased with your designed prompt, you can export it to code by clicking on the <> VIEW CODE link on the top right.

The post Vertex AI PaLM: Intro to text-bison appeared first on My Shitty Code.