GCSFuse + Docker: “Error while creating mount source path ‘/a’: mkdir /a: file exists.”

This post illustrates how you can mount a GCS bucket using GCSFuse on your host machine and expose it as a volume to a Docker container.

PROBLEM

You want to volume mount a FUSE-mounted directory to a container, for example:

When attempting to run the container…

docker run -it --rm -v /my-kfc-bucket:/home busybox

… an error occurred:

docker: Error response from daemon: error while creating
mount source path '/my-kfc-bucket': mkdir /my-kfc-bucket: 
file exists.

SOLUTION

Unmount the existing FUSE-mounted directory.

sudo umount /my-kfc-bucket

Mount it back with the following option. Because this command with -o allow_other must be executed with sudo privilege, you will need to change the root ownership to yourself (via –uid and –gid) so that you can easily read/write within the directory.

sudo gcsfuse \
  -o allow_other \
  --uid $(id -u) \
  --gid $(id -g) \
  gcs-bucket /my-kfc-bucket  

If it is successful, the output should look like this:

Start gcsfuse/0.40.0 (Go version go1.17.6) for app "" using mount point: /my-kfc-bucket
Opening GCS connection...
Mounting file system "gcs-bucket"...
File system has been successfully mounted.

Rerun the docker container.

docker run -it --rm -v /my-kfc-bucket:/home busybox

Now, you can read/write the GCS bucket’s data from the container. In this example, the GCS bucket’s data is located in /home.

GCP: Pushing Codebase from IntelliJ IDEA to VM Instance

OBJECTIVE

To push codebase from IntelliJ IDEA (or any JetBrains products) on a local machine to a VM instance in Google Cloud Platform.

To run the codebase remotely.

WHY DO THIS

You want to leverage all the power of a modern IDE on your 4K screen.

You do not want to use remote desktop tools such as VNC or NoMachine due to performance and screen lag problems.

Your team members make fun of your VIM skills.

SOLUTION

Configuring VM Port Forwarding

Log into GCP.

gcloud auth login

Perform port forwarding over SSH using your running VM.

# SYNTAX
gcloud compute ssh VM_NAME \
    --project PROJECT_ID \
    --zone ZONE \
    -- -NL LOCAL_PORT:localhost:REMOTE_PORT

# EXAMPLE
gcloud compute ssh shitty_vm \
    --project shitty_project \
    --zone us-central1-b \
    -- -NL 8888:localhost:22

Note: If you choose to listen to local port 22, you will most likely to get this error because your local SSH server may already be using it:

bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: 22
Could not request local forwarding.

If this is your first SSH into your VM, you will be prompted to create the SSH key pair. In this case, keep pressing the “Enter” key until it is created.

WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/shitty_user/.ssh/google_compute_engine.
Your public key has been saved in /Users/shitty_user/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:gmwGL9bfJLi/FYnebZLL0vVBYoZ3XeT/ivSSFCmiRT8 shitty_user@shitty_machine
The key's randomart image is:
+---[RSA 3072]----+
|               ..|
|        .      ..|
|  .    . o   o ..|
|   = o .+.E = . .|
|  o O +oS= * .  .|
| . + +.* +. o   .|
|    . o.*.oo.o  .|
|     ..o.+ .+o . |
|      ooo   ..o  |
+----[SHA256]-----+
External IP address was not found; defaulting to using IAP tunneling.
Writing 3 keys to /Users/shitty_user/.ssh/google_compute_known_hosts

Upon a successful port forwarding, the command will hang with the following text:

External IP address was not found; defaulting to using IAP tunneling.
Existing host keys found in /Users/shitty_user/.ssh/google_compute_known_hosts

That is an expected behavior because the SSH tunnel is now established between your local machine and the VM.

Configuring IntelliJ IDEA

In IntelliJ IDEA, select Tools > Deployment > Browser Remote Host

Under Remote Host panel, select button.

Under Add Server dialog:

  • Name: <A Memorable Name… ex: shitty_server>
  • Type: SFTP

Click OK button.

Under Deployment dialog, select button on SSH Configurations.

Under SSH Configurations dialog:

  • Host: localhost
  • Port: 8888 (or the local port you specified)
  • User name: <Your VM’s user name>
  • Authentication type: Key pair
  • Private key file: /<PATH>/.ssh/google_compute_engine

Click on Test Connection button and ensure it is successful.

Click OK button.

Under Deployment dialog, select Mappings tab.

Under Mappings tab, click on the folder icon and specify a location to deploy the codebase to.

Click OK button.

Under Remote Host panel, you can now browse and access the files in your VM remotely.

Pushing Codebase from IntelliJ IDEA to VM

To deploy codebase to the VM, right click on the directory, select Deployment > Upload to [VM_NAME].

The codebase should be copied to the location you specified.

Tips: If you makes changes in both your local machine and VM, select Deployment > Sync with Deployed to [VM_NAME]. This allows you to synchronize the changes on both sides.

Running Codebase Remotely

To run the codebase remotely, select Tools > Start SSH Session.

Select the configured host.

Run the codebase.

GCP + Container Registry: Pushing/Pulling Images

PROBLEM

You want to push a new image to Google Container Registry (GCR) or pull an existing image from GCR.

SOLUTION

Pushing a New Image to GCR

Prepare your Dockerfile.

FROM alpine:3.7

# some content...

Create an image and tag it with a path pointing to GCR within a project.

There are several variations of GCR’s hostname (ex: gcr.io, us.gcr.io, eu.gcr.io, etc) depending on the data center’s location.

The GCR path has the following format: [HOSTNAME]/[PROJECT-ID]/[IMAGE].

docker build -t gcr.io/shitty-project/shitty-repo .

Log into GCP.

gcloud auth login

Register gcloud as a Docker credential helper.

gcloud auth configure-docker

Push the image to GCR.

docker push gcr.io/shitty-project/shitty-repo

View pushed image.

gcloud container images list-tags gcr.io/shitty-project/shitty-repo

DIGEST        TAGS    TIMESTAMP
78b36c0b456d  latest  2019-03-07T16:19:53

The repository and image can also be viewed in GCP Console.

Image in GCR

Pulling an Existing Image from GCR

Proceed with the authentication process if it is not done already.

gcloud auth login
gcloud auth configure-docker

Pull the image from GCR.

docker pull gcr.io/shitty-project/shitty-repo