GCP: Pushing Codebase from IntelliJ IDEA to VM Instance

OBJECTIVE

To push codebase from IntelliJ IDEA (or any JetBrains products) on a local machine to a VM instance in Google Cloud Platform.

To run the codebase remotely.

WHY DO THIS

You want to leverage all the power of a modern IDE on your 4K screen.

You do not want to use remote desktop tools such as VNC or NoMachine due to performance and screen lag problems.

Your team members make fun of your VIM skills.

SOLUTION

Configuring VM Port Forwarding

Log into GCP.

gcloud auth login

Perform port forwarding over SSH using your running VM.

# SYNTAX
gcloud compute ssh VM_NAME \
    --project PROJECT_ID \
    --zone ZONE \
    -- -NL LOCAL_PORT:localhost:REMOTE_PORT

# EXAMPLE
gcloud compute ssh shitty_vm \
    --project shitty_project \
    --zone us-central1-b \
    -- -NL 8888:localhost:22

Note: If you choose to listen to local port 22, you will most likely to get this error because your local SSH server may already be using it:

bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: 22
Could not request local forwarding.

If this is your first SSH into your VM, you will be prompted to create the SSH key pair. In this case, keep pressing the “Enter” key until it is created.

WARNING: The private SSH key file for gcloud does not exist.
WARNING: The public SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/shitty_user/.ssh/google_compute_engine.
Your public key has been saved in /Users/shitty_user/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:gmwGL9bfJLi/FYnebZLL0vVBYoZ3XeT/ivSSFCmiRT8 shitty_user@shitty_machine
The key's randomart image is:
+---[RSA 3072]----+
|               ..|
|        .      ..|
|  .    . o   o ..|
|   = o .+.E = . .|
|  o O +oS= * .  .|
| . + +.* +. o   .|
|    . o.*.oo.o  .|
|     ..o.+ .+o . |
|      ooo   ..o  |
+----[SHA256]-----+
External IP address was not found; defaulting to using IAP tunneling.
Writing 3 keys to /Users/shitty_user/.ssh/google_compute_known_hosts

Upon a successful port forwarding, the command will hang with the following text:

External IP address was not found; defaulting to using IAP tunneling.
Existing host keys found in /Users/shitty_user/.ssh/google_compute_known_hosts

That is an expected behavior because the SSH tunnel is now established between your local machine and the VM.

Configuring IntelliJ IDEA

In IntelliJ IDEA, select Tools > Deployment > Browser Remote Host

Under Remote Host panel, select button.

Under Add Server dialog:

  • Name: <A Memorable Name… ex: shitty_server>
  • Type: SFTP

Click OK button.

Under Deployment dialog, select button on SSH Configurations.

Under SSH Configurations dialog:

  • Host: localhost
  • Port: 8888 (or the local port you specified)
  • User name: <Your VM’s user name>
  • Authentication type: Key pair
  • Private key file: /<PATH>/.ssh/google_compute_engine

Click on Test Connection button and ensure it is successful.

Click OK button.

Under Deployment dialog, select Mappings tab.

Under Mappings tab, click on the folder icon and specify a location to deploy the codebase to.

Click OK button.

Under Remote Host panel, you can now browse and access the files in your VM remotely.

Pushing Codebase from IntelliJ IDEA to VM

To deploy codebase to the VM, right click on the directory, select Deployment > Upload to [VM_NAME].

The codebase should be copied to the location you specified.

Tips: If you makes changes in both your local machine and VM, select Deployment > Sync with Deployed to [VM_NAME]. This allows you to synchronize the changes on both sides.

Running Codebase Remotely

To run the codebase remotely, select Tools > Start SSH Session.

Select the configured host.

Run the codebase.

GCP + Container Registry: Pushing/Pulling Images

PROBLEM

You want to push a new image to Google Container Registry (GCR) or pull an existing image from GCR.

SOLUTION

Pushing a New Image to GCR

Prepare your Dockerfile.

FROM alpine:3.7

# some content...

Create an image and tag it with a path pointing to GCR within a project.

There are several variations of GCR’s hostname (ex: gcr.io, us.gcr.io, eu.gcr.io, etc) depending on the data center’s location.

The GCR path has the following format: [HOSTNAME]/[PROJECT-ID]/[IMAGE].

docker build -t gcr.io/shitty-project/shitty-repo .

Log into GCP.

gcloud auth login

Register gcloud as a Docker credential helper.

gcloud auth configure-docker

Push the image to GCR.

docker push gcr.io/shitty-project/shitty-repo

View pushed image.

gcloud container images list-tags gcr.io/shitty-project/shitty-repo

DIGEST        TAGS    TIMESTAMP
78b36c0b456d  latest  2019-03-07T16:19:53

The repository and image can also be viewed in GCP Console.

Image in GCR

Pulling an Existing Image from GCR

Proceed with the authentication process if it is not done already.

gcloud auth login
gcloud auth configure-docker

Pull the image from GCR.

docker pull gcr.io/shitty-project/shitty-repo

GCP Logging Agent: Converting Unstructured to Structured Logging

BACKGROUND

The GCP logging agent uses modified fluentd, which allows us to do either unstructured logging or structured logging. The structured logging relies on JSON payload while the unstructured logging can be any texts. The advantage of structured logging is we can leverage log features in GCP Log Viewer.

UNSTRUCTURED LOGGING

Installing the unstructured logging is straightforward:-

curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
sudo bash add-logging-agent-repo.sh
sudo yum install -y google-fluentd
sudo yum install -y google-fluentd-catch-all-config

To capture a specific log file, create a config file under /etc/google-fluentd/config.d dir, ex:-

# /etc/google-fluentd/config.d/test.conf

&lt;source&gt;
  @type tail
  format none
  path /tmp/test.log
  pos_file /var/lib/google-fluentd/pos/test.pos
  tag test
  read_from_head true
&lt;/source&gt;

Finally, start the service.

sudo systemctl restart google-fluentd 

format none indicates no additional processing will be done. So, the logs appear as “flat” texts.

Using unstructured logging, the severity indicator never gets set, so we lose the color coding and filtering capabilities in GCP Log Viewer, ex:-

gcp-unstructured-logging

JOURNEY TO STRUCTURED LOGGING

Installing structured logging is very similar to unstructured logging with one small change:-

curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
sudo bash add-logging-agent-repo.sh
sudo yum install -y google-fluentd
sudo yum install -y google-fluentd-catch-all-config-structured

Create a config file, but the content is slightly more complicated depending on how granular we want to capture the structured log data:-

# /etc/google-fluentd/config.d/test.conf

&lt;source&gt;
  @type tail
  format multiline
  format_firstline /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/
  format1 /^(?&lt;time&gt;[^ ]*) \[(?&lt;severity&gt;[^\] ]*).*?\] - (?&lt;message&gt;.*)$/
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  path /tmp/test.log
  pos_file /var/lib/google-fluentd/pos/test.pos
  tag test
  read_from_head true
&lt;/source&gt;

Given the unstructured logs may span multiple lines, multiline parser is ideal. Here’s an example of logs that span multiple lines:-

2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report:
Number of binaries:      0
Total execution time:    4 millis
2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles
2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup
2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions

format_firstline indicates when to stop parsing when scanning through multiple lines.

There can be multiple format[N], depending on how granular we want to capture the structured log data.

Under format1, there are several reserved captured group names used in this example. time indicates log time, severity activates color coding in GCP Log Viewer and message indicates the log message.

time_format formats values from time so that GCP Log Viewer can convert the UTC timezone to the local timezone.

Custom captured group names can also be used, and they will appear in under jsonPayload in GCP Log Viewer, ex:-

{
  insertId: "6un3n1flz5dd3y2pv"  
  jsonPayload: {
    class_line_num: "o.j.x.c.h.XrayHeartbeatImpl:55"   
    thread: "http-nio-8081-exec-8"   
    trace_id: "42998b6c3c8d8a8"   
  }
  labels: {…}  
  logName: "projects/ml-mps-cpl-ajfrog-p-ea94/logs/jfrog_artifactory%2Fartifactory_service_log"  
  receiveTimestamp: "2020-06-20T23:46:45.123156405Z"  
  resource: {…}  
  severity: "INFO"  
  timestamp: "2020-06-20T23:46:43.188Z"  
}

Here’s an example of a fully converted structured logging in GCP Log Viewer:-

gcp-structured-logging

Now, we can easily filter the logs by severity:-

gcp-structured-logging-filtering

A COUPLE OF HELPFUL TIPS

Building Regular Expression

Building a robust regular expression is very painful, but you can use tools such as https://regex101.com to construct and test the regular expressions first before pasting in the config file.

Checking for Errors

When tweaking the config file, the quickest way to verify the correctness of it is to restart the service. Then, run this command, check for any errors, rinse and repeat:-

tail /var/log/google-fluentd/google-fluentd.log

Speed Up Testing

Instead of waiting for the app to push logs into the log file, you can manually append the mock data to the log file, ex:-

cat &lt;&lt;EOF &gt;&gt;/tmp/myshittycode_structured.log
2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report:
Number of binaries:      0
Total execution time:    4 millis
2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles
2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup
2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions
EOF 	

Then, run this command to ensure there are no “pattern not match” errors:-

tail /var/log/google-fluentd/google-fluentd.log

If there are no errors, the logs will eventually appear in GCP Log Viewer.