Git: Querying Tags Without Cloning the Repository

PROBLEM

A typical way to get a list of tags from a repository is to clone it before running git tag:-

git clone git@ssh.dev.azure.com:v3/test/my-shitty-repo
cd my-shitty-repo
git -c 'versionsort.suffix=-' tag --sort='v:refname'

# output
1.0.0-b20200317174203
1.0.0
1.0.1-b20200318174753
1.0.1-b20200318174841
1.0.1-b20200407185909
1.0.1
1.0.2-b20200413205910
1.0.2

versionsort.suffix=- ensures 1.0.0-XXXXXX comes after 1.0.0.

To retrieve the latest tag:-

git clone git@ssh.dev.azure.com:v3/test/my-shitty-repo
cd my-shitty-repo
git -c 'versionsort.suffix=-' tag --sort='v:refname' |
tail -n1

# output
1.0.2

While it works, it requires us to clone the repository first, and if we want to retrieve tags from multiple repositories, we are quickly filling our hard drive space.

SOLUTION

Git has a way to perform a remote query through git ls-remote.

To perform the same task without cloning the repository, we can run this:-

git -c 'versionsort.suffix=-' ls-remote \
--tags \
--sort='v:refname' \
git@ssh.dev.azure.com:v3/test/my-shitty-repo

# output
b90df3d12413db22d051db1f7c7286cdd2f00b66	refs/tags/1.0.0-b20200317174203
e355a58829a2d2895ab2d7610fad1ab26dc0c1e7	refs/tags/1.0.0
345153c39a588a6ab2782772ee9dcf9f9123efa9	refs/tags/1.0.1-b20200318174753
efc40f0bd68bb8c7920be7700cab81db0e6bdf83	refs/tags/1.0.1-b20200318174841
efc40f0bd68bb8c7920be7700cab81db0e6bdf83	refs/tags/1.0.1-b20200407185909
c5ed5fe30cba621f40daa542c0613fa2c1c1a47d	refs/tags/1.0.1
7205ada5d8bd4318f82e58e8752ba651211f9d82	refs/tags/1.0.2-b20200413205910
6ba62a0f06f831812cbb13a6d1e83602ffe9e8d3	refs/tags/1.0.2

To retrieve the latest tag:-

git -c 'versionsort.suffix=-' ls-remote \
--tags \
--sort='v:refname' \
git@ssh.dev.azure.com:v3/test/my-shitty-repo |
tail -n1 |
sed -E 's|.*refs/tags/(.+)|\1|'

# output
1.0.2

RPM: Performing Offline Installation

PROBLEM

To perform an offline (or airgapped) installation, sometimes it’s not sufficient to download just the needed RPM package. In most cases, this package requires a list of dependencies to be installed too.

For example, Nginx requires at least 20 different packages in order for its installation to be successful:-

$ sudo yum deplist nginx
package: nginx.x86_64 1:1.16.1-1.el7
  dependency: /bin/sh
   provider: bash.x86_64 4.2.46-34.el7
  dependency: libc.so.6(GLIBC_2.17)(64bit)
   provider: glibc.x86_64 2.17-307.el7.1
  dependency: libcrypt.so.1()(64bit)
   provider: glibc.x86_64 2.17-307.el7.1
  ...
  ...
  ...
  dependency: system-logos
   provider: redhat-logos.noarch 70.7.0-1.el7
  dependency: systemd
   provider: systemd.x86_64 219-73.el7_8.8

SOLUTION

The first step to spin up an instance using the same Linux distro/version and this instance must have internet access.

Download the following yum plugin that allows the package(s) to be downloaded without installing them:-

sudo yum install yum-plugin-downloadonly

Download the needed package (in this case, Nginx) and its dependencies into a directory:-

sudo yum install --downloadonly --downloaddir=/tmp/nginx nginx

To inspect what’s being downloaded:-

$ ls -a /tmp/nginx | sort
.
..
dejavu-fonts-common-2.33-6.el7.noarch.rpm
dejavu-sans-fonts-2.33-6.el7.noarch.rpm
fontconfig-2.13.0-4.3.el7.x86_64.rpm
fontpackages-filesystem-1.44-8.el7.noarch.rpm
gd-2.0.35-26.el7.x86_64.rpm
gperftools-libs-2.6.1-1.el7.x86_64.rpm
libjpeg-turbo-1.2.90-8.el7.x86_64.rpm
libX11-1.6.7-2.el7.x86_64.rpm
libX11-common-1.6.7-2.el7.noarch.rpm
libXau-1.0.8-2.1.el7.x86_64.rpm
libxcb-1.13-1.el7.x86_64.rpm
libXpm-3.5.12-1.el7.x86_64.rpm
nginx-1.16.1-1.el7.x86_64.rpm
nginx-all-modules-1.16.1-1.el7.noarch.rpm
nginx-filesystem-1.16.1-1.el7.noarch.rpm
nginx-mod-http-image-filter-1.16.1-1.el7.x86_64.rpm
nginx-mod-http-perl-1.16.1-1.el7.x86_64.rpm
nginx-mod-http-xslt-filter-1.16.1-1.el7.x86_64.rpm
nginx-mod-mail-1.16.1-1.el7.x86_64.rpm
nginx-mod-stream-1.16.1-1.el7.x86_64.rpm
redhat-indexhtml-7-13.el7.noarch.rpm	

Now, this directory can be safely compressed into a tarball to be copied to the target instance without internet access.

After extracting the tarball in the target instance, it’s time to install the package:-

sudo yum -y --disablerepo=* localinstall /tmp/nginx/*.rpm

This command is smart enough to figure out all the dependencies within the directory and install them in the proper order.

Nginx: Requesting and Configuring SSL Certificate

INTRO

Step by step instructions for my future self to obtain the SSL certificate and to configure it in Nginx because my fragile little brain cannot retain them at the moment.

INSTRUCTIONS

Generate a private key and store it in a safe place.

openssl genrsa -out myshittycode_com.key 2048  

Generate a certificate signing request (CSR).

openssl req -new -sha256 \
  -key myshittycode_com.key \
  -out myshittycode_com.csr	\
  -subj "/C=US/ST=ShittyState/L=ShittyCity/OU=ShittyUnit/O=ShittyCompany/CN=myshittycode.com"	

Request a SSL certificate from Certificate Authority (ex: Sectigo) using the generated CSR.

If approved, you will receive an email similar to the below.

ssl-certificate

To configure the SSL certificate in Nginx, don’t use Certificate (w/ chain), PEM encoded because it contains the certificates in the following order:-

  • Root CA certificate
  • Intermediate CA certificate
  • Certificate

Nginx wants them in this order:-

  • Certificate
  • Root CA certificate
  • Intermediate CA certificate

To pull this off, download Certificate only, PEM encoded and Root/Intermediate(s) only, PEM encoded.

Then combine them into one file in the proper order.

cat myshittycode_com_cert.cer myshittycode_com_interm.cer > myshittycode_com_bundle.crt

In nginx.conf, add the bundle certificate, private key and server name.

server {
  listen               443;
  ssl                  on;
  ssl_certificate      /etc/pki/tls/certs/myshittycode_com_bundle.crt;
  ssl_certificate_key  /etc/pki/tls/private/myshittycode_com.key;
  server_name          myshittycode.com;

  location / {
    ...
  }
}

Restart Nginx service.

GCP Logging Agent: Converting Unstructured to Structured Logging

BACKGROUND

The GCP logging agent uses modified fluentd, which allows us to do either unstructured logging or structured logging. The structured logging relies on JSON payload while the unstructured logging can be any texts. The advantage of structured logging is we can leverage log features in GCP Log Viewer.

UNSTRUCTURED LOGGING

Installing the unstructured logging is straightforward:-

curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
sudo bash add-logging-agent-repo.sh
sudo yum install -y google-fluentd
sudo yum install -y google-fluentd-catch-all-config

To capture a specific log file, create a config file under /etc/google-fluentd/config.d dir, ex:-

# /etc/google-fluentd/config.d/test.conf

<source>
  @type tail
  format none
  path /tmp/test.log
  pos_file /var/lib/google-fluentd/pos/test.pos
  tag test
  read_from_head true
</source>

Finally, start the service.

sudo systemctl restart google-fluentd 

format none indicates no additional processing will be done. So, the logs appear as “flat” texts.

Using unstructured logging, the severity indicator never gets set, so we lose the color coding and filtering capabilities in GCP Log Viewer, ex:-

gcp-unstructured-logging

JOURNEY TO STRUCTURED LOGGING

Installing structured logging is very similar to unstructured logging with one small change:-

curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh
sudo bash add-logging-agent-repo.sh
sudo yum install -y google-fluentd
sudo yum install -y google-fluentd-catch-all-config-structured

Create a config file, but the content is slightly more complicated depending on how granular we want to capture the structured log data:-

# /etc/google-fluentd/config.d/test.conf

<source>
  @type tail
  format multiline
  format_firstline /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/
  format1 /^(?<time>[^ ]*) \[(?<severity>[^\] ]*).*?\] - (?<message>.*)$/
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  path /tmp/test.log
  pos_file /var/lib/google-fluentd/pos/test.pos
  tag test
  read_from_head true
</source>

Given the unstructured logs may span multiple lines, multiline parser is ideal. Here’s an example of logs that span multiple lines:-

2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report:
Number of binaries:      0
Total execution time:    4 millis
2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles
2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup
2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions

format_firstline indicates when to stop parsing when scanning through multiple lines.

There can be multiple format[N], depending on how granular we want to capture the structured log data.

Under format1, there are several reserved captured group names used in this example. time indicates log time, severity activates color coding in GCP Log Viewer and message indicates the log message.

time_format formats values from time so that GCP Log Viewer can convert the UTC timezone to the local timezone.

Custom captured group names can also be used, and they will appear in under jsonPayload in GCP Log Viewer, ex:-

{
  insertId: "6un3n1flz5dd3y2pv"  
  jsonPayload: {
    class_line_num: "o.j.x.c.h.XrayHeartbeatImpl:55"   
    thread: "http-nio-8081-exec-8"   
    trace_id: "42998b6c3c8d8a8"   
  }
  labels: {…}  
  logName: "projects/ml-mps-cpl-ajfrog-p-ea94/logs/jfrog_artifactory%2Fartifactory_service_log"  
  receiveTimestamp: "2020-06-20T23:46:45.123156405Z"  
  resource: {…}  
  severity: "INFO"  
  timestamp: "2020-06-20T23:46:43.188Z"  
}

Here’s an example of a fully converted structured logging in GCP Log Viewer:-

gcp-structured-logging

Now, we can easily filter the logs by severity:-

gcp-structured-logging-filtering

A COUPLE OF HELPFUL TIPS

Building Regular Expression

Building a robust regular expression is very painful, but you can use tools such as https://regex101.com to construct and test the regular expressions first before pasting in the config file.

Checking for Errors

When tweaking the config file, the quickest way to verify the correctness of it is to restart the service. Then, run this command, check for any errors, rinse and repeat:-

tail /var/log/google-fluentd/google-fluentd.log

Speed Up Testing

Instead of waiting for the app to push logs into the log file, you can manually append the mock data to the log file, ex:-

cat <<EOF >>/tmp/myshittycode_structured.log
2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report:
Number of binaries:      0
Total execution time:    4 millis
2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles
2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup
2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions
EOF 	

Then, run this command to ensure there are no “pattern not match” errors:-

tail /var/log/google-fluentd/google-fluentd.log

If there are no errors, the logs will eventually appear in GCP Log Viewer.

GCP Stackdriver Agent: “write_gcm: can not take infinite value” Error

PROBLEM

When running the stackdriver-agent (v6.x), the log file contains the following errors:-

collectd[1496]: write_gcm: can not take infinite value
collectd[1496]: write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing.
collectd[1496]: write_gcm: can not take infinite value
collectd[1496]: write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing.
collectd[1496]: write_gcm: can not take infinite value
collectd[1496]: write_gcm: wg_typed_value_create_from_value_t_inline failed for swap/percent/value! Continuing.

SOLUTION

By default, the swap metric is enabled.

To verify, go to /etc/stackdriver/collectd.conf and locate the following configuration:-

LoadPlugin swap
<Plugin "swap">
  ValuesPercentage true
</Plugin>

This error occurs because the VM instance does not have swap memory, hence this metric plugin tries to divide by 0.

To verify the VM’s swap memory:-

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3536         191        3197           8         147        3151
Swap:             0           0           0

To fix this, remove this configuration and restart stackdriver-agent.