Gone from our sight, but never from our hearts.
– Unknown
Embracing the Messiness in Search of Epic Solutions
– Unknown
A typical way to get a list of tags from a repository is to clone it before running git tag
:-
git clone git@ssh.dev.azure.com:v3/test/my-shitty-repo cd my-shitty-repo git -c 'versionsort.suffix=-' tag --sort='v:refname' # output 1.0.0-b20200317174203 1.0.0 1.0.1-b20200318174753 1.0.1-b20200318174841 1.0.1-b20200407185909 1.0.1 1.0.2-b20200413205910 1.0.2
versionsort.suffix=-
ensures 1.0.0-XXXXXX
comes after 1.0.0
.
To retrieve the latest tag:-
git clone git@ssh.dev.azure.com:v3/test/my-shitty-repo cd my-shitty-repo git -c 'versionsort.suffix=-' tag --sort='v:refname' | tail -n1 # output 1.0.2
While it works, it requires us to clone the repository first, and if we want to retrieve tags from multiple repositories, we are quickly filling our hard drive space.
Git has a way to perform a remote query through git ls-remote
.
To perform the same task without cloning the repository, we can run this:-
git -c 'versionsort.suffix=-' ls-remote \ --tags \ --sort='v:refname' \ git@ssh.dev.azure.com:v3/test/my-shitty-repo # output b90df3d12413db22d051db1f7c7286cdd2f00b66 refs/tags/1.0.0-b20200317174203 e355a58829a2d2895ab2d7610fad1ab26dc0c1e7 refs/tags/1.0.0 345153c39a588a6ab2782772ee9dcf9f9123efa9 refs/tags/1.0.1-b20200318174753 efc40f0bd68bb8c7920be7700cab81db0e6bdf83 refs/tags/1.0.1-b20200318174841 efc40f0bd68bb8c7920be7700cab81db0e6bdf83 refs/tags/1.0.1-b20200407185909 c5ed5fe30cba621f40daa542c0613fa2c1c1a47d refs/tags/1.0.1 7205ada5d8bd4318f82e58e8752ba651211f9d82 refs/tags/1.0.2-b20200413205910 6ba62a0f06f831812cbb13a6d1e83602ffe9e8d3 refs/tags/1.0.2
To retrieve the latest tag:-
git -c 'versionsort.suffix=-' ls-remote \ --tags \ --sort='v:refname' \ git@ssh.dev.azure.com:v3/test/my-shitty-repo | tail -n1 | sed -E 's|.*refs/tags/(.+)|\1|' # output 1.0.2
To perform an offline (or airgapped) installation, sometimes it’s not sufficient to download just the needed RPM package. In most cases, this package requires a list of dependencies to be installed too.
For example, Nginx requires at least 20 different packages in order for its installation to be successful:-
$ sudo yum deplist nginx package: nginx.x86_64 1:1.16.1-1.el7 dependency: /bin/sh provider: bash.x86_64 4.2.46-34.el7 dependency: libc.so.6(GLIBC_2.17)(64bit) provider: glibc.x86_64 2.17-307.el7.1 dependency: libcrypt.so.1()(64bit) provider: glibc.x86_64 2.17-307.el7.1 ... ... ... dependency: system-logos provider: redhat-logos.noarch 70.7.0-1.el7 dependency: systemd provider: systemd.x86_64 219-73.el7_8.8
The first step to spin up an instance using the same Linux distro/version and this instance must have internet access.
Download the following yum plugin that allows the package(s) to be downloaded without installing them:-
sudo yum install yum-plugin-downloadonly
Download the needed package (in this case, Nginx) and its dependencies into a directory:-
sudo yum install --downloadonly --downloaddir=/tmp/nginx nginx
To inspect what’s being downloaded:-
$ ls -a /tmp/nginx | sort . .. dejavu-fonts-common-2.33-6.el7.noarch.rpm dejavu-sans-fonts-2.33-6.el7.noarch.rpm fontconfig-2.13.0-4.3.el7.x86_64.rpm fontpackages-filesystem-1.44-8.el7.noarch.rpm gd-2.0.35-26.el7.x86_64.rpm gperftools-libs-2.6.1-1.el7.x86_64.rpm libjpeg-turbo-1.2.90-8.el7.x86_64.rpm libX11-1.6.7-2.el7.x86_64.rpm libX11-common-1.6.7-2.el7.noarch.rpm libXau-1.0.8-2.1.el7.x86_64.rpm libxcb-1.13-1.el7.x86_64.rpm libXpm-3.5.12-1.el7.x86_64.rpm nginx-1.16.1-1.el7.x86_64.rpm nginx-all-modules-1.16.1-1.el7.noarch.rpm nginx-filesystem-1.16.1-1.el7.noarch.rpm nginx-mod-http-image-filter-1.16.1-1.el7.x86_64.rpm nginx-mod-http-perl-1.16.1-1.el7.x86_64.rpm nginx-mod-http-xslt-filter-1.16.1-1.el7.x86_64.rpm nginx-mod-mail-1.16.1-1.el7.x86_64.rpm nginx-mod-stream-1.16.1-1.el7.x86_64.rpm redhat-indexhtml-7-13.el7.noarch.rpm
Now, this directory can be safely compressed into a tarball to be copied to the target instance without internet access.
After extracting the tarball in the target instance, it’s time to install the package:-
sudo yum -y --disablerepo=* localinstall /tmp/nginx/*.rpm
This command is smart enough to figure out all the dependencies within the directory and install them in the proper order.
Step by step instructions for my future self to obtain the SSL certificate and to configure it in Nginx because my fragile little brain cannot retain them at the moment.
Generate a private key and store it in a safe place.
openssl genrsa -out myshittycode_com.key 2048
Generate a certificate signing request (CSR).
openssl req -new -sha256 \ -key myshittycode_com.key \ -out myshittycode_com.csr \ -subj "/C=US/ST=ShittyState/L=ShittyCity/OU=ShittyUnit/O=ShittyCompany/CN=myshittycode.com"
Request a SSL certificate from Certificate Authority (ex: Sectigo) using the generated CSR.
If approved, you will receive an email similar to the below.
To configure the SSL certificate in Nginx, don’t use Certificate (w/ chain), PEM encoded because it contains the certificates in the following order:-
Nginx wants them in this order:-
To pull this off, download Certificate only, PEM encoded and Root/Intermediate(s) only, PEM encoded.
Then combine them into one file in the proper order.
cat myshittycode_com_cert.cer myshittycode_com_interm.cer > myshittycode_com_bundle.crt
In nginx.conf, add the bundle certificate, private key and server name.
server { listen 443; ssl on; ssl_certificate /etc/pki/tls/certs/myshittycode_com_bundle.crt; ssl_certificate_key /etc/pki/tls/private/myshittycode_com.key; server_name myshittycode.com; location / { ... } }
Restart Nginx service.
The GCP logging agent uses modified fluentd, which allows us to do either unstructured logging or structured logging. The structured logging relies on JSON payload while the unstructured logging can be any texts. The advantage of structured logging is we can leverage log features in GCP Log Viewer.
Installing the unstructured logging is straightforward:-
curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh sudo bash add-logging-agent-repo.sh sudo yum install -y google-fluentd sudo yum install -y google-fluentd-catch-all-config
To capture a specific log file, create a config file under /etc/google-fluentd/config.d
dir, ex:-
# /etc/google-fluentd/config.d/test.conf <source> @type tail format none path /tmp/test.log pos_file /var/lib/google-fluentd/pos/test.pos tag test read_from_head true </source>
Finally, start the service.
sudo systemctl restart google-fluentd
format none
indicates no additional processing will be done. So, the logs appear as “flat” texts.
Using unstructured logging, the severity indicator never gets set, so we lose the color coding and filtering capabilities in GCP Log Viewer, ex:-
Installing structured logging is very similar to unstructured logging with one small change:-
curl -sSO https://dl.google.com/cloudagents/add-logging-agent-repo.sh sudo bash add-logging-agent-repo.sh sudo yum install -y google-fluentd sudo yum install -y google-fluentd-catch-all-config-structured
Create a config file, but the content is slightly more complicated depending on how granular we want to capture the structured log data:-
# /etc/google-fluentd/config.d/test.conf <source> @type tail format multiline format_firstline /\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z/ format1 /^(?<time>[^ ]*) \[(?<severity>[^\] ]*).*?\] - (?<message>.*)$/ time_format %Y-%m-%dT%H:%M:%S.%NZ path /tmp/test.log pos_file /var/lib/google-fluentd/pos/test.pos tag test read_from_head true </source>
Given the unstructured logs may span multiple lines, multiline parser is ideal. Here’s an example of logs that span multiple lines:-
2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report: Number of binaries: 0 Total execution time: 4 millis 2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles 2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup 2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions
format_firstline
indicates when to stop parsing when scanning through multiple lines.
There can be multiple format[N]
, depending on how granular we want to capture the structured log data.
Under format1
, there are several reserved captured group names used in this example. time
indicates log time, severity
activates color coding in GCP Log Viewer and message
indicates the log message.
time_format
formats values from time
so that GCP Log Viewer can convert the UTC timezone to the local timezone.
Custom captured group names can also be used, and they will appear in under jsonPayload
in GCP Log Viewer, ex:-
{ insertId: "6un3n1flz5dd3y2pv" jsonPayload: { class_line_num: "o.j.x.c.h.XrayHeartbeatImpl:55" thread: "http-nio-8081-exec-8" trace_id: "42998b6c3c8d8a8" } labels: {…} logName: "projects/ml-mps-cpl-ajfrog-p-ea94/logs/jfrog_artifactory%2Fartifactory_service_log" receiveTimestamp: "2020-06-20T23:46:45.123156405Z" resource: {…} severity: "INFO" timestamp: "2020-06-20T23:46:43.188Z" }
Here’s an example of a fully converted structured logging in GCP Log Viewer:-
Now, we can easily filter the logs by severity:-
Building a robust regular expression is very painful, but you can use tools such as https://regex101.com to construct and test the regular expressions first before pasting in the config file.
When tweaking the config file, the quickest way to verify the correctness of it is to restart the service. Then, run this command, check for any errors, rinse and repeat:-
tail /var/log/google-fluentd/google-fluentd.log
Instead of waiting for the app to push logs into the log file, you can manually append the mock data to the log file, ex:-
cat <<EOF >>/tmp/myshittycode_structured.log 2020-06-20T20:00:00.107Z [INFO ] - Storage garbage collector report: Number of binaries: 0 Total execution time: 4 millis 2020-06-20T20:14:25.525Z [INFO ] - Starting to cleanup incomplete Release Bundles 2020-06-20T20:14:25.533Z [WARN ] - Finished incomplete Release Bundles cleanup 2020-06-20T20:31:00.167Z [ERROR ] - Start cleaning expired sessions EOF
Then, run this command to ensure there are no “pattern not match” errors:-
tail /var/log/google-fluentd/google-fluentd.log
If there are no errors, the logs will eventually appear in GCP Log Viewer.