Deploy Loki on Kubernetes, and monitor the logs of your pods

Jan 20, 2022 by Thibault Debatty | 12750 views

Kubernetes Monitoring DevOps

https://cylab.be/blog/197/deploy-loki-on-kubernetes-and-monitor-the-logs-of-your-pods

Loki is a log database developed by Grafana Labs. In a previous blog post we have shown how to run Loki with docker-compose. In this blog post we will deploy Loki on a Kubernetes cluster, and we will use it to monitor the log of our pods.

grafana-apache.png

Installation

The easiest way to deploy Loki is using Helm. So first make sure you installed Helm.

Then you can add the helm repository of Grafana:

helm repo add grafana https://grafana.github.io/helm-charts

A typical Loki stack consists of:

  • Loki itself, the log database (this would be the equivalent of Elasticsearch);
  • Grafana, the visualisation web interface (equivalent of Kibana);
  • Promtail, that allows to scrape log files and send the logs to Loki (equivalent of Logstash).

You can install the complete stack in a dedicated namespace (loki) with:

helm install loki grafana/loki-stack--namespace loki --create-namespace --set grafana.enabled=true

When Grafana is deployed, a random password is generated, so now you should extract the password of Grafana:

kubectl get secret --namespace loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Furthermore, the grafana interface is not exposed to the outside world, so you must use kubectl port-forward to create a tunnel from your computer to the loki-grafana service:

kubectl port-forward --namespace loki service/loki-grafana 3000:80

Grafana will now be available at http://localhost:3000 using following credentials:

  • Username: admin
  • Password: the password you extracted at the previous step...

First steps

The default installation settings of the loki stack are pretty complete:

  • the data source is correctly configured in Grafana
  • promtail is configured to scrape the logs of the pods running on your cluster

This means you can directly head to the Explore menu to check the logs of your pods:

  1. click on the compass on the left;
  2. at the top of the screen, select the loki;
  3. the main field at the top allows to type a LogQL query.

grafana-01.png

You can try with the following query, that will show you the logs from the pods of the loki namespace:

{namespace="loki"}

grafana-02.png

This field has an autocomplete feature, so it is actually pretty simple to type your queries. You can find the details of LogQL at https://grafana.com/docs/loki/latest/logql/

Apache dashboard

Grafana also allows to create dashboards, to quickly visualize and monitor the state of your application. To create a new dashboard, click on the + on the left.

Then you can add multiple panels to your dashboard. For each panel, you should at least fill:

  1. the type of data (time series, gauge, logs etc.);
  2. the data source (is always Loki for our examples);
  3. the query to execute.

grafana-new-panel.png

Here are a few examples...

To show the number of requests received per minute (for pods in the default namespace):

  • type: time series
  • query: count_over_time({namespace="default"}[1m])

To show the number of login attempts per minute:

  • type: time series
  • query: count_over_time({namespace="default"}|="POST /login"[1m])

To show the requests that caused a server error (code 5xx):

  • type: logs
  • query: {namespace="default"}|~" 5.. "

grafana-apache.png

Install Kubernetes on Ubuntu 22.04 with MicroK8s
Kubernetes is a complex beast, with lots of available drivers and plugins to handle different types of clusters. For example, to handle networking between pods in the cluster, you must install a CNI (Container Network Interface) plugin like Flannel, Calico, Weave Net, Cilium or other. To handle storage and volumes, you can install one of multiple CSI (Container Storage Interface) plugins like CephFS, GlusterFS, Google Cloud Storage etc. In this blog post we show how to use MicroK8s to simplify the installation process.
Getting started with Helm
Helm is a template engine that can be used to deploy applications on a Kubernetes cluster. It allows to build a customized kubernetes resources definition, that can be deployed on your cluster. But helm has actually many other functionalities, to handle the whole process of building, distributing, installing and managing Kubernetes applications.
Apache : log real IP addresses behind a reverse proxy
It's quite common now to run your web application behind a reverse proxy or a loadbalancer. This is typically the case if you are running your application in a Kubernetes cluster. In this case, the IP address that is logged by Apache is the IP of the proxy server, which is quite misleading and useless. To get Apache to log the real IP address of the clients, you will have to enable and configure the module remoteip.