Observe your charm with COS Lite¶

From Zero to Hero: Write your first Kubernetes charm > Observe your charm with COS Lite

See previous: Expose your charm’s operational tasks via actions

Important

This document is part of a series, and we recommend you follow it in sequence. However, you can also jump straight in by checking out the code from the previous chapter:

git clone https://github.com/canonical/operator.git
cd operator/examples/k8s-4-action

In a production deployment it is essential to observe and monitor the health of your application. A charm user will want to be able to collect real time metrics and application logs, set up alert rules, and visualise any collected data in a neat form on a dashboard.

Our application is prepared for that – as you might recall, it uses starlette-exporter to generate real-time application metrics and to expose them via a /metrics endpoint that is designed to be scraped by Prometheus. As a charm developer, you’ll want to use that to make your charm observable.

In the charming universe, what you would do is deploy the existing Canonical Observability Stack (COS) lite bundle – a convenient collection of charms that includes all of Prometheus, Loki, and Grafana – and then integrate your charm with Prometheus to collect real-time application metrics; with Loki to collect application logs; and with Grafana to create dashboards and visualise collected data.

In this part of the tutorial we will follow this process to collect various metrics and logs about your application and visualise them on a dashboard.

Fetch libraries¶

Your charm will need several more libraries from Charmhub.

Ensure you’re in your Multipass Ubuntu VM, in your charm project directory. Then, in charmcraft.yaml, extend the charm-libs section:

charm-libs:
  - lib: data_platform_libs.data_interfaces
    version: "0"
  - lib: grafana_k8s.grafana_dashboard
    version: "0"
  - lib: loki_k8s.loki_push_api
    version: "1"
  - lib: observability_libs.juju_topology
    version: "0"
  - lib: prometheus_k8s.prometheus_scrape
    version: "0"

Next, run the following command to download the libraries:

ubuntu@juju-sandbox-k8s:~/fastapi-demo$ charmcraft fetch-libs

You might see a warning that Charmcraft cannot get a keyring. You can ignore the warning.

After Charmcraft has downloaded the libraries, your project’s lib directory contains:

lib
└── charms
    ├── data_platform_libs
    │   └── v0
    │       └── data_interfaces.py
    ├── grafana_k8s
    │   └── v0
    │       └── grafana_dashboard.py
    ├── loki_k8s
    │   └── v1
    │       └── loki_push_api.py
    ├── observability_libs
    │   └── v0
    │       └── juju_topology.py
    └── prometheus_k8s
        └── v0
            └── prometheus_scrape.py

Add dependencies from libraries¶

When you use libraries from Charmhub, you must check whether the libraries have any dependencies apart from ops.

If you open lib/charms/grafana_k8s/v0/grafana_dashboard.py and the other library files, you’ll see that some of the libraries depend on the cosl package:

grafana_dashboard.py specifies PYDEPS = ["cosl >= 0.0.50"]
loki_push_api.py specifies PYDEPS = ["cosl"]
prometheus_scrape.py specifies PYDEPS = ["cosl>=0.0.53"]

This means that you need to add cosl>=0.0.53 to your charm’s dependencies.

To update your charm’s dependencies in pyproject.toml, run:

uv add 'cosl>=0.0.53'

Integrate with Prometheus¶

Follow the steps below to make your charm capable of integrating with the existing Prometheus charm. This will enable your charm user to collect real-time metrics about your application.

Define the Prometheus relation interface¶

In your charmcraft.yaml file, after the requires block, add a provides endpoint with relation name metrics-endpoint and interface name prometheus_scrape, as below. This declares that your charm can offer services to other charms over the prometheus-scrape interface. In short, that your charm is open to integrating with, for example, the official Prometheus charm. (Note: metrics-endpoint is the default relation name recommended by the prometheus_scrape interface library.)

provides:
  metrics-endpoint:
    interface: prometheus_scrape
    optional: true

Import the Prometheus interface libraries and set up Prometheus scraping¶

In your src/charm.py file, do the following:

First, at the top of the file, import the prometheus_scrape library:

from charms.prometheus_k8s.v0.prometheus_scrape import MetricsEndpointProvider

Now, in your charm’s __init__ method, initialise the MetricsEndpointProvider instance with the desired scrape target, as below. Note that this uses the relation name that you specified earlier in the charmcraft.yaml file. Also, reflecting the fact that you’ve made your charm’s port configurable (see previous chapter Make the charm configurable), the target job is set to be consumed from config. The URL path is not included because it is predictable (defaults to /metrics), so the Prometheus library uses it automatically. The last line, which sets the refresh_event to the config_change event, ensures that the Prometheus charm will change its scraping target every time someone changes the port configuration. Overall, this code will allow your application to be scraped by Prometheus once they’ve been integrated.

# Provide a metrics endpoint for Prometheus to scrape.
try:
    config = self.load_config(FastAPIConfig)
except ValueError as e:
    logger.warning("Unable to add metrics: invalid configuration: %s", e)
else:
    self._prometheus_scraping = MetricsEndpointProvider(
        self,
        relation_name="metrics-endpoint",
        jobs=[{"static_configs": [{"targets": [f"*:{config.server_port}"]}]}],
        refresh_event=self.on.config_changed,
    )

Congratulations, your charm is ready to be integrated with Prometheus!

Integrate with Loki¶

Follow the steps below to make your charm capable of integrating with the existing Loki charm. This will enable your charm user to collect application logs.

Define the Loki relation interface¶

In your charmcraft.yaml file, beneath your existing requires endpoint, add another requires endpoint with relation name logging and interface name loki_push_api. This declares that your charm can optionally make use of services from other charms over the loki_push_api interface. In short, that your charm is open to integrating with, for example, the official Loki charm. (Note: logging is the default relation name recommended by the loki_push_api interface library.)

requires:
  database:
    interface: postgresql_client
    limit: 1
    optional: false
  logging:
    interface: loki_push_api
    optional: true

Import the Loki interface libraries and set up the Loki API¶

In your src/charm.py file, do the following:

First, import the loki_push_api lib:

from charms.loki_k8s.v1.loki_push_api import LogForwarder

Then, in your charm’s __init__ method, initialise the LogForwarder instance as shown below. The logging relation name comes from the charmcraft.yaml file. Overall this code ensures that your application can push logs to Loki (or any other charms that implement the loki_push_api interface).

# Enable pushing application logs to Loki.
self._logging = LogForwarder(self, relation_name="logging")

Congratulations, your charm can now also integrate with Loki!

Integrate with Grafana¶

Follow the steps below to make your charm capable of integrating with the existing Grafana charm. This will allow your charm user to visualise the data collected from Prometheus and Loki.

Define the Grafana relation interface¶

In your charmcraft.yaml file, add another provides endpoint with relation name grafana-dashboard and interface name grafana_dashboard, as below. This declares that your charm can offer services to other charms over the grafana-dashboard interface. In short, that your charm is open to integrations with, for example, the official Grafana charm. (Note: Here grafana-dashboard endpoint is the default relation name recommended by the grafana_dashboard library.)

provides:
  metrics-endpoint:
    interface: prometheus_scrape
    optional: true
  grafana-dashboard:
    interface: grafana_dashboard
    optional: true

Import the Grafana interface libraries and set up the Grafana dashboards¶

In your src/charm.py file, do the following:

First, at the top of the file, import the grafana_dashboard lib:

from charms.grafana_k8s.v0.grafana_dashboard import GrafanaDashboardProvider

Now, in your charm’s __init__ method, initialise the GrafanaDashboardProvider instance, as below. The grafana-dashboard is the relation name you defined earlier in your charmcraft.yaml file. Overall, this code states that your application supports the Grafana interface.

# Provide grafana dashboards over a relation interface.
self._grafana_dashboards = GrafanaDashboardProvider(
    self, relation_name="grafana-dashboard"
)

Now, in your src directory, create a subdirectory called grafana_dashboards and, in this directory, create a file called FastAPI-Monitoring.json.tmpl with the following content: FastAPI-Monitoring.json.tmpl. Once your charm has been integrated with Grafana, the GrafanaDashboardProvider you defined just before will take this file as well as any other files defined in this directory and put them into a Grafana files tree to be read by Grafana.

Important

How to build a Grafana dashboard is beyond the scope of this tutorial. However, if you’d like to get a quick idea: The dashboard template file was created by manually building a Grafana dashboard using the Grafana web UI, then exporting it to a JSON file and updating the datasource uid for Prometheus and Loki from constant values to the dynamic variables "${prometheusds}" and "${lokids}", respectively.

Validate your charm¶

Open a shell in your Multipass Ubuntu VM, navigate inside your project directory, and run all of the following.

First, repack and refresh your charm:

charmcraft pack
juju refresh \
  --path="./fastapi-demo_amd64.charm" \
  fastapi-demo --force-units --resource \
  demo-server-image=ghcr.io/canonical/api_demo_server:1.0.2

Next, test your charm’s ability to integrate with Prometheus, Loki, and Grafana by following the steps below.

Deploy COS Lite¶

Create a Juju model called cos-lite and, to this model, deploy the Canonical Observability Stack bundle cos-lite, as below. This will deploy all the COS applications (alertmanager, catalogue, grafana, loki, prometheus, traefik), already suitably integrated with one another. Note that these also include the applications that you’ve been working to make your charm integrate with – Prometheus, Loki, and Grafana.

juju add-model cos-lite
juju deploy cos-lite --trust

Important

Why put COS Lite in a separate model? Because (1) it is always a good idea to separate logically unrelated applications in different models and (2) this way you can observe applications across all your models. PS In a production-grade scenario you would actually even want to put your COS Lite in a separate cloud (i.e., Kubernetes cluster). This is recommended, for example, to ensure proper hardware resource allocation.

Expose the application relation endpoints¶

Once all the COS Lite applications are deployed and settled down (you can monitor this by using juju status --watch 2s), expose the relation points you are interested in for your charm – loki:logging, grafana-dashboard, and metrics-endpoint – as below.

juju offer prometheus:metrics-endpoint
juju offer loki:logging
juju offer grafana:grafana-dashboard

Validate that the offers have been successfully created by running:

juju find-offers cos-lite

You should see something similar to the output below:

Store          URL                        Access  Interfaces
concierge-k8s  admin/cos-lite.loki        admin   loki_push_api:logging
concierge-k8s  admin/cos-lite.prometheus  admin   prometheus_scrape:metrics-endpoint
concierge-k8s  admin/cos-lite.grafana     admin   grafana_dashboard:grafana-dashboard

As you might notice from your knowledge of Juju, this is essentially preparing these endpoints, which exist in the cos-lite model, for a cross-model relation with your charm, which you’ve deployed to the testing model.

Integrate your charm with COS Lite¶

Now switch back to the charm model and integrate your charm with the exposed endpoints, as below. This effectively integrates your application with Prometheus, Loki, and Grafana.

juju switch testing
juju integrate fastapi-demo admin/cos-lite.grafana
juju integrate fastapi-demo admin/cos-lite.loki
juju integrate fastapi-demo admin/cos-lite.prometheus

Simulate API requests¶

Before we monitor the health of our application, let’s simulate a continuous load of API requests. We’ll set up the simulation so that a proportion of requests fail because of a server error.

First, create a file called simulate.sh in your project directory:

#!/bin/sh

unit_location="10.1.157.94:8000"  # Get the IP address from 'juju status'

while true; do
    for i in {1..3}; do
        curl "http://$unit_location/names"
        echo
        sleep 5
    done

    curl "http://$unit_location/error"
    echo
    sleep 5
done

This script repeatedly sends requests to our application’s API endpoints. Our application’s /error endpoint deliberately returns HTTP status code 500 (Internal Server Error). When we monitor the health of our application, we’ll see a sustained error rate of 25%.

Replace 10.1.157.94 by the IP address of your fastapi-demo unit, which you can get from the output of juju status.

Note

simulate.sh isn’t intended to show how to benchmark a real application. Sending requests to a Juju unit is convenient as a one-off local simulation, but for a real application you’d send requests through an ingress integrator such as Traefik. You’d also use a benchmarking tool such as ab.

Next, open a new terminal in your virtual machine:

multipass shell juju-sandbox-k8s

Then run the script:

chmod +x ~/fastapi-demo/simulate.sh
. ~/fastapi-demo/simulate.sh

The output should look like:

{"names":{}}
{"names":{}}
{"names":{}}
Internal server error
{"names":{}}
...

Leave the script running for the rest of the tutorial. To stop the script later, press Ctrl + C.

Access Grafana from your host machine¶

Grafana allows you to visualise metrics on a dashboard. We’ll now open Grafana’s web UI to monitor the health of our application.

COS Lite exposes Grafana through a load balancer that is provided by the Traefik ingress integrator. In a production deployment, you’d access Grafana by connecting to the external endpoint that Traefik exposes. We don’t have a production deployment, so we’ll access Grafana by connecting to the load balancer’s Kubernetes service.

To access Grafana from your host machine, you’ll need:

Grafana’s admin password
The HTTP port of the load balancer’s Kubernetes service
Your virtual machine’s IP address

To get Grafana’s admin password, run the following command in your virtual machine:

juju run grafana/0 -m cos-lite get-admin-password --wait 1m

The output should look like:

Running operation 3 with 1 task
  - task 4 on unit-grafana-0

Waiting for task 4...
admin-password: eEJOix1zkrJ6
url: http://10.43.45.0/cos-lite-grafana

In our example, the admin password is eEJOix1zkrJ6.

Next, to get the HTTP port of the load balancer’s Kubernetes service, run the following command in your virtual machine:

kubectl -n cos-lite get svc traefik-lb

The output should look like:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
traefik-lb   LoadBalancer   10.152.183.166   10.43.45.0    80:31471/TCP,443:31548/TCP   10m

The port we want is given by 80:<port>/TCP. In our example, the port is 31471.

Finally, to get your virtual machine’s IP address, run the following command on your host machine:

multipass info juju-sandbox-k8s

The output should look like:

Name:           juju-sandbox-k8s
State:          Running
Snapshots:      1
IPv4:           10.112.13.157
                10.49.132.1
                10.1.157.64
Release:        Ubuntu 24.04.3 LTS
Image hash:     2b5f90ffe818 (Ubuntu 24.04 LTS)
CPU(s):         4
Load:           0.31 0.25 0.28
Disk usage:     19.4GiB out of 48.4GiB
Memory usage:   3.2GiB out of 7.7GiB
Mounts:         /home/me/k8s-tutorial => ~/fastapi-demo
                    UID map: 1000:default
                    GID map: 1000:default

The IP address we want is the first IPv4 address listed. In our example, the IP address is 10.112.13.157.

We can now combine the IP address and port to obtain the URL of Grafana’s web UI:

http://10.112.13.157:31471/cos-lite-grafana

Your Grafana URL will be similar.

Open your Grafana URL in your browser, then log in using the username admin and the password you got from juju run.

Inspect the Grafana dashboard¶

In the Grafana web UI, navigate to the Dashboards page, then click General > FastAPI Monitoring. This opens the dashboard that you put in the grafana_dashboards directory of your charm.

Next, in the “Juju model” drop down field, select “testing”.

You should see the following data on the dashboard:

HTTP request duration percentiles - This graph shows Prometheus data. It tracks the duration below which 60% of requests fall (p60) and the duration below which 90% of requests fall (p90).
Percentage of failed requests - This graph should be flat at 25% because simulate.sh sends 25% of requests to /error.
FastAPI logs from the workload container - These logs were captured from our application by Pebble, sent to Loki, then sent to Grafana. You can see info and error messages as FastAPI handles each request to /names and /error, including exception tracebacks.

Application monitoring dashboard in Grafana

Inspect metrics in Prometheus¶

Let’s use Prometheus to explore the metrics that our application provides from its /metrics endpoint. These metrics are the data source for the graphs on the Grafana dashboard.

The URL of Prometheus’s web UI is:

http://10.112.13.157:31471/cos-lite-prometheus-0/graph

Where 10.112.13.157 is your virtual machine’s IP address and 31471 is the HTTP port of the load balancer’s Kubernetes service, as with Grafana.

Open your Prometheus URL in your browser, then enter a search expression and click Execute. For example, use the following expression to see how many requests the /names endpoint has received:

starlette_requests_total{path="/names"}

The search result should look like:

Application metrics in Prometheus

Which means that /names has received 150 requests so far.

Review the final code¶

For the full code, see our example charm for this chapter.

Next steps¶

Congratulations on reaching the end of the tutorial!

For suggestions of what to explore next, see Next steps.