Darren Rabbitt

Writing /

Implementing Observability with Prometheus, VictoriaMetrics, and Tilt

Kubernetes, Observability, Prometheus, Victoria Metrics, Tilt

Introduction

In the world of cloud-native applications, observability is crucial for understanding the health and performance of services. As I delved into the topic, I decided to explore how to set up a robust observability stack using Prometheus and VictoriaMetrics, orchestrated with Tilt on a local Kubernetes cluster. This post outlines my learnings and the practical steps I took to implement this setup, making it easier to monitor and analyze metrics from a microservice.

Exploring Prometheus and VictoriaMetrics

Before diving into the setup, it’s important to revisit the roles that Prometheus and VictoriaMetrics play within an observability stack.

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. It’s known for its powerful query language (PromQL) and ability to scrape metrics from various targets at configurable intervals. In this setup:

  • Scraping Metrics: Prometheus scrapes metrics exposed by the microservice, such as http_requests_total, which tracks the number of HTTP requests.
  • Data Storage and Queries: Prometheus stores these metrics in a time-series database and provides a query language to retrieve and analyze the data.

VictoriaMetrics

VictoriaMetrics is a high-performance, cost-effective monitoring solution that excels at handling large volumes of time-series data. I chose VictoriaMetrics as the long-term storage solution for metrics scraped by Prometheus. In this setup:

  • Receiving Data: Prometheus forwards the scraped metrics to VictoriaMetrics using the remote_write configuration.
  • Storing and Querying: VictoriaMetrics stores the data for long-term analysis and provides an interface for querying the stored metrics, either directly or via a UI.

Implementing the Observability Stack

Here’s how I structured the implementation using Tilt to manage the local Kubernetes environment.

Setting Up the Project

I started by setting up a simple microservice that exposes metrics, followed by configuring Prometheus and VictoriaMetrics.

git clone https://github.com/darrenrabbs/microservice-observability.git
cd microservice-observability

To get the environment running:

Ensure you have docker running with Kubernetes enabled If you haven’t installed Tilt yet, follow the Tilt installation guide.

kind create cluster --name observability-cluster
tilt up

This command builds the Docker images, deploys the microservice, and sets up Prometheus and VictoriaMetrics on the local Kubernetes cluster.

Project Structure

Here’s the structure of the project:

microservice-observability/
├── charts/
│   ├── microservice/
│   │   ├── templates/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   └── Chart.yaml
│   │   └── values.yaml
├── k8s/
│   ├── prometheus.yaml
│   └── victoria-metrics.yaml
├── src/
│   ├── main.go
│   ├── Dockerfile
├── tests/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── test_metrics.py
│   └── entrypoint.sh
├── Tiltfile
└── README.md

Configuring Prometheus and VictoriaMetrics

The core of this setup involves configuring Prometheus to scrape metrics from the microservice and forward them to VictoriaMetrics.

Prometheus Configuration (prometheus.yaml):

scrape_configs:
  - job_name: 'microservice'
    static_configs:
      - targets: ['microservice:8080']
remote_write:
  - url: 'http://victoria-metrics:8428/api/v1/write'

This configuration tells Prometheus to scrape the microservice at http://microservice:8080 and forward the data to VictoriaMetrics.

VictoriaMetrics Setup (victoria-metrics.yaml):

VictoriaMetrics is configured to receive and store the time-series data from Prometheus. The setup is straightforward and can be scaled as needed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: victoria-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: victoria-metrics
  template:
    metadata:
      labels:
        app: victoria-metrics
    spec:
      containers:
        - name: victoria-metrics
          image: victoriametrics/victoria-metrics
          ports:
            - containerPort: 8428
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "500m"

Testing the Setup

Manual steps

  1. Send a request to the microservice . Use Curl to send the request, this will increase the http_requests_total metric
curl http://localhost:8080/
  1. Verify Prometheus is Scraping Metrics:

Verify Victoria Metrics Interface (VMUI):

  • Access the Victoria Metrics User Interface (VMUI) to ensure it is functioning correctly.
  • Open your web browser and enter the URL for the VMUI - http://localhost:8428/vmui
  • Explore the interface to verify that the metrics are being stored and can be queried effectively.

To ensure the observability stack is working, I wrote a simple Python script that sends a request to the microservice and checks if the http_requests_total metric is captured by VictoriaMetrics.

Running the Test

The test is automated using a Kubernetes job managed by Tilt. The script performs the following steps:

  • Send Request to Microservice: It triggers an HTTP request to ensure the microservice is generating metrics.
  • Query VictoriaMetrics: It then queries VictoriaMetrics to verify the metric exists and is correctly stored.

Example Test Output

Here’s an example output from the test:

2024-08-28 21:27:20,866 - INFO - http_requests_total has value: 6
2024-08-28 21:27:20,867 - INFO - Success: http_requests_total is present with a value of 6.

This confirms that the observability pipeline is functioning as expected.

Observability in Action

Using Tilt to manage a local Kubernetes cluster allows you to quickly gain insights into the health of your microservice. It provides a convenient environment to define and monitor key metrics, enabling you to assess the performance and stability of your application in real time.

Conclusion

This exercise provided valuable hands-on experience in setting up and managing a complete observability stack using Prometheus, VictoriaMetrics, and Tilt. By implementing this setup, I demonstrated how Prometheus is crucial for real-time monitoring of microservices, while VictoriaMetrics offers a scalable solution for long-term storage and in-depth analysis of time-series data. Together, these tools create a robust observability framework that is essential for maintaining the health and performance of applications in a cloud-native environment.

If you’re interested in replicating this setup or exploring the code further, you can check out the repository at microservice-observability.