Create Real-Time Monitoring with Elasticsearch and Grafana

In this article, we will explore how to create a real-time monitoring and visualization system using Elasticsearch, Grafana, Filebeat, and Metricbeat. We will coordinate all components using Docker, enabling an efficient and scalable system.

Introduction

Monitoring systems are essential for understanding your applications’ performance, health, and activity. By combining tools such as Elasticsearch, Grafana, Filebeat, and Metricbeat, we can create a robust, easily deployable monitoring stack that scales across multiple services.

Prerequisites

Before starting, you’ll need:

Docker installed on your machine.
Basic knowledge of Docker and Docker Compose.
Familiarity with Elasticsearch, Grafana, and Beats.

Project Architecture

Our project will consist of:

Elasticsearch: For storing the logs, metrics, and performance data.
Grafana: For visualizing the data from Elasticsearch.
Filebeat: For collecting and shipping logs.
Metricbeat: For collecting metrics from the system.
Docker Compose: To orchestrate all services together.

Step 1: Setting Up the Environment

First, let’s set up the environment by creating a docker-compose.yml file that orchestrates Elasticsearch, Grafana, Filebeat, and Metricbeat.

version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.10
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - network.host=0.0.0.0
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - esdata:/usr/share/elasticsearch/data

  grafana:
    image: grafana/grafana:9.4.7
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"
    depends_on:
      - elasticsearch

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.17.10
    container_name: filebeat
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /var/log:/var/log
    depends_on:
      - elasticsearch

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:7.17.10
    container_name: metricbeat
    volumes:
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
    depends_on:
      - elasticsearch

volumes:
  esdata:
    driver: local

Step 2: Configure Filebeat

Filebeat is responsible for shipping logs to Elasticsearch. You can customize filebeat.yml for monitoring logs from different containers or services.

Create a filebeat.yml file in the root directory:

filebeat.inputs:
  - type: container
    paths:
      - /var/lib/docker/containers/*/*.log

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

setup.kibana:
  host: "http://kibana:5601"

This configuration collects all logs from /var/log/*.log and sends them to Elasticsearch.

Step 3: Configure Metricbeat

Metricbeat will be responsible for collecting system metrics and sending them to Elasticsearch. Create a metricbeat.yml file:

metricbeat.modules:
  - module: system
    period: 10s
    metricsets:
      - cpu
      - memory
      - network
      - diskio
      - filesystem
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]

setup.kibana:
  host: "http://kibana:5601"

This configuration collects system metrics such as CPU, memory, and network every 10 seconds and sends the data to Elasticsearch.

Step 4: Set Up Grafana

Grafana will be used for visualizing the data stored in Elasticsearch. We have already mapped Grafana to port 3005. Once the containers are running, you can access Grafana at http://localhost:3000.

Go to Data Sources → Add Data Source.
Select Elasticsearch and provide the URL http://elasticsearch:9200.

Grafana will automatically fetch the data from Elasticsearch for visualization.

Step 5: Running the Stack

Run the entire stack using Docker Compose:

docker-compose up -d

This will start Elasticsearch, Grafana, Filebeat, and Metricbeat in detached mode. You can verify that the containers are running by checking their status:

docker ps

Step 6: Visualizing Data in Grafana

Now, you can visualize logs and metrics in Grafana. Here’s how you can set up dashboards for different metrics:

Navigate to Dashboards in Grafana.
Choose Import and either create a new dashboard or use pre-built dashboards for system metrics and logs.

You should be able to visualize real-time system metrics and logs from your application or server. For example:

Filebeat Dashboard: Shows logs from your applications.
Metricbeat Dashboard: Displays CPU usage, memory, and network metrics.

Conclusion

Using Docker, we have established a robust real-time monitoring and logging system with Elasticsearch, Grafana, Filebeat, and Metricbeat. This setup enables you to monitor logs and metrics in real-time, providing you with the necessary tools to maintain the health and performance of your applications.

Whether you’re monitoring a single application or a distributed system, this setup provides a scalable, containerized solution. With Elasticsearch’s search capabilities and Grafana’s visualization power, you’ll have full visibility into your systems’ health.

The project code is available on my GitHub.

#Grafana #docker #kubernetes #devops #elasticsearch #observability #search #apm #APM #integration #dashbaord

Feel Free to Reach Out at Linkedin: