Monitoring WildFly with Prometheus and Grafana - Mastertheboss (2024)

This is the third tutorial about how to use Prometheus server to capture metrics from a Microprofile compatible server like WildFly or Quarkus. In the first two tutorials, we have discussed how to set up Prometheus to connect to WildFly (Monitoring WildFly with Prometheus) and Quarkus( Monitoring Quarkus with Prometheus) to capture Microprofile metrics. We will now learn how to use the Alert Manager to capture, group or export our alerts. Then, we will learn how to export our metrics in a Grafana dashboard.

Step 1: Set up the environment

We will be working with the following components:

  • Prometheus: collects metrics from various sources and stores them in a time-series database. It uses PromQL for data analysis and alerting.
  • Alertmanager: receives alerts from Prometheus and handles them appropriately, such as sending notifications. It can group, prioritize, and de-duplicate alerts.
  • Grafana: creates dashboards of metrics from Prometheus using drag-and-drop and various chart types. It also supports alerting and data integration.

To speed up the environment set up, we will use Docker compose to bootstrap all them with just a main configuration file.

Firstly, create the following docker-compose.yml file:

version: '3'services: prometheus: image: prom/prometheus:v2.42.0 user: 0:0 ports: - 9090:9090 volumes: - ./prometheus:/etc/prometheus - prometheus-data:/prometheus extra_hosts: host.docker.internal: host-gateway command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml grafana: image: grafana/grafana:8.2.2 user: 0:0 ports: - 3000:3000 restart: unless-stopped volumes: - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources - grafana-data:/var/lib/grafana alertmanager: image: prom/alertmanager:v0.23.0 user: 0:0 restart: unless-stopped ports: - "9093:9093" volumes: - "./alertmanager:/config" - alertmanager-data:/data command: --config.file=/config/alertmanager.yml --log.level=debugvolumes: prometheus-data: grafana-data: alertmanager-data:

As you can see, the above file starts the docker image of the three components. Since we will scrape metrics from a WildFly server running outside of Docker, we needed to add the host.docker.internal to Prometheus external hosts.

Next, let’s add the configuration for each component.

Adding Prometheus configuration

The following prometheus.yml file will scrape metrics from WildFly on port 9990:

global: scrape_interval: 10s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 10s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s).# Alertmanager configurationalerting: alertmanagers: - scheme: http static_configs: - targets: [ 'alertmanager:9093' ]# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.rule_files: - rules.ymlscrape_configs: # this is the configuration to poll metrics from WildFly 15 - job_name: 'metrics' scrape_interval: 10s static_configs: - targets: ['host.docker.internal:9990']

Place the above file in the prometheus folder.

Additionally, in the prometheus folder add also the rules.yml file which defines an alert for WildFly service down:

groups:- name: example rules: - alert: service_down expr: up{job="metrics"} == 0 labels: severity: major annotations: description: Service {{ $labels.instance }} is unavailable. value: DOWN ({{ $value }})

Adding the Alert Manager configuration

Next, within the alertmanager folder, add the following alertmanager.yml file configuration:

route: group_by: [ alertname ] receiver: 'mail' # default receiver repeat_interval: 24h routes: - receiver: 'teams' repeat_interval: 12h matchers: - severity="medium" - receiver: 'teams' repeat_interval: 4h matchers: - severity="high"receivers: - name: 'mail' email_configs: - smarthost: 'yourmailhost.com:465' auth_username: '[emailprotected]' auth_password: "your mail password" from: '[emailprotected]' to: '[emailprotected]' require_tls: false

This configuration file defines how Alertmanager will handle alerts based on their severity and grouping. The default receiver is “mail”, which sends alerts to an email address. If an alert has a medium severity, it will also be sent to teams using the “teams” receiver. If an alert has a high severity, it will be sent to teams more frequently using the “teams” receiver.

Adding Grafana configuration

Finally, we will add the Grafana configuration in the folder grafana/provisioning/datasource:

datasources:- name: Prometheus access: proxy type: prometheus url: http://prometheus:9090 isDefault: true

This configuration defines a data source named “Prometheus” for Grafana to connect to the Prometheus server running at “http://prometheus:9090”. The data source is configured as a proxy, which means that Grafana will connect to the Prometheus server directly and proxy the queries to it. This is the most common way to connect Grafana to Prometheus. The data source is also set as the default, which means that it will be used for all queries unless a specific data source is specified.

Step 2: Start the Docker Containers

Firstly, here is an overview of our Docker Compose project:

├── alertmanager│ └── alertmanager.yml├── docker-compose.yml├── grafana│ └── provisioning│ └── datasources│ └── prometheus_ds.yml├── prometheus│ ├── alert.yml│ ├── prometheus.yml│ └── rules.yml

Next, start the docker-compose file as follows:

docker-compose up

Verify that Prometheus and Grafana are up and running:

prometheus_1 | ts=2023-12-09T18:58:07.598Z caller=main.go:978 level=info msg="Server is ready to receive web requests."prometheus_1 | ts=2023-12-09T18:58:07.598Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."grafana_1 | t=2023-12-09T18:58:07+0000 lvl=info msg="Starting DB migrations" logger=migratorgrafana_1 | t=2023-12-09T18:58:07+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=346 duration=504.378µsgrafana_1 | t=2023-12-09T18:58:07+0000 lvl=info msg="Starting plugin search" logger=pluginsgrafana_1 | t=2023-12-09T18:58:07+0000 lvl=info msg="Registering plugin" logger=plugins id=inputgrafana_1 | t=2023-12-09T18:58:07+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http

Then, as first step, head to the Alert manager console: http://localhost:9093/#/alerts

Since WildFly is still not running, you should be able to see the following Alert:

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (1)

Step 3: Start WildFly and check connectivity

We are now ready to start WildFly. Let’s start all available interfaces of the application server as follows:

./standalone.sh -b 0.0.0.0 -bmanagement=0.0.0.0

Firstly, if you move to the Alert Manager console at http://localhost:9093/#/alerts you should no longer see the Alert for the “metrics” Prometheus job.

Then, let’s move to Prometheus Console at http://localhost:9090/targets and check that you are able to scrape metrics from WildFly:

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (2)

Step 4: Visualize your application Metrics with Grafana

The last step will be visualizing the Metrics collected by Prometheus with Grafana.

Log into Grafana console available at localhost:3000. Login with the default user (admin/admin).

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (3)

As soon as you are logged in, add a DataSource to it:

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (4)

Choose to add a Prometheus Datasource:

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (5)

Next, use the default Prometheus host/port (localhost:9090) for your Datasource:

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (6)

Next, we will add a new Dashboard.

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (7)

A Dashboard is a set of one or more panels organized and arranged into one or more rows. Within your Dashboard, bind it to your Prometheus Datasource and choose a Query expression. The Query expression can be any metric which has been published by your Enterprise service. For example, choose the “base_memory_usedHeap_bytes” which shows the amount of Heap you are using (in bytes):

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (8)

As you can see from the above picture, the amount of Memory of your Enterprise application will be visualized in a nice Grafana Dashboard, which will be updating based on the chosen timing factor.

Conclusion

We have discussed how to use the Alert Manager to group/forward the Alerts for our Enterprise Application collected by Prometheus. We have then learnt how to export our Metrics in the Grafana opensource tool using a Datasource and check them in a Dashboard

You can find the Docker Compose project here: https://github.com/fmarchioni/mastertheboss/tree/master/monitoring/prometheus-grafana

Given your interest in utilizing Prometheus to capture metrics from a Microprofile-compatible server such as WildFly or Quarkus, it's evident you're seeking a robust monitoring and visualization setup for your applications. I've been extensively involved in system monitoring and observability, including using Prometheus for metric collection, Alertmanager for handling alerts, and Grafana for visualizing data. Here's a breakdown of the concepts used in the provided article:

  1. Prometheus: It's a monitoring and alerting toolkit that collects metrics from configured targets, stores them, and allows querying and analysis using PromQL (Prometheus Query Language). Key components include:

    • Configuration: In the article, a prometheus.yml file is used to define scraping configurations (scrape_configs) for targets such as WildFly. It specifies job names, scraping intervals, and target endpoints to fetch metrics.
    • Alerting: Rules for generating alerts based on certain conditions are defined in rules.yml. These rules are evaluated at specific intervals to trigger alerts when predefined criteria are met.
  2. Alertmanager: This component receives alerts from Prometheus and manages their handling, grouping, and forwarding to various receivers such as email or external services. Configuration is done through alertmanager.yml, where routing and receiver settings are defined based on alert severity and conditions.

  3. Grafana: A visualization and analytics platform that creates interactive dashboards for metrics collected by Prometheus. It connects to Prometheus as a data source, allowing users to build custom dashboards using various visualization options and supports alerting based on defined thresholds.

  4. Docker Compose: A tool to define and run multi-container Docker applications. The provided docker-compose.yml file orchestrates the setup of Prometheus, Alertmanager, and Grafana services, configuring ports, volumes, and dependencies between these services.

  5. WildFly: A Java-based application server. Configuration for Prometheus scraping is done for WildFly in the prometheus.yml file, specifying where to scrape metrics (such as host.docker.internal:9990).

The provided tutorial walks through setting up these components, configuring Prometheus to scrape metrics from WildFly, defining rules for alerting, configuring Alertmanager for handling alerts based on severity, and setting up Grafana to visualize the collected metrics using a specified data source.

For a practical demonstration and to follow along, the tutorial provides a Docker Compose project on GitHub, facilitating the setup and configuration of the entire monitoring and visualization stack using Docker containers.

Feel free to ask if you need further details or assistance with any specific aspect of this setup!

Monitoring WildFly with Prometheus and Grafana - Mastertheboss (2024)
Top Articles
Latest Posts
Article information

Author: Trent Wehner

Last Updated:

Views: 6417

Rating: 4.6 / 5 (76 voted)

Reviews: 83% of readers found this page helpful

Author information

Name: Trent Wehner

Birthday: 1993-03-14

Address: 872 Kevin Squares, New Codyville, AK 01785-0416

Phone: +18698800304764

Job: Senior Farming Developer

Hobby: Paintball, Calligraphy, Hunting, Flying disc, Lapidary, Rafting, Inline skating

Introduction: My name is Trent Wehner, I am a talented, brainy, zealous, light, funny, gleaming, attractive person who loves writing and wants to share my knowledge and understanding with you.