You can check out my videos about LoadRunner integration with Grafana in my YouTube channel. Recently I have started working extensively on JMeter, primarily in HTTP, MQTT, and JDBC Sampler and will be working on NeoLoad pretty soon. I got a challenge in my projects where developers would like to monitor the critical metrics when the performance test is on. I started sharing the Dynatrace dashboard link, but for non-technical members Dynatrace information is overwhelming. Our stack has Prometheus – time series database with Grafana in the front end. Then I started exploring about the integration and successfully implemented JMeter > Prometheus > Grafana integration and shared the link with my team. It is well-received. In this blog post, we will see about JMeter, Prometheus, and Grafana Integration.
Table of Contents
Little Background
In JMeter Plugins repository, there are backend listeners available for Kafka, ElasticSearch, and Azure. By default, JMeter supports Graphite and InfluxDB. But there is no backend listener available for Prometheus.
Fortunately, I found this excellent open source project to export the JMeter metrics for Prometheus. Thanks to Jeff Ohrstrom.
Now we shall see high-level diagram about the integration.
As you observed in the above diagram, JMeter + Prometheus plugin exports the JMeter metrics to Prometheus database. If you want to see the visualization, you need to configure the data source in Grafana.
The main intention of the plugin is to export the results to database (Prometheus).
What is Prometheus?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
What is Grafana?
Grafana is the open source analytics and monitoring solution for every database.
What is JMeter-Prometheus Plugin?
It is an open source project which helps to export the JMeter metrics to Prometheus.
Prerequisites
- Apache JMeter
- JMeter-Prometheus Plugin
- Prometheus
- Grafana
Recommended versions for above components is latest and stable release. I am in Windows 10 Home Edition.
Installing JMeter-Prometheus Plugin
There are multiple ways to place the jar file for JMeter. Very simple method is to download the jar file from the latest releases in GitHub and place it in the <JMETER_HOME>\lib\ext.
Download JMeter-Prometheus Plugin from here
Validate the installation by creating a simple script in JMeter, and then navigate to Right Click on Thread Group > Add > Listener > Prometheus Listener
Or Right Click on Thread Group > Add > Config Element > Prometheus Metrics
Install Prometheus
It is very straight forward to install Prometheus. Download Prometheus for your OS and extract the contents and keep it ready. In few moments, we will fire up Prometheus.
Install Grafana
Like Prometheus, it is easy to install Grafana. Head to this download section and install it. By default, it uses the port 3000.
Configuring JMeter-Prometheus Plugin in JMeter
After designing your test, you need to configure JMeter-Prometheus plugin in your test plan. Detailed instruction has already available in the GitHub page. Please read it twice for better understanding. I am not going to reiterate again in here :)
To demonstrate, I have configured the listener as shown below.
Configuring Prometheus
Now, it is time to configure Prometheus so that JMeter-Prometheus plugin will export the results to Prometheus database.
Go to Prometheus installation folder and open prometheus.yml in your favorite editor.
Add the below job details to your yml file. Below configuration helps to scrape the metrics from the port 9270.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'jmeter'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9270']Launch Prometheus
Once the configuration is completed, launch command prompt and change the directory to the prometheus installation folder and fire up the below command.
prometheus.exe --config.file=prometheus.ymlNow launch, Prometheus by launching http://localhost:9090/graph as shown below.
Launch JMeter Test
We are good with Prometheus. Now it is time to validate the JMeter test plan. Configure your thread group with couple threads and loop infinite and hit run.
Launch, http://localhost:9270/metrics as shown below. If you are seeing the below metrics, everything is working fine. If not, please let me know the issue you are facing.
Viewing JMeter metrics in Prometheus
As highlighted in the above metrics snapshot, we will see how to view the metrics in Prometheus. Just copy jvm_classes_loaded and go to Prometheus and paste it in the query text box and hit enter. You can see the value as shown below.
Click on Graph tab to view the visualization. You can customize how you want to visualize. Below is the graph which displays 95 percentile response time of each transaction.
Where is Grafana :) ?
If you do not want to integrate Grafana, you can stop here. But Grafana is more powerful than the Prometheus visualization features.
Now, let us configure Grafana to fetch the metrics from Prometheus and create beautiful visualization.
Launch Grafana by navigating to http://localhost:3000, you may need to change the password if you are logging in for first time.
Hover your mouse over the Settings icon and click on Data Sources as shown below.
Click on Add data source as shown below.
Fill up the details as shown below and hit Save & Test
Let us validate the Prometheus data source in Grafana
Hover your mouse over Explore icon and click on it.
Enter jmeter_threads{} and hit enter the query text box. If the configuration is correct, you will see the graph. Note that your JMeter test is still running. If stopped, you need to change the time picker value at top right corner.
Creating Grafana Dashboard
Hover your mouse over plus icon and click on Dashboard.
Click on Add Query button.
Change the Query dropdown to Prometheus as shown below.
Enter the query jmeter_summary{quantile="0.95"} and hit enter. You can customize the graph as desired.
Here is my simple dashboard.
Sample Grafana Dashboard
https://github.com/johrstrom/jmeter-prometheus-plugin/tree/master/docs/examples
Adding an excerpt from one of the author.
Another point of the plugin is that it does not need to send metrics to the JMeter controller to aggregate results as this is done by Prometheus itself.
Conclusion
Now you have successfully integrated JMeter with Prometheus and Grafana. If you are satisfied with Prometheus visualization, no need to configure Grafana. But Grafana has more powerful visualization features than Prometheus. You need to setup JMeter-Prometheus plugin where ever your are running JMeter and streamline the data flow into centralized Prometheus database. This will help you to monitor the runtime metrics.
You need to carefully craft the query in Grafana, otherwise you will end up in viewing false metrics.
Please let me know in the comments, if you face any issues. Happy to help you out :)
Hi!
thank you for your great article, I’m one of the contributor of the Prometheus plugin for JMeter. Your guide does a terrific job of simplifying what is needed to run the whole package.
In the latest releases, we shared the Grafana dashboard we usually use when running performance tests with this setup. You can try to use this dashboard as a starting point it already includes most-used metrics and also some monitoring to monitor JMeter as well.
Another point of the plugin is that it does not need to send metrics to the JMeter controller to aggregate results as this is done by Prometheus itself. This is a long-lasting issue of Jmeter when running in a distributed fashion to generate a high workload.
Thank you again for your guide!
Many thanks Giovanni! Sure, let me add your points to the article :)
Hi Naveen,
Thanks for the great article. I have a query, like Influx DB listener, does the Prometheus plugin sends data to HTTP end point ?
Also, when i see Jmeter Aggregate report, as and when a data sample is recorded, does this sample saves to Prometheus( meaning prometheus Jmeter job scrape_interval set to 1sec and for every 1 sec, lets say in jmeter we have 50 data samples observed in aggregate report, does all these sent to Prometheus ?). Because using InfluxDB listner, i tried and seen a difference that some samples are missing for high throughput test.
Please let me know if any of my query above is not clear, will explain again.
Regards,
Sai
Nice article.
Can you create/add a video on how to use the plugin: JMeter-Prometheus Plugin ?
This is helpful for many people.
All the details about the plugin is mentioned here https://github.com/johrstrom/jmeter-prometheus-plugin, I will put up a video soon :)
Hi Naveen,
I followed the step. but unfortunately, I am not able to open the port 9270 which means, the data from jemter is not forwarded to Promethus.
PS : Prometheus successfull launched with 9090 port.
i am also not seeing metrics at port 9270 . I did not understand how jmeter communicate with prometheus . In the artical ..please post complete version as well
This is the complete tutorial. I have put all the steps which are required to get started with Prometheus and JMeter. If you are not able to see 9270, make sure that you have started the JMeter test plan. Only after starting the test plan, you will get the port assigned.
When i use http://localhost:9090/ i get page not found error
When i use http://localhost:9270/ i see the data but i dont see the menu for graph
When i tried with grafana integration its giving me the bad gateway error
Hi Naveen,
Thanks for the article. Its really good. I am running the promethues as docker image instead of .exe file. Jmeter is not sending the metrics to promethues server if I run as docker image. Please let me know the reasons. Both are running in the same machine.
Here is Yaml file:
# A scrape configuration containing exactly one endpoint to scrape:
# Here it’s Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries scraped from this config.
– job_name: ‘prometheus’
# metrics_path defaults to ‘/metrics’
# scheme defaults to ‘http’.
static_configs:
– targets: [‘localhost:9090’]
– job_name: ‘jmeter’
# metrics_path defaults to ‘/metrics’
# scheme defaults to ‘http’.
static_configs:
– targets: [‘192.168.1.74:9270’]
— 192.168.1.74 is my inet address.
docker run -d -p 9090:9090 -v ~/Documents/JmeterProm/prometheus1.yml:/etc/prometheus/prometheus.yml prom/prometheus
— Tested the https://github.com/johrstrom/jmeter-prometheus-plugin/blob/master/docs/examples/simple_prometheus_example.jmx with Jmeter.
If I am trying to access the targets in promethues using this URL http://192.168.1.74:9090/targets is showing Jmeter target as down with below error:
Get http://192.168.1.74:9270/metrics: dial tcp 192.168.1.74:9270: connect: connection refused
If I am trying to access http://192.168.1.74:9270/ is giving “Site can’t be reached”
I am running the promethues as docker image instead of .exe file. Jmeter is not sending the metrics to promethues server if I run as docker image. Please let me know the reasons. Both are running in the same machine.
Here is Yaml file:
scrape_configs:
config.
job_name: ‘prometheus’
static_configs:
targets: [‘localhost:9090’]
job_name: ‘jmeter’
static_configs:
targets: [‘192.168.1.74:9270’]
— 192.168.1.74 is my inet address.
docker run -d -p 9090:9090 -v ~/Documents/JmeterProm/prometheus1.yml:/etc/prometheus/prometheus.yml prom/prometheus
— Tested the https://github.com/johrstrom/jmeter-prometheus-plugin/blob/master/docs/examples/simple_prometheus_example.jmx with Jmeter.
If I am trying to access the targets in promethues using this URL http://192.168.1.74:9090/targets is showing Jmeter target as down with below error:
Get http://192.168.1.74:9270/metrics: dial tcp 192.168.1.74:9270: connect: connection refused
If I am trying to access http://192.168.1.74:9270/ is giving “Site can’t be reached”
Sorry, I do not have a Docker environment to reproduce the issue. Can you please raise the issue in GitHub?
hello,
i had the same problem
did you find solution?
The problem is due to loopback IP. In this project Jeff used loopback IP as Prometheus server IP and remote server IP. Change that IP to the IP of your system(better use static IP) the create the jar file and use it in your JMeter. It will work and Prometheus can scrape the metrics available in that address.
Jeff used loopback IP for Prometheus server and remote server. This IP is not accessible by the Prometheus running inside Docker. To solve this issue download the project and replace the loopback IP with your system IP(better use static IP) and create jar file from the edited project then use that jar in your JMeter app.
Hi Naveen,
Thanks for the article. It’s really good. I am running the Prometheus in different Linux server and JMeter executions in different Linux server how I need to integrate it.
Hi Naveen Sir,
Thanks for the Information,
I have Set the Following Config in .yml File
– job_name: ‘JMeter’
# metrics_path defaults to ‘/metrics’
# scheme defaults to ‘http’.
static_configs:
– targets: [‘localhost:9270’]
I am unable, to view Metrics on “http://localhost:9270/”, even same issue comes when i replace localhost with my IPV4 Address. Can you help me out with this??
Please try 127.0.0.1:9270
Thanks for the post.
Can you mention the difference between using Jmeter-Prometheus-Grafana and Jmeter-InfluxDB-Grafana. Also, one doubt I have the Prometheus and grafana are installed in a separte instance and I will be executing the load test from a different instance So how is it possible and where I need to provide the IP for the Jmeter LG
Hi, I see you provided the details on viewing JMeter metrics in Prometheus-Grafana. But in my case, there are details of CPU, Memory and network related stats already exist in Prometheus-Grafana. What I do right now is, login to the grafana and view those resource usage graphs based on time or some other identifications. Instead of manually doing this, I am looking for some kind of way to retrieve with JMeter calls. Hope you understand my question. Please let me know if you have any ideas to achieve this.
thanks
Thanks for the Post Naveen,
It worked fine for me. Wanted to know how do we integrate multiple Jmeter Servers with Prometheus?
Thanks in Advance
HI Naveen,
Can you please provide insights if we are running Jmeter using distributed load env. Prometheus is able to capture only the Master machine stats, how to retrieve slave machine stats.
Reply
Will it works in distributed system?
I am facing issue will trying it.
Thanks for this. I am hosting the Prometheus server on AKS and will be running the performance tests on demand in a pod. How can I scrape the logs of Jmeter tests from the pod?