Looking for the Perfect Dashboard: InfluxDB, Telegraf and Grafana – Part XIX (Monitoring Veeam with Enterprise Manager) Shell Script

Greetings friends, I bring you a new entry about Grafana and Veeam, which I’m sure you’ll like and put in your labs. Back in 2017 I told you how they monitor Veeam using the PowerShell CMDlets and also how to do it using PowerShell and Enterprise Manager RESTful API. These entries have had tens of thousands of hits, but they were more of a proof of concept as they were missing a lot of detail.

Today, I am pleased to bring you a complete and finished Dashboard to monitor Veeam Backup & Replication, without limit of servers, jobs, VMs or Repos, thanks to Veeam Enterprise Manager and its RESTful API (only in Enterprise Plus)

Veeam Enterprise Manager Dashboard

When you finish the entry you will have something similar to that Dashboard that will allow you to visualize:

Enterprise Manager Section – Summary

  • Total number of Veeam Backup & Replication Servers managed by Enterprise Manager
  • Total number of Veeam Backup Repositories that have Veeam Backup & Replication Servers managed by Enterprise Manager
  • Total number of Veeam Backup Proxies that have Veeam Backup & Replication Servers managed by Enterprise Manager
  • Total number of Veeam Jobs that have Veeam Backup & Replication Servers managed by Enterprise Manager
  • Total number of protected VMs that have Veeam Backup & Replication Servers managed by Enterprise Manager
  • Disk space consumption per Veeam Backup Repository

VBR section – Veeam Backup Performance

  • CPU consumption of the Veeam Backup Server
  • Veeam Backup Server RAM Consumption
  • Veeam Backup Server Disk Space Consumption
  • Veeam Backup Server Network Consumption

VBR Section – Backup Jobs

  • Name of each Backup Job, with the last known status in the period selected in the Dashboard

VBR Section – Replica Jobs

  • Name of each Replica Job, with the last known status in the period selected in the Dashboard

VBR Section – Protected VMs

  • Name of each VM protected by Veeam that includes a backup job, with the last known status in the period selected in the Dashboard

VBR Section – Replicated VMs

  • Name of each VM replicated by Veeam that includes a replica job, with the last known status in the period selected in the Dashboard

VBR Section – Job Historical Performance and Duration

  • Table with status information of all backup jobs, ordered by date, includes name and last known status in the period selected in the Dashboard.
  • Table with information of the duration of each backup and replication job, ordered by date, includes duration and name of the job according to the period selected in the Dashboard.

Topology with all logical components

This entry is a little different from the previous ones since in this case, we will use a combination of telegraf for Microsoft Windows to collect metrics from the Veeam Backup servers and a shell script to collect the Enterprise Manager metrics using RESTful API. The design would be something similar to this:As we can see, the telegraf agent will collect all counters with metrics, and send them to InfluxDB, in turn, the shell script will download metrics from Enterprise Manager using RESTful API, which will send to InfluxDB as well, from where we can comfortably view them with Grafana.

How to Download and install as a telegraf service for Microsoft Windows

Although I already told you the steps a few years ago, let’s see the updated steps, the first thing will be to download the latest version of Telegraf for Windows from GitHub:Create a folder in C:/Program Files/Telegraf and extract the files inside:From a PowerShell or CMD console, as an administrator, we will launch the following:

And with this we would have telegraph installed, to start it, we could launch it trying the config with the following command:

Let’s start the service now:

Now that we have everything ready, let’s go to the next step to see what we need to add to the configuration file.

Configuring telegraf.conf for Veeam Enterprise Manager

The telegraf configuration file is quite dense, the first thing we will have to edit as always is the output, where we will have to configure the path of the InfluxDB where we are sending the data, in my case:

Besides this basic, this is the recommended configuration to be able to ingest well the hostname of the VBR:

With this telegraf.conf file ready, please edit if you have the logs in another location or something, and we can start the service:

We can do a more of the telegraph log to see if there are any errors. We have everything, let’s go to the next step.

Download and configure the veeam-enterprisemanager.sh script

We have almost everything ready, we have one last step, the script that will make all this work, we will download the latest version from the Github repository:

This shell script, can be downloaded and run from the telegraf server, or influxDB, or any other Linux. We will have to edit the configuration parameters:

Once the changes are done, make the script executable with chmod:

The output of the command should be something like the next, without errors:

If so, please now add this script to your crontab, like for example every 30 minutes:

We are ready to move to the next step.

Checking that we are ingesting information with Chronograf

The normal thing at this point, if we have done all the steps well, is that we are already sending information collected by the script to InfluxDB, if we perform a search using the wonderful Chronograf, we can check that we have information:All the variables in this new Veeam script for Enterprise Manager are stored in veeam_em_* so it’s really easy to find them.

Grafana Dashboards

I created a Dashboard from scratch by selecting the best requests to the database, finishing off colors, thinking about graphics and how to display them, and everything is automated so that it fits our environment without any problem and without having to edit anything manually. The Dashboard can be found here, once imported, you can use the top drop-down menus to select between organizations, backups, repositories and proxies, VMs, etc.:

Import Grafana Dashboards easily

So that you don’t have to waste hours configuring a new Dashboard, and ingesting and debugging you want, I have already created four wonderful Dashboards with everything you need to monitor our environment in a very simple way, you will be like the image I showed you above. Select the name you want and enter the ID: 10349, which is the unique ID of the Dashboard, or the URL:

With the menus above you can display and adjust the Dashboard to your VBR, Repositories, Jobs, VMs, etc.:Please leave your feedback in the comments, or on GitHub.
If you want to see them working without installing anything, here is the link to my environment:

That’s all folks, if you want to follow the full Blog series about Grafana, InfluxDB, Telegraf, please click on the next links:

Author: jorgeuk

Father, writing in https://www.jorgedelacruz.es and https://jorgedelacruz.uk Blogger, Systems Engineer @veeam - vExpert 2014/2020 & NTC 2018/19

26 Thoughts

  1. When I run the script, I get an errors:

    [[email protected] home]# sh veeam_enterprisemanager.sh
    veeam_enterprisemanager.sh: line 37: jq: command not found
    veeam_enterprisemanager.sh: line 47: jq: command not found
    veeam_enterprisemanager.sh: line 48: jq: command not found
    veeam_enterprisemanager.sh: line 49: jq: command not found
    veeam_enterprisemanager.sh: line 50: jq: command not found
    veeam_enterprisemanager.sh: line 51: jq: command not found
    veeam_enterprisemanager.sh: line 52: jq: command not found
    veeam_enterprisemanager.sh: line 53: jq: command not found
    veeam_enterprisemanager.sh: line 54: jq: command not found
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    Request-Id: a6c03a96-3da7-11ea-9cfe-005056bac63f
    X-Influxdb-Build: OSS
    X-Influxdb-Error: unable to parse ‘veeam_em_overview,host=vbem_ip_address veeamBackupServers=,veeamProxyServers=,veeamRepositoryServers=,veeamRunningJobs=,veeamScheduledJobs=,veeamSuccessfulVmLastestStates=,veeamWarningVmLastestStates=,veeamFailedVmLastestStates=’: missing field value
    X-Influxdb-Version: 1.7.9
    X-Request-Id: a6c03a96-3da7-11ea-9cfe-005056bac63f
    Date: Thu, 23 Jan 2020 06:14:45 GMT
    Content-Length: 284

  2. I have installed jq-1.5-1.0.1.el6.x86_64 but:
    [[email protected] home]# sh veeam_enterprisemanager.sh
    parse error: Invalid numeric literal at line 1, column 10
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    Request-Id: 54c11662-3e5f-11ea-8066-005056bac63f
    X-Influxdb-Build: OSS
    X-Influxdb-Error: unable to parse ‘veeam_em_overview,host=vbem_ip_address veeamBackupServers=null,veeamProxyServers=null,veeamRepositoryServers=null,veeamRunningJobs=null,veeamScheduledJobs=null,veeamSuccessfulVmLastestStates=null,veeamWarningVmLastestStates=null,veeamFailedVmLastestStates=null’: invalid number
    X-Influxdb-Version: 1.7.9
    X-Request-Id: 54c11662-3e5f-11ea-8066-005056bac63f
    Date: Fri, 24 Jan 2020 04:09:35 GMT
    Content-Length: 311

  3. Hello, do not run it with sh, but with ./veeam_enterprisemanager.sh

    Have you made it executable with chmod +x veeam_enterprisemanager.sh ?

  4. Hi, Jorgeuk.
    It makes no difference how to run a script:
    [[email protected] home]# ./veeam_enterprisemanager.sh
    parse error: Invalid numeric literal at line 1, column 10
    HTTP/1.1 400 Bad Request
    Content-Type: application/json
    Request-Id: 7afaeb0b-3e91-11ea-8944-005056bac63f
    X-Influxdb-Build: OSS
    X-Influxdb-Error: unable to parse ‘veeam_em_overview,host=vbem_ip_address veeamBackupServers=null,veeamProxyServers=null,veeamRepositoryServers=null,veeamRunningJobs=null,veeamScheduledJobs=null,veeamSuccessfulVmLastestStates=null,veeamWarningVmLastestStates=null,veeamFailedVmLastestStates=null’: invalid number
    X-Influxdb-Version: 1.7.9
    X-Request-Id: 7afaeb0b-3e91-11ea-8944-005056bac63f
    Date: Fri, 24 Jan 2020 10:08:34 GMT

    Sure. Rights assigned:
    -rwxr-xr-x 1 root root 26554 Jan 24 07:09 veeam_enterprisemanager.sh

  5. Okay, so a RHEL 6 at the end. Have you filled all your InfluxDB and EM settings, and you have all the components listening and all?

  6. Yes, I edited the fields in the script veeam_enterprisemanager.sh and in the config file telegraf.conf

  7. Hi, jorgeuk.
    From :9399/web/#/api/reports/summary/job_statistics I get empty field:
    <JobStatisticsReportFrame

    output from script:
    [[email protected]]# echo $veeamEMOJobUrl
    {“RunningJobs”:0,”ScheduledJobs”:24,”ScheduledBackupJobs”:24,”ScheduledReplicaJobs”:0,”TotalJobRuns”:21,”SuccessfulJobRuns”:21,”WarningsJobRuns”:0,”FailedJobRuns”:0,”MaxJobDuration”:2760,”MaxBackupJobDuration”:2760,”MaxReplicaJobDuration”:0,”MaxDurationBackupJobName”:”VM-NAME”,”MaxDurationReplicaJobName”:””,”BackupJobStatusReportLink”:”Workspace/ViewReport.aspx?definition=7962844d-db6c-4d29-8b6e-4e0f7db0785f&ShowParams=1″}

    and as result:
    X-Influxdb-Error: unable to parse ‘veeam_em_overview_jobs,host=vbem_ip_address,veeamMaxDurationBackupJobName=VM-NAME,veeamMaxDurationReplicaJobName= veeamRunningJobs=0,veeamScheduledJobs=24,veeamScheduledBackupJobs=24,veeamScheduledReplicaJobs=0,veeamTotalJobRuns=21,veeamSuccessfulJobRuns=21,veeamWarningsJobRuns=0,veeamFailedJobRuns=0,veeamMaxJobDuration=2760,veeamMaxBackupJobDuration=2760,veeamMaxReplicaJobDuration=0’: missing tag value

  8. No, there are no such tasks. I understand that this is the problem, but how to exclude from the request?

  9. Can you please replace this on the Script:
    veeamMaxDurationBackupJobName=$(echo "$veeamEMOJobUrl" | jq --raw-output ".MaxDurationBackupJobName" | awk '{gsub(/ /,"\\ ");print}')
    veeamMaxDurationReplicaJobName=$(echo "$veeamEMOJobUrl" | jq --raw-output ".MaxDurationReplicaJobName" | awk '{gsub(/ /,"\\ ");print}')

    For this:
    veeamMaxDurationBackupJobName=$(echo "$veeamEMOJobUrl" | jq --raw-output ".MaxDurationBackupJobName" | awk '{gsub(/ /,"\\ ");print}')
    [[ ! -z "$veeamMaxDurationBackupJobName" ]] || veeamMaxDurationBackupJobName="None"
    veeamMaxDurationReplicaJobName=$(echo "$veeamEMOJobUrl" | jq --raw-output ".MaxDurationReplicaJobName" | awk '{gsub(/ /,"\\ ");print}')
    [[ ! -z "$veeamMaxDurationReplicaJobName" ]] || veeamMaxDurationReplicaJobName="None"

  10. jorgeuk, thanks for fixes. The last error remained:
    jq: error (at : 1): Cannot iterate over null (null)

  11. Hi, jorgeuk. This is the full error:

    HTTP/1.1 204 No Content
    Content-Type: application/json
    Request-Id: 7bbf1501-43d6-11ea-a5aa-000c29ed7e64
    X-Influxdb-Build: OSS
    X-Influxdb-Version: 1.7.9
    X-Request-Id: 7bbf1501-43d6-11ea-a5aa-000c29ed7e64
    Date: Fri, 31 Jan 2020 03:05:06 GMT

    jq: error (at :1): Cannot iterate over null (null)
    jq: error (at :1): Cannot iterate over null (null)
    jq: error (at :1): Cannot iterate over null (null)
    jq: error (at :1): Cannot iterate over null (null)
    [[email protected]]#

  12. Hi, jorgeuk.
    My mistake. veeam_enterprisemanager.sh works only with edition Veeam Enterprise Plus. It worked.

  13. Hey! I have the same issue with :

    HTTP/1.1 204 No Content
    Content-Type: application/json
    Request-Id: 5d92a6e8-4cbc-11ea-97da-0050528ea220
    X-Influxdb-Build: OSS
    X-Influxdb-Version: 1.7.10
    X-Request-Id: 5d92a6e8-4cbc-11ea-97da-0050528ea220
    Date: Tue, 11 Feb 2020 10:50:49 GMT

    jq: error (at :1): Cannot iterate over null (null)
    jq: error (at :1): Cannot iterate over null (null)

    Only the last step it will be and yes we have Replication Jobs

  14. Weird, do you have all the configuration properly configured and all, with user and pass? Backups and Replicas I am assuming? And VBR added and all?

  15. I have just tried on my lab, and all worked without an issue. I really need to add some debug into it 🙂

    Let me know

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.