Grafana on Azure – Azure MySQL Storage

In this multi-part series you’ll learn how to host Grafana safely, and cheaply on Azure, and how to get some decent visbility from Azure Monitor/App Insights through it.

In the last post, you’ve setup a basic Grafana VM on Azure that is using a database on the VM’s disk as it’s datastore.

In this post, we’ll move that database to Azure PaaS.

What is Azure Database for MySQL?

This is Azure’s managed version of MySQL. It’s cheap (~£25/month) and supports everything you’d expect an Azure PaaS platform to do. There’s no patch management, backups, etc. just an endpoint for you to hit. What’s not to love?

This makes it great for our database for Grafana as the point of a monitoring system is be always up, and you don’t want to be monitoring your monitoring system with your monitoring system. Having a single VM, with the database, the frontend, etc. as a single point of failure just isn’t great. Having then to add backups of all your data regularly, and suddenly the quick and easy Grafana instance is now a management overhead in itself.

Step 1 – Azure MySQL Setup

Grafana stores a very small amount of data, and can be stored in a shared server if you already have one. We’ll be setting up a server from scratch for this, but most of the steps will work on an existing server if you have one.

You can get away with the smallest possible instance, which a B1 at ~£20/month, with the minimum base storage (5GB). This will give you the ability to have backups, SSL, and DB maintainance. I’m going to guess that this is likely nothing in comparison to the other things you’re running, and also nothing in terms of the ~1day/month it would take you to verify all backups, etc. if you ran your own.

You’ll need to create a database and a user for Grafana. Later in this post, we’ll look at locking down the Azure MySQL instance, for now, you’ll need to manually add your IP.

Click the link “Add current client IP Address”, and then click save. This will add your public IP Address to the SQL Server.

Now connect to the MySQL instance from the command line (if you don’t want to install mysql, running the docker image is a nice alternative).

mysql -h {servername}.mysql.database.azure.com -u {adminuser}@{servername} -p

Your user will need full privileges to manage the database as grafana has builtin database migrations.

CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
CREATE DATABASE grafana_data;
GRANT ALL ON grafana_data.* TO '<username>'@'%';

That’s it for MySQL. You’ve created the database, and it’s setup so you can access it. Next we’ll make grafana use it.

Step 2 – Configuring Grafana’s DB

The first thing we’ll need to do is allow Grafana to access the database. For now, we’ll enable the “Allow access to Azure Services”, this will enable access from Azure IPs which will work for now, and we can secure this later.

Next we’ll need to tell grafana where to find our database, and how to log in. SSH to your VM and open up the grafana config file

sudo nano /etc/grafana/grafana.ini

In there, find the section labelled [Database] and set the following values, leaving all the others at their defaults.

Note: the ini file uses ; to comment out the settings, remove it to set them.

type = mysqlhost = <servername>.mysql.database.azure.com:3306name = <database-name>user = <username>@<server-name>

ssl_mode = skip-verify

ca_cert_path = /etc/ssl/certs/ca-certificates.crt

Now restart grafana, and you’ll be need to login again with the default credentials.

sudo systemctl restart grafana-server

If you end up with your url timing out, check the logs.

sudo tail -f /var/log/grafana/grafana.log

A common error is accessing the MySQL instance. This will manifest as the log tail above stalling at “Starting DB Migrations”. Likely causes are:

  • Username hasn’t include @<server-name>
  • ssl_mode and ca_cert_path aren’t set
  • MySQL firewall rules haven’t been setup.

If all has gone well, you’ll be presented with the Grafana logon screen, and be asked to input the default admin password and reset it.

Step 3 – Securing MySQL connection

Right now the MySQL instance has been setup so that your public IP and literally anything that runs on Azure can access it. This is not a desireable situation, so we’ll need to secure that further.

There are a few things you can do to secure the MySQL instance.

Option 1: IP restrictions

You can restrict your MySQL instance to just the IP of your VM. This is the simplest approach, but isn’t really scalable as if you kill the VM, you’ll need to update the IP. It also isn’t scalable if you bring on multiple instances/scaling.

That said, to get started, this is a completely viable approach. You can set this by going to you MySQL Server > Connection Security and adding your VM public IP, and then disabling “Allow access to Azure services”

Option 2: VNet Security

This is generally my preferred option. This allows you to make your MySQL Server only accessible from within a specific VNET’s subnet. Unfortuately, this is only available on Standard edition MySQL servers, which come in at ~£50/month.

Rant on “Allow Access to Azure Services”

A quick gripe with this setting. It can be tempting to just enable this setting as Azure Managed services can then access your MySQL instance. However, this is NOT limited to the services in your subscription. Therefore, anyone running their app in Azure can access your MySQL instance. Given the amount of people running in Azure, this doesn’t really provide much security. Use with caution.

Conclusion

In this post we’ve connected our grafana instance to an Azure MySQL instance so that we get backups, security and a persistent database. This means that should our Grafana VM become corrupt, or otherwise become compromised, we can just terminate it and bring up a new one from scratch.

In addition, you can also setup as manage VMs for grafana as you want, add load balancers, host it in containers, anything.

In the next post, we’ll look at securing the frontend using SSL, and generating certificates using LetsEncrypt

15

Tags:

AzureObservabilityAzuremysqlGrafana