15
Grafana on Azure – Azure MySQL Storage
In this multi-part series you’ll learn how to host Grafana safely, and cheaply on Azure, and how to get some decent visbility from Azure Monitor/App Insights through it.
- Part 1 – Hosting/Configuration
- Part 2 – Azure MySQL Storage (this post)
- Part 3 – SSL with LetsEncrypt
- Part 4 – Azure AD Login
- Part 5 – Azure Monitor Datasource (Coming Soon)
In the last post, you’ve setup a basic Grafana VM on Azure that is using a database on the VM’s disk as it’s datastore.
In this post, we’ll move that database to Azure PaaS.
This is Azure’s managed version of MySQL. It’s cheap (~£25/month) and supports everything you’d expect an Azure PaaS platform to do. There’s no patch management, backups, etc. just an endpoint for you to hit. What’s not to love?
This makes it great for our database for Grafana as the point of a monitoring system is be always up, and you don’t want to be monitoring your monitoring system with your monitoring system. Having a single VM, with the database, the frontend, etc. as a single point of failure just isn’t great. Having then to add backups of all your data regularly, and suddenly the quick and easy Grafana instance is now a management overhead in itself.
Grafana stores a very small amount of data, and can be stored in a shared server if you already have one. We’ll be setting up a server from scratch for this, but most of the steps will work on an existing server if you have one.
You can get away with the smallest possible instance, which a B1 at ~£20/month, with the minimum base storage (5GB). This will give you the ability to have backups, SSL, and DB maintainance. I’m going to guess that this is likely nothing in comparison to the other things you’re running, and also nothing in terms of the ~1day/month it would take you to verify all backups, etc. if you ran your own.
You’ll need to create a database and a user for Grafana. Later in this post, we’ll look at locking down the Azure MySQL instance, for now, you’ll need to manually add your IP.
Click the link “Add current client IP Address”, and then click save. This will add your public IP Address to the SQL Server.
Now connect to the MySQL instance from the command line (if you don’t want to install mysql, running the docker image is a nice alternative).
mysql -h {servername}.mysql.database.azure.com -u {adminuser}@{servername} -pYour user will need full privileges to manage the database as grafana has builtin database migrations.
CREATE USER '<username>'@'%' IDENTIFIED BY '<password>';
CREATE DATABASE grafana_data;
GRANT ALL ON grafana_data.* TO '<username>'@'%';That’s it for MySQL. You’ve created the database, and it’s setup so you can access it. Next we’ll make grafana use it.
The first thing we’ll need to do is allow Grafana to access the database. For now, we’ll enable the “Allow access to Azure Services”, this will enable access from Azure IPs which will work for now, and we can secure this later.
Next we’ll need to tell grafana where to find our database, and how to log in. SSH to your VM and open up the grafana config file
sudo nano /etc/grafana/grafana.iniIn there, find the section labelled [Database] and set the following values, leaving all the others at their defaults.
Note: the ini file uses ; to comment out the settings, remove it to set them.
type = mysqlhost = <servername>.mysql.database.azure.com:3306name = <database-name>user = <username>@<server-name>
ssl_mode = skip-verify
ca_cert_path = /etc/ssl/certs/ca-certificates.crt
Now restart grafana, and you’ll be need to login again with the default credentials.
sudo systemctl restart grafana-serverIf you end up with your url timing out, check the logs.
sudo tail -f /var/log/grafana/grafana.logA common error is accessing the MySQL instance. This will manifest as the log tail above stalling at “Starting DB Migrations”. Likely causes are:
- Username hasn’t include
@<server-name> -
ssl_modeandca_cert_patharen’t set - MySQL firewall rules haven’t been setup.
If all has gone well, you’ll be presented with the Grafana logon screen, and be asked to input the default admin password and reset it.
Right now the MySQL instance has been setup so that your public IP and literally anything that runs on Azure can access it. This is not a desireable situation, so we’ll need to secure that further.
There are a few things you can do to secure the MySQL instance.
You can restrict your MySQL instance to just the IP of your VM. This is the simplest approach, but isn’t really scalable as if you kill the VM, you’ll need to update the IP. It also isn’t scalable if you bring on multiple instances/scaling.
That said, to get started, this is a completely viable approach. You can set this by going to you MySQL Server > Connection Security and adding your VM public IP, and then disabling “Allow access to Azure services”
This is generally my preferred option. This allows you to make your MySQL Server only accessible from within a specific VNET’s subnet. Unfortuately, this is only available on Standard edition MySQL servers, which come in at ~£50/month.
A quick gripe with this setting. It can be tempting to just enable this setting as Azure Managed services can then access your MySQL instance. However, this is NOT limited to the services in your subscription. Therefore, anyone running their app in Azure can access your MySQL instance. Given the amount of people running in Azure, this doesn’t really provide much security. Use with caution.
In this post we’ve connected our grafana instance to an Azure MySQL instance so that we get backups, security and a persistent database. This means that should our Grafana VM become corrupt, or otherwise become compromised, we can just terminate it and bring up a new one from scratch.
In addition, you can also setup as manage VMs for grafana as you want, add load balancers, host it in containers, anything.
In the next post, we’ll look at securing the frontend using SSL, and generating certificates using LetsEncrypt
15