Metrics Monitoring App with Anomaly Detection using MongoDB

Overview of My Submission

Schedule metric collectors to gather measurements from different external services and optionally run anomaly detection on the new values.

The project uses MongoDB as a storage system, storing data as time series entries.

To make the experience complete, a visualization tool is needed. Fortunately, Mongo gives us Mongo Charts which can be used directly from Mongo Atlas.

Submission Category:

Prime Time

Link to Code

The code can be found here:

GitHub logo madalinfasie / monitoringapp_mongodb

Metric Monitoring App Using Mongo Time Series

Description

Schedule metrics collectors for different services and run anomaly detection on new values.

Getting started

Prerequisites

Make sure you have Docker and Docker Compose installed on your machine.

Create a MongoDB Atlas cluster with MongoDB 5.0 or above installed.

For security reasons, the .env was not included in the repository. The available configuration variables are provided inside .env.sample. Make sure to set all the configuration variables before running the app.

Running the app

Running this app is as easy as running:

$ docker-compose build
$ docker-compose up

Overview

Starting with the docker-compose.yml file, we can see that this application is split into a main webapp service and the scheduling service (celery in this case).

The metrics collection is done mainly by the celery workers. The metrics_web service providing only a way of pushing metrics from external service (when scheduling is…

View on GitHub

Additional Info

To test the project, I created three metrics, two of them are randomly generated (as their name suggests) and another one is the number of Dev.to posts in the last 15 minutes (I chose 15 minutes just so I don't have to wait so long for the data).

In the first screenshot, we can see the number of posts in the last 15 minutes and anther graph, plotting the values returned by random.randrange(1, 100)

In the second screenshot, we can see the second randomly generated metric (on the right) and a plot of the anomaly prediction (on the left). It is considered an anomaly if the score is -1.

As you can see, due to the metric's randomness and few data points, the anomaly prediction is not giving us any real insight.

Built with

34

Categorieën:

PythonMongodb

Tags:

AtlashackathonPythonMongodbMonitoring