Join our upcoming webinar 'Learn how to Leverage Plugin APIs on Hyperledger Besu' - February 4, 2020 at 2PM ET

 

December 19, 2019

Setting up Monitoring for Hyperledger Besu in AWS

by, Joshua Fernandes

This post was written by PegaSys Protocol Engineer, Joshua Fernandes. To learn more about setting up Hyperledger Besu in AWS, read this blog.

This guide is a walkthrough of setting up Prometheus and Grafana on AWS to automatically monitor Besu nodes.

Outcome:



We will be setting up a single EC2 instance that will run Prometheus & Grafana via docker-compose.

An alternative is to set the two up manually and apply the config for each as described below.

You can also put this configuration into the likes of a Launch Configuration that is used by an AutoScaling Group or make use of containers directly via ECS or AKS

Steps:

1. We use the Prometheus’  EC2 Scraper which allows retrieving data from targets on EC2 instances.

The private IP address is used by used default, but can be changed to the public IP with relabelling.
Following best practices we will avoid storing AWS keys in the config and will setup a role first which will be used by the instance

2. Create an IAM role for the instance by going to the IAM service in the AWS console. 

Set the trusted entity to EC2 (ie the assume role): AWS service: ec2.amazonaws.com

Set the IAM policy to AmazonEC2ReadOnlyAccess. This is an AWS managed policy.

Note: if you attach the instance to a load balancer or similar system, you will need to attach a custom policy.

3. Create a security group that allows access to port 3000 & port 9090 from your CIDR range eg: “0.0.0.0/0”. Ideally lock this down to selected trusted IPs 

4. Create an instance using the AmazonLinux2 AMI or the Ubuntu 18.04 AMI or equivalent. We will use the Ubuntu AMI for the rest of this tutorial. For this example, we are using an instance type of t3.small and setting the volume size to 50GB. Select the IAM role and security group from steps 2 & 3. Ensure that you use a public IP or set the instance to come up in a public subnet

Note: In a real production setup, use a large second volume that will persist if the instance/AZ fails. The size will depend on the frequency of logging, the number of nodes and the retention period (default is 14 days). 

5. Once the instance is up, ssh into the machine and install packages:

sudo apt-get update && sudo apt-get install -y apparmor apt-transport-https ca-certificates curl build-essential
# install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add –
sudo sh -c ‘echo “deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable” > /etc/apt/sources.list.d/docker.list’
sudo apt-get update && sudo apt-get -y install docker-ce
sudo usermod -aG docker $USER
# install docker-compose
sudo curl -L https://github.com/docker/compose/releases/download/1.24.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

6. Create the following folder structure to store the data and config for prometheus and grafana. Use any location that works for you, we will be using /data

sudo mkdir -p /data/prometheus/config && sudo mkdir -p /data/prometheus/datasudo mkdir -p /data/grafana/config && sudo mkdir -p /data/grafana/data
# 472 is the respective container user:group pairssudo mkdir -p /var/log/grafana/ && sudo chown -R 472:472 /var/log/grafana/sudo chown -R $USER:$USER /data

7. Create the prometheus config file called prometheus.yml in /data/prometheus/config/ with the contents below:

Note:

  • The relabel config below sets the instance’s name to show up instead of the private IP. Customize this to suit your requirements.
  • Update the aws-region parameter based on the region you are using
  • Update the regex parameter based on the besu node prefix you use
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

# Alertmanager configuration
alerting:

# Load rules once and evaluate them according to the global ‘evaluation_interval’.
rule_files:

scrape_configs:
  – job_name: prometheus
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      – targets: [ localhost:9090 ]

  – job_name: besu-nodes
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    ec2_sd_configs:
      – region: ap-southeast-2
        port: 9545
    relabel_configs:
      – source_labels: [__meta_ec2_tag_Name]
        regex: besu-.*
        action: keep
        # Use the name as the instance label
      – source_labels: [__meta_ec2_tag_Name]
        target_label: instance


8. Create the following docker-compose.yml file in /data

Note:

Update the aws-region parameter based on the region you are using

Grafana supports a number of authentication mechanisms for secure login. Select one that works best for your needs. The example below uses a simple admin:password scheme where password is what is defined in the GF_SECURITY_ADMIN_PASSWORD env var:

—version “3.4”
services:

  prometheus:
    container_name: prometheus
    labels:
      service_name: prometheus
    image: “registry.hub.docker.com/prom/prometheus:v2.10.0”
    restart: always
    network_mode: “host”
    ports:
      – “9090:9090”
    volumes:
      – /var/run/docker.sock:/var/run/docker.sock:ro
      – /data/prometheus/config/prometheus.yml:/etc/prometheus/prometheus.yml
      – /data/prometheus/data/:/prometheus    command:
      – ‘–log.format=json’
      – ‘–config.file=/etc/prometheus/prometheus.yml’
    logging:
      driver: “json-file”
      options:
        max-size: “200k”
        max-file: “10”
    environment:
      AWS_REGION: “ap-southeast-2”
    deploy:
      resources:
        limits:
          cpus: ‘0.50’
          memory: 1024M
        reservations:
          cpus: ‘0.25’
          memory: 512M
    cap_add:
      – NET_ADMIN

  grafana:
    container_name: grafana
    labels:
      service_name: grafana
    image: “registry.hub.docker.com/grafana/grafana:6.2.4”
    restart: always
    network_mode: “host”
    ports:
      – “3000:3000”
    volumes:
      – /data/grafana/data/:/var/lib/grafana/
      – /var/run/docker.sock:/var/run/docker.sock:ro
      – /var/log/grafana/:/var/log/grafana/
    environment:
      AWS_REGION: “ap-southeast-2”
      GF_SECURITY_ADMIN_PASSWORD: “SuperSecretPassword”
      GF_LOG_MODE: “file”
    deploy:
      resources:
        limits:
          cpus: ‘0.50’
          memory: 1024M
        reservations:
          cpus: ‘0.25’
          memory: 512M
    cap_add:
      – NET_ADMIN

9. Update permissions of the folder so the containers can read and write.

sudo chown -R $USER:$USER /data
sudo chmod -R 777 /data/grafana /data/prometheus

10. Start docker-compose

cd /data && docker-compose up -d

11. Add the following options to your Besu startup command to enable metrics collection:

–metrics-enabled –metrics-host=0.0.0.0 –metrics-port=9545

12. Setup Grafana datasource 

Once compose has started both services, go to http://<IP>:3000 and login with the credentials from above of admin:SuperSecretPassword 

Select Add a Datasource -> Prometheus and put the details in like so:

13. Setup Besu Dashboard

https://grafana.com/grafana/dashboards/10273

Select Dashboard -> Manage -> Import and

Enter 10273 for Grafana.dashboard & click Load

select the Prometheus datasource you just created and Import 

That will take you to the main dashboard and you will see the Besu dashboard and any Besu nodes that have metrics enabled

14. Preconfiguring grafana to read from prometheus datasource & install the Besu dashboard – this can be done by saving the yml and json data and putting them in a provisioning folder which is mounted at runtime. Please refer to the official documentation for details

15. Logrotate for grafana (optional but recommended)

We recommend doing this for Besu too, if it’s not already done.

Create the config file /etc/logrotate.d/grafana with the contents below:

/var/log/grafana/*.log {
    rotate 7
    missingok
    daily
    compress
    copytruncate
}

16. Kubernetes SD config

For kubernetes related setups please use the kuberenetes scraper instead. For a working example please see our besu-kubernetes repo 

17. Additional system metrics eg: disk space, cpu etc. (optional but recommended)

You can also add other system metrics to this setup so you capture a lot more information and can alert of that. Install the prometheus node exporter and add an extra scrape_config to the prometheus list in step 7

18. Load balancer & https config

Create an Application Load Balancer and target groups. Use ACM to generate ssl certs.

Add any instances to the target groups and pick whichever service you wish to expose. Generally this is only grafana with some auth mechanism, and prometheus is kept internal only.


Recent Posts

Key Factors to Consider When Choosing a Blockchain Consensus Protocol

This post was authored by David Hyland-Wood, with input from Roberto Saltini, Franck Cassez, and Joanne Fuller.

A blockchain …

Introducing Plugin APIs in Hyperledger Besu

As part of its 1.4 release (set for February 26), Hyperledger Besu will expose a Plugin API. …

PegaSys in 2020

2019 has come to an end, and we’re already a week into 2020. …