Node Monitoring Tools for Stewards¶
- Plugin Manager
- Email Plugin
- AWS SNS Plugin
Currently, indy-node emits different events via the Plugin Manager when certain criteria are met. The Plugin Manager tries to import all pip packages which names start with “indynotifier*”. Each of these packages is required to expose
send_message; interface which is used to pass the event with the associated message to the package for further handling.
The Plugin Manager code is located at here.
- .nodeRequestSpike : NodeRequestSuspiciousSpike
- .clusterThroughputSpike : ClusterThroughputSuspiciousSpike
- .clusterLatencyTooHigh : ClusterLatencyTooHigh
- .nodeUpgradeScheduled : NodeUpgradeScheduled
- .nodeUpgradeComplete : NodeUpgradeComplete
- .nodeUpgradeFail : NodeUpgradeFail
- .poolUpgradeCancel : PoolUpgradeCancel
- SMTP server must be running on localhost.
- Install SMTP server (if you don’t have one already)
The most simple way on Ubuntu is to use
$ sudo apt-get install sendmail
To check that it’s working execute:
echo "Subject: sendmail test" | sendmail -v email@example.com -f firstname.lastname@example.org
If you get a email on your email@example.com then
sendmail is working.
# pip3 install indynotifieremail
The spike detection and notification mechanisms should be enabled by appending of the following line to
indy_config.py configuration file:
The package depends on two environment variables:
Add these variables to
/etc/indy/indy.env environment file as you are required to set such system environment
variables for indy-node service in form described below.
INDY_NOTIFIER_EMAIL_RECIPIENTS should be a string in a format of:
firstname.lastname@example.org [optional list of events the recipient is going to get],email@example.com [event list]
If no list was provided the recipient is going to get notifications for all events. Example:
firstname.lastname@example.org event1 event2, email@example.com, firstname.lastname@example.org event3
This way steward1 is going to get notifications for event1 and event2, steward2 is going to get all possible notifications and steward3 is going to get notifications for event3 only.
The current list of events can be found above.
By default every email notification is going to be from email@example.com. You can change this by setting
INDY_NOTIFIER_EMAIL_SENDER. May be useful for email filters.
Email delivery frequency¶
By default you will not get a email with the same topic more than once an hour. This is defined by
SILENCE_TIMEOUT. It can be overridden by setting
INDY_NOTIFIER_SILENCE_TIMEOUT environment variable in
/etc/indy/indy.env file. Emails regarding update procedure are always delivered.
AWS SNS Plugin¶
- .A AWS SNS topic created with permissions to publish to it.
- .A installed Sovrin Validator instance.
Install the python package for sovrin-notifier-awssns. This should be only be installed using pip3.
pip3 install sovrinnotifierawssns
To configure AWS Credentials you will need to know the values for:
aws_secret_access_key. Follow the steps documented here Boto3 Configuring Credentials.
Use either of the following ways:
- .Environment variables
- .Shared credential file (~/.aws/credentials)
- .Boto2 config file (/etc/boto.cfg and ~/.boto)
Configure AWS Region you will need to know the value where the SNS Topic is hosted e.g. us-west-1, us-west-2, sa-east-1
To achieve this:
- .Set a Environment variable AWS_DEFAULT_REGION
- .Set region using file (~/.aws/config)
Define environment variable
SOVRIN_NOTIFIER_AWSSNS_TOPICARN on the Validator and set valid AWS SNS TopicARN as the value.
Events that cause a notification:
The home directory for the account that runs
sovrin-node.service on a Validator is
/home/sovrin/. So the aws credentials/config files must be created in
To set an environment variable on the Validator you must add it to the file
/home/sovrin/.sovrin/sovrin.env and restart the Validator. The TopicARN must be defined in this file.
To restart the Validator on a Ubuntu system you must execute the command
sudo systemctl restart sovrin-node.service while not logged in as a sovrin user.
This simple script will complete the setup, assuming that the sovrinnotifierawssns package is already installed:
#!/bin/bash sudo mkdir /home/sovrin/.aws sudo sh -c "printf \"[default]\nregion=us-west-2\" > /home/sovrin/.aws/config" sudo sh -c "printf \" .[default]\naws_access_key_id=AKIAIGKGW3CKRXKKWPZA\naws_secret_access_key=<YOUR_SECRET_KEY>\" > /home/sovrin/.aws/credentials" sudo sh -c "printf \"SOVRIN_NOTIFIER_AWSSNS_TOPICARN=arn:aws:sns:us-west-2:034727365312:validator-health-monitor-STN\" >> /home/sovrin/.sovrin/sovrin.env" sudo chown -R sovrin:sovrin /home/sovrin/.aws /home/sovrin/.sovrin/sovrin.env sudo systemctl restart sovrin-node