Ops Center monitoring information

There are 4 main components in the monitoring system: InfluxDB, Heapster, Grafana, and Kapacitor.

InfluxDB

Is an open source time series database which is used for the main data store for monitoring time series data. Provides the Kubernetes service influxdb.monitoring.svc.cluster.local.

Heapster

Monitors Kubernetes components in generating a collection of not only performance metrics about workloads, nodes, and pods, but also events generated by Clusters. The statistics captured are reported to InfluxDB.

Grafana

Is an open source metrics suite which provides the dashboard in the Gravity monitoring and alerts system. The dashboard provides a visual to the information stored in InfluxDB, which is exposed as the service grafana.monitoring.svc.cluster.local. Credentials generated are placed into a secret grafana in the monitoring namespace

Gravity is shipped with 2 pre-configured dashboards providing a visual of machine and pod-level overview of the installed cluster. Within the Gravity control panel, you can access the dashboard by navigating to the Monitoring page.

By default, Grafana is running in anonymous read-only mode. Anyone who logs into Gravity can view but not modify the dashboards.

Kapacitor

Is the data processing engine for InfluxDB, which streams data from InfluxDB and sends alerts to the end user exposed as the service kapacitor.monitoring.svc.cluster.local.

Metric Retention Policy & Rollups

Let's now talk about durations the measurements are stored for. During initial installation Gravity pre-configures InfluxDB with the following retention policies:

• default = 24 hours - is used for high precision metrics.
• medium = 4 weeks - is used for medium precision metrics.
• long = 52 weeks - keeps metrics aggregated over even larger intervals.

All metrics sent to InfluxDB by Heapster are saved using the default retention policy which means that all the high-resolution metrics collected are kept intact for 24 hours.

To provide historical overview some of the most commonly helpful metrics (such as CPU/memory usage, network transfer rates) are rolled up to lower resolutions and stored using the longer retention policies mentioned above.

In order to provide such downsampled metrics, Gravity uses InfluxDB “continuous queries” which are programmed to run automatically and aggregate metrics over a certain interval.

The Gravity monitoring system allows two types of rollup configurations for collecting metrics:

• medium = aggregates data over 5 minute intervals
• long = aggregates data over 1 hour intervals

Each of the two rollups mentioned above, continue to their respective retention policy following. For example the long rollup aggregates data over 1 hour interval and goes into the long retention policy.

Preconfigured rollups that Gravity clusters come with are stored in the rollups-default config map in the monitoring namespace:

$ kubectl -nmonitoring get configmaps/rollups-default -oyaml

The configuration of retention policies and rollups is handled by a “watcher” service that runs in a container as a part of the InfluxDB pod so all these configurations can be seen in its logs:

$ kubectl -nmonitoring logs influxdb-599c5f5c45-6hqmc watcher

Kapacitor Email Configuration

In order to configure email alerts via Kapacitor you will need to create Gravity resources of type smtp and alerttarget.

An example of the configuration is shown below:

kind: smtp
version: v2
metadata:
  name: smtp
spec:
  host: smtp.host
  port: <smtp port> # 465 by default
  username: <username>
  password: <password>
---
kind: alerttarget
version: v2
metadata:
  name: email-alerts
spec:
  email: triage@example.com # Email address of the alert recipient

Creating these resources will accordingly update and reload Kapacitor configuration:

$ gravity resource create -f smtp.yaml

In order to view the current SMTP settings or alert target:

$ gravity resource get smtp
$ gravity resource get alerttarget

Only a single alert target can be configured. To remove the current alert target, you can execute the following kapacitor command inside the designated pod:

$ kapacitor delete alerttarget email-alerts