Cerberus on Krkn

Installation

Thu, 05 Jan 2017 00:00:00 +0000

Following ways are supported to run Cerberus:

Standalone python program through Git or python package
Containerized version using either Podman or Docker as the runtime
Kubernetes or OpenShift deployment

Note

Only OpenShift 4.x versions are tested.

Git

Pick the latest stable release to install here.

$ git clone https://github.com/redhat-chaos/cerberus.git --branch <release>

Install the dependencies

NOTE: Recommended to use a virtual environment(pyenv,venv) so as to prevent conflicts with already installed packages.

Config

Thu, 05 Jan 2017 00:00:00 +0000

Cerberus Config Components Explained

Config

Set the components to monitor and the tunings like duration to wait between each check in the config file located at config/config.yaml. A sample config looks like:

cerberus:
 distribution: openshift  # Distribution can be kubernetes or openshift
 kubeconfig_path: /root/.kube/config  # Path to kubeconfig
 port: 8081 # http server port where cerberus status is published
 watch_nodes: True # Set to True for the cerberus to monitor the cluster nodes
 watch_cluster_operators: True # Set to True for cerberus to monitor cluster operators
 watch_terminating_namespaces: True # Set to True to monitor if any namespaces (set below under 'watch_namespaces' start terminating
 watch_url_routes:
 # Route url's you want to monitor, this is a double array with the url and optional authorization parameter
 watch_master_schedulable: # When enabled checks for the schedulable master nodes with given label.
 enabled: True
 label: node-role.kubernetes.io/master
 watch_namespaces: # List of namespaces to be monitored
 -  openshift-etcd
 -  openshift-apiserver
 -  openshift-kube-apiserver
 -  openshift-monitoring
 -  openshift-kube-controller-manager
 -  openshift-machine-api
 -  openshift-kube-scheduler
 -  openshift-ingress
 -  openshift-sdn  # When enabled, it will check for the cluster sdn and monitor that namespace
 watch_namespaces_ignore_pattern: [] # Ignores pods matching the regex pattern in the namespaces specified under watch_namespaces
 cerberus_publish_status: True # When enabled, cerberus starts a light weight http server and publishes the status
 inspect_components: False # Enable it only when OpenShift client is supported to run
 # When enabled, cerberus collects logs, events and metrics of failed components

 prometheus_url: # The prometheus url/route is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes.
 prometheus_bearer_token: # The bearer token is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes. This is needed to authenticate with prometheus.
 # This enables Cerberus to query prometheus and alert on observing high Kube API Server latencies.

 slack_integration: False # When enabled, cerberus reports the failed iterations in the slack channel
 # The following env vars needs to be set: SLACK_API_TOKEN ( Bot User OAuth Access Token ) and SLACK_CHANNEL ( channel to send notifications in case of failures )
 # When slack_integration is enabled, a watcher can be assigned for each day. The watcher of the day is tagged while reporting failures in the slack channel. Values are slack member ID's.
 watcher_slack_ID: # (NOTE: Defining the watcher id's is optional and when the watcher slack id's are not defined, the slack_team_alias tag is used if it is set else no tag is used while reporting failures in the slack channel.)
 Monday:
 Tuesday:
 Wednesday:
 Thursday:
 Friday:
 Saturday:
 Sunday:
 slack_team_alias: # The slack team alias to be tagged while reporting failures in the slack channel when no watcher is assigned

 custom_checks:
 -  custom_checks/custom_check_sample.py  # Relative paths of files containing additional user defined checks

tunings:
 timeout: 20 # Number of seconds before requests fail
 iterations: 1 # Iterations to loop before stopping the watch, it will be replaced with infinity when the daemon mode is enabled
 sleep_time: 3 # Sleep duration between each iteration
 kube_api_request_chunk_size: 250 # Large requests will be broken into the specified chunk size to reduce the load on API server and improve responsiveness.
 daemon_mode: True # Iterations are set to infinity which means that the cerberus will monitor the resources forever
 cores_usage_percentage: 0.5 # Set the fraction of cores to be used for multiprocessing

database:
 database_path: /tmp/cerberus.db  # Path where cerberus database needs to be stored
 reuse_database: False # When enabled, the database is reused to store the failures

Watch Nodes

This flag returns any nodes where the KernelDeadlock is not set to False and does not have a Ready status

Example Report

Thu, 05 Jan 2017 00:00:00 +0000

2020-03-26 22:05:06,393 [INFO] Starting cerberus
2020-03-26 22:05:06,401 [INFO] Initializing client to talk to the Kubernetes cluster
2020-03-26 22:05:06,434 [INFO] Fetching cluster info
2020-03-26 22:05:06,739 [INFO] Publishing cerberus status at http://0.0.0.0:8080
2020-03-26 22:05:06,753 [INFO] Starting http server at http://0.0.0.0:8080
2020-03-26 22:05:06,753 [INFO] Daemon mode enabled, cerberus will monitor forever
2020-03-26 22:05:06,753 [INFO] Ignoring the iterations set

2020-03-26 22:05:25,104 [INFO] Iteration 4: Node status: True
2020-03-26 22:05:25,133 [INFO] Iteration 4: Etcd member pods status: True
2020-03-26 22:05:25,161 [INFO] Iteration 4: OpenShift apiserver status: True
2020-03-26 22:05:25,546 [INFO] Iteration 4: Kube ApiServer status: True
2020-03-26 22:05:25,717 [INFO] Iteration 4: Monitoring stack status: True
2020-03-26 22:05:25,720 [INFO] Iteration 4: Kube controller status: True
2020-03-26 22:05:25,746 [INFO] Iteration 4: Machine API components status: True
2020-03-26 22:05:25,945 [INFO] Iteration 4: Kube scheduler status: True
2020-03-26 22:05:25,963 [INFO] Iteration 4: OpenShift ingress status: True
2020-03-26 22:05:26,077 [INFO] Iteration 4: OpenShift SDN status: True
2020-03-26 22:05:26,077 [INFO] HTTP requests served: 0
2020-03-26 22:05:26,077 [INFO] Sleeping for the specified duration: 5


2020-03-26 22:05:31,134 [INFO] Iteration 5: Node status: True
2020-03-26 22:05:31,162 [INFO] Iteration 5: Etcd member pods status: True
2020-03-26 22:05:31,190 [INFO] Iteration 5: OpenShift apiserver status: True
127.0.0.1 - - [26/Mar/2020 22:05:31] "GET / HTTP/1.1" 200 -
2020-03-26 22:05:31,588 [INFO] Iteration 5: Kube ApiServer status: True
2020-03-26 22:05:31,759 [INFO] Iteration 5: Monitoring stack status: True
2020-03-26 22:05:31,763 [INFO] Iteration 5: Kube controller status: True
2020-03-26 22:05:31,788 [INFO] Iteration 5: Machine API components status: True
2020-03-26 22:05:31,989 [INFO] Iteration 5: Kube scheduler status: True
2020-03-26 22:05:32,007 [INFO] Iteration 5: OpenShift ingress status: True
2020-03-26 22:05:32,118 [INFO] Iteration 5: OpenShift SDN status: False
2020-03-26 22:05:32,118 [INFO] HTTP requests served: 1
2020-03-26 22:05:32,118 [INFO] Sleeping for the specified duration: 5
+--------------------------------------------------Failed Components--------------------------------------------------+
2020-03-26 22:05:37,123 [INFO] Failed openshift sdn components: ['sdn-xmqhd']

2020-05-23 23:26:43,041 [INFO] ------------------------- Iteration Stats ---------------------------------------------
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_nodes in iteration 1: 0.0996248722076416 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_cluster_operators in iteration 1: 0.3672499656677246 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_namespaces in iteration 1: 1.085144281387329 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run entire_iteration in iteration 1: 4.107403039932251 seconds
2020-05-23 23:26:43,041 [INFO] ---------------------------------------------------------------------------------------

Usage

Thu, 05 Jan 2017 00:00:00 +0000

Config

Set the supported components to monitor and the tunings like number of iterations to monitor and duration to wait between each check in the config file located at config/config.yaml. A sample config looks like:

cerberus:
 distribution: openshift  # Distribution can be kubernetes or openshift
 kubeconfig_path: ~/.kube/config  # Path to kubeconfig
 port: 8080 # http server port where cerberus status is published
 watch_nodes: True # Set to True for the cerberus to monitor the cluster nodes
 watch_cluster_operators: True # Set to True for cerberus to monitor cluster operators. Parameter is optional, will set to True if not specified
 watch_url_routes: # Route url's you want to monitor
 - - https://...
 - Bearer ****  # This parameter is optional, specify authorization need for get call to route
 - - http://...
 watch_master_schedulable: # When enabled checks for the schedulable
 enabled: True master nodes with given label.
 label: node-role.kubernetes.io/master
 watch_namespaces: # List of namespaces to be monitored
 -  openshift-etcd
 -  openshift-apiserver
 -  openshift-kube-apiserver
 -  openshift-monitoring
 -  openshift-kube-controller-manager
 -  openshift-machine-api
 -  openshift-kube-scheduler
 -  openshift-ingress
 -  openshift-sdn
 cerberus_publish_status: True # When enabled, cerberus starts a light weight http server and publishes the status
 inspect_components: False # Enable it only when OpenShift client is supported to run.
 # When enabled, cerberus collects logs, events and metrics of failed components

 prometheus_url: # The prometheus url/route is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes.
 prometheus_bearer_token: # The bearer token is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes. This is needed to authenticate with prometheus.
 # This enables Cerberus to query prometheus and alert on observing high Kube API Server latencies.

 slack_integration: False # When enabled, cerberus reports status of failed iterations in the slack channel
 # The following env vars need to be set: SLACK_API_TOKEN ( Bot User OAuth Access Token ) and SLACK_CHANNEL ( channel to send notifications in case of failures )
 # When slack_integration is enabled, a watcher can be assigned for each day. The watcher of the day is tagged while reporting failures in the slack channel. Values are slack member ID's.
 watcher_slack_ID: # (NOTE: Defining the watcher id's is optional and when the watcher slack id's are not defined, the slack_team_alias tag is used if it is set else no tag is used while reporting failures in the slack channel.)
 Monday:
 Tuesday:
 Wednesday:
 Thursday:
 Friday:
 Saturday:
 Sunday:
 slack_team_alias: # The slack team alias to be tagged while reporting failures in the slack channel when no watcher is assigned

 custom_checks: # Relative paths of files containing additional user defined checks
 -  custom_checks/custom_check_sample.py
 -  custom_check.py

tunings:
 iterations: 5 # Iterations to loop before stopping the watch, it will be replaced with infinity when the daemon mode is enabled
 sleep_time: 60 # Sleep duration between each iteration
 kube_api_request_chunk_size: 250 # Large requests will be broken into the specified chunk size to reduce the load on API server and improve responsiveness.
 daemon_mode: True # Iterations are set to infinity which means that the cerberus will monitor the resources forever
 cores_usage_percentage: 0.5 # Set the fraction of cores to be used for multiprocessing

database:
 database_path: /tmp/cerberus.db  # Path where cerberus database needs to be stored
 reuse_database: False # When enabled, the database is reused to store the failures

Note

watch_namespaces support regex patterns. Any valid regex pattern can be used to watch all the namespaces matching the regex pattern. For example, ^openshift-.*$ can be used to watch all namespaces that start with openshift- or openshift can be used to watch all namespaces that have openshift in it.

Note

The current implementation can monitor only one cluster from one host. It can be used to monitor multiple clusters provided multiple instances of Cerberus are launched on different hosts.

Note

The components especially the namespaces needs to be changed depending on the distribution i.e Kubernetes or OpenShift. The default specified in the config assumes that the distribution is OpenShift. A config file for Kubernetes is located at config/kubernetes_config.yaml

Alerts

Thu, 05 Jan 2017 00:00:00 +0000

Cerberus consumes the metrics from Prometheus deployed on the cluster to report the alerts.

When provided the prometheus url and bearer token in the config, Cerberus reports the following alerts:

KubeAPILatencyHigh: alerts at the end of each iteration and warns if 99th percentile latency for given requests to the kube-apiserver is above 1 second. It is the official SLI/SLO defined for Kubernetes.
High number of etcd leader changes: alerts the user when an increase in etcd leader changes are observed on the cluster. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.

Node Problem Detector

Thu, 05 Jan 2017 00:00:00 +0000

node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack.

Installation

Please follow the instructions in the installation section to setup Node Problem Detector on Kubernetes. The following instructions are setting it up on OpenShift:

Create openshift-node-problem-detector namespace ns.yaml with oc create -f ns.yaml
Add cluster role with oc adm policy add-cluster-role-to-user system:node-problem-detector -z default -n openshift-node-problem-detector
Add security context constraints with oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-node-problem-detector:default
Edit node-problem-detector.yaml to fit your environment.
Edit node-problem-detector-config.yaml to configure node-problem-detector.
Create the ConfigMap with oc create -f node-problem-detector-config.yaml
Create the DaemonSet with oc create -f node-problem-detector.yaml

Once installed you will see node-problem-detector pods in openshift-node-problem-detector namespace. Now enable openshift-node-problem-detector in the config.yaml. Cerberus just monitors KernelDeadlock condition provided by the node problem detector as it is system critical and can hinder node performance.

Slack Integration

Thu, 05 Jan 2017 00:00:00 +0000

The user has the option to enable/disable the slack integration ( disabled by default ). To use the slack integration, the user has to first create an app and add a bot to it on slack. SLACK_API_TOKEN and SLACK_CHANNEL environment variables have to be set. SLACK_API_TOKEN refers to Bot User OAuth Access Token and SLACK_CHANNEL refers to the slack channel ID the user wishes to receive the notifications. Make sure the Slack Bot Token Scopes contains this permission [calls:read] [channels:read] [chat:write] [groups:read] [im:read] [mpim:read]

Contribute

Thu, 05 Jan 2017 00:00:00 +0000

How to contribute

Contributions are always appreciated.

How to:

Pull request

In order to submit a change or a PR, please fork the project and follow instructions:

$ git clone http://github.com/<me>/cerberus
$ cd cerberus
$ git checkout -b <branch_name>
$ <make change>
$ git add <changes>
$ git commit -a
$ <insert good message>
$ git push

Fix Formatting

Cerberus uses pre-commit framework to maintain the code linting and python code styling. The CI would run the pre-commit check on each pull request. We encourage our contributors to follow the same pattern, while contributing to the code.