Krkn - Chaos and Resiliency Testing for Kubernetes on Krkn

Installation

Thu, 05 Jan 2017 00:00:00 +0000

Following ways are supported to run Cerberus:

Standalone python program through Git or python package
Containerized version using either Podman or Docker as the runtime
Kubernetes or OpenShift deployment

Note

Only OpenShift 4.x versions are tested.

Git

Pick the latest stable release to install here.

$ git clone https://github.com/redhat-chaos/cerberus.git --branch <release>

Install the dependencies

NOTE: Recommended to use a virtual environment(pyenv,venv) so as to prevent conflicts with already installed packages.

Krkn

Thu, 05 Jan 2017 00:00:00 +0000

Installation

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to krkn in anyway in the future we recommend forking the repository first before cloning.

See the latest release version here

$ git clone https://github.com/krkn-chaos/krkn.git --branch <release version>
$ cd krkn

Fork and Clone the Repository

Fork the repository

Krkn-Hub All Scenarios Variables

Thu, 05 Jan 2017 00:00:00 +0000

These variables are to be used for the top level configuration template that are shared by all the scenarios in Krkn-hub

See the description and default values below

Supported parameters for all scenarios in Krkn-Hub

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

example: export <parameter_name>=<value>

Parameter	Description	Default
CERBERUS_ENABLED	Set this to true if cerberus is running and monitoring the cluster	False
CERBERUS_URL	URL to poll for the go/no-go signal	http://0.0.0.0:8080
WAIT_DURATION	Duration in seconds to wait between each chaos scenario	60
ITERATIONS	Number of times to execute the scenarios	1
DAEMON_MODE	Iterations are set to infinity which means that the kraken will cause chaos forever	False
PUBLISH_KRAKEN_STATUS	If you want	True
SIGNAL_ADDRESS	Address to print kraken status to	0.0.0.0
PORT	Port to print kraken status to	8081
SIGNAL_STATE	Waits for the RUN signal when set to PAUSE before running the scenarios, refer docs for more details	RUN
DEPLOY_DASHBOARDS	Deploys mutable grafana loaded with dashboards visualizing performance metrics pulled from in-cluster prometheus. The dashboard will be exposed as a route.	False
CAPTURE_METRICS	Captures metrics as specified in the profile from in-cluster prometheus. Default metrics captures are listed here	False
ENABLE_ALERTS	Evaluates expressions from in-cluster prometheus and exits 0 or 1 based on the severity set. Default profile.	False
ALERTS_PATH	Path to the alerts file to use when ENABLE_ALERTS is set	config/alerts
ELASTIC_SERVER	Be able to track telemtry data in elasticsearch, this is the url of the elasticsearch data storage	blank
ELASTIC_INDEX	Elastic search index pattern to post results to	blank
HEALTH_CHECK_URL	URL to continually check and detect downtimes	blank
HEALTH_CHECK_INTERVAL	Interval at which to get	2
HEALTH_CHECK_BEARER_TOKEN	Bearer token used for authenticating into health check URL	blank
HEALTH_CHECK_AUTH	Tuple of (username,password) used for authenticating into health check URL	blank
HEALTH_CHECK_EXIT_ON_FAILURE	If value is True exits when health check failed for application, values can be True/False	blank
HEALTH_CHECK_VERIFY	Health check URL SSL validation; can be True/False	False
KUBE_VIRT_CHECK_INTERVAL	Interval at which to test kubevirt connections	2
KUBE_VIRT_NAMESPACE	Namespace to find VMIs in and watch	blank
KUBE_VIRT_NAME	Regex style name to match VMIs to watch	blank
KUBE_VIRT_FAILURES	If value is True exits will only report when ssh connections fail to vmi, values can be True/False	blank
KUBE_VIRT_DISCONNECTED	Use disconnected check by passing cluster API, can be True/False	False
KUBE_VIRT_NODE_NAME	If set, will filter vms further to only track ones that are on specified node name	blank
KUBE_VIRT_EXIT_ON_FAIL	Fails run if VMs still have false status at end of run, can be True/False	False
KUBE_VIRT_SSH_NODE	If set, will be a backup way to ssh to a node. Will want to set to a node that isn’t targeted in chaos	blank
CHECK_CRITICAL_ALERTS	When enabled will check prometheus for critical alerts firing post chaos	False
TELEMETRY_ENABLED	Enable/disables the telemetry collection feature	False
TELEMETRY_API_URL	telemetry service endpoint	https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production
TELEMETRY_USERNAME	telemetry service username	redhat-chaos
TELEMETRY_PASSWORD		No default
TELEMETRY_PROMETHEUS_BACKUP	enables/disables prometheus data collection	True
TELEMTRY_FULL_PROMETHEUS_BACKUP	if is set to False only the /prometheus/wal folder will be downloaded	False
TELEMETRY_BACKUP_THREADS	number of telemetry download/upload threads	5
TELEMETRY_ARCHIVE_PATH	local path where the archive files will be temporarly stored	/tmp
TELEMETRY_MAX_RETRIES	maximum number of upload retries (if 0 will retry forever)	0
TELEMETRY_RUN_TAG	if set, this will be appended to the run folder in the bucket (useful to group the runs	chaos
TELEMETRY_GROUP	if set will archive the telemetry in the S3 bucket on a folder named after the value	default
TELEMETRY_ARCHIVE_SIZE	the size of the prometheus data archive size in KB. The lower the size of archive is	1000
TELEMETRY_LOGS_BACKUP	Logs backup to s3	False
TELEMETRY_FILTER_PATTER	Filter logs based on certain time stamp patterns	["(\w{3}\s\d{1,2}\s\d{2}:\d{2}:\d{2}\.\d+).+",“kinit (\d+/\d+/\d+\s\d{2}:\d{2}:\d{2})\s+”,"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z).+"]
TELEMETRY_CLI_PATH	OC Cli path, if not specified will be search in $PATH	blank

Note

For setting the TELEMETRY_ARCHIVE_SIZE,the higher the number of archive files will be produced and uploaded (and processed by backup_thread simultaneously).For unstable/slow connection is better to keep this value low increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the failed chunk without affecting the whole upload.

Krkn-lib

Thu, 05 Jan 2017 00:00:00 +0000

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

The Library contains Classes, Models and helper functions used in Kraken to interact with Kubernetes, Openshift and other external APIS. The goal of this library is to give to developers the building blocks to realize new Chaos Scenarios and to increase the testability and the modularity of the Krkn codebase.

Krknctl All Scenarios Variables

Thu, 05 Jan 2017 00:00:00 +0000

These variables are to be used for the top level configuration template that are shared by all the scenarios in Krknctl

See the description and default values below

Supported parameters for all scenarios in KrknCtl

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Usage example: --<parameter> <value>

Parameter	Description	Type	Possible Values	Default
–cerberus-enabled	Enables Cerberus Support	enum	True/False	False
–cerberus-url	Cerberus http url	string	-	http://0.0.0.0:8080
–distribution	Selects the orchestrator distribution	enum	openshift/kubernetes	openshift
–krkn-kubeconfig	Sets the path where krkn will search for kubeconfig in container	string	-	/home/krkn/.kube/config
–wait-duration	Waits for a certain amount of time after the scenario	number	-	1
–iterations	Number of times the same chaos scenario will be executed	number	-	1
–daemon-mode	If set the scenario will execute forever	enum	True/False	False
–uuid	Sets krkn run uuid instead of generating it	string	-	-
–capture-metrics	Enables metrics capture	enum	True/False	False
–enable-alerts	Enables cluster alerts check	enum	True/False	False
–alerts-path	Allows to specify a different alert file path	string	-	config/alerts.yaml
–metrics-path	Allows to specify a different metrics file path	string	-	config/metrics-aggregated.yaml
–enable-es	Enables elastic search data collection	enum	True/False	False
–es-server	Elasticsearch instance URL	string	-	http://0.0.0.0
–es-port	Elasticsearch instance port	number	-	443
–es-username	Elasticsearch instance username	string	-	elastic
–es-password	Elasticsearch instance password	string	-	-
–es-verify-certs	Enables elasticsearch TLS certificate verification	enum	True/False	False
–es-metrics-index	Index name for metrics in Elasticsearch	string	-	krkn-metrics
–es-alerts-index	Index name for alerts in Elasticsearch	string	-	krkn-alerts
–es-telemetry-index	Index name for telemetry in Elasticsearch	string	-	krkn-telemetry
–check-critical-alerts	Enables checking for critical alerts	enum	True/False	False
–telemetry-enabled	Enables telemetry support	enum	True/False	False
–telemetry-api-url	API endpoint for telemetry data	string	-	https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production
–telemetry-username	Username for telemetry authentication	string	-	redhat-chaos
–telemetry-password	Password for telemetry authentication	string	-	-
–telemetry-prometheus-backup	Enables Prometheus backup for telemetry	enum	True/False	True
–telemetry-full-prometheus-backup	Enables full Prometheus backup for telemetry	enum	True/False	False
–telemetry-backup-threads	Number of threads for telemetry backup	number	-	5
–telemetry-archive-path	Path to save telemetry archive	string	-	/tmp
–telemetry-max-retries	Maximum retries for telemetry operations	number	-	0
–telemetry-run-tag	Tag for telemetry run	string	-	chaos
–telemetry-group	Group name for telemetry data	string	-	default
–telemetry-archive-size	Maximum size for telemetry archives	number	-	1000
–telemetry-logs-backup	Enables logs backup for telemetry	enum	True/False	False
–telemetry-filter-pattern	Filter pattern for telemetry logs	string	-	["\w{3}\s\d{1,2}\s\d{2}:\d{2}:\d{2}\.\d+", “kinit (\d+/\d+/\d+\s\d{2}:\d{2}:\d{2}”, “\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z”]
–telemetry-cli-path	Path to telemetry CLI tool (oc)	string	-	-
–telemetry-events-backup	Enables events backup for telemetry	enum	True/False	True
–health-check-interval	How often to check the health check urls (seconds)	number	-	2
–health-check-url	URL to check the health of	string	-	-
–health-check-auth	Authentication tuple to authenticate into health check URL	string	-	-
–health-check-bearer-token	Bearer token to authenticate into health check URL	string	-	-
–health-check-exit	Exit on failure when health check URL is not able to connect	string	-	-
–health-check-verify	SSL Verification to authenticate into health check URL	string	-	false
–kubevirt-check-interval	How often to check the KubeVirt VMs SSH status (seconds)	number	-	2
–kubevirt-namespace	KubeVirt namespace to check the health of	string	-	-
–kubevirt-name	KubeVirt regex names to watch	string	-	-
–kubevirt-only-failures	KubeVirt checks only report if failure occurs	enum	True/False	false
–kubevirt-disconnected	KubeVirt checks in disconnected mode, bypassing the cluster’s API	enum	True/False	false
–kubevirt-ssh-node	KubeVirt backup node to SSH into when checking VMI IP address status	string	-	false
–kubevirt-exit-on-failure	KubeVirt fails run if VMs still have false status	enum	True/False	false
–kubevirt-node-node	Only track VMs in KubeVirt on given node name	string	-	false
–krkn-debug	Enables debug mode for Krkn	enum	True/False	False

Note

For setting the TELEMETRY_ARCHIVE_SIZE,the higher the number of archive files will be produced and uploaded (and processed by backup_thread simultaneously| .For unstable/slow connection is better to keep this value low increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the failed chunk without affecting the whole upload.

Network Chaos API

Thu, 05 Jan 2017 00:00:00 +0000

`AbstractNetworkChaosModule` abstract module class

All the plugins must implement the AbstractNetworkChaosModule abstract class in order to be instantiated and ran by the Netwok Chaos NG plugin. This abstract class implements two main abstract methods:

run(self, target: str, kubecli: KrknTelemetryOpenshift, error_queue: queue.Queue = None) is the entrypoint for each Network Chaos module. If the module is configured to be run in parallel error_queue must not be None
- target: param is the name of the resource (Pod, Node etc.) that will be targeted by the scenario
- kubecli: the KrknTelemetryOpenshift needed by the scenario to access to the krkn-lib methods
- error_queue: a queue that will be used by the plugin to push the errors raised during the execution of parallel modules
get_config(self) -> (NetworkChaosScenarioType, BaseNetworkChaosConfig) returns the common subset of settings shared by all the scenarios BaseNetworkChaosConfig and the type of Network Chaos Scenario that is running (Pod Scenario or Node Scenario)

`BaseNetworkChaosConfig` base module configuration

Is the base class that contains the common parameters shared by all the Network Chaos NG modules.

Supported Cloud Providers

Thu, 05 Jan 2017 00:00:00 +0000

AWS

NOTE: For clusters with AWS make sure AWS CLI is installed and properly configured using an AWS account. This should set a configuration file at $HOME/.aws/config for your the AWS account. If you have multiple profiles configured on AWS, you can change the profile by setting export AWS_DEFAULT_PROFILE=<profile-name>

export AWS_DEFAULT_REGION=<aws-region>

This configuration will work for self managed AWS, ROSA and Rosa-HCP

Adding scenarios via plugin api

Mon, 01 Jan 0001 00:00:00 +0000

Scenario Plugin API:

This API enables seamless integration of Scenario Plugins for Krkn. Plugins are automatically detected and loaded by the plugin loader, provided they extend the AbstractPluginScenario abstract class, implement the required methods, and adhere to the specified naming conventions.

Plugin folder:

The plugin loader automatically loads plugins found in the krkn/scenario_plugins directory, relative to the Krkn root folder. Each plugin must reside in its own directory and can consist of one or more Python files. The entry point for each plugin is a Python class that extends the AbstractPluginScenario abstract class and implements its required methods.

Evolutionary Algorithm

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI uses an online learning approach by leveraging an evolutionary algorithm, where an agent runs tests on the actual cluster and gathers feedback by measuring various KPIs for your cluster and application. The algorithm begins by creating random population samples that contain Chaos scenarios. These scenarios are executed on the cluster, feedback is collected, and then the best samples (parents) are selected to undergo crossover and mutation operations to generate the next set of samples (offspring). The algorithm relies on heuristics to guide the exploration and exploitation of scenarios.

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started with Krkn-AI

This documentation details how to deploy a sample microservice application on Kubernetes Cluster and run Krkn-AI test.

Prerequisites

Follow this guide to install Krkn-AI CLI.
Krkn-AI uses Thanos Querier to fetch SLO metrics by PromQL. You can easily install it by setting up prometheus-operator in your cluster.

Deploy Sample Microservice

For demonstration purpose, we will deploy a sample microservice called robot-shop on the cluster:

Krkn Config Explanations

Mon, 01 Jan 0001 00:00:00 +0000

Config

Set the scenarios to inject and the tunings like duration to wait between each scenario in the config file located at config/config.yaml.

NOTE: config can be used if leveraging the automated way to install the infrastructure pieces.

Config components:

Kraken

This section defines scenarios and specific data to the chaos run

Distribution

The distribution is now automatically set based on some verification points. Depending on which distribution, either openshift or kubernetes other parameters will be automatically set. The prometheus url/route and bearer token are automatically obtained in case of OpenShift, please be sure to set it when the distribution is Kubernetes.

Usage

Mon, 01 Jan 0001 00:00:00 +0000

Commands:

Commands are grouped by action and may include one or more subcommands to further define the specific action.

`list <subcommand>`:

`available`:

Builds a list of all the available scenarios in krkn-hub

% krknctl list available

Name	Size	Digest	Last Modified
network-chaos	**	sha256:**	2025-01-01 00:00:00+0000 +0000
service-disruption-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
node-memory-hog	**	sha256:**	2025-01-01 00:00:00+0000 +0000
application-outages	**	sha256:**	2025-01-01 00:00:00+0000 +0000
node-cpu-hog	**	sha256:**	2025-01-01 00:00:00+0000 +0000
time-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
node-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
service-hijacking	**	sha256:**	2025-01-01 00:00:00+0000 +0000
pvc-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
chaos-recommender	**	sha256:**	2025-01-01 00:00:00+0000 +0000
syn-flood	**	sha256:**	2025-01-01 00:00:00+0000 +0000
container-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
pod-network-chaos	**	sha256:**	2025-01-01 00:00:00+0000 +0000
pod-scenarios	**	sha256:**	2025-01-01 00:00:00+0000 +0000
node-io-hog	**	sha256:**	2025-01-01 00:00:00+0000 +0000
power-outages	**	sha256:**	2025-01-01 00:00:00+0000 +0000
zone-outages	**	sha256:**	2025-01-01 00:00:00+0000 +0000
dummy-scenario	**	sha256:**	2025-01-01 00:00:00+0000 +0000

running:

Builds a list of all the scenarios currently running in the system. The scenarios are filtered based on the tool’s naming conventions.

Config

Thu, 05 Jan 2017 00:00:00 +0000

Cerberus Config Components Explained

Config

Set the components to monitor and the tunings like duration to wait between each check in the config file located at config/config.yaml. A sample config looks like:

cerberus:
 distribution: openshift  # Distribution can be kubernetes or openshift
 kubeconfig_path: /root/.kube/config  # Path to kubeconfig
 port: 8081 # http server port where cerberus status is published
 watch_nodes: True # Set to True for the cerberus to monitor the cluster nodes
 watch_cluster_operators: True # Set to True for cerberus to monitor cluster operators
 watch_terminating_namespaces: True # Set to True to monitor if any namespaces (set below under 'watch_namespaces' start terminating
 watch_url_routes:
 # Route url's you want to monitor, this is a double array with the url and optional authorization parameter
 watch_master_schedulable: # When enabled checks for the schedulable master nodes with given label.
 enabled: True
 label: node-role.kubernetes.io/master
 watch_namespaces: # List of namespaces to be monitored
 -  openshift-etcd
 -  openshift-apiserver
 -  openshift-kube-apiserver
 -  openshift-monitoring
 -  openshift-kube-controller-manager
 -  openshift-machine-api
 -  openshift-kube-scheduler
 -  openshift-ingress
 -  openshift-sdn  # When enabled, it will check for the cluster sdn and monitor that namespace
 watch_namespaces_ignore_pattern: [] # Ignores pods matching the regex pattern in the namespaces specified under watch_namespaces
 cerberus_publish_status: True # When enabled, cerberus starts a light weight http server and publishes the status
 inspect_components: False # Enable it only when OpenShift client is supported to run
 # When enabled, cerberus collects logs, events and metrics of failed components

 prometheus_url: # The prometheus url/route is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes.
 prometheus_bearer_token: # The bearer token is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes. This is needed to authenticate with prometheus.
 # This enables Cerberus to query prometheus and alert on observing high Kube API Server latencies.

 slack_integration: False # When enabled, cerberus reports the failed iterations in the slack channel
 # The following env vars needs to be set: SLACK_API_TOKEN ( Bot User OAuth Access Token ) and SLACK_CHANNEL ( channel to send notifications in case of failures )
 # When slack_integration is enabled, a watcher can be assigned for each day. The watcher of the day is tagged while reporting failures in the slack channel. Values are slack member ID's.
 watcher_slack_ID: # (NOTE: Defining the watcher id's is optional and when the watcher slack id's are not defined, the slack_team_alias tag is used if it is set else no tag is used while reporting failures in the slack channel.)
 Monday:
 Tuesday:
 Wednesday:
 Thursday:
 Friday:
 Saturday:
 Sunday:
 slack_team_alias: # The slack team alias to be tagged while reporting failures in the slack channel when no watcher is assigned

 custom_checks:
 -  custom_checks/custom_check_sample.py  # Relative paths of files containing additional user defined checks

tunings:
 timeout: 20 # Number of seconds before requests fail
 iterations: 1 # Iterations to loop before stopping the watch, it will be replaced with infinity when the daemon mode is enabled
 sleep_time: 3 # Sleep duration between each iteration
 kube_api_request_chunk_size: 250 # Large requests will be broken into the specified chunk size to reduce the load on API server and improve responsiveness.
 daemon_mode: True # Iterations are set to infinity which means that the cerberus will monitor the resources forever
 cores_usage_percentage: 0.5 # Set the fraction of cores to be used for multiprocessing

database:
 database_path: /tmp/cerberus.db  # Path where cerberus database needs to be stored
 reuse_database: False # When enabled, the database is reused to store the failures

Watch Nodes

This flag returns any nodes where the KernelDeadlock is not set to False and does not have a Ready status

krkn-hub

Thu, 05 Jan 2017 00:00:00 +0000

Hosts container images and wrapper for running scenarios supported by Krkn, a chaos testing tool for Kubernetes clusters to ensure it is resilient to failures. All we need to do is run the containers with the respective environment variables defined as supported by the scenarios without having to maintain and tweak files!

Set Up

You can use docker or podman to run kraken-hub

Node Scenarios on Bare Metal using Krkn-Hub

Thu, 05 Jan 2017 00:00:00 +0000

Node Scenarios Bare Metal

This scenario disrupts the node(s) matching the label on a bare metal Kubernetes/OpenShift cluster. Actions/disruptions supported are listed here

Run

Unlike other krkn-hub scenarios, this one requires a specific configuration due to its unique structure. You must set up the scenario in a local file following the scenario syntax, and then pass this file’s base64-encoded content to the container via the SCENARIO_BASE64 variable.

Adding New Scenario to Krkn-hub

Mon, 01 Jan 0001 00:00:00 +0000

Adding/Editing a New Scenario to Krkn-hub

Create folder with scenario name under krkn-hub
Create generic scenario template with environment variables

a. See scenario.yaml for example

b. Almost all parameters should be set using a variable (these will be set in the env.sh file or through the command line environment variables)
Add defaults for any environment variables in an “env.sh” file

a. See env.sh for example
Create script to run.sh chaos scenario a. See run.sh for example

Cluster Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI uses a genetic algorithm to generate Chaos scenarios. These scenarios require information about the components available in the cluster, which is obtained from the cluster_components YAML field of the Krkn-AI configuration.

CLI Usage

$ uv run krkn_ai discover --help
Usage: krkn_ai discover [OPTIONS]

 Discover components for Krkn-AI tests

Options:
 -k, --kubeconfig TEXT Path to cluster kubeconfig file.
 -o, --output TEXT Path to save config file.
 -n, --namespace TEXT Namespace(s) to discover components in. Supports
 Regex and comma separated values.
 -pl, --pod-label TEXT Pod Label Keys(s) to filter. Supports Regex and
 comma separated values.
 -nl, --node-label TEXT Node Label Keys(s) to filter. Supports Regex and
 comma separated values.
 -v, --verbose Increase verbosity of output.
 --skip-pod-name TEXT Pod name to skip. Supports comma separated values
 with regex.
 --help Show this message and exit.

Example

The example below filters cluster components from namespaces that match the patterns robot-.* and etcd. In addition to namespaces, we also provide filters for pod labels and node labels. This allows us to narrow down the necessary components to consider when running a Krkn-AI test.

Fitness Function

Mon, 01 Jan 0001 00:00:00 +0000

The fitness function is a crucial element in the Krkn-AI algorithm. It evaluates each Chaos experiment and generates a score. These scores are then used during the selection phase of the algorithm to identify the best candidate solutions in each generation.

The fitness function can be defined as an SLO or as cluster metrics using a Prometheus query.
Fitness scores are calculated for the time range during which the Chaos scenario is executed.

Example

Let’s look at a simple fitness function that calculates the total number of restarts in a namespace:

Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

Health Checks

Health checks provide real-time visibility into the impact of chaos scenarios on application availability and performance. They run continuously in the background throughout the chaos run, detecting outages, measuring downtime duration, and recording results in telemetry.

Krkn supports multiple health check types through a plugin-based architecture:

http_health_check — monitors HTTP/HTTPS endpoints (documented on this page)
virt_health_check — monitors KubeVirt VMI SSH connectivity (see Kube Virt Checks)
Custom plugins — extend the system with your own health check logic (see Health Check Plugins)

Health checks are configured in the config.yaml under the health_checks key.

Krkn RBAC

Mon, 01 Jan 0001 00:00:00 +0000

RBAC Configurations

Krkn supports two types of RBAC configurations:

Ns-Privileged RBAC: Provides namespace-scoped permissions for scenarios that only require access to resources within a specific namespace.
Privileged RBAC: Provides cluster-wide permissions for scenarios that require access to cluster-level resources like nodes.

INFO

The examples below use placeholders such as target-namespace and krkn-namespace which should be replaced with your actual namespaces. The service account name krkn-sa is also a placeholder that you can customize.

RBAC YAML Files

Ns-Privileged Role

The ns-privileged role provides permissions limited to namespace-scoped resources:

Krkn Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Krkn Roadmap

Following are a list of enhancements that we are planning to work on adding support in Krkn. Of course any help/contributions are greatly appreciated.

Ability to run multiple chaos scenarios in parallel under load to mimic real world outages
Centralized storage for chaos experiments artifacts
Support for causing DNS outages
Chaos recommender to suggest scenarios having probability of impacting the service under test using profiling results
Chaos AI integration to improve test coverage while reducing fault space to save costs and execution time krkn-chaos-ai
Support for pod level network traffic shaping
Ability to visualize the metrics that are being captured by Kraken and stored in Elasticsearch
Support for running all the scenarios of Kraken on Kubernetes distribution - see https://github.com/krkn-chaos/krkn/issues/185, https://github.com/redhat-chaos/krkn/issues/186
Continue to improve Chaos Testing Guide in terms of adding best practices, test environment recommendations and scenarios to make sure the OpenShift platform, as well the applications running on top it, are resilient and performant under chaotic conditions.
Switch documentation references to Kubernetes
OCP and Kubernetes functionalities segregation
Krknctl - client for running Krkn scenarios with ease
AI Chat bot to help get started with Krkn and commands
Ability to roll back cluster to original state if chaos fails
Add recovery time metrics to each scenario for better regression analysis
Add resiliency scoring to chaos scenarios ran on cluster

krknctl

Mon, 01 Jan 0001 00:00:00 +0000

Install krknctl (Recommended)

Use the official install script as the primary installation method:

Install using the official script

curl -fsSL https://raw.githubusercontent.com/krkn-chaos/krknctl/refs/heads/main/install.sh | bash

Verify installation:

krknctl --version

Alternative installation methods

Binary distribution

The krknctl binary is available for download from GitHub releases for supported operating systems and architectures. Extract the tarball and add the binary to your $PATH.

Build from source

Fork and clone the repository

Fork the repository:

Kube Virt Checks

Mon, 01 Jan 0001 00:00:00 +0000

Kube Virt Checks

Virt checks provide real-time visibility into the impact of chaos scenarios on VMI ssh connectivity and performance. Virt checks are configured in the config.yaml here

The system periodically checks the VMI’s in the provided namespace based on the defined interval and records the results in Telemetry. The checks will run continuously from the very beginning of krkn until all scenarios are done and wait durations are complete. The telemetry data includes:

Resiliency Scoring

Mon, 01 Jan 0001 00:00:00 +0000

WARNING

Beta Feature: Resiliency Scoring is currently in Beta. The API, configuration format, and scoring behavior may change in future releases.

If you encounter any issues or unexpected behavior, please open a bug report at github.com/krkn-chaos/krkn/issues. Include your configuration, krkn version, and any relevant logs or output.

Introduction

What is the Resiliency Score? The Resiliency Score is a percentage (0-100%) that represents the health and stability of your Kubernetes cluster during a chaos scenario. It is calculated by evaluating a set of Service Level Objectives (SLOs) against live Prometheus data.

Running a Chaos Scenario with Krkn

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios

Config

Instructions on how to setup the config and all the available options supported can be found at Config.

In all the examples below you’ll replace the scenario_type with the scenario plugin type that can be found in the second column here

Running a Single Scenario

To run a single scenario, you’ll edit the krkn config file and only have 1 item in the list of chaos_scenarios

Setting Up Disconnected Enviornment

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios in a Disconnected Enviornment

Mirror following images on the bastion host

quay.io/krkn-chaos/krkn-hub:node-scenarios-bm - Master/worker node disruptions on baremetal
quay.io/krkn-chaos/krkn-hub:network-chaos - Network disruptions/traffic shaping
quay.io/krkn-chaos/krkn-hub:pod-scenarios - Pod level disruptions and evaluating recovery time/availability
quay.io/krkn-chaos/krkn-hub:syn-flood - Generates substantial amount of traffic/half open connections targeting a service
quay.io/krkn-chaos/krkn-hub:node-cpu-hog - Hogs CPU on the target nodes
quay.io/krkn-chaos/krkn-hub:node-io-hog - Hogs IO on the target nodes
quay.io/krkn-chaos/krkn-hub:node-memory-hog - Hogs memory on the target nodes
quay.io/krkn-chaos/krkn-hub:pvc-scenarios - Fills up a given PersistentVolumeClaim
quay.io/krkn-chaos/krkn-hub:service-hijacking - Simulates fake HTTP response for a service
quay.io/krkn-chaos/krkn-hub:power-outages - Shuts down the cluster and turns it back on
quay.io/krkn-chaos/krkn-hub:container-scenarios - Kills a container via provided kill signal
quay.io/krkn-chaos/krkn-hub:application-outages - Isolates application Ingress/Egress traffic
quay.io/krkn-chaos/krkn-hub:time-scenarios - Tweaks time/date on the nodes
quay.io/krkn-chaos/krkn-hub:pod-network-chaos - Introduces network chaos at pod level
quay.io/krkn-chaos/krkn-hub:node-network-filter - Node ip traffic filtering
quay.io/krkn-chaos/krkn-hub:pod-network-filter - DNS, internal/external service outages

Will also need these mirrored images inside the cluster

Network disruptions - quay.io/krkn-chaos/krkn:tools
Hog scenarios ( CPU, Memory and IO ) - quay.io/krkn-chaos/krkn-hog
SYN flood - quay.io/krkn-chaos/krkn-syn-flood:latest
Pod network filter scenarios - quay.io/krkn-chaos/krkn-network-chaos:latest
Service hijacking scenarios - quay.io/krkn-chaos/krkn-service-hijacking:v0.1.3

How to Mirror

The strategy is simple:

Signaling to Krkn

Mon, 01 Jan 0001 00:00:00 +0000

This functionality allows a user to be able to pause or stop the Krkn run at any time no matter the number of iterations or daemon_mode set in the config.

If publish_kraken_status is set to True in the config, Krkn will start up a connection to a url at a certain port to decide if it should continue running.

By default, it will get posted to http://0.0.0.0:8081/

An example use case for this feature would be coordinating Krkn runs based on the status of the service installation or load on the cluster.

SLO Validation

Mon, 01 Jan 0001 00:00:00 +0000

SLOs validation

Krkn has a few different options that give a Pass/fail based on metrics captured from the cluster is important in addition to checking the health status and recovery. Krkn supports:

Checking for critical alerts post chaos

If enabled, the check runs at the end of each scenario ( post chaos ) and Krkn exits in case critical alerts are firing to allow user to debug. You can enable it in the config:

Stopping Criteria

Mon, 01 Jan 0001 00:00:00 +0000

The stopping criteria framework lets users define when the genetic algorithm should terminate, allowing for more flexible control beyond strictly generation count or time limits. By configuring these parameters, you can ensure the algorithm stops once it achieves a target fitness or if it reaches a state of saturation where no further improvements or discoveries are being made.

Configurations

You can configure the following options under the stopping_criteria section of the Krkn-AI config file. All fields are optional and, with the exception of saturation_threshold, default to disabled (null).

Telemetry

Mon, 01 Jan 0001 00:00:00 +0000

Telemetry Details

We wanted to gather some more insights regarding our Krkn runs that could have been post processed (eg. by a ML model) to have a better understanding about the behavior of the clusters hit by krkn, so we decided to include this as an opt-in feature that, based on the platform (Kubernetes/OCP), is able to gather different type of data and metadata in the time frame of each chaos run. The telemetry service is currently able to gather several scenario and cluster metadata: A json named telemetry.json containing:

What is krkn-hub?

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started

Checkout how to clone the repo and get started using this documentation page

Example Report

Thu, 05 Jan 2017 00:00:00 +0000

2020-03-26 22:05:06,393 [INFO] Starting cerberus
2020-03-26 22:05:06,401 [INFO] Initializing client to talk to the Kubernetes cluster
2020-03-26 22:05:06,434 [INFO] Fetching cluster info
2020-03-26 22:05:06,739 [INFO] Publishing cerberus status at http://0.0.0.0:8080
2020-03-26 22:05:06,753 [INFO] Starting http server at http://0.0.0.0:8080
2020-03-26 22:05:06,753 [INFO] Daemon mode enabled, cerberus will monitor forever
2020-03-26 22:05:06,753 [INFO] Ignoring the iterations set

2020-03-26 22:05:25,104 [INFO] Iteration 4: Node status: True
2020-03-26 22:05:25,133 [INFO] Iteration 4: Etcd member pods status: True
2020-03-26 22:05:25,161 [INFO] Iteration 4: OpenShift apiserver status: True
2020-03-26 22:05:25,546 [INFO] Iteration 4: Kube ApiServer status: True
2020-03-26 22:05:25,717 [INFO] Iteration 4: Monitoring stack status: True
2020-03-26 22:05:25,720 [INFO] Iteration 4: Kube controller status: True
2020-03-26 22:05:25,746 [INFO] Iteration 4: Machine API components status: True
2020-03-26 22:05:25,945 [INFO] Iteration 4: Kube scheduler status: True
2020-03-26 22:05:25,963 [INFO] Iteration 4: OpenShift ingress status: True
2020-03-26 22:05:26,077 [INFO] Iteration 4: OpenShift SDN status: True
2020-03-26 22:05:26,077 [INFO] HTTP requests served: 0
2020-03-26 22:05:26,077 [INFO] Sleeping for the specified duration: 5


2020-03-26 22:05:31,134 [INFO] Iteration 5: Node status: True
2020-03-26 22:05:31,162 [INFO] Iteration 5: Etcd member pods status: True
2020-03-26 22:05:31,190 [INFO] Iteration 5: OpenShift apiserver status: True
127.0.0.1 - - [26/Mar/2020 22:05:31] "GET / HTTP/1.1" 200 -
2020-03-26 22:05:31,588 [INFO] Iteration 5: Kube ApiServer status: True
2020-03-26 22:05:31,759 [INFO] Iteration 5: Monitoring stack status: True
2020-03-26 22:05:31,763 [INFO] Iteration 5: Kube controller status: True
2020-03-26 22:05:31,788 [INFO] Iteration 5: Machine API components status: True
2020-03-26 22:05:31,989 [INFO] Iteration 5: Kube scheduler status: True
2020-03-26 22:05:32,007 [INFO] Iteration 5: OpenShift ingress status: True
2020-03-26 22:05:32,118 [INFO] Iteration 5: OpenShift SDN status: False
2020-03-26 22:05:32,118 [INFO] HTTP requests served: 1
2020-03-26 22:05:32,118 [INFO] Sleeping for the specified duration: 5
+--------------------------------------------------Failed Components--------------------------------------------------+
2020-03-26 22:05:37,123 [INFO] Failed openshift sdn components: ['sdn-xmqhd']

2020-05-23 23:26:43,041 [INFO] ------------------------- Iteration Stats ---------------------------------------------
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_nodes in iteration 1: 0.0996248722076416 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_cluster_operators in iteration 1: 0.3672499656677246 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run watch_namespaces in iteration 1: 1.085144281387329 seconds
2020-05-23 23:26:43,041 [INFO] Time taken to run entire_iteration in iteration 1: 4.107403039932251 seconds
2020-05-23 23:26:43,041 [INFO] ---------------------------------------------------------------------------------------

ManagedCluster Scenarios

Wed, 04 Jan 2017 00:00:00 +0000

ManagedCluster scenarios provide a way to integrate kraken with Open Cluster Management (OCM) and Red Hat Advanced Cluster Management for Kubernetes (ACM).

ManagedCluster scenarios leverage ManifestWorks to inject faults into the ManagedClusters.

The following ManagedCluster chaos scenarios are supported:

managedcluster_start_scenario: Scenario to start the ManagedCluster instance.
managedcluster_stop_scenario: Scenario to stop the ManagedCluster instance.
managedcluster_stop_start_scenario: Scenario to stop and then start the ManagedCluster instance.
start_klusterlet_scenario: Scenario to start the klusterlet of the ManagedCluster instance.
stop_klusterlet_scenario: Scenario to stop the klusterlet of the ManagedCluster instance.
stop_start_klusterlet_scenario: Scenario to stop and start the klusterlet of the ManagedCluster instance.

ManagedCluster scenarios can be injected by placing the ManagedCluster scenarios config files under managedcluster_scenarios option in the Kraken config. Refer to managedcluster_scenarios_example config file.

Adding New Scenario to Krknctl

Mon, 01 Jan 0001 00:00:00 +0000

Adding a New Scenario to Krknctl

For krknctl to find the parameters of the scenario it uses a krknctl input json file. Once this file is added to krkn-hub, krknctl will be able to find it along with the details of how to run the scenario.

Add KrknCtl Input Json

This file adds every environment variable that is set up for krkn-hub to be defined as a flag to the krknctl cli command. There are a number of different type of variables that you can use, each with their own required fields. See below for an example of the different variable types

Application Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

When defining the Chaos Config, you can provide details about your application endpoints. Krkn-AI can access these endpoints during the Chaos experiment to evaluate how the application’s uptime is impacted.

Note

Application endpoints must be accessible from the system where Krkn-AI is running in order to reach the service.

Configuration

The following configuration options are available when defining an application for health checks:

name: Name of the service.
url: Service endpoint; supports parameterization with “$”.
status_code: Expected status code returned when accessing the service.
timeout: Timeout period after which the request is canceled.
interval: How often to check the endpoint.
stop_watcher_on_failure: This setting allows you to stop the health check watcher for an endpoint after it encounters a failure.

Example

health_checks:
 stop_watcher_on_failure: false
 applications:
 - name: cart
 url: "$HOST/cart/add/1/Watson/1"
 status_code: 200
 timeout: 10
 interval: 2
 - name: catalogue
 url: "$HOST/catalogue/categories"
 - name: shipping
 url: "$HOST/shipping/codes"
 - name: payment
 url: "$HOST/payment/health"
 - name: user
 url: "$HOST/user/uniqueid"
 - name: ratings
 url: "$HOST/ratings/api/fetch/Watson"

URL Parameterization

When defining Krkn-AI config files, the URL entry for an application may vary depending on the cluster. To make the URL configuration more manageable, you can specify the values for these parameters at runtime using the --param flag.

Git Help For Contributions

Mon, 01 Jan 0001 00:00:00 +0000

How to contribute

Contributions are always appreciated.

How to:

Pull request

In order to submit a change or a PR, please fork the project and follow instructions:

$ git clone http://github.com/<me>/krkn-hub
$ cd krkn-hub
$ git checkout -b <branch_name>
$ <make change>
$ git add <changes>
$ git commit -a
$ <insert good message>
$ git push

Squash Commits

If there are multiple commits, please rebase/squash multiple commits before creating the PR by following:

Run Krkn-AI

Mon, 01 Jan 0001 00:00:00 +0000

The run command executes automated resilience and chaos testing using Krkn-AI. It initializes a random population samples containing Chaos Experiments based on your Krkn-AI configuration file, then starts the evolutionary algorithm to run the experiments, gather feedback, and continue evolving existing scenarios until stopping criteria is met.

CLI Usage

$ uv run krkn_ai run --help
Usage: krkn_ai run [OPTIONS]

 Run Krkn-AI tests

Options:
 -c, --config TEXT Path to Krkn-AI config file.
 -o, --output TEXT Directory to save results.
 -f, --format [json|yaml] Format of the output file. [default: yaml]
 -r, --runner-type [krknctl|krknhub] Type of chaos engine to use.
 -p, --param TEXT Additional parameters for config file in key=value format.
 -v, --verbose Increase verbosity of output. [default: 0]
 --help Show this message and exit.

Example

The following command runs Krkn-AI with verbose output (-vv), specifies the configuration file (-c), sets the output directory for results (-o), and passes an additional parameter (-p) to override the HOST variable in the config file:

Usage

Thu, 05 Jan 2017 00:00:00 +0000

Config

Set the supported components to monitor and the tunings like number of iterations to monitor and duration to wait between each check in the config file located at config/config.yaml. A sample config looks like:

cerberus:
 distribution: openshift  # Distribution can be kubernetes or openshift
 kubeconfig_path: ~/.kube/config  # Path to kubeconfig
 port: 8080 # http server port where cerberus status is published
 watch_nodes: True # Set to True for the cerberus to monitor the cluster nodes
 watch_cluster_operators: True # Set to True for cerberus to monitor cluster operators. Parameter is optional, will set to True if not specified
 watch_url_routes: # Route url's you want to monitor
 - - https://...
 - Bearer ****  # This parameter is optional, specify authorization need for get call to route
 - - http://...
 watch_master_schedulable: # When enabled checks for the schedulable
 enabled: True master nodes with given label.
 label: node-role.kubernetes.io/master
 watch_namespaces: # List of namespaces to be monitored
 -  openshift-etcd
 -  openshift-apiserver
 -  openshift-kube-apiserver
 -  openshift-monitoring
 -  openshift-kube-controller-manager
 -  openshift-machine-api
 -  openshift-kube-scheduler
 -  openshift-ingress
 -  openshift-sdn
 cerberus_publish_status: True # When enabled, cerberus starts a light weight http server and publishes the status
 inspect_components: False # Enable it only when OpenShift client is supported to run.
 # When enabled, cerberus collects logs, events and metrics of failed components

 prometheus_url: # The prometheus url/route is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes.
 prometheus_bearer_token: # The bearer token is automatically obtained in case of OpenShift, please set it when the distribution is Kubernetes. This is needed to authenticate with prometheus.
 # This enables Cerberus to query prometheus and alert on observing high Kube API Server latencies.

 slack_integration: False # When enabled, cerberus reports status of failed iterations in the slack channel
 # The following env vars need to be set: SLACK_API_TOKEN ( Bot User OAuth Access Token ) and SLACK_CHANNEL ( channel to send notifications in case of failures )
 # When slack_integration is enabled, a watcher can be assigned for each day. The watcher of the day is tagged while reporting failures in the slack channel. Values are slack member ID's.
 watcher_slack_ID: # (NOTE: Defining the watcher id's is optional and when the watcher slack id's are not defined, the slack_team_alias tag is used if it is set else no tag is used while reporting failures in the slack channel.)
 Monday:
 Tuesday:
 Wednesday:
 Thursday:
 Friday:
 Saturday:
 Sunday:
 slack_team_alias: # The slack team alias to be tagged while reporting failures in the slack channel when no watcher is assigned

 custom_checks: # Relative paths of files containing additional user defined checks
 -  custom_checks/custom_check_sample.py
 -  custom_check.py

tunings:
 iterations: 5 # Iterations to loop before stopping the watch, it will be replaced with infinity when the daemon mode is enabled
 sleep_time: 60 # Sleep duration between each iteration
 kube_api_request_chunk_size: 250 # Large requests will be broken into the specified chunk size to reduce the load on API server and improve responsiveness.
 daemon_mode: True # Iterations are set to infinity which means that the cerberus will monitor the resources forever
 cores_usage_percentage: 0.5 # Set the fraction of cores to be used for multiprocessing

database:
 database_path: /tmp/cerberus.db  # Path where cerberus database needs to be stored
 reuse_database: False # When enabled, the database is reused to store the failures

Note

watch_namespaces support regex patterns. Any valid regex pattern can be used to watch all the namespaces matching the regex pattern. For example, ^openshift-.*$ can be used to watch all namespaces that start with openshift- or openshift can be used to watch all namespaces that have openshift in it.

Note

The current implementation can monitor only one cluster from one host. It can be used to monitor multiple clusters provided multiple instances of Cerberus are launched on different hosts.

Note

The components especially the namespaces needs to be changed depending on the distribution i.e Kubernetes or OpenShift. The default specified in the config assumes that the distribution is OpenShift. A config file for Kubernetes is located at config/kubernetes_config.yaml

Adding to Krkn Test Suite

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers how to add both unit tests and functional tests to the krkn project. Tests are essential for ensuring code quality and preventing regressions.

Unit Tests

Unit tests in krkn are located in the tests/ directory and use Python’s unittest framework with comprehensive mocking to avoid requiring external dependencies like cloud providers or Kubernetes clusters.

Creating a Unit Test

1. File Location and Naming

Place your test file in the tests/ directory with the naming pattern test_<feature>.py:

Krkn-AI

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Prerequisites

Python 3.11+
Podman or Docker Container Runtime
krknctl
uv package manager (recommended) or pip

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to Krkn-AI in anyway in the future we recommend forking the repository first before cloning.

$ git clone https://github.com/krkn-chaos/krkn-ai.git
$ cd krkn-ai

Fork and Clone the Repository

Fork the repository

Run Krkn-AI (Container)

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI can be run inside containers, which simplifies integration with continuous testing workflows.

Container Image

A pre-built container image is available on Quay.io:

podman pull quay.io/krkn-chaos/krkn-ai:latest

Running the Container

The container supports two modes controlled by the MODE environment variable:

1. Discovery Mode

Discovers cluster components and generates a configuration file.

Usage:

# create a folder
mkdir -p ./tmp/container/

# copy kubeconfig to ./tmp/container

# execute discover command
podman run --rm \
 --net="host" \
 -v ./tmp/container:/mount:Z \
 -e MODE="discover" \
 -e KUBECONFIG="/mount/kubeconfig.yaml" \
 -e OUTPUT_DIR="/mount" \
 -e NAMESPACE="robot-shop" \
 -e POD_LABEL="service" \
 -e NODE_LABEL="kubernetes.io/hostname" \
 -e SKIP_POD_NAME="nginx-proxy.*" \
 -e VERBOSE="2" \
 quay.io/krkn-chaos/krkn-ai:latest

Environment Variables (Discovery):

Scenarios

Mon, 01 Jan 0001 00:00:00 +0000

The following Krkn scenarios are currently supported by Kkrn-AI.

At least one scenario must be enabled for the Kkrn-AI experiment to run.

Scenario	Kkrn-AI Config (YAML)
Pod Scenario	scenario.pod-scenarios
Application Outages	scenario.application-outages
Container Scenario	scenario.container-scenarios
Node CPU Hog	scenario.node-cpu-hog
Node Memory Hog	scenario.node-memory-hog
Node IO Hog	scenario.node-io-hog
Syn Flood	scenario.syn-flood
Time Scenario	scenario.time-scenarios
Network Scenarios	scenario.network-scenarios
DNS Outage	scenario.dns-outage
PVC Scenario	scenario.pvc-scenarios

By default, scenarios are not enabled. Depending on your use case, you can enable or disable these scenarios in the krkn-ai.yaml config file by setting the enable field to true or false.

Testing your changes

Mon, 01 Jan 0001 00:00:00 +0000

This page gives details about how you can get a kind cluster configured to be able to run on krkn-lib (the lowest level of krkn-chaos repos) up through krknctl (our easiest way to run and highest level repo)

Configure Kind Testing Environment

Install kind
Create cluster using kind-config.yml under krkn-lib base folder

Health Check Plugins

Mon, 01 Jan 2024 00:00:00 +0000

Health Check Plugins

Health check plugins allow Krkn to continuously monitor the health of your services and infrastructure during chaos experiments. They run in background threads alongside the chaos scenario, detecting outages, tracking downtime duration, and collecting telemetry data.

Overview

The health check system uses a plugin architecture:

HealthCheckFactory — automatically discovers and loads all plugins from the krkn.health_checks package
AbstractHealthCheckPlugin — base class all plugins must extend
Plugins run in separate threads and write telemetry to a shared queue
The factory tracks all active plugin instances and provides lifecycle management (increment_all_iterations, stop_all)

Exit Codes

Code	Meaning
`0`	Success — all health checks passed
`2`	Critical alert detected during the run
`3`	Health check failure (e.g. `exit_on_failure: true` triggered)

Built-in Plugins

HTTP Health Check (`health_checks`)

Monitors HTTP endpoints by making periodic GET requests. Tracks status changes, measures downtime duration, and records telemetry for each state transition.

Alerts

Thu, 05 Jan 2017 00:00:00 +0000

Cerberus consumes the metrics from Prometheus deployed on the cluster to report the alerts.

When provided the prometheus url and bearer token in the config, Cerberus reports the following alerts:

KubeAPILatencyHigh: alerts at the end of each iteration and warns if 99th percentile latency for given requests to the kube-apiserver is above 1 second. It is the official SLI/SLO defined for Kubernetes.
High number of etcd leader changes: alerts the user when an increase in etcd leader changes are observed on the cluster. Frequent elections may be a sign of insufficient resources, high network latency, or disruptions by other components and should be investigated.

Krkn Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

The Krkn Dashboard is a web UI for running and observing Krkn chaos scenarios. You can run it locally (Node.js on your machine) or containerized (Podman/Docker).

Prerequisites (both methods)

Kubernetes cluster — You need a cluster and a kubeconfig so that the dashboard can target it for chaos runs. If you don’t have one, see Kubernetes, minikube, K3s, or OpenShift.
Podman or Docker — The dashboard starts chaos runs by launching krkn-hub containers; the host must have Podman (or Docker) installed and available.

Local installation

Run the dashboard on your machine with Node.js.

Output

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI generates various output files during the execution of chaos experiments, including scenario YAML files, graph visualizations, and log files. By default, these files follow a standard naming convention, but you can customize the file names using format strings in the configuration file.

Available Parameters

The output section in your krkn-ai.yaml configuration file allows you to customize the naming format for different output file types:

`result_name_fmt`

Specifies the naming format for scenario result YAML files. These files contain the complete scenario configuration and execution results for each generated scenario.

Elastic Search

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI supports integration with Elasticsearch to store scenario configurations, run results, and metrics. This allows you to centralize and query experiment data using Elasticsearch’s search and visualization capabilities (e.g., with Kibana).

Configuration Parameters

enable (bool): Set to true to enable saving results to Elasticsearch. Default: false.
server (string): URL or address of your Elasticsearch server (e.g., http://localhost).
port (int): Port to connect to Elasticsearch (default: 9200).
username (string): Username for Elasticsearch authentication (can reference environment variables).
password (string): Password for Elasticsearch authentication. If using environment substitution, prefix with __ to treat as private.
verify_certs (bool): Set to true to verify SSL certificates. Default: true.
index (string): Name prefix for the Elasticsearch index where Krkn-AI results will be stored (e.g., krkn-ai).

Example Configuration

elastic:
 enable: true # Enable Elasticsearch integration
 server: "http://localhost" # Elasticsearch server URL
 port: 9200 # Elasticsearch port
 username: "$ES_USER" # Username (environment substitution supported)
 password: "$__ES_PASSWORD" # Password (start with __ for sensitive/private handling)
 verify_certs: true # Verify SSL certificates
 index: "krkn-ai" # Index prefix for storing results

In addition to the standard Krkn telemetry and metrics indices, Krkn-AI creates two dedicated Elasticsearch indices to store detailed run information:

Performance Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Performance dashboards

Krkn-chaos repository collection gives you a way to install a grafana on the cluster with dashboards loaded to help with monitoring the cluster for things like resource usage to find the outliers, API stats, Etcd health, Critical alerts etc. As well as specific metrics related to your krkn runs if elasticsearch is enabled.

https://github.com/krkn-chaos/visualize

git clone https://github.com/krkn-chaos/visualize
cd krkn-visualize

./deploy.sh 

OR 

./deploy.sh -c oc # openshift installation

The dashboards can be edited/modified to include your queries of interest by logging in as the admin user. New dasbhoards can be imported using the import script and following directions defined here

Node Problem Detector

Thu, 05 Jan 2017 00:00:00 +0000

node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack.

Installation

Please follow the instructions in the installation section to setup Node Problem Detector on Kubernetes. The following instructions are setting it up on OpenShift:

Create openshift-node-problem-detector namespace ns.yaml with oc create -f ns.yaml
Add cluster role with oc adm policy add-cluster-role-to-user system:node-problem-detector -z default -n openshift-node-problem-detector
Add security context constraints with oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-node-problem-detector:default
Edit node-problem-detector.yaml to fit your environment.
Edit node-problem-detector-config.yaml to configure node-problem-detector.
Create the ConfigMap with oc create -f node-problem-detector-config.yaml
Create the DaemonSet with oc create -f node-problem-detector.yaml

Once installed you will see node-problem-detector pods in openshift-node-problem-detector namespace. Now enable openshift-node-problem-detector in the config.yaml. Cerberus just monitors KernelDeadlock condition provided by the node problem detector as it is system critical and can hinder node performance.

Slack Integration

Thu, 05 Jan 2017 00:00:00 +0000

The user has the option to enable/disable the slack integration ( disabled by default ). To use the slack integration, the user has to first create an app and add a bot to it on slack. SLACK_API_TOKEN and SLACK_CHANNEL environment variables have to be set. SLACK_API_TOKEN refers to Bot User OAuth Access Token and SLACK_CHANNEL refers to the slack channel ID the user wishes to receive the notifications. Make sure the Slack Bot Token Scopes contains this permission [calls:read] [channels:read] [chat:write] [groups:read] [im:read] [mpim:read]

Chaos Recommendation Tool

Mon, 01 Jan 0001 00:00:00 +0000

This tool, designed for Kraken, operates through the command line and offers recommendations for chaos testing. It suggests probable chaos test cases that can disrupt application services by analyzing their behavior and assessing their susceptibility to specific fault types.

This tool profiles an application and gathers telemetry data such as CPU, Memory, and Network usage, analyzing it to suggest probable chaos scenarios. For optimal results, it is recommended to activate the utility while the application is under load.

Krkn Debugging Tips

Mon, 01 Jan 0001 00:00:00 +0000

Common Debugging Issues

SSL Certification

Error

...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.***.io', port=6443): Max retries exceeded with url: /apis/config.openshift.io/v1/clusterversions (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1147)')))

Fix

The user needs to have tls verification by logging in using

$ oc login [-u=<username>] \
 [-p=<password>] \
 [-s=<server>] \
 [-n=<project>] \
 --insecure-skip-tls-verify

Also verify insecure-skip-tls-verify: true is in the kubeconfig:

Using the UI

Mon, 01 Jan 0001 00:00:00 +0000

Using the UI

Once the dashboard is running, open http://localhost:3000 (or the port shown in the terminal) in your browser. The dashboard has a side menu with other dashboard views (Overview and Metrics). Each is described below.

Overview

The Overview page is the default landing page. It has two tabs at the top:

Kraken tab

Scenarios card — A set of scenario tiles (e.g. Pod Scenarios, Node CPU hog, Node IO hog, Node Memory hog). Click a scenario to select it for the next run.
Supported Parameters — Set your parameters for the selected scenario and either enter a kubeconfig path or upload a kubeconfig file. Use Start Kraken to launch the krkn-hub container for that scenario.
Pod Details — A table of all krkn-hub containers known to the dashboard. Use this to see which chaos runs are active or finished.

Logs tab

Logs viewer — A dropdown to select a running or past container (from the same list as Pod Details). Once selected, the panel shows that container’s live or captured logs so you can watch chaos output without using the terminal.

Contribute

Thu, 05 Jan 2017 00:00:00 +0000

How to contribute

Contributions are always appreciated.

How to:

Pull request

In order to submit a change or a PR, please fork the project and follow instructions:

$ git clone http://github.com/<me>/cerberus
$ cd cerberus
$ git checkout -b <branch_name>
$ <make change>
$ git add <changes>
$ git commit -a
$ <insert good message>
$ git push

Fix Formatting

Cerberus uses pre-commit framework to maintain the code linting and python code styling. The CI would run the pre-commit check on each pull request. We encourage our contributors to follow the same pattern, while contributing to the code.

Security self-assessment

Mon, 01 Jan 0001 00:00:00 +0000

Krkn Security Self-Assessment

Security reviewers: Tullio Sebastiani, Paige Patton, Naga Ravi Elluri

This document is intended to aid in roadmapping, and the onboarding of new maintainers.

Metadata


Software	https://github.com/krkn-chaos/krkn https://github.com/krkn-chaos/krkn-lib https://github.com/krkn-chaos/krkn-hub https://github.com/krkn-chaos/krknctl
Security Provider?	No. Krkn is designed to measure application or cluster resilience injecting controlled resource disruptions, but it should not be considered a security provider.
Languages	Python, Go, Bash
Software Bill of Materials	🔹 Krkn : https://github.com/krkn-chaos/krkn/releases/latest/download/sbom.json 🔹 krkn-lib : https://github.com/krkn-chaos/krkn-lib/releases/latest/download/sbom.json 🔹 krknctl : https://github.com/krkn-chaos/krknctl/releases/latest/download/sbom.json
Security Links	Known Weakness. Creation of a security-insights.yml should be added to the roadmap.

Overview

krkn is a chaos and resiliency testing tool for Kubernetes. Krkn injects deliberate failures into Kubernetes clusters to check if it is resilient to turbulent conditions.

Randomized chaos testing

Mon, 01 Jan 0001 00:00:00 +0000

The random subcommand is valuable for generating chaos tests on a large scale with ease and speed. The random scaffold command, when used with the --seed-file and --number-of-scenarios flags, allows you to expand a pre-existing random or graph plan as a template (or seed). The tool randomly distributes scenarios from the seed-file to meet the specified number-of-scenarios. The resulting output is compatible exclusively with the random run command, which generates a random graph from it.

Mon, 01 Jan 0001 00:00:00 +0000

ManagedCluster Scenarios

ManagedCluster scenarios provide a way to integrate kraken with Open Cluster Management (OCM) and Red Hat Advanced Cluster Management for Kubernetes (ACM).

ManagedCluster scenarios leverage ManifestWorks to inject faults into the ManagedClusters.

The following ManagedCluster chaos scenarios are supported:

managedcluster_start_scenario: Scenario to start the ManagedCluster instance.
managedcluster_stop_scenario: Scenario to stop the ManagedCluster instance.
managedcluster_stop_start_scenario: Scenario to stop and then start the ManagedCluster instance.
start_klusterlet_scenario: Scenario to start the klusterlet of the ManagedCluster instance.
stop_klusterlet_scenario: Scenario to stop the klusterlet of the ManagedCluster instance.
stop_start_klusterlet_scenario: Scenario to stop and start the klusterlet of the ManagedCluster instance.

Articles

Mon, 01 Jan 0001 00:00:00 +0000

Presentations and Blogs

Blog post on introduction to Kraken: https://www.openshift.com/blog/introduction-to-kraken-a-chaos-tool-for-openshift/kubernetes
Discussion and demo on how Kraken can be leveraged to ensure OpenShift is reliable, performant and scalable: https://www.youtube.com/watch?v=s1PvupI5sD0&ab_channel=OpenShift
Blog post emphasizing the importance of making Chaos part of Performance and Scale runs to mimic the production environments: https://www.openshift.com/blog/making-chaos-part-of-kubernetes/openshift-performance-and-scalability-tests
Blog post on findings from Chaos test runs: https://cloud.redhat.com/blog/openshift/kubernetes-chaos-stories
Discussion with CNCF TAG App Delivery on Krkn workflow, features and addition to CNCF sandbox: Github, Tracker, recording
Blog post on supercharging chaos testing using AI integration in Krkn: https://www.redhat.com/en/blog/supercharging-chaos-testing-using-ai
Blog post announcing Krkn joining CNCF Sandbox: https://www.redhat.com/en/blog/krknchaos-joining-cncf-sandbox
Blog post on OpenShift Chaos Stories - findings from test runs: https://cloud.redhat.com/blog/openshift/kubernetes-chaos-stories
Blog post on unleashing controlled chaos with krknctl: https://developers.redhat.com/articles/2025/08/21/unleash-controlled-chaos-krknctl
Blog post on enhancing the system resilience with krkn chaos dashboard: https://developers.redhat.com/articles/2025/08/14/enhancing-system-resilience-krkn-chaos-dashboard
Blog post on LFX Mentorship Term 2 2025 experience working with the krkn team on the rollback feature: https://blog.zhu424.dev/open-source-contribution/lfx-mentorship-2025-cncf-krkn
Blog post on creating custom chaos scenarios in krkn: https://medium.com/@abhinavs1920/krkn-custom-chaos-engineering-plugins-70d73c7e2118

Chaos Testing Guide

Guide covering the best practices, recommendations that an OpenShift platform as well as the applications running on top it should follow to be resilient, performant and reliable in addition to instructions on how to leverage tooling to achieve it: https://redhat-chaos.github.io/krkn/.

Search Results

Mon, 01 Jan 0001 00:00:00 +0000

Krkn - Chaos and Resiliency Testing for Kubernetes on Krkn

Installation

Note

Git

Install the dependencies

Krkn

Installation

Clone the Repository

Fork and Clone the Repository

Krkn-Hub All Scenarios Variables

Supported parameters for all scenarios in Krkn-Hub

Note

Krkn-lib

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

Contents

Krknctl All Scenarios Variables

Supported parameters for all scenarios in KrknCtl

Note

Network Chaos API

AbstractNetworkChaosModule abstract module class

BaseNetworkChaosConfig base module configuration

Supported Cloud Providers

AWS

Adding scenarios via plugin api

Scenario Plugin API:

Plugin folder:

Evolutionary Algorithm

Getting Started

Getting Started with Krkn-AI

Prerequisites

Deploy Sample Microservice

Krkn Config Explanations

Config

Kraken

Distribution

Usage

Commands:

list <subcommand>:

available:

running:

Config

Config

Watch Nodes

krkn-hub

Set Up

Node Scenarios on Bare Metal using Krkn-Hub

Node Scenarios Bare Metal

Run

Adding New Scenario to Krkn-hub

Adding/Editing a New Scenario to Krkn-hub

Cluster Discovery

CLI Usage

Example

Fitness Function

Example

Health Checks

Health Checks

Krkn RBAC

RBAC Configurations

RBAC YAML Files

Ns-Privileged Role

Krkn Roadmap

Krkn Roadmap

krknctl

Install krknctl (Recommended)

Alternative installation methods

Binary distribution

Build from source

Fork and clone the repository

Kube Virt Checks

Kube Virt Checks

Resiliency Scoring

Introduction

Running a Chaos Scenario with Krkn

Getting Started Running Chaos Scenarios

Config

Running a Single Scenario

Setting Up Disconnected Enviornment

Getting Started Running Chaos Scenarios in a Disconnected Enviornment

`AbstractNetworkChaosModule` abstract module class

`BaseNetworkChaosConfig` base module configuration

`list <subcommand>`:

`available`:

`running`:

HTTP Health Check (`health_checks`)

`result_name_fmt`