What is Krkn? on Krkn

Krkn Config Explanations

Mon, 01 Jan 0001 00:00:00 +0000

Config

Set the scenarios to inject and the tunings like duration to wait between each scenario in the config file located at config/config.yaml.

NOTE: config can be used if leveraging the automated way to install the infrastructure pieces.

Config components:

Kraken

This section defines scenarios and specific data to the chaos run

Distribution

The distribution is now automatically set based on some verification points. Depending on which distribution, either openshift or kubernetes other parameters will be automatically set. The prometheus url/route and bearer token are automatically obtained in case of OpenShift, please be sure to set it when the distribution is Kubernetes.

Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

Health Checks

Health checks provide real-time visibility into the impact of chaos scenarios on application availability and performance. They run continuously in the background throughout the chaos run, detecting outages, measuring downtime duration, and recording results in telemetry.

Krkn supports multiple health check types through a plugin-based architecture:

http_health_check — monitors HTTP/HTTPS endpoints (documented on this page)
virt_health_check — monitors KubeVirt VMI SSH connectivity (see Kube Virt Checks)
Custom plugins — extend the system with your own health check logic (see Health Check Plugins)

Health checks are configured in the config.yaml under the health_checks key.

Krkn RBAC

Mon, 01 Jan 0001 00:00:00 +0000

RBAC Configurations

Krkn supports two types of RBAC configurations:

Ns-Privileged RBAC: Provides namespace-scoped permissions for scenarios that only require access to resources within a specific namespace.
Privileged RBAC: Provides cluster-wide permissions for scenarios that require access to cluster-level resources like nodes.

INFO

The examples below use placeholders such as target-namespace and krkn-namespace which should be replaced with your actual namespaces. The service account name krkn-sa is also a placeholder that you can customize.

RBAC YAML Files

Ns-Privileged Role

The ns-privileged role provides permissions limited to namespace-scoped resources:

Krkn Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Krkn Roadmap

Following are a list of enhancements that we are planning to work on adding support in Krkn. Of course any help/contributions are greatly appreciated.

Ability to run multiple chaos scenarios in parallel under load to mimic real world outages
Centralized storage for chaos experiments artifacts
Support for causing DNS outages
Chaos recommender to suggest scenarios having probability of impacting the service under test using profiling results
Chaos AI integration to improve test coverage while reducing fault space to save costs and execution time krkn-chaos-ai
Support for pod level network traffic shaping
Ability to visualize the metrics that are being captured by Kraken and stored in Elasticsearch
Support for running all the scenarios of Kraken on Kubernetes distribution - see https://github.com/krkn-chaos/krkn/issues/185, https://github.com/redhat-chaos/krkn/issues/186
Continue to improve Chaos Testing Guide in terms of adding best practices, test environment recommendations and scenarios to make sure the OpenShift platform, as well the applications running on top it, are resilient and performant under chaotic conditions.
Switch documentation references to Kubernetes
OCP and Kubernetes functionalities segregation
Krknctl - client for running Krkn scenarios with ease
AI Chat bot to help get started with Krkn and commands
Ability to roll back cluster to original state if chaos fails
Add recovery time metrics to each scenario for better regression analysis
Add resiliency scoring to chaos scenarios ran on cluster

Kube Virt Checks

Mon, 01 Jan 0001 00:00:00 +0000

Kube Virt Checks

Virt checks provide real-time visibility into the impact of chaos scenarios on VMI ssh connectivity and performance. Virt checks are configured in the config.yaml here

The system periodically checks the VMI’s in the provided namespace based on the defined interval and records the results in Telemetry. The checks will run continuously from the very beginning of krkn until all scenarios are done and wait durations are complete. The telemetry data includes:

Resiliency Scoring

Mon, 01 Jan 0001 00:00:00 +0000

WARNING

Beta Feature: Resiliency Scoring is currently in Beta. The API, configuration format, and scoring behavior may change in future releases.

If you encounter any issues or unexpected behavior, please open a bug report at github.com/krkn-chaos/krkn/issues. Include your configuration, krkn version, and any relevant logs or output.

Introduction

What is the Resiliency Score? The Resiliency Score is a percentage (0-100%) that represents the health and stability of your Kubernetes cluster during a chaos scenario. It is calculated by evaluating a set of Service Level Objectives (SLOs) against live Prometheus data.

Signaling to Krkn

Mon, 01 Jan 0001 00:00:00 +0000

This functionality allows a user to be able to pause or stop the Krkn run at any time no matter the number of iterations or daemon_mode set in the config.

If publish_kraken_status is set to True in the config, Krkn will start up a connection to a url at a certain port to decide if it should continue running.

By default, it will get posted to http://0.0.0.0:8081/

An example use case for this feature would be coordinating Krkn runs based on the status of the service installation or load on the cluster.

SLO Validation

Mon, 01 Jan 0001 00:00:00 +0000

SLOs validation

Krkn has a few different options that give a Pass/fail based on metrics captured from the cluster is important in addition to checking the health status and recovery. Krkn supports:

Checking for critical alerts post chaos

If enabled, the check runs at the end of each scenario ( post chaos ) and Krkn exits in case critical alerts are firing to allow user to debug. You can enable it in the config:

Telemetry

Mon, 01 Jan 0001 00:00:00 +0000

Telemetry Details

We wanted to gather some more insights regarding our Krkn runs that could have been post processed (eg. by a ML model) to have a better understanding about the behavior of the clusters hit by krkn, so we decided to include this as an opt-in feature that, based on the platform (Kubernetes/OCP), is able to gather different type of data and metadata in the time frame of each chaos run. The telemetry service is currently able to gather several scenario and cluster metadata: A json named telemetry.json containing: