Docs on Krkn

Krkn

Thu, 05 Jan 2017 00:00:00 +0000

Installation

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to krkn in anyway in the future we recommend forking the repository first before cloning.

See the latest release version here

$ git clone https://github.com/krkn-chaos/krkn.git --branch <release version>
$ cd krkn

Fork and Clone the Repository

Fork the repository

Krkn-lib

Thu, 05 Jan 2017 00:00:00 +0000

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

The Library contains Classes, Models and helper functions used in Kraken to interact with Kubernetes, Openshift and other external APIS. The goal of this library is to give to developers the building blocks to realize new Chaos Scenarios and to increase the testability and the modularity of the Krkn codebase.

Adding scenarios via plugin api

Mon, 01 Jan 0001 00:00:00 +0000

Scenario Plugin API:

This API enables seamless integration of Scenario Plugins for Krkn. Plugins are automatically detected and loaded by the plugin loader, provided they extend the AbstractPluginScenario abstract class, implement the required methods, and adhere to the specified naming conventions.

Plugin folder:

The plugin loader automatically loads plugins found in the krkn/scenario_plugins directory, relative to the Krkn root folder. Each plugin must reside in its own directory and can consist of one or more Python files. The entry point for each plugin is a Python class that extends the AbstractPluginScenario abstract class and implements its required methods.

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started with Krkn-AI

This documentation details how to deploy a sample microservice application on Kubernetes Cluster and run Krkn-AI test.

Prerequisites

Follow this guide to install Krkn-AI CLI.
Krkn-AI uses Thanos Querier to fetch SLO metrics by PromQL. You can easily install it by setting up prometheus-operator in your cluster.

Deploy Sample Microservice

For demonstration purpose, we will deploy a sample microservice called robot-shop on the cluster:

krkn-hub

Thu, 05 Jan 2017 00:00:00 +0000

Hosts container images and wrapper for running scenarios supported by Krkn, a chaos testing tool for Kubernetes clusters to ensure it is resilient to failures. All we need to do is run the containers with the respective environment variables defined as supported by the scenarios without having to maintain and tweak files!

Set Up

You can use docker or podman to run kraken-hub

Adding New Scenario to Krkn-hub

Mon, 01 Jan 0001 00:00:00 +0000

Adding/Editing a New Scenario to Krkn-hub

Create folder with scenario name under krkn-hub
Create generic scenario template with environment variables

a. See scenario.yaml for example

b. Almost all parameters should be set using a variable (these will be set in the env.sh file or through the command line environment variables)
Add defaults for any environment variables in an “env.sh” file

a. See env.sh for example
Create script to run.sh chaos scenario a. See run.sh for example

Running a Chaos Scenario with Krkn

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios

Config

Instructions on how to setup the config and all the available options supported can be found at Config.

In all the examples below you’ll replace the scenario_type with the scenario plugin type that can be found in the second column here

Running a Single Scenario

To run a single scenario, you’ll edit the krkn config file and only have 1 item in the list of chaos_scenarios

Setting Up Disconnected Enviornment

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started Running Chaos Scenarios in a Disconnected Enviornment

Mirror following images on the bastion host

quay.io/krkn-chaos/krkn-hub:node-scenarios-bm - Master/worker node disruptions on baremetal
quay.io/krkn-chaos/krkn-hub:network-chaos - Network disruptions/traffic shaping
quay.io/krkn-chaos/krkn-hub:pod-scenarios - Pod level disruptions and evaluating recovery time/availability
quay.io/krkn-chaos/krkn-hub:syn-flood - Generates substantial amount of traffic/half open connections targeting a service
quay.io/krkn-chaos/krkn-hub:node-cpu-hog - Hogs CPU on the target nodes
quay.io/krkn-chaos/krkn-hub:node-io-hog - Hogs IO on the target nodes
quay.io/krkn-chaos/krkn-hub:node-memory-hog - Hogs memory on the target nodes
quay.io/krkn-chaos/krkn-hub:pvc-scenarios - Fills up a given PersistentVolumeClaim
quay.io/krkn-chaos/krkn-hub:service-hijacking - Simulates fake HTTP response for a service
quay.io/krkn-chaos/krkn-hub:power-outages - Shuts down the cluster and turns it back on
quay.io/krkn-chaos/krkn-hub:container-scenarios - Kills a container via provided kill signal
quay.io/krkn-chaos/krkn-hub:application-outages - Isolates application Ingress/Egress traffic
quay.io/krkn-chaos/krkn-hub:time-scenarios - Tweaks time/date on the nodes
quay.io/krkn-chaos/krkn-hub:pod-network-chaos - Introduces network chaos at pod level
quay.io/krkn-chaos/krkn-hub:node-network-filter - Node ip traffic filtering
quay.io/krkn-chaos/krkn-hub:pod-network-filter - DNS, internal/external service outages

Will also need these mirrored images inside the cluster

Network disruptions - quay.io/krkn-chaos/krkn:tools
Hog scenarios ( CPU, Memory and IO ) - quay.io/krkn-chaos/krkn-hog
SYN flood - quay.io/krkn-chaos/krkn-syn-flood:latest
Pod network filter scenarios - quay.io/krkn-chaos/krkn-network-chaos:latest
Service hijacking scenarios - quay.io/krkn-chaos/krkn-service-hijacking:v0.1.3

How to Mirror

The strategy is simple:

Adding New Scenario to Krknctl

Mon, 01 Jan 0001 00:00:00 +0000

Adding a New Scenario to Krknctl

For krknctl to find the parameters of the scenario it uses a krknctl input json file. Once this file is added to krkn-hub, krknctl will be able to find it along with the details of how to run the scenario.

Add KrknCtl Input Json

This file adds every environment variable that is set up for krkn-hub to be defined as a flag to the krknctl cli command. There are a number of different type of variables that you can use, each with their own required fields. See below for an example of the different variable types

Git Help For Contributions

Mon, 01 Jan 0001 00:00:00 +0000

How to contribute

Contributions are always appreciated.

How to:

Pull request

In order to submit a change or a PR, please fork the project and follow instructions:

$ git clone http://github.com/<me>/krkn-hub
$ cd krkn-hub
$ git checkout -b <branch_name>
$ <make change>
$ git add <changes>
$ git commit -a
$ <insert good message>
$ git push

Squash Commits

If there are multiple commits, please rebase/squash multiple commits before creating the PR by following:

Adding to Krkn Test Suite

Mon, 01 Jan 0001 00:00:00 +0000

This guide covers how to add both unit tests and functional tests to the krkn project. Tests are essential for ensuring code quality and preventing regressions.

Unit Tests

Unit tests in krkn are located in the tests/ directory and use Python’s unittest framework with comprehensive mocking to avoid requiring external dependencies like cloud providers or Kubernetes clusters.

Creating a Unit Test

1. File Location and Naming

Place your test file in the tests/ directory with the naming pattern test_<feature>.py:

Getting Started with Running Scenarios

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start with krknctl (Recommended)

Recommended Approach

krknctl is the recommended and easiest way to run krkn scenarios. It provides command auto-completion, input validation, and abstracts the complexities of the container environment so you can focus on chaos engineering.

Why krknctl?

krknctl is a dedicated CLI tool that streamlines running chaos scenarios by providing:

Command auto-completion - Quick access to all available commands
Input validation - Catch errors before they happen
Scenario descriptions - Built-in documentation and instructions
Simple workflow - No need to manage config files or containers

Get Started in 3 Steps:

Install krknctl - Follow the installation guide
Explore features - Learn about krknctl usage and how to execute chaos scenarios
Run scenarios - Check out each scenario’s documentation for krknctl examples

Alternative Methods

The following alternative methods are available for advanced use cases:

Installation

Mon, 01 Jan 0001 00:00:00 +0000

Choose Your Installation Method

Krkn provides multiple ways to run chaos scenarios. Choose the method that best fits your needs:

Tool	What is it?	Best For
krknctl	CLI tool with auto-completion	Complex workflow orchestration, querying and running scenarios, ease of use
krkn-hub	Pre-built container images	CI/CD pipelines, automation
krkn Dashboard	Web UI for running scenarios	Visual runs, demos, teams that prefer a GUI
krkn	Standalone Python program	Full control, development, and customization

Recommendation

krknctl is the recommended way to run Krkn. It provides the simplest path to chaos testing with powerful capabilities including complex workflow orchestration, built-in scenario discovery, and interactive query support — all without managing configuration files.

Recommendation
Look for features marked with [BETA] (e.g., [BETA] Krkn Resilience Score). Beta features provide early access to new capabilities for experimentation and feedback and may not yet meet the stability, performance, or compatibility guarantees of GA features. Please refer to the Beta feature policy for more details.

Installation Methods

krknctl (Recommended)

What is it? A dedicated command-line interface (CLI) tool that simplifies running Krkn chaos scenarios while providing powerful orchestration capabilities.

Krkn-AI

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Prerequisites

Python 3.11+
Podman or Docker Container Runtime
krknctl
uv package manager (recommended) or pip

Clone the Repository

To clone and use the latest krkn version follow the directions below. If you’re wanting to contribute back to Krkn-AI in anyway in the future we recommend forking the repository first before cloning.

$ git clone https://github.com/krkn-chaos/krkn-ai.git
$ cd krkn-ai

Fork and Clone the Repository

Fork the repository

Testing your changes

Mon, 01 Jan 0001 00:00:00 +0000

This page gives details about how you can get a kind cluster configured to be able to run on krkn-lib (the lowest level of krkn-chaos repos) up through krknctl (our easiest way to run and highest level repo)

Configure Kind Testing Environment

Install kind
Create cluster using kind-config.yml under krkn-lib base folder

Health Check Plugins

Mon, 01 Jan 2024 00:00:00 +0000

Health Check Plugins

Health check plugins allow Krkn to continuously monitor the health of your services and infrastructure during chaos experiments. They run in background threads alongside the chaos scenario, detecting outages, tracking downtime duration, and collecting telemetry data.

Overview

The health check system uses a plugin architecture:

HealthCheckFactory — automatically discovers and loads all plugins from the krkn.health_checks package
AbstractHealthCheckPlugin — base class all plugins must extend
Plugins run in separate threads and write telemetry to a shared queue
The factory tracks all active plugin instances and provides lifecycle management (increment_all_iterations, stop_all)

Exit Codes

Code	Meaning
`0`	Success — all health checks passed
`2`	Critical alert detected during the run
`3`	Health check failure (e.g. `exit_on_failure: true` triggered)

Built-in Plugins

HTTP Health Check (`health_checks`)

Monitors HTTP endpoints by making periodic GET requests. Tracks status changes, measures downtime duration, and records telemetry for each state transition.

Krkn Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

The Krkn Dashboard is a web UI for running and observing Krkn chaos scenarios. You can run it locally (Node.js on your machine) or containerized (Podman/Docker).

Prerequisites (both methods)

Kubernetes cluster — You need a cluster and a kubeconfig so that the dashboard can target it for chaos runs. If you don’t have one, see Kubernetes, minikube, K3s, or OpenShift.
Podman or Docker — The dashboard starts chaos runs by launching krkn-hub containers; the host must have Podman (or Docker) installed and available.

Local installation

Run the dashboard on your machine with Node.js.

Cerberus

Mon, 01 Jan 0001 00:00:00 +0000

Cerberus

Guardian of Kubernetes and OpenShift Clusters

Cerberus watches the Kubernetes/OpenShift clusters for dead nodes, system component failures/health and exposes a go or no-go signal which can be consumed by other workload generators or applications in the cluster and act accordingly.

Workflow

Installation

Instructions on how to setup, configure and run Cerberus can be found at Installation.

Developers Guide

Mon, 01 Jan 0001 00:00:00 +0000

This document describes how to develop and add to Krkn. Before you start, it is recommended that you read the following documents first:

Be sure to properly install Krkn. Then you can start to develop krkn. The following documents will help you get started:

Add k8s functionality to krkn-lib
Add a New Chaos Scenario using Plugin API: Adding a new scenario into krkn
Add a Health Check Plugin: Monitor services during chaos using the health check plugin system
Test your changes

NOTE: All base kubernetes functionality should be added into krkn-lib and called from krkn

Docs on Krkn

Krkn

Installation

Clone the Repository

Fork and Clone the Repository

Krkn-lib

krkn-lib

Krkn Chaos and resiliency testing tool Foundation Library

Contents

Adding scenarios via plugin api

Scenario Plugin API:

Plugin folder:

Getting Started

Getting Started with Krkn-AI

Prerequisites

Deploy Sample Microservice

krkn-hub

Set Up

Adding New Scenario to Krkn-hub

Adding/Editing a New Scenario to Krkn-hub

Running a Chaos Scenario with Krkn

Getting Started Running Chaos Scenarios

Config

Running a Single Scenario

Setting Up Disconnected Enviornment

Getting Started Running Chaos Scenarios in a Disconnected Enviornment

Mirror following images on the bastion host

Will also need these mirrored images inside the cluster

How to Mirror

Adding New Scenario to Krknctl

Adding a New Scenario to Krknctl

Add KrknCtl Input Json

Git Help For Contributions

How to contribute

Pull request

Squash Commits

Adding to Krkn Test Suite

Unit Tests

Creating a Unit Test

1. File Location and Naming

Getting Started with Running Scenarios

Quick Start with krknctl (Recommended)

Recommended Approach

Why krknctl?

Get Started in 3 Steps:

Alternative Methods

Installation

Choose Your Installation Method

Recommendation

Recommendation

Installation Methods

krknctl (Recommended)

Krkn-AI

Installation

Prerequisites

Clone the Repository

Fork and Clone the Repository

Testing your changes

Configure Kind Testing Environment

Health Check Plugins

Health Check Plugins

Overview

Exit Codes

Built-in Plugins

HTTP Health Check (health_checks)

Krkn Dashboard

Prerequisites (both methods)

Local installation

Cerberus

Cerberus

Workflow

Installation

Developers Guide

HTTP Health Check (`health_checks`)