Configuration on Krkn

Evolutionary Algorithm

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI uses an online learning approach by leveraging an evolutionary algorithm, where an agent runs tests on the actual cluster and gathers feedback by measuring various KPIs for your cluster and application. The algorithm begins by creating random population samples that contain Chaos scenarios. These scenarios are executed on the cluster, feedback is collected, and then the best samples (parents) are selected to undergo crossover and mutation operations to generate the next set of samples (offspring). The algorithm relies on heuristics to guide the exploration and exploitation of scenarios.

Fitness Function

Mon, 01 Jan 0001 00:00:00 +0000

The fitness function is a crucial element in the Krkn-AI algorithm. It evaluates each Chaos experiment and generates a score. These scores are then used during the selection phase of the algorithm to identify the best candidate solutions in each generation.

The fitness function can be defined as an SLO or as cluster metrics using a Prometheus query.
Fitness scores are calculated for the time range during which the Chaos scenario is executed.

Example

Let’s look at a simple fitness function that calculates the total number of restarts in a namespace:

Stopping Criteria

Mon, 01 Jan 0001 00:00:00 +0000

The stopping criteria framework lets users define when the genetic algorithm should terminate, allowing for more flexible control beyond strictly generation count or time limits. By configuring these parameters, you can ensure the algorithm stops once it achieves a target fitness or if it reaches a state of saturation where no further improvements or discoveries are being made.

Configurations

You can configure the following options under the stopping_criteria section of the Krkn-AI config file. All fields are optional and, with the exception of saturation_threshold, default to disabled (null).

Application Health Checks

Mon, 01 Jan 0001 00:00:00 +0000

When defining the Chaos Config, you can provide details about your application endpoints. Krkn-AI can access these endpoints during the Chaos experiment to evaluate how the application’s uptime is impacted.

Note

Application endpoints must be accessible from the system where Krkn-AI is running in order to reach the service.

Configuration

The following configuration options are available when defining an application for health checks:

name: Name of the service.
url: Service endpoint; supports parameterization with “$”.
status_code: Expected status code returned when accessing the service.
timeout: Timeout period after which the request is canceled.
interval: How often to check the endpoint.
stop_watcher_on_failure: This setting allows you to stop the health check watcher for an endpoint after it encounters a failure.

Example

health_checks:
 stop_watcher_on_failure: false
 applications:
 - name: cart
 url: "$HOST/cart/add/1/Watson/1"
 status_code: 200
 timeout: 10
 interval: 2
 - name: catalogue
 url: "$HOST/catalogue/categories"
 - name: shipping
 url: "$HOST/shipping/codes"
 - name: payment
 url: "$HOST/payment/health"
 - name: user
 url: "$HOST/user/uniqueid"
 - name: ratings
 url: "$HOST/ratings/api/fetch/Watson"

URL Parameterization

When defining Krkn-AI config files, the URL entry for an application may vary depending on the cluster. To make the URL configuration more manageable, you can specify the values for these parameters at runtime using the --param flag.

Scenarios

Mon, 01 Jan 0001 00:00:00 +0000

The following Krkn scenarios are currently supported by Kkrn-AI.

At least one scenario must be enabled for the Kkrn-AI experiment to run.

Scenario	Kkrn-AI Config (YAML)
Pod Scenario	scenario.pod-scenarios
Application Outages	scenario.application-outages
Container Scenario	scenario.container-scenarios
Node CPU Hog	scenario.node-cpu-hog
Node Memory Hog	scenario.node-memory-hog
Node IO Hog	scenario.node-io-hog
Syn Flood	scenario.syn-flood
Time Scenario	scenario.time-scenarios
Network Scenarios	scenario.network-scenarios
DNS Outage	scenario.dns-outage
PVC Scenario	scenario.pvc-scenarios

By default, scenarios are not enabled. Depending on your use case, you can enable or disable these scenarios in the krkn-ai.yaml config file by setting the enable field to true or false.

Output

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI generates various output files during the execution of chaos experiments, including scenario YAML files, graph visualizations, and log files. By default, these files follow a standard naming convention, but you can customize the file names using format strings in the configuration file.

Available Parameters

The output section in your krkn-ai.yaml configuration file allows you to customize the naming format for different output file types:

`result_name_fmt`

Specifies the naming format for scenario result YAML files. These files contain the complete scenario configuration and execution results for each generated scenario.

Elastic Search

Mon, 01 Jan 0001 00:00:00 +0000

Krkn-AI supports integration with Elasticsearch to store scenario configurations, run results, and metrics. This allows you to centralize and query experiment data using Elasticsearch’s search and visualization capabilities (e.g., with Kibana).

Configuration Parameters

enable (bool): Set to true to enable saving results to Elasticsearch. Default: false.
server (string): URL or address of your Elasticsearch server (e.g., http://localhost).
port (int): Port to connect to Elasticsearch (default: 9200).
username (string): Username for Elasticsearch authentication (can reference environment variables).
password (string): Password for Elasticsearch authentication. If using environment substitution, prefix with __ to treat as private.
verify_certs (bool): Set to true to verify SSL certificates. Default: true.
index (string): Name prefix for the Elasticsearch index where Krkn-AI results will be stored (e.g., krkn-ai).

Example Configuration

elastic:
 enable: true # Enable Elasticsearch integration
 server: "http://localhost" # Elasticsearch server URL
 port: 9200 # Elasticsearch port
 username: "$ES_USER" # Username (environment substitution supported)
 password: "$__ES_PASSWORD" # Password (start with __ for sensitive/private handling)
 verify_certs: true # Verify SSL certificates
 index: "krkn-ai" # Index prefix for storing results

In addition to the standard Krkn telemetry and metrics indices, Krkn-AI creates two dedicated Elasticsearch indices to store detailed run information: