Scenarios on Krkn

Krkn-Hub All Scenarios Variables

Thu, 05 Jan 2017 00:00:00 +0000

These variables are to be used for the top level configuration template that are shared by all the scenarios in Krkn-hub

See the description and default values below

Supported parameters for all scenarios in Krkn-Hub

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

example: export <parameter_name>=<value>

Parameter	Description	Default
CERBERUS_ENABLED	Set this to true if cerberus is running and monitoring the cluster	False
CERBERUS_URL	URL to poll for the go/no-go signal	http://0.0.0.0:8080
WAIT_DURATION	Duration in seconds to wait between each chaos scenario	60
ITERATIONS	Number of times to execute the scenarios	1
DAEMON_MODE	Iterations are set to infinity which means that the kraken will cause chaos forever	False
PUBLISH_KRAKEN_STATUS	If you want	True
SIGNAL_ADDRESS	Address to print kraken status to	0.0.0.0
PORT	Port to print kraken status to	8081
SIGNAL_STATE	Waits for the RUN signal when set to PAUSE before running the scenarios, refer docs for more details	RUN
DEPLOY_DASHBOARDS	Deploys mutable grafana loaded with dashboards visualizing performance metrics pulled from in-cluster prometheus. The dashboard will be exposed as a route.	False
CAPTURE_METRICS	Captures metrics as specified in the profile from in-cluster prometheus. Default metrics captures are listed here	False
ENABLE_ALERTS	Evaluates expressions from in-cluster prometheus and exits 0 or 1 based on the severity set. Default profile.	False
ALERTS_PATH	Path to the alerts file to use when ENABLE_ALERTS is set	config/alerts
ELASTIC_SERVER	Be able to track telemtry data in elasticsearch, this is the url of the elasticsearch data storage	blank
ELASTIC_INDEX	Elastic search index pattern to post results to	blank
HEALTH_CHECK_URL	URL to continually check and detect downtimes	blank
HEALTH_CHECK_INTERVAL	Interval at which to get	2
HEALTH_CHECK_BEARER_TOKEN	Bearer token used for authenticating into health check URL	blank
HEALTH_CHECK_AUTH	Tuple of (username,password) used for authenticating into health check URL	blank
HEALTH_CHECK_EXIT_ON_FAILURE	If value is True exits when health check failed for application, values can be True/False	blank
HEALTH_CHECK_VERIFY	Health check URL SSL validation; can be True/False	False
KUBE_VIRT_CHECK_INTERVAL	Interval at which to test kubevirt connections	2
KUBE_VIRT_NAMESPACE	Namespace to find VMIs in and watch	blank
KUBE_VIRT_NAME	Regex style name to match VMIs to watch	blank
KUBE_VIRT_FAILURES	If value is True exits will only report when ssh connections fail to vmi, values can be True/False	blank
KUBE_VIRT_DISCONNECTED	Use disconnected check by passing cluster API, can be True/False	False
KUBE_VIRT_NODE_NAME	If set, will filter vms further to only track ones that are on specified node name	blank
KUBE_VIRT_EXIT_ON_FAIL	Fails run if VMs still have false status at end of run, can be True/False	False
KUBE_VIRT_SSH_NODE	If set, will be a backup way to ssh to a node. Will want to set to a node that isn’t targeted in chaos	blank
CHECK_CRITICAL_ALERTS	When enabled will check prometheus for critical alerts firing post chaos	False
TELEMETRY_ENABLED	Enable/disables the telemetry collection feature	False
TELEMETRY_API_URL	telemetry service endpoint	https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production
TELEMETRY_USERNAME	telemetry service username	redhat-chaos
TELEMETRY_PASSWORD		No default
TELEMETRY_PROMETHEUS_BACKUP	enables/disables prometheus data collection	True
TELEMTRY_FULL_PROMETHEUS_BACKUP	if is set to False only the /prometheus/wal folder will be downloaded	False
TELEMETRY_BACKUP_THREADS	number of telemetry download/upload threads	5
TELEMETRY_ARCHIVE_PATH	local path where the archive files will be temporarly stored	/tmp
TELEMETRY_MAX_RETRIES	maximum number of upload retries (if 0 will retry forever)	0
TELEMETRY_RUN_TAG	if set, this will be appended to the run folder in the bucket (useful to group the runs	chaos
TELEMETRY_GROUP	if set will archive the telemetry in the S3 bucket on a folder named after the value	default
TELEMETRY_ARCHIVE_SIZE	the size of the prometheus data archive size in KB. The lower the size of archive is	1000
TELEMETRY_LOGS_BACKUP	Logs backup to s3	False
TELEMETRY_FILTER_PATTER	Filter logs based on certain time stamp patterns	["(\w{3}\s\d{1,2}\s\d{2}:\d{2}:\d{2}\.\d+).+",“kinit (\d+/\d+/\d+\s\d{2}:\d{2}:\d{2})\s+”,"(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z).+"]
TELEMETRY_CLI_PATH	OC Cli path, if not specified will be search in $PATH	blank

Note

For setting the TELEMETRY_ARCHIVE_SIZE,the higher the number of archive files will be produced and uploaded (and processed by backup_thread simultaneously).For unstable/slow connection is better to keep this value low increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the failed chunk without affecting the whole upload.

Krknctl All Scenarios Variables

Thu, 05 Jan 2017 00:00:00 +0000

These variables are to be used for the top level configuration template that are shared by all the scenarios in Krknctl

See the description and default values below

Supported parameters for all scenarios in KrknCtl

The following environment variables can be set on the host running the container to tweak the scenario/faults being injected:

Usage example: --<parameter> <value>

Parameter	Description	Type	Possible Values	Default
–cerberus-enabled	Enables Cerberus Support	enum	True/False	False
–cerberus-url	Cerberus http url	string	-	http://0.0.0.0:8080
–distribution	Selects the orchestrator distribution	enum	openshift/kubernetes	openshift
–krkn-kubeconfig	Sets the path where krkn will search for kubeconfig in container	string	-	/home/krkn/.kube/config
–wait-duration	Waits for a certain amount of time after the scenario	number	-	1
–iterations	Number of times the same chaos scenario will be executed	number	-	1
–daemon-mode	If set the scenario will execute forever	enum	True/False	False
–uuid	Sets krkn run uuid instead of generating it	string	-	-
–capture-metrics	Enables metrics capture	enum	True/False	False
–enable-alerts	Enables cluster alerts check	enum	True/False	False
–alerts-path	Allows to specify a different alert file path	string	-	config/alerts.yaml
–metrics-path	Allows to specify a different metrics file path	string	-	config/metrics-aggregated.yaml
–enable-es	Enables elastic search data collection	enum	True/False	False
–es-server	Elasticsearch instance URL	string	-	http://0.0.0.0
–es-port	Elasticsearch instance port	number	-	443
–es-username	Elasticsearch instance username	string	-	elastic
–es-password	Elasticsearch instance password	string	-	-
–es-verify-certs	Enables elasticsearch TLS certificate verification	enum	True/False	False
–es-metrics-index	Index name for metrics in Elasticsearch	string	-	krkn-metrics
–es-alerts-index	Index name for alerts in Elasticsearch	string	-	krkn-alerts
–es-telemetry-index	Index name for telemetry in Elasticsearch	string	-	krkn-telemetry
–check-critical-alerts	Enables checking for critical alerts	enum	True/False	False
–telemetry-enabled	Enables telemetry support	enum	True/False	False
–telemetry-api-url	API endpoint for telemetry data	string	-	https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production
–telemetry-username	Username for telemetry authentication	string	-	redhat-chaos
–telemetry-password	Password for telemetry authentication	string	-	-
–telemetry-prometheus-backup	Enables Prometheus backup for telemetry	enum	True/False	True
–telemetry-full-prometheus-backup	Enables full Prometheus backup for telemetry	enum	True/False	False
–telemetry-backup-threads	Number of threads for telemetry backup	number	-	5
–telemetry-archive-path	Path to save telemetry archive	string	-	/tmp
–telemetry-max-retries	Maximum retries for telemetry operations	number	-	0
–telemetry-run-tag	Tag for telemetry run	string	-	chaos
–telemetry-group	Group name for telemetry data	string	-	default
–telemetry-archive-size	Maximum size for telemetry archives	number	-	1000
–telemetry-logs-backup	Enables logs backup for telemetry	enum	True/False	False
–telemetry-filter-pattern	Filter pattern for telemetry logs	string	-	["\w{3}\s\d{1,2}\s\d{2}:\d{2}:\d{2}\.\d+", “kinit (\d+/\d+/\d+\s\d{2}:\d{2}:\d{2}”, “\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z”]
–telemetry-cli-path	Path to telemetry CLI tool (oc)	string	-	-
–telemetry-events-backup	Enables events backup for telemetry	enum	True/False	True
–health-check-interval	How often to check the health check urls (seconds)	number	-	2
–health-check-url	URL to check the health of	string	-	-
–health-check-auth	Authentication tuple to authenticate into health check URL	string	-	-
–health-check-bearer-token	Bearer token to authenticate into health check URL	string	-	-
–health-check-exit	Exit on failure when health check URL is not able to connect	string	-	-
–health-check-verify	SSL Verification to authenticate into health check URL	string	-	false
–kubevirt-check-interval	How often to check the KubeVirt VMs SSH status (seconds)	number	-	2
–kubevirt-namespace	KubeVirt namespace to check the health of	string	-	-
–kubevirt-name	KubeVirt regex names to watch	string	-	-
–kubevirt-only-failures	KubeVirt checks only report if failure occurs	enum	True/False	false
–kubevirt-disconnected	KubeVirt checks in disconnected mode, bypassing the cluster’s API	enum	True/False	false
–kubevirt-ssh-node	KubeVirt backup node to SSH into when checking VMI IP address status	string	-	false
–kubevirt-exit-on-failure	KubeVirt fails run if VMs still have false status	enum	True/False	false
–kubevirt-node-node	Only track VMs in KubeVirt on given node name	string	-	false
–krkn-debug	Enables debug mode for Krkn	enum	True/False	False

Note

For setting the TELEMETRY_ARCHIVE_SIZE,the higher the number of archive files will be produced and uploaded (and processed by backup_thread simultaneously| .For unstable/slow connection is better to keep this value low increasing the number of backup_threads, in this way, on upload failure, the retry will happen only on the failed chunk without affecting the whole upload.

Supported Cloud Providers

Thu, 05 Jan 2017 00:00:00 +0000

AWS

NOTE: For clusters with AWS make sure AWS CLI is installed and properly configured using an AWS account. This should set a configuration file at $HOME/.aws/config for your the AWS account. If you have multiple profiles configured on AWS, you can change the profile by setting export AWS_DEFAULT_PROFILE=<profile-name>

export AWS_DEFAULT_REGION=<aws-region>

This configuration will work for self managed AWS, ROSA and Rosa-HCP

ManagedCluster Scenarios

Wed, 04 Jan 2017 00:00:00 +0000

ManagedCluster scenarios provide a way to integrate kraken with Open Cluster Management (OCM) and Red Hat Advanced Cluster Management for Kubernetes (ACM).

ManagedCluster scenarios leverage ManifestWorks to inject faults into the ManagedClusters.

The following ManagedCluster chaos scenarios are supported:

managedcluster_start_scenario: Scenario to start the ManagedCluster instance.
managedcluster_stop_scenario: Scenario to stop the ManagedCluster instance.
managedcluster_stop_start_scenario: Scenario to stop and then start the ManagedCluster instance.
start_klusterlet_scenario: Scenario to start the klusterlet of the ManagedCluster instance.
stop_klusterlet_scenario: Scenario to stop the klusterlet of the ManagedCluster instance.
stop_start_klusterlet_scenario: Scenario to stop and start the klusterlet of the ManagedCluster instance.

ManagedCluster scenarios can be injected by placing the ManagedCluster scenarios config files under managedcluster_scenarios option in the Kraken config. Refer to managedcluster_scenarios_example config file.