Testground
GitHubGo SDKInfra
master
master
  • README
  • Table of contents
    • What is Testground?
      • Community
    • Concepts and architecture
      • Test plans and test cases
      • Daemon and client
      • Synchronization service
      • Networking
      • Sidecar
      • Builders
      • Runners
      • Runtime environment (runenv)
      • Client-Server communication
    • Getting started
    • Writing test plans
      • Quick start
      • Understanding the test plan manifest
      • Parameters and test cases
      • Keeping instances in sync
      • Communication between instances
      • Observability, assets and metrics
    • Managing test plans
    • Running test plans
    • Traffic shaping
    • Analyzing test run results
      • Capturing profiles
    • Debugging test plans
    • Docker Settings
    • Featured projects
  • Runner library
    • local:exec
    • local:docker
      • System overview
      • Runner flags
      • Troubleshooting
    • cluster:k8s
      • System overview
      • How to create a Kubernetes cluster for Testground
      • Monitoring and Observability
      • Understanding Testground performance on Kubernetes
      • Troubleshooting
  • Builder Library
    • docker:go
    • exec:go
    • docker:generic
Powered by GitBook
On this page
  • Delete all pods related to a test plan
  • Restart the sidecar
  • Review running, completed, failed pods
  • Get logs from a given pod
  • Get access to the Redis shell
  1. Runner library
  2. cluster:k8s

Troubleshooting

Testground is still in early stage of development, so it is possible that:

  • Testground crashes

  • One of the underlying systems that Testground uses crashes (Kubernetes, weave , redis, etc.)

  • Testground doesn't properly clean-up after a test run

  • etc.

Here are a few commands that could be helpful for you to inspect the state of your Kubernetes cluster and clean up after Testground:

Delete all pods related to a test plan

Delete all pods that have the testground.plan=dht label. This is useful in case you used the --run-cfg keep_service=true setting on Testground.

$ kubectl delete pods -l testground.plan=dht --grace-period=0 --force

Restart the sidecar

Restart the sidecar daemon which manages networks for all testplans

$ kubectl delete pods -l name=testground-sidecar --grace-period=0 --force

Review running, completed, failed pods

You can check all running pods with

$ kubectl get pods -o wide

Another useful combination is watching for pods that are not in Running state or that are failing their health checks, with:

# watch all non-running pods
watch 'kubectl get pods --all-namespaces -o wide | grep -v Running'

# watch all not-ready pods
watch 'kubectl get pods --all-namespaces -o wide | grep "0\/1"'

Get logs from a given pod

$ kubectl logs <pod-id, e.g. tg-dht-c95b5>

Get access to the Redis shell

$ kubectl port-forward svc/testground-infra-redis-master 6379:6379 &
$ redis-cli -h localhost -p 6379
PreviousUnderstanding Testground performance on KubernetesNextBuilder Library

Last updated 5 years ago