# Troubleshooting

Testground is still in early stage of development, so it is possible that:

* Testground crashes
* One of the underlying systems that Testground uses crashes (`Kubernetes`, `weave` , `redis`, etc.)
* Testground doesn't properly clean-up after a test run
* etc.

Here are a few commands that could be helpful for you to inspect the state of your Kubernetes cluster and clean up after Testground:

### Delete all pods related to a test plan

Delete all pods that have the `testground.plan=dht` label. This is useful in case you used the `--run-cfg keep_service=true` setting on Testground.

```
$ kubectl delete pods -l testground.plan=dht --grace-period=0 --force
```

### Restart the sidecar

Restart the `sidecar` daemon which manages networks for all testplans

```
$ kubectl delete pods -l name=testground-sidecar --grace-period=0 --force
```

### Review running, completed, failed pods

You can check all running pods with

```
$ kubectl get pods -o wide
```

Another useful combination is `watching` for pods that are not in `Running` state or that are failing their health  checks, with:

```bash
# watch all non-running pods
watch 'kubectl get pods --all-namespaces -o wide | grep -v Running'

# watch all not-ready pods
watch 'kubectl get pods --all-namespaces -o wide | grep "0\/1"'
```

### Get logs from a given pod

```
$ kubectl logs <pod-id, e.g. tg-dht-c95b5>
```

### Get access to the Redis shell

```
$ kubectl port-forward svc/testground-infra-redis-master 6379:6379 &
$ redis-cli -h localhost -p 6379
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://protocol-labs.gitbook.io/testground/master/runner-library/cluster-k8s/troubleshooting.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.