Testground
GitHubGo SDKInfra
master
master
  • README
  • Table of contents
    • What is Testground?
      • Community
    • Concepts and architecture
      • Test plans and test cases
      • Daemon and client
      • Synchronization service
      • Networking
      • Sidecar
      • Builders
      • Runners
      • Runtime environment (runenv)
      • Client-Server communication
    • Getting started
    • Writing test plans
      • Quick start
      • Understanding the test plan manifest
      • Parameters and test cases
      • Keeping instances in sync
      • Communication between instances
      • Observability, assets and metrics
    • Managing test plans
    • Running test plans
    • Traffic shaping
    • Analyzing test run results
      • Capturing profiles
    • Debugging test plans
    • Docker Settings
    • Featured projects
  • Runner library
    • local:exec
    • local:docker
      • System overview
      • Runner flags
      • Troubleshooting
    • cluster:k8s
      • System overview
      • How to create a Kubernetes cluster for Testground
      • Monitoring and Observability
      • Understanding Testground performance on Kubernetes
      • Troubleshooting
  • Builder Library
    • docker:go
    • exec:go
    • docker:generic
Powered by GitBook
On this page
  • What is a test plan?
  • The test plan <> Testground contract
  • Test plan manifest
  • Where do test plans live?
  • Test cases
  • Test runs
  1. Table of contents
  2. Concepts and architecture

Test plans and test cases

PreviousConcepts and architectureNextDaemon and client

Last updated 2 years ago

What is a test plan?

A test plan is a collection of test cases that exercise, benchmark, or verify a particular component, subsystem, or API of the system under test.

Test plans are the unit of deployment that Testground deals with.

Each test plan is a world of its own. In other words, test plans are opaque to Testground: they behave like black boxes.

Testground does not care what the test plan actually does, the language it's written in, nor the runtime it targets. As long as the source code is accessible, and a builder exists to compile that code into a runnable artifact, such as an executable or a Docker image, you're good to go 🚀

At the time of writing, Testground offers two builders:

  • exec:go compiles a Go test plan into a platform executable using the system Go installation.

  • docker:go compiles a Go test plan into a Docker image.

The test plan <> Testground contract

While test plans are opaque to the eyes of Testground, test plans and Testground promise to satisfy a contract. That contract is inspired by the , and facilitates deployment on cloud infrastructure when it's time to scale. The contract is as follows:

  1. Execution: Test plans expose a single point of entry, i.e. a main() function.

  2. Input: Test plans consume a , in the form of environment variables.

  3. Output: Test plans record events, results, and optional diagnostics in a predefined JSON schema on stdout and specific files. Any additional output assets they want harvested (e.g. event trails, traces, generated files, etc.) are written to a path received in the runtime environment.

?> Success The Testground community offers SDKs that make it easy for user-written test plans to adhere to the test plan contract, as well as facilitating interactions with the sync service and the emission of metrics.

Test plan manifest

Every test plan must contain a manifest.toml file at its root. This is a specification file that declares:

  • the name of the test plan, and authorship metadata.

  • which builders and runners it can work with.

  • the test cases it contains.

  • for each test case, the parameters it accepts, including the data type, default value, and a description.

The manifest.toml is used by tools such as the Testground CLI, or the upcoming Jupyter Notebooks integration, to enable a better user experience. Without this manifest file, it would be impossible to know the contents and behaviour of a test plan without inspecting its source.

Where do test plans live?

Test plans can be hosted and version anywhere—either on the local filesystem, or on public or private Git repositories: Testground does not care.

What's important is that the test plans source is available to the Testground client during runtime, under the $TESTGROUND_HOME/plans directory, where $TESTGROUND_HOME defaults to $HOME/testground if not set.

Test cases

While the unit of deployment in Testground is the test plan, test plans nest one or many test cases inside.

Think of a test plan as a family of tests that, together, exercises a given component or subsystem. Test cases evaluate concrete use cases that we wish to reproduce consistently, in order to capture variations in the observed behaviour, as the source code of the relevant components evolve over time.

Testground offers first-class support for dealing with test cases inside test plans:

  1. When inspecting a test plan, the Testground CLI allows you to enumerate all test cases within all test plans:

    $ testground plan list --testcases
  2. When scheduling a test run, the Testground CLI allows you to specify the test case out of a test plan that you want to run, e.g.:

    $ testground run single \
                     --plan=network \
                     --testcase=ping-pong \
                     --builder=docker:go \
                     --runner=local:docker \
                     --instances=2 \
                     --wait
  3. When developing a test plan, the Testground SDK allows you to select the test case that will run, based on an environment variable.

Test runs

Every time we execute a test plan, we generate a test run. Each test run is assigned a unique ID. That ID is used to identify the run when collecting outputs, or exploring results or diagnostics.

For more information on the format of the manifest, see .

!> Please note that $TESTGROUND_HOMEis not the same place where you clone the Testground .

The Testground client CLI offers a series of simple commands to manage test plan sources. Refer to the section for more information.

12-factor principles
formal, standardized runtime environment
Writing test plans > Test plan manifest
git repository
Managing test plans