Introduction to Spectest

Spectest is a testing framework and platform designed to test systems in their entirety, enabling you to build faster and remain confident that nothing breaks. In a Spectest suite, your product is tested from the outside through the same surfaces as your users, for example browsers, mobile apps, CLIs and APIs, with full determinism.

To make this work, we run a full copy of your code and infrastructure inside the testing environment. This includes your own code, like web apps and backends, but also all backing services, such as databases, object storage and external APIs.

Spectest can be added to an existing project in a few hours, without any changes to your code or access to production data.

Build your product through tests

Spectest is designed both for regression testing and to inspect how your product works without manual testing. Tests build on top of each other and form a tree of scenarios that closely match how your users use your product. Through our UI, you can then inspect exactly the user flows and any assertions.

Here’s an example of inspecting one test that drives a dashboard interaction. At the end, the test also asserts that the database was updated based on the changes made in the dashboard.

Test anything and everything

Spectest is not limited to just browsers but can test any surface of your product. We support testing web apps, mobile apps, CLIs, APIs, databases, and more. All these run as part of your full environment, meaning everything is tested end-to-end and multiple surfaces can be tested at the same time.

The following example shows testing of a CLI tool for deploying apps. Once deployed, the tests also drive a browser to ensure the app is reachable at the URL the CLI returned.

Work faster with your coding agent

One of the most important points of Spectest is to let you build even faster with coding agents, without breaking anything. When your coding agent makes a change, Spectest can produce a diff of tests instead of code. This lets you visually inspect what changed in your product without manual testing, and with full confidence that nothing else broke.

In this example, an agent has opened a pull request and Spectest calculates a diff based on the tests. The user can jump straight in to review the new feature through the test replay.

Full environments for full coverage

Spectest test environments are full Linux VMs, meaning you can (and should) run anything inside to match your production environment as closely as possible. We have users running a wide variety of architectures: like Kubernetes, Supabase, MQTT brokers, edge devices, Temporal, just to name a few.

For external APIs that can’t be run inside, for example an LLM API, we support creating fakes and replaying real API interactions.

This example shows an agent being tested. The test uses a deterministic replay of real responses from the OpenAI API, with tool calls running for each test and deploying a user app inside the test environment.

FAQ

Do I need to use Specific to use Spectest?

No. Spectest is a separate product and platform that can be used with any infrastructure and architecture.

If you are using Specific for your project, it’s very simple to get a Spectest environment set up.

Can I run tests locally?

At this point, no. Spectest relies on virtualization to run the isolated test environments and on macOS we have noticed this is too slow to run a test suite effectively. On Linux, we may support it in the future, but it will still likely be much faster to run tests on our powerful servers.

Can I self-host Spectest?

Yes, Spectest is architected to run test suites on our infrastructure or on servers you bring. The caveat is that they must be bare-metal servers and we recommend they have a large number of cores to run the test suite at full parallelism and speed.