Introduction
SolveAI is an autonomous testing platform. You describe what a user should be able to do, SolveAI turns it into a reviewable workflow, runs it in an isolated worker against an environment you control, and explains failures with evidence. When a test is out of date with your product, SolveAI proposes a repair you can approve — it never rewrites a test to hide a real regression.
Core concepts
| Concept | What it is |
|---|---|
| Project | A product or service under test. Owns environments, workflows, secrets and members. |
| Environment | A verified target such as staging or production, with its base URL, variables and encrypted secrets. |
| Workflow | A named test: the natural-language instruction, the generated steps, run configuration, schedule and version history. |
| Run | One execution of a workflow version against an environment, with logs, artifacts, a result and (on failure) a diagnosis. |
| Report | The structured, shareable output of a run — failed step, error, evidence, repro steps, diagnosis and remediation. |
Workflow modes
Each workflow chooses how it executes:
| Mode | Behaviour | Best for |
|---|---|---|
| Pinned | SolveAI generates explicit steps once. Every run replays the same steps, so results are comparable run to run. Eligible for reviewed repairs. | Regression suites, CI gating |
| Adaptive | An agent pursues the stated goal on each run and records the path it took. No stored script to repair. | Exploratory checks, flows that legitimately vary |
Workflow lifecycle
| State | Meaning |
|---|---|
draft | Instruction written; steps not generated or not yet reviewed. |
ready | Steps reviewed and saved. Can run manually, on a schedule or from CI. |
running | A run is in progress. |
passed | The latest run succeeded. |
failed | The latest run failed and was diagnosed as a product or environment problem. |
needs_attention | The latest failure looks like test drift; a repair is proposed and awaits approval. |
disabled | Kept, but excluded from schedules and CI. |
archived | Hidden and cannot run. History is retained. |
Runs & artifacts
Runs move through queued → running → passed | failed | cancelled | errored. errored is reserved for SolveAI infrastructure problems and does not count against your plan. Every result shown in the dashboard comes from a recorded run — SolveAI never displays a pass that did not execute.
| Artifact | Captured when |
|---|---|
| Step screenshots | Every browser step |
| Playwright trace | Browser runs that fail or are marked for tracing |
| Console & network log | Browser runs; failed requests are highlighted |
| Request / response bodies | API steps (secrets redacted) |
| stdout / stderr | CLI steps |
| Reproduction steps | Every failed run |
Failure diagnosis
Failed runs of both modes are classified from collected evidence into one class, with a confidence level and the evidence that supports it:
| Class | Typical evidence | What happens next |
|---|---|---|
| Application regression | Assertions fail on correct elements; 5xx responses; changed behaviour | Alert. No repair. |
| Selector / layout change | Target missing but equivalent element present | Repair proposed (Pinned only). |
| Environment / config | Missing variable, expired secret, DNS or auth errors | Alert with config hint. |
| Timing / synchronisation | Element appears after timeout; slow responses | Suggest wait or timeout change. |
| API contract change | Schema or status mismatch against expectation | Alert; optional contract update for review. |
| Intermittent | Passes on identical retry; history of flips | Mark flaky; track rate. |
| Infrastructure | Worker crash, browser launch failure | Auto-retry; run marked errored. |
Repairs
Only Pinned workflows whose failure is classified as selector/layout drift (or a timing fix) are eligible. SolveAI proposes the smallest change that restores the original intent and shows it as a diff against the current version. Approving creates a new version and re-runs the workflow; rejecting keeps the original. Every proposal, approval and rejection is written to the project audit log. Assertions are never weakened by a repair.
Security model
- Runs can only target domains verified for the project (DNS TXT or file challenge).
- Workers are ephemeral, isolated, time-boxed and resource-limited, with no access to the host.
- Egress to private IP ranges, link-local addresses and cloud metadata endpoints is blocked, including after redirects and DNS rebinding.
- CLI steps run only allow-listed binaries with arguments passed as arrays — never through a shell string.
- Secrets are encrypted at rest, injected at run time, redacted from logs/reports and never sent to the AI model in plain text.
- Authorization is enforced on the server for every request; roles are scoped per team and project.
- Sensitive operations (secret changes, repairs, role changes, deletions) are audit-logged.
API & CLI
The planned REST API and CLI share one model. Shapes below are a design preview and may change.
# Run a workflow and wait for the result
solveai run billing-change-plan --env staging --wait
# REST
POST /v1/workflows/{id}/runs { "environment": "staging" }
GET /v1/runs/{run_id} → status, steps[], artifacts[], diagnosis
GET /v1/runs/{run_id}/report → structured report (JSON or Markdown)Roadmap
- Available
Phase 1–3
Reference research, brand identity, marketing site and docs
- In development
Phase 4
Accounts, teams, roles, dashboard shell and project management
- In development
Phase 5
AI Test Builder: natural language → reviewed, versioned steps
- In development
Phase 6
Execution workers (Playwright, HTTP), reports and AI diagnosis
- Planned
Phase 7
Database, secrets, integrations (GitHub, Slack, CI) and deployment
- Planned
Phase 8
Automated tests, accessibility audit, error handling, hardening
- Planned
Phase 9
End-to-end verification and general availability