Playwright 1.64 lock and --shuffle for a Sume API test suite

Yes, you can test the Sume API with Playwright's built-in request fixture, and Playwright 1.64 makes the suite safer to run in parallel. The release adds a lock option to test.describe.configure() and a --shuffle flag. Put the tests that read or change the workspace's credit balance behind one named lock, and let the free validation tests run in parallel with a random order.

Everything below uses only documented Sume behavior from the Video Generation guide, the balance read, and the error envelope.

What did Playwright 1.64 add that matters for API tests?

The 1.64 release notes list three runner changes. lock in test.describe.configure() declares named locks for every test in the scope. The API page says it prevents tests from running concurrently when they share a lock. --shuffle [seed] schedules tests in a random order, and the seed is printed so you can replay a failing order. testProject.default keeps a project in the config without running it by default, which is handy for a paid-API project.

Which Sume tests cost nothing?

Validation failures are free. Sume's /v1/videos surface rejects size with 400 unsupported_parameter, because every model reports supported_sizes: null. The same happens for seed and a non-empty provider.options. The API rejects these fields instead of ignoring them, so the test pins the contract without a generation. Add a second test per field, and a 401 test with a bad key, and you have a suite that never touches your balance.

How do you lock the tests that change credits?

GET /v1/balance returns data.balance.available_credits. A test that compares the balance before and after a submit breaks if another worker submits at the same moment. Give those tests one lock. The config sets a base URL and the key header once:

import { defineConfig } from "@playwright/test";

export default defineConfig({
  fullyParallel: true,
  use: {
    baseURL: process.env.SUME_BASE_URL ?? "https://api.sume.com",
    extraHTTPHeaders: { Authorization: `Bearer ${process.env.SUME_API_KEY}` },
  },
});

Then the spec has a free test and a locked paid test. Run it with SUME_API_KEY=... npx playwright test --shuffle --workers=4:

import { test, expect } from "@playwright/test";

const credits = async (request: any) =>
  (await (await request.get("/v1/balance")).json()).data.balance.available_credits;

test("size is rejected with a 400", async ({ request }) => {
  const res = await request.post("/v1/videos", {
    data: { model: "wan-3.0", prompt: "a lighthouse at dusk", size: "1280x720" },
  });
  expect(res.status()).toBe(400);
  expect((await res.json()).error.code).toBe("unsupported_parameter");
});

test.describe("paid", () => {
  test.describe.configure({ lock: "sume-credits" });

  test("a 2 s job is accepted and credits are reserved", async ({ request }) => {
    const before = await credits(request);
    const res = await request.post("/v1/videos", {
      headers: { "Idempotency-Key": `pw-${Date.now()}` },
      data: { model: "wan-3.0", prompt: "a lighthouse at dusk", duration: 2, resolution: "480p" },
    });
    expect(res.status()).toBe(202);
    expect(await credits(request)).toBeLessThan(before);
  });
});

The submit returns 202 with a job in pending status; Sume reserves the job's credits at submit, so the available balance drops. Send an Idempotency-Key so a Playwright retry replays the original job and does not reserve twice. Keep the paid job short: wan-3.0 accepts 2 to 30 seconds, and duration: 2 at 480p is the smallest request in this example.

What should you not assert?

Do not assert exact credit amounts. Read them from the rate card when you need them, and compare before and after. Do not wait for completion inside the same test. A video usually takes from 30 seconds to several minutes, so poll in a separate long-timeout project, or receive a callback_url webhook. Do not let a shuffled order matter: each test should create its own job and its own idempotency key.