Assisted redaction

A detector proposes regions on the image, you accept or reject them on the stage, and each accepted proposal becomes an ordinary redaction. The first detector finds licence plates. Nothing leaves the browser.

What it does

1. On selecting the Assist tool — or right after an image loads, with run: 'load' — the editor renders the unedited working copy and hands it to the detector. Proposals below minScore and outside classes are dropped.
2. The proposals appear as dashed boxes with a score. Each has an accept and a reject handle; the panel has Accept all and Reject all. A box can be moved before it is accepted.
3. Accepting dispatches a plain redact.add with the redact plugin's defaultMode and defaultAmount — or the assist options' own mode and amount — grown by padding. The edit log cannot tell it from a hand-drawn rectangle: the operation has no field for a model, a score or a label, and never will. Rejecting leaves no trace. When the redact options refuse the operation, for example an amount outside redact.amountRange with enforce: true, nothing is dispatched, the proposal stays open and the panel shows the reason (assist.refused).
4. The toolbar button carries the number of open proposals as a badge, so a detection that ran on load is visible while another tool is active.

The proposals live in the tool, not in the log. A stored log renders in five years without the model, the runtime or any record of either.

Privacy and the local boundary

Detection runs on the working copy, in a worker, against a model and a runtime you serve from your own origin. The image is not sent anywhere.

A local export needs the original in memory. The editor keeps it up to limits.maxClientMegapixels — 16 megapixels by default, set on the element as limits — and above that export() throws NOT_RESIDENT. A host that handles larger originals raises the limit or renders the original on its own server with @sealpixel/render, which this feature does not use.

Install

assist is published and supported but not in defaultPlugins, like sticker and watermark: register it. It declares no operations, so the bundled render worker still applies to [...defaultPlugins, assistPlugin]. The plate detector needs onnxruntime-web, an optional peer dependency:

pnpm add onnxruntime-web

Serve two things yourself:

• the runtime's WebAssembly: copy node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded.wasm and ort-wasm-simd-threaded.mjs to a directory of your site, say /ort/;
• the model, an ONNX file, and its sha256. The SDK ships no weights.

Then wire the tool and point the options at both:

import { createAssistTool, defaultPlugins } from '@sealpixel/element';
import { assistPlugin } from '@sealpixel/plugins';
import { createPlateDetector } from '@sealpixel/plugin-assist/plates';

editor.plugins = [...defaultPlugins, assistPlugin];
editor.tools = { assist: createAssistTool(createPlateDetector()) };
editor.pluginOptions = {
  assist: {
    model: { url: '/models/plate-384.onnx', sha256: '3f9c…' },
    wasmPaths: '/ort/',
  },
};

In React:

<SealpixelEditor
  src={file}
  plugins={[...defaultPlugins, assistPlugin]}
  tools={{ assist: createAssistTool(createPlateDetector()) }}
  pluginOptions={{ assist: { model: { url, sha256 }, wasmPaths: '/ort/' } }}
/>

The runtime is imported and the model fetched on the first detection, not on page load. A host that imports @sealpixel/plugin-assist/plates without installing onnxruntime-web gets a build-time resolution error; at runtime a missing or unhashable model surfaces in the panel as assist.error.

A failed load is not kept. The next detection, on the next selection of the tool or on Detect again, loads the model again, so a detection that failed offline works once the network is back. Leaving the tool stops a detection in flight, and selecting it again starts a new one.

Options

Everything here shapes the proposal and the review. None of it reaches the render: an accepted proposal is a plain redact.add whatever the options say.

Option	Default	What it does
run	'select'	When the detector runs: select (the tool is chosen), load (right after an image loads; fetches runtime and model then), manual (only the panel button).
timeoutMs	20000	A detection that takes longer fails with a message instead of waiting forever. The time includes loading the runtime and the model.
classes	[]	Labels to ask for and keep; empty means everything the detector offers. The plate detector knows plate.
minScore	0.6	Proposals below this score are dropped. The default sits between the lowest true plate (0.72) and the highest false positive (0.59) of the photo set the detector was measured on.
maxSuggestions	50	At most this many proposals, best first.
model	null	{ url, sha256, inputSize? } of the model file. Required by the plate detector; inputSize picks the 256, 384 or 512 export, default 384.
wasmPaths	null	Directory the runtime loads its .wasm from, with a trailing slash. Required by the plate detector.
padding	0.02	Grown around each proposed rect before it becomes a redaction, in image units.
mode	null	Mode of an accepted proposal; null takes redact.defaultMode. Subject to redact.enforce. See Pick a mode that cannot be undone.
amount	null	Amount of an accepted proposal in source px, at least 2; null takes redact.defaultAmount.
acceptAbove	null	Proposals at or above this score are accepted without review. Off by default: the tool is built for a person to decide. A compliance workflow that wants every plate gone sets it.
badge	true	Count of open proposals on the toolbar button.
showScore	true	Score chip on each proposed box.
adjust	true	Proposals can be moved before they are accepted.
bulk	true	"Accept all" and "Reject all" in the panel.

Set them like any plugin option, through pluginOptions.assist on the element and the wrappers, or assistPlugin.configure({ … }).

Pick a mode that cannot be undone

The redact plugin has three modes, and they protect differently.

• pixelate and blur compute the result from the pixels under the region. The result therefore still carries information about what it covers. Text from a known set of characters, such as a licence plate, can be recovered by pixelating candidate texts the same way and comparing. Faces can be matched against a set of candidates. The renderer is deterministic and its source is public, so an attacker can reproduce either mode exactly.
• mosaic looks like pixelation, but no cell reads a pixel inside the region. Each cell takes its colour from the pixels just around the region, with a brightness variation that depends only on the cell's position on the grid. It also covers every pixel the region touches in full, with no blended edge.

The test suite asserts the difference: it replaces every pixel inside a region with other content, renders again and expects the same bytes for mosaic, and different bytes for pixelate and blur. A mosaic region also never samples a pixel that another region of the same image touches, and mosaic regions are applied before pixelate and blur regions. A pixelate cell or the blur reaches past its own region, and applied first it could carry what a mosaic covers into visible pixels.

The colour stage, finetune.clarity included, runs after every redaction. It sharpens the redacted pixels as the redaction left them, so it cannot bring back what a region hides: a mosaic still does not depend on the pixels under it, and a pixelated region still depends only on its cell means. An annotation drawn over something to hide it is not a redaction.

For licence plates, faces and anything else that identifies a person, use mosaic. To make it the only choice, and to reject logs that use another mode on the client and on the server:

editor.pluginOptions = {
  redact: { modes: ['mosaic'], defaultMode: 'mosaic', enforce: true },
};

An accepted proposal then becomes a mosaic region too. mosaic hides what the region covers, not what surrounds it. Draw the region around the whole object, because a stroke that reaches past the edge stays visible. The shape of a car, the colour of a coat or the place stay visible too, and what to cover stays an editorial decision.

Your own detector

Anything that returns rectangles in image space works:

import type { Detector } from '@sealpixel/plugins';

const detector: Detector = {
  async detect(image, { signal, options }) {
    // image: { width, height, data: Uint8Array } — RGBA8, the unedited working copy
    return [{ rect: { x: 0.1, y: 0.2, width: 0.3, height: 0.1 }, score: 0.9, label: 'thing' }];
  },
};

editor.tools = { assist: createAssistTool(detector) };

rect is 0..1 over the source after EXIF orientation — the same space the redact regions use, so a box found on the working copy needs no conversion. Honour signal: the editor aborts a detection when the source changes, the tool is left or timeoutMs passes. The tool stops waiting at that moment even if the detector does not return; honouring the signal stops the work itself.

What the plate detector is

@sealpixel/plugin-assist/plates runs a YOLOv9-t "end2end" export from open-image-models on onnxruntime-web, single-threaded, in a dedicated worker. On the photo set it was measured against it found 15 of 16 plates at 384 px with one false positive, in about 90 ms after the first detection and about 300 ms including runtime and session start-up; small, distant plates are what it misses.

The author publishes those weights under MIT. Whether their training base permits that has not been confirmed by the author, so the SDK ships no weights. Host your own and check the licence of what you host.

Keyboard and screen readers

The accept and reject handles are buttons: Tab reaches them, Enter and Space trigger them. The panel's status line has role="status", and the badge has a visually hidden label with the count.