Prompts, goldens & evaluations

An evaluation should answer two questions: did the assistant produce the expected behavior, and did it use information that this specific person was allowed to see? Huck runs the same answering path used by chat against your saved identities and knowledge. The reference answer is given to the judges, never to the answering assistant.

This guide walks through a release from a saved prompt to a passing current-context evaluation. Use separate automation credentials with the relevant prompts:* and evaluations:* scopes. A 202 evaluation response means work was accepted; it does not mean it passed.

Prepare representative data

Before creating a release suite:

1. Publish the access policy you intend to use.
2. Save complete identities for a member, a manager, and any administrator you need to test. Use authoritative data from your backend.
3. Ingest representative permitted and forbidden records and wait for indexing to finish.
4. Add cases for normal answers, unavailable information, cross-organization denial, role boundaries, and prompt-injection text inside retrieved content.

The evaluation subject is selected with subject_external_id, not browser-supplied gate claims. A malformed stored identity cannot skip authorization merely because a test has empty evidence or the administrator role.

Save an immutable prompt version

POST /prompts with prompts:write creates a new version. It never replaces the active prompt by itself.

{
  "content": "Help customers understand their orders. State the order status and cite the authorized source. If no source supports an answer, explain that the information is unavailable. Do not guess shipping dates.",
  "change_note": "Explain order status without inventing shipping estimates",
  "model_settings": {"temperature": 0, "max_output_tokens": 700},
  "retrieval_settings": {"limit": 6}
}

curl --fail-with-body "$HUCK_API_URL/api/v1/prompts" \
  -H "Authorization: Bearer $HUCK_RELEASE_KEY" \
  -H 'Content-Type: application/json' --data-binary @prompt.json

The 201 response contains data.id, the API identifier, and data.version, the human-readable version number. Use data.id in subsequent API paths. To describe lineage, supply parent_id from an earlier version. Optional model selects a configured model; omission uses the configured default. A model must have a reviewed rate card and working provider configuration before generation can run. Output settings are also bounded by the platform's configured maximum.

Prompt content is limited to 16,000 characters, change notes to 1,000. temperature supports 0–2, max_output_tokens 1–4096, and retrieval limit 1–20. Saving a version does not imply the selected provider supports every possible model/setting combination; test the saved candidate.

Your inner prompt is wrapped by Huck's versioned outer instructions. Access is enforced by application code, not by telling the model to obey gates. The current wrapper treats sources, user messages, and action results as untrusted data, requires supported answers, and uses the configured assistant name. You can call your assistant something other than Huck in Appearance; evaluate and publish the resulting behavior.

GET /prompts returns saved versions in data, the current active prompt, and recent releases. The console's Prompt workbench shows the wrapper, version history, a test against a stored subject, and the release selector.

Create a golden case

POST /goldens requires evaluations:write and an existing active subject. This example assumes order 42 is accessible to subject alice:

{
  "name": "Alice sees her paid order",
  "question": "What is the status of order 42?",
  "subject_external_id": "alice",
  "required_facts": ["paid"],
  "forbidden_facts": ["delivered tomorrow"],
  "expected_sources": [{"external_type":"order","external_id":"42"}],
  "expected_abstention": false,
  "judge_config": {"match":"none","require_citations":true,"max_length":1200}
}

Expected sources match the complete type/ID pair. A plain string such as "refund-policy" means the default document type; it cannot stand in for order/refund-policy.

For exact wording, add expected_answer and select judge_config.match: "exact". "normalized" ignores case and extra whitespace; it does not establish semantic equivalence. With "none", the expected answer is not used for a deterministic equality check. Required and forbidden facts are normalized substring checks, not a proof that every factual claim is correct.

A denial case can use expected_abstention: true and forbidden facts from another organization's record. Do not add a forbidden record to expected_sources: that would require a citation you intentionally want to prohibit.

Update a case with PUT /goldens/{goldenId} and its current version plus the complete desired case. The saved version increments automatically. A stale version returns 409. This is not a partial field patch: omitted optional expectations reset to their defaults. GET /evaluations returns current goldens and recent runs for inspection.

Know what each judge proves

Each case needs a meaningful expectation. The required suite must contain 1–50 cases. A case can contain up to 30 required facts, 30 forbidden facts, and 20 expected sources. Goldens use a question up to 4,000 characters and an expected answer up to 16,000.

An optional rubric belongs in judge_config:

{
  "rubric": {
    "criteria": "The answer must explain the status supported by the authorized order, distinguish unknown dates from confirmed dates, and avoid unsupported promises.",
    "model": "gpt-4.1-mini",
    "threshold": 0.85
  }
}

Use a model supported by your deployment's rate card. Rubric judgments consume the evaluation allowance. Malformed, refused, or inconclusive judgments cannot pass. A model judge cannot override a failed deterministic check. Passing a suite is evidence about these cases and these inputs; it is not a guarantee that every possible future answer is true.

Request, poll, and inspect a run

POST /evaluations requires evaluations:run and an Idempotency-Key of 1–120 non-whitespace ASCII characters. Persist the key for the logical run. Reuse it with the identical request after an uncertain response; use a new key for a deliberate new run.

curl --fail-with-body "$HUCK_API_URL/api/v1/evaluations" \
  -H "Authorization: Bearer $HUCK_RELEASE_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: release-order-prompt-2026-10-14-01' \
  --data '{"prompt_version_id":123}'

Replace 123 with the prompt's returned API ID. The response is 202 with data, status_url, a Location header, and Retry-After: 2. Poll GET /evaluations/{runId} using evaluations:read until the run leaves running; back off between polls. finished_cases and total_cases show progress. Inspect every entry in results, including its checks, provider information, and error code when present.

Current run states include running, completed, failed, stale, and privacy-invalidated state. Only completed with passed: true is a passing outcome. Workers recheck requester authorization and the input fingerprint before protected work. Revoking the requesting key or changing knowledge, identities, access policy, prompt settings, or required goldens can make work stale.

An interrupted provider attempt is not automatically rerun: its billing outcome may be uncertain. Review the run and usage first, then explicitly request a new run with a new key. Queue workers and the scheduler must be running for background cases and recovery.

Publish exactly what was tested

curl --fail-with-body "$HUCK_API_URL/api/v1/prompts/$PROMPT_ID/publish" \
  -H "Authorization: Bearer $HUCK_RELEASE_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"evaluation_run_id":456}'

Use the passing run ID in place of 456. Publishing requires prompts:publish. Huck rechecks that the run belongs to this prompt, covers all required cases, passed every check, retains usable context, and still matches the current fingerprint. A 409 means you must resolve the change and run the current suite again. It does not mean “force publish.”

The fingerprint includes the prompt and wrapper, model/retrieval settings, policy and identity state, knowledge revisions and structured source hashes, goldens, and relevant runtime configuration. Metadata or relationship changes can therefore invalidate an earlier approval even when the underlying retrieval text did not require new embeddings.

To roll back behavior, select an earlier saved prompt, evaluate it against today's data and permissions, and publish that passing result. There is no bypass that makes old approval evidence current again.

Frozen comparisons and retention

To compare a prompt against the retained context from an earlier run, supply snapshot_run_id with prompt_version_id when requesting an evaluation. Frozen context is still reauthorized; it cannot be used to retrieve revoked information. Snapshots expire after seven days and can become unavailable sooner after source changes or privacy erasure.

Frozen comparisons are diagnostic and cannot approve a release. Always run a final evaluation using current retrieval before publishing. Stored evaluation references and results may contain sensitive customer information; keep evaluation-read credentials on your server and follow your organization's retention requirements.

Continue with knowledge ingestion, access testing, or API errors.