Evaris
log insign up

Guide

Publish eval logs

Deep dive on how publishing works: the publishInspectLogs function, the evaris CLI, and the run manifest + checksum contract. For the authoring side, see Inspect AI integration.

Two ways to publish

CLI: simplest, covers the one-log-at-a-time case:

evaris publish <log-dir-or-file> --project <project-id> [--name <run-name>]

Reads EVARIS_API_URL and EVARIS_API_TOKEN from the environment. See CLI reference.

Library: when you want to publish from a script or CI step with custom control:

import { publishInspectLogs } from "evaris";

const result = await publishInspectLogs({
  apiUrl: "https://api.your-evaris-deploy.example",
  apiToken: process.env.EVARIS_API_TOKEN!,
  projectId: "support-agent",
  logDir: "./logs",
  runName: "nightly-run-123",
  tags: { branch: "main", sha: "abc1234" },
});

console.log(result.run_id, result.status);

Both paths call the same function under the hood.

What publishInspectLogs does

For a single .eval / .json(.gz) log file, it performs six steps:

  1. Create the run: POST /v1/runs with execution_mode: external, the project id, optional name, and tags. The run starts in created status.
  2. Discover the log file: scans logDir. Accepts .eval, .eval.gz, .json, .json.gz. Throws if it finds zero or more than one file.
  3. Checksum + stat: computes SHA-256 of the file bytes and reads its size.
  4. Request an upload target: POST /v1/runs/{run_id}/artifacts/upload-url with the file name, content type, size, and checksum. The backend returns a pre-signed PUT URL (R2 in production, filesystem in tests).
  5. Upload the bytes: PUT the raw file to the returned URL with the returned headers.
  6. Build + send the manifest, complete the run: POST /v1/runs/{run_id}/complete with a RunManifest containing the artifact manifest, a checksum of that manifest, source info (inspect-ai name + detected version), and timing. Completing enqueues ingestion on the backend.

If any step after run creation fails, publishInspectLogs deletes the run (best-effort) so it does not orphan in created status, then re-throws the original error.

The run manifest

The manifest is the signed summary the publisher sends when completing a run. Current schema version: 2026-06-11.

type RunManifest = {
  schema_version: "2026-06-11";
  run_id: string;
  project_id: string;
  execution_mode: "external" | "managed";
  producer: { name: string; version: string };   // "evaris-inspect-publish", "0.1.0"
  source: {                                       // What produced the artifact
    name: "inspect_ai";
    version: string;                              // detected inspect-ai version
    log_format?: string;
    python_version?: string;
  };
  status: "succeeded" | "failed";
  started_at: string;
  completed_at: string;
  artifact_manifest: ArtifactManifest;
  summary: { artifact_count: number; artifact_bytes: number };
  checksums: { artifact_manifest_sha256: string };
};

The checksums.artifact_manifest_sha256 is the SHA-256 of the serialized artifact_manifest JSON. The backend uses it to verify the manifest arrived intact.

Detecting the inspect-ai version

publishInspectLogs records the inspect-ai version in source.version. The detection order is:

  1. INSPECT_AI_VERSION env var, if set (use this in CI to pin/override).
  2. python3 -c "from importlib.metadata import version; print(version('inspect-ai'))".
  3. Same via python.
  4. The string "unknown".

The backend may emit a RunWarning with code unsupported_inspect_version or invalid_inspect_version if the version is outside a known-good range. The run still completes (status ingested_with_warnings); warnings are surfaced on the run and in the UI.

Log directory discipline

publishInspectLogs requires exactly one log file in logDir. This is a deliberate constraint, not a parser limitation:

  • It prevents accidentally publishing the wrong run when a directory contains logs from multiple eval invocations.
  • Each Inspect run should map to exactly one Evaris run.

If you have multiple logs to publish, call publishInspectLogs once per file (pass the file path, not its parent dir, as logDir; the function accepts either).

Tags

tags is a flat Record<string, string> on the run. Use it for filtering/grouping in the UI: branch, commit sha, eval config name, etc. Tags are not indexed for search yet; keep keys small and bounded.

Idempotency

Publishing is not idempotent today: calling publishInspectLogs twice for the same log file creates two runs with two artifacts. If you need deduplication, track the returned run_id externally. This is a known gap.

Next