1. Habit
  2. Skills
  3. Arize Evaluator skill
arize-ai · 3K installs

Arize Evaluator skill
Runs by itself, every Monday morning.

Score new spans every week.

Install the Arize Evaluator skill and make it a habit: follow https://gethabit.dev/skills/arize-ai/arize-evaluator.md

  1. 1Copy
  2. 2Paste into Claude Code, Codex or Cursor
  3. 3It runs by itself

3K installs65 starsMIT licenceSource on GitHub ↗

Arize Evaluator

by arize-ai · 3K installs

The skill

The arize evaluator skill creates LLM-as-judge and code evaluators on Arize with the ax CLI, wires column mappings, and runs evaluation tasks on spans or experiments.

Only when you ask.

The habit

Score new spans every week.

↻ Runs by itselfevery Monday morning

How the habit works.
You don't have to.

Three questionsWhen does it start? How does it check its work? What does it remember?

01

It starts by itself

every Monday morning. Runs in Claude Code routine or cron.

  • ↻ every Monday morning
  • 0 7 * * 1

02

It checks its work

spans scored, not zero. If a check fails, it tells you.

  • ✓ax tasks get-run shows the run completed
  • ✓scored span count is above zero
  • ✓every template variable has a column mapping

03

It remembers

eval task run log. The next run reads it first: task id, last data window and last scores.

loops/arize-eval-check-loop/run-log.md

One run, step by step.
Then it stops.

Stop ruleStops after one task run of at most 100 spans and one retry.

  1. 01

    Read the log

    Your agent loads the task id and the window it scored last time.

  2. 02

    Pick evaluator once

    On the first run it samples spans with ax spans export and lets you choose one of a few evaluator ideas.

  3. 03

    Check mappings

    It exports a few spans and confirms each template variable maps to a real field.

  4. 04

    Trigger the run

    It calls ax tasks trigger-run with --max-spans and --wait on a window that ended at least two hours ago.

  5. 05

    Check the result

    ax tasks get-run must show a completed run with scored spans.

  6. 06

    Report honestly

    If the run fails or scores nothing, the agent reports the error instead of inventing scores.

What it may do.
Nothing more.

Safe by defaultIt writes only in its own folder and asks you before anything goes out.

Is it for you?
Use it when it fits.

Quick checkRead both lists. If the left one sounds like you, the Arize Evaluator skill helps.

Use it when

  • ✓ You need a hallucination or correctness judge
  • ✓ You want to score spans or experiments
  • ✓ You set up continuous monitoring

Skip it when

  • – You do not use Arize
  • – You only need a one-time manual review

Questions

Arize Evaluator skill, answered.

Why does the window end two hours ago?

The skill notes that the Arize eval index can lag, so a window ending now may score zero spans.

Can the agent read my .env for keys?

No. The skill uses ax profiles and ax ai-integrations for credentials and never reads .env files.

How do I install the Arize Evaluator skill?

Copy the prompt at the top into your agent. It installs the skill from arize-ai/arize-skills and sets up the habit. Or run npx skills add arize-ai/arize-skills --skill arize-evaluator.

Is the Arize Evaluator skill free?

Yes. It is published under MIT. Habit is free too.

Arize Evaluator. Now as a habit.

Install the Arize Evaluator skill and make it a habit: follow https://gethabit.dev/skills/arize-ai/arize-evaluator.md

  • Free
  • No account
  • Undo in one line

All arize-ai skills →More code skills →

Works in

  • Claude Code
  • Codex
  • Cursor
  • ChatGPT
  • Claude app

Made by arize-ai, published under MIT. Habit links to the original and never changes it. Is this your skill?

Copying didn't work here. Select the line and copy it by hand.