Quality checks: test your AI’s answers
Use a quality check to send example questions through one AI step and see how many of its answers were good.
A quality check tests one AI step in a workflow. It sends a list of example questions to that step, and a grader scores each answer. Then you see a simple result, like 8 of 10 passed.
Use a check before you change a prompt or a model, and again after. If the number drops, the change made your AI worse.
Quality checks live under AI → Quality checks in your workspace sidebar.
The words you will see
| Word | What it means |
|---|---|
| Check | A test for one AI step. It has a name, the step it tests, and its example questions. |
| Example questions | The test cases. Each one is a question your AI step should answer well. |
| Grader | An AI model that reads each answer and gives it a score. It uses your own AI key. |
| Result | What you get each time you run a check, for example 6 of 10 answers passed. |
| Grading instructions | What the grader, or a person, follows to decide if an answer is good. |
| Review answers | A list of answers for a person to score by hand, to make sure the grader agrees with real people. |
Make your first check
Go to AI → Quality checks and click New check. It takes about 5 minutes. You can also start from the workflow editor: open the menu on an AI step and choose Check answer quality.
Pick the AI step
Pick the AI step you want to check. Steps are listed by workflow.
Add example questions
Add at least 5. Real questions from your customers work best. You can:
- From recent runs: tick runs where the step did its job. Each one becomes a question. Private details like names and emails are taken out first.
- Type them in: write a question, and what a good answer looks like if you want.
- Upload a spreadsheet: use a CSV file with a
questioncolumn and an optionalgood_answercolumn. Download the template from the same screen. - Let AI write some: handy for trying a check, but these questions don't count toward pass or fail.
Choose how to score answers
Pick one:
- Should match the good answer: the grader compares each answer to the good answer you gave.
- Should follow these instructions: you write the rules, and the grader checks each answer against them.
- Should pick the right label: for a step that sorts things into groups. The label must be the same as the good answer.
Then pick a pass mark. At 70%, an answer passes when it scores 70% or higher, and the check passes when at least 70% of answers pass.
Check and run
Look over your check, then click Run check. You land on the result. It fills in as the grader scores each answer.
To add more questions later, open the check and click See or add example questions.
When a check passes
Each answer gets a score from the grader. A check passes when enough of its answers pass. New checks start at 70%, so with 10 example questions, at least 7 answers must pass.
Checks are advice. A failed check does not stop you from publishing a workflow.
What a check costs
Each run asks the grader to score every example question once. The grader uses your own AI key, so the cost goes on your AI provider's bill, not on TaskJuice. The check's page shows a rough cost per run. Open Settings on that page to see the math, or to pick a different key or model for the grader.
If you can't run a check
The Run check button is turned off and the page says why. The common reasons:
| What the page says | What to do |
|---|---|
| You need at least one example question | Add a real example question. Questions made by AI help you try a check, but they don't count. |
| The grader has no AI key yet | Open Settings on the check's page and pick an AI key and model. |
| This check uses an old copy of its example questions | Make a new check. It will use the latest questions. |