Calibration

Finic's answers are calibrated: a score of 250, or a probability of 0.25, means about a 25% chance. Of records scored near 250, about 1 in 4 turn out bad; of answers near 0.8, about 8 in 10 are right. And they're calibrated to your distribution:

• A score is built from your labeled examples, and recalibrated each time you send outcomes.
• A decision shifts to your rates when a question carries a base_rate.

Calibrate a scoreCopy Anchor Link

A score is calibrated to the outcomes it knows: your examples, then every outcome you send. If your examples have more bad outcomes than production does (say, you included every bad case you have), scores run higher than the true rate. Records still rank the same way, so thresholds chosen on your own records still work, and calibrating with production outcomes corrects it.

Send outcomes with POST /v1/calibrate: rows of the score_id you sent to POST /v1/score and is_bad, as JSON or CSV.

score_id,is_bad
pay_88213,true
pay_88214,false

• It runs as a job. Scores already returned don't change; score calls after the job completes use the updated score.
• Send outcomes in batches as you learn them, good and bad alike: up to 10 calibrations per score per day, one at a time. Each uses every outcome sent so far.
• The response counts the rows that matched a score call (matched) and those that didn't (unmatched, skipped). If none match, it fails with 400 invalid_request.
• Sending an outcome again replaces it, so you can correct a label. A row can carry model_id too, as long as it's the score you're calibrating.

Base rates for decisionsCopy Anchor Link

Any decide question can carry base_rate: how common each answer is in your cases. Finic then shifts that question's probabilities from their usual average to your rate. Cases keep their order, so the same cases rank highest; only the probability levels move. Use it when your rate differs from what the answers average: for example, fraud is confirmed on 0.3% of your alerts, but answers to is_fraud average 2%.

"questions": {
  "is_fraud": { "type": "noul", "instructions": "Is this payment fraudulent?", "base_rate": 0.003 },
  "claim_type": {
    "type": "choice",
    "instructions": "What kind of claim is this?",
    "criteria": { "ato": "Account takeover", "first_party": "First-party fraud", "none": "Not fraud" },
    "base_rate": { "ato": 0.1, "first_party": 0.05, "none": 0.85 }
  },
  "risk": {
    "type": "score",
    "instructions": "How likely is this account to be closed for fraud within 90 days?",
    "criteria": ["Very unlikely", "Unlikely", "Uncertain", "Likely", "Very likely"],
    "base_rate": [0.6, 0.2, 0.1, 0.07, 0.03]
  }
}

A share of 0 makes that answer all but impossible: use it only for answers that never happen.

How it worksCopy Anchor Link

• Finic keeps a running summary of its answers to each of your organization's questions: how they're spread, over roughly the latest 200, weighted to the most recent. Every answer updates it, with or without a base rate.
• A base rate applies once the question has 30 answers. The answer then has calibrated: true; before that, calibrated: false and the probabilities are unchanged. Questions without base_rate have no calibrated field.
• Every answer moves by the same amount on the log-odds scale, chosen so the recent answers average your rate (per outcome for a choice or score, then renormalized). An answer already at your rate is left alone.
• A question is recognized by its name, type and option keys (for a score, its number of levels). Rewording the instructions or descriptions keeps its summary; renaming it, changing its type or changing its options starts a new one, which needs 30 answers again.
• Each request uses the base rate it sends, so update it as your rates drift.

"is_fraud": { "type": "noul", "noul": 0.0041, "calibrated": true }

Choosing a base rate: measure it on your own data, on the same cases you send (if you only send alerts a rule flagged, the rate among those alerts). A rate close to the truth improves the probabilities; one several times off makes them worse than none.

What's stored: the summary, per organization, under a one-way hash of the organization and the question's name, type and option keys. It holds only aggregates of the answers: no single answer, question text, data or base rate. Unused summaries are deleted after 180 days.

Errors: an invalid base rate gets 400 invalid_request with param questions.<name>.base_rate:

The playground has base rates too: turn on Add base rate on a question.

Choosing thresholdsCopy Anchor Link

Finic returns probabilities, not decisions: you choose where to act.

• Choose on your own labeled records, each cut at the time of the decision: the threshold that gives the review volume, catch rate or losses you want. Check calibration on that sample before relying on it.
• Use one threshold per action: for example, decline a score above 800 and review above 300, or block a decision above 0.9 and review above 0.3.
• Add base rates first. They change the probabilities (not the order), so a threshold chosen without one catches a different share with it.
• For a score question, add up the levels you act on (P(Likely) + P(Very likely)). For a choice, look at the margin: 0.48 against 0.45 is a close call.