Base rates
Tell Grayson how common each answer is in your own cases, so its probabilities match your rates.
Any question can include base_rate: how common each answer really is in your cases. Grayson then shifts that question's probabilities from its own average for the question to your rate. Cases keep their order, so the same cases rank highest; only the probability levels move.
Use it when your rates differ from what Grayson's answers average. For example, fraud is confirmed on 0.3% of your alerts, but Grayson's answers to is_fraud average 2% on them. It's optional: without base_rate, answers are unchanged.
{
"model": "grayson-1",
"state": { "...": "..." },
"questions": {
"is_fraud": {
"type": "noul",
"instructions": "Is this payment fraudulent?",
"base_rate": 0.003
},
"claim_type": {
"type": "choice",
"instructions": "What kind of claim is this?",
"criteria": { "ato": "Account takeover", "first_party": "First-party fraud", "none": "Not fraud" },
"base_rate": { "ato": 0.1, "first_party": 0.05, "none": 0.85 }
},
"risk": {
"type": "score",
"instructions": "How likely is this account to be closed for fraud within 90 days?",
"criteria": ["Very unlikely", "Unlikely", "Uncertain", "Likely", "Very likely"],
"base_rate": [0.6, 0.2, 0.1, 0.07, 0.03]
}
}
}The shape for each type
| Type | base_rate | Rules |
|---|---|---|
noul | number | The share of cases that are yes. Strictly between 0 and 1 |
choice | object | Every option's key, with the share of cases it's the right answer. Each share is from 0 to 1, and they add up to 1 (within 0.02; Grayson scales them to exactly 1) |
multi_choice | object | Any of the option keys, with the share of cases each option applies to. Each is strictly between 0 and 1, on its own: they don't need to add up to 1. Options you leave out aren't shifted |
score | array of numbers | The share of cases at each level, lowest level first, one per level. Each is from 0 to 1, and they add up to 1 (within 0.02) |
Option keys are the keys of the question's criteria. Use a share of 0 for a choice option or a score level only if that answer never happens in your cases: it makes the answer all but impossible.
How it works
- Grayson keeps a running average of its own answers to each of your organization's questions, over roughly your latest 200 answers to that question, weighted toward the most recent. Every answer updates it, whether or not the request sent a base rate.
- A base rate is applied once the question has 30 answers. Until then, its answers come back unchanged.
- The answer says whether it was applied. An answer to a question that sent
base_rateincludescalibrated:truewhen the base rate was applied,falsewhen the question doesn't have 30 answers yet. Questions withoutbase_ratehave nocalibratedfield. - Every answer moves by the same amount. For a yes/no, and for each option of a multi_choice, each answer's probability shifts by the same amount on the log-odds scale. For a choice or a score, each outcome's probability is scaled by your rate over Grayson's average for it, then the answer is renormalized. An answer whose average already matches your rate is left as it is.
- A question is recognized by its name, type and option keys (for a score, its number of levels). Rewording its instructions or option descriptions keeps the same average. Renaming it, changing its type, or adding or removing an option starts a new average, which needs 30 answers again.
- Each request uses the base rate it sends, so you can update it as your rates drift.
"is_fraud": { "type": "noul", "noul": 0.0041, "calibrated": true }Choosing a base rate
- Use a number measured on your own data, such as last quarter's confirmed-fraud rate on the same alerts you send. A rate close to the truth improves the probabilities. One that's several times off makes them worse than sending none.
- Measure it on the cases you send. If you only send alerts that a rule flagged, use the rate among those alerts, not across all transactions.
- Choose thresholds after you add a base rate. The order of cases doesn't change, but the probabilities do, so a threshold chosen without the base rate will catch a different share of cases with it. See Setting thresholds.
What's stored
The running average is stored per organization, under a one-way hash of the organization and the question's name, type and option keys. The average itself holds no question text, state or base rate. Requests, including their questions and any base rate, are kept only as your organization's data retention setting says. Averages that go unused for 180 days are deleted.
Errors
An invalid base rate returns 400 invalid_request with param set to questions.<name>.base_rate and a message that says what's wrong, for example:
| Cause | message |
|---|---|
| A yes/no base rate of 0 or 1, or not a number | A yes/no question's base_rate is the share of cases that are yes, between 0 and 1 (not 0 or 1). |
| A choice's shares don't add up to 1 | A choice's base_rate shares must add up to 1. |
| A choice's base rate leaves out an option | A choice's base_rate gives every option a share, and the shares add up to 1. |
| A key that isn't one of the question's options | base_rate names "wire", which isn't one of the question's options. |
| A score's list has the wrong number of levels | A score's base_rate lists the share of cases at each level, lowest first: 5 numbers that add up to 1. |
The playground has base rates too: turn on Add base rate on a question and enter percentages. The answer says whether it was applied.