Scoring models

A scoring model rates the risk of one kind of decision, such as a payment, a sign-up or a loan application. You create it once, from a description of the decision or from past records whose outcomes you know, score each new record with POST /v1/score, and calibrate it with outcomes as you learn them so it keeps improving. Each scoring model is your own. It scores a record by asking Grayson a handful of yes/no questions about it and weighing the answers (Questions and weights).

The score is an integer from 0 (lowest risk) to 1000 (highest). Once the scoring model is calibrated, divided by 1,000 it's about the probability that the outcome is bad: of records scored near 250, about 1 in 4 turn out bad. Before then, scores rank records, but their scale is an estimate. See Calibration for what that rests on and how to choose thresholds.

Creating a scoring modelCopy Anchor Link

Create one with POST /v1/scoring_models, from either:

curl https://api.finic.ai/v1/scoring_models \
  -H "Authorization: Bearer $FINIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Card-not-present payment fraud", "criteria": "Card-not-present payments at an online store. Bad: the payment was charged back as fraud within 90 days."}'

A scoring model created from its criteria isn't calibrated yet:

• Its weights are Finic's estimates, and calibrated_at is null. Its scores still run from 0 to 1000 and rank records, but their scale is an estimate, not calibrated to your rate of bad outcomes: choose thresholds once it's calibrated.
• It takes any record, since it has no schema until you calibrate it with examples.
• You can add and remove its questions freely (Changing questions).
• Calibrate it with labeled examples, or with the outcomes of at least 100 records it scored, 20 of each (Calibration).

Example dataCopy Anchor Link

To create a scoring model from examples, or calibrate one with them, send past records, each with is_bad: true when the outcome was bad, false when it was good. Send them as a JSON array of objects (nested objects and arrays are fine) or as CSV with a header row, as text or a file:

amount_usd,merchant_category,account_age_days,new_device,billing_country,ip_country,is_bad
2400.00,electronics,19,true,US,NG,true
58.12,grocery,1260,false,US,US,false
310.00,gift_cards,3,true,US,US,true

RulesCopy Anchor Link

• At least 100 rows, with at least 20 of each outcome. Up to 2,000 are used; a larger set is sampled down, keeping its mix of outcomes. The request can be up to 2 MB.
• is_bad is true or false, 1 or 0, or the text "true", "false", "1", "0", "yes" or "no", in any case.
• Every row has at least one other field, and is up to 60,000 bytes as compact JSON without is_bad.
• CSV: every row has as many columns as the header, which names each column once. An empty cell is null, true and false are booleans, plain numbers are numbers, and the rest is text (leading zeros, as in ZIP code 02139, stay text). A dotted column name is a nested field: merchant.category is {"merchant": {"category": …}}, so CSV examples match JSON records scored later. A dotted name that clashes with another column stays one field.
• name (required, up to 100 characters): a few words for people to recognize the scoring model by, such as "Card-not-present payment fraud". You can rename it in the Finic Portal.
• criteria (up to 4,000 characters): what the decision is and what is_bad means, such as "Card-not-present payments. is_bad: the payment was charged back as fraud within 90 days." Required without data, since Finic writes the questions from it; with data, optional, and used as context.
• add_questions (optional, up to 20, 500 characters each): your own yes/no questions to use alongside Finic's. See Changing questions.

What to includeCopy Anchor Link

• Customer identity or event data in JSON: sign-ups, logins, payments, transactions, applications. One row per past decision, with its real outcome as is_bad.
• The fields you'll send when you score, with the same names and shapes, from the same systems.
• Only what was known at the time of the decision. A later chargeback, fraud flag or analyst verdict makes the scoring model look better than it will be on new records.
• Readable, consistent fields: names in words (amount_usd, not amt), units and currencies, codes expanded, the same type for a field in every row.
• Recent, representative records from the population you'll score.

SchemaCopy Anchor Link

Creating a scoring model from examples, or calibrating one with examples for the first time, records its schema: every field in the examples and its type (text, number, boolean, object or array). Nested fields have paths such as account.age_days, and fields in arrays paths such as logins[].country. A field every example has, never null, is required.

Each record you score is checked against it and rejected with 400 schema_mismatch if a required field is missing or null, a field is new, or a field has a type the examples never had. Text counts as the same type as numbers and booleans, so "58.12" fits a number field. The error lists up to 20 differences:

{ "error": { "type": "schema_mismatch", "message": "data doesn't match the scoring model's schema: missing field: account_age_days; unexpected field: account_age. Send skip_schema_validation to score it anyway, or update_schema to make this the new schema.", "param": "data" } }

A scoring model created from its criteria has no schema, and takes any record, until you calibrate it with examples (or a score call sends update_schema).

A scoring model is most reliable on records like its examples. If your data changes a lot, create a new one.

Score idsCopy Anchor Link

Every score call carries a score_id: your own id for that record, such as the payment's id, up to 200 characters. It's unique within the scoring model: sending it again gets 409 duplicate_score_id. Keep it with the record, and send it back with the outcome to calibrate the scoring model.

Questions and weightsCopy Anchor Link

Finic writes questions and uses the ones that best predict your outcomes, and you can add your own. GET /v1/scoring_models/{model_id} lists the questions a scoring model uses:

"questions": [
  { "id": "q004", "text": "This record is a card payment. Is the IP address in a different country from the billing address?", "source": "finic", "weight": 0.91 },
  { "id": "q022", "text": "This record is a card payment. Is this a typical amount for the merchant category?", "source": "finic", "weight": -0.52 },
  { "id": "q051", "text": "Was the account opened less than a week before the payment?", "source": "user", "weight": 0.38 }
]

• The strongest is listed first. Only the questions it uses are listed; its page in the Finic Portal also lists your own questions it doesn't use, under Not used.
• weight is how much a question moves the score, as of the latest calibration (before one, Finic's estimate): positive when a likely yes raises the score, negative when it lowers it. The larger it is either way, the more the question counts.
• source is finic for a question Finic wrote, user for one you added.

To see what's behind one record's score, send include_questions: true to POST /v1/score. The response then lists each question the score used, with Grayson's probability of yes for that record and the question's weight.

Changing questionsCopy Anchor Link

Write each question of your own as a yes/no question about one record. Add and remove questions at any time with add_questions and remove_question, or as you create or calibrate the scoring model:

• Before it's calibrated, a question you add counts right away, with a weight Finic estimates. You can remove any question, as long as one is left.
• Once it's calibrated, a question you add is asked on every score call and tried at the next calibration, which uses it if it helps: with examples, or with outcomes once 30 of the records it was asked about have one. Until then, and if the calibration doesn't use it, it's listed as not used on the scoring model's page in the Finic Portal, and tried again at later calibrations.
• Removing a question the calibrated score uses happens in a calibration, with remove_questions and its id, so the score is refit without it (remove_question gets 409 question_in_use). Your questions it doesn't use can be removed at any time. A removed question isn't asked or tried again.
• In a calibration with examples, add_questions adds questions too, and the scoring model uses the ones that help. Adding in a calibration needs examples: Finic doesn't keep the records it has seen, so it answers new questions on new labeled records.
• Up to 50 changes per scoring model a day, counting calibrations and creating it.

You can have up to 50 questions of your own per scoring model, used or not, adding up to 20 per request; each is up to 500 characters. A question that's blank, too long or already one of the scoring model's, or an id in remove_questions it doesn't have, gets 400 invalid_request.

JobsCopy Anchor Link

Creating and calibrating return 202 with a job_id at once; the work runs in a job.

• Creating takes less than a minute from its criteria, and about two minutes from examples (longer with more). Until then, score calls get 409 scoring_model_not_ready.
• Check the job with GET /v1/jobs/{job_id} every 10 seconds or so, until complete is true.
• A failed job has an error saying what to do. A failed creation leaves the scoring model unusable: create it again. A failed calibration leaves it as it was.
• job_interrupted means maintenance interrupted creating the scoring model or calibrating it with examples, which are held only in memory: create it again, or send the examples again.
• During a calibration, the scoring model keeps scoring with its current version.

New scoring models are limited to 100 per organization per day, and calibrations to 10 per scoring model per day; see Limits.

In the Finic PortalCopy Anchor Link

The Scoring models page lists your scoring models. There you can:

• Create a scoring model from its name and criteria.
• See its questions and their weights on its page, with a Not used yet section for your own questions it doesn't use, and add or remove questions there.
• Calibrate a scoring model from a CSV or JSON file of labeled examples or of outcomes, removing questions the score uses if you like.
• See how each calibration did as a precision-recall curve (how it's measured).
• Archive a scoring model to stop calls to it without deleting anything: score and calibrate return 409 scoring_model_archived until you unarchive it.
• Delete a scoring model (admins only). Its records and outcomes are deleted, and calls to it return 404 scoring_model_not_found. This can't be undone.

What's storedCopy Anchor Link

A scoring model keeps its name, its criteria, its schema (field names and types) and its questions, including the ones you added. Each score call keeps its score_id, its score, Grayson's answers to the questions and, once you calibrate, the outcome. Each example, whether sent to create the scoring model or to calibrate it, is kept the same way: its answers and its label. The examples and records themselves are processed in memory and aren't stored, whatever your data retention. Your scoring models are listed by list_scoring_models, and in the Finic Portal under Scoring models.