Limits
Request size, question and option counts, name lengths and rate limits.
| Limit | Value |
|---|---|
| Tokens per request (state + all questions + markers) | 32,768 |
| Request body | 2 MB |
| Questions per request | 1–128 |
Options per choice or multi_choice | 2–255 |
Levels per score | 2–32 |
| Name length (questions and option keys) | 1–64 characters |
A request over a limit fails with 400 invalid_request, or with 413 request_too_large for the token and body-size limits. See Errors.
How tokens are counted
usage.input_tokens in each response counts everything the model reads:
- the state: a string as sent, or an object or array as compact JSON
- for a
noul,choiceorscore: its instructions and each option description or level, plus marker tokens, 2 per question and 1 per option or level - for a
multi_choice: the model asks about each option separately, so each option costs the instructions, the list of all option descriptions (when there are 16 options or fewer), the option itself and 2 marker tokens
A multi_choice therefore grows with the number of options: one with 8 options reads its instructions 8 times. Question names and option keys aren't shown to the model, so they don't count.
To size your requests, send a representative one and read usage.input_tokens. A request over 32,768 tokens fails with 413 request_too_large, and the message gives its size. State has ways to fit a long history.
Rate limits
Each API key is rate limited. Too many requests in a short time return 429 rate_limited, which doesn't use credit. Responses carry no Retry-After or rate-limit headers, so retry with exponential backoff; see Retrying. To plan for higher volume, contact support@finic.ai.