Reliax1The two guarantees
1.1Reliability of the inference
Reliax builds on a family of methods called conformal prediction. The idea is simple enough to state in four steps.
- Take past decisions with known outcomes that the model did not train on. This is the calibration data.
- Measure how wrong the model is on each of them. For each past decision, one minus the probability the model gave to what actually happened.
- Pick the threshold from those errors. Sort them from smallest to largest and take the value that 95% of them stay below. The 95% is a parameter the lender sets.
- For a new decision, check each possible outcome against the threshold. An outcome whose would-be error stays below the threshold is kept; call the kept outcomes the guaranteed set. If exactly one outcome is kept, that is the model's answer, and it is guaranteed. Otherwise a person decides.
What is proven: on data like the calibration data, the guaranteed set contains the true outcome at least 95% of the time, for any model and any size of data (Vovk et al., 2005). That is a property of the procedure over many decisions, never a probability for one applicant. We do not take it on faith: we count it, on held-out applicants, in section 2.
The lender chooses the target. A higher target means fewer wrong automatic answers, but more decisions sent to a person. The figure shows that trade-off on three credit datasets.
ShowHide the numbers
| Target | Give Me Some Credit | Home Credit 2018 | Taiwan |
|---|---|---|---|
| 82% | · | · | 0.6% |
| 83% | · | · | 2.7% |
| 84% | · | · | 4.8% |
| 85% | · | · | 7.2% |
| 86% | · | · | 10.2% |
| 87% | · | · | 13.2% |
| 88% | · | · | 15.9% |
| 89% | · | · | 18.6% |
| 90% | · | · | 22.6% |
| 91% | · | · | 26.3% |
| 92% | · | · | 30.6% |
| 93% | · | 3% | 36% |
| 94% | 0.9% | 7% | 41.3% |
| 95% | 3.4% | 11.9% | 47% |
| 96% | 6.4% | 18.4% | 54.2% |
| 97% | 10.9% | 27.8% | 61.9% |
| 98% | 19.9% | 39.4% | 71.5% |
| 99% | 35.8% | 58.1% | 81.9% |
Share of decisions sent to a person, mean of five runs. "·": target not shown, because some cases would get no answer at all.
The guarantee holds over everyone, not automatically for each group. So the rate is also measured inside each audited group, such as age, sex or ethnicity. Attributes the lender may not use to decide are used only for this check, never in the decision.
1.2The lender's cut-off
A lender approves when the predicted risk is below its break-even point: with a margin M on a good loan and a loss L on a default, lending pays when the default probability is below PD* = M / (M + L). With M = 600 and L = 5,000, PD* is 10.7%. The lender keeps that cut-off. Reliax proves which decisions are on the right side of it.
The calibration scores are cut into 20 equal slices. For each slice, the default rate gets an exact lower and upper bound (Clopper and Pearson, 1934). Two tests are then run in a fixed order, each at 2.5%: from the safest slice up, each slice is approved automatically only if its upper bound is below PD*; from the riskiest slice down, each is refused automatically only if its lower bound is above PD*. Each test stops at the first slice that fails. The slices neither test can settle go to a person. Jointly, with 95% confidence, every automatic approval comes from a slice that defaults at most PD*, and every automatic refusal from one that defaults at least PD* (fixed-sequence testing, in the Learn-then-Test framework of Angelopoulos et al., 2021).
1.3The checks around them
- The input check. Each input's distance to the calibration data is compared with the distances of the calibration decisions themselves. An input that sits farther than almost all of them is one the guarantee does not cover, and a person decides.
- The drift test. The same distances are watched across many decisions. If new inputs keep landing far from the calibration data, the population has moved, and the test raises an alarm. If nothing has moved, the chance of a false alarm over the whole life of the stream is at most 1% (Ville, 1939).
- The outcome check. When outcomes land, the guaranteed rate is recomputed on the new decisions and compared with what was claimed. Those decisions can then refresh the calibration data; that is how the guarantee is recalibrated after a drift.
2Results
2.1Taiwan and German
| The question | Taiwan · 30,000 applicants | German · 1,000 applicants |
|---|---|---|
| Can this answer be trusted? | Guarantee held: 0.9515 against a 0.95 target, with 54% of decisions sent to a person. | Guarantee held: 0.9532 against 0.95. |
| Does it hold for every group? | Worst audited group (age 46+) at 0.945, within noise of the target. | Women at 0.943. 75 test cases per run: indicative. |
| What happens when the world changes? | 0 false alarms on 5 stable streams. A real shift caught in 4 of 5 streams. The share sent to a person rose by itself from 26% to 48%. | Not measured: too few applicants to form streams. |
On 4.7 million US mortgage decisions (HMDA 2025), the guarantee held over everyone at 0.950, but was 0.924 for Black applicants and 0.936 for Hispanic applicants, against 0.956 for White applicants. The model never saw race; the group check is what shows the gap.
2.2The test fixed in advance
Two datasets, Give Me Some Credit and Home Credit Default Risk, were not used until a test, its pass mark and its code had been written down and published. They were then opened and the test run once. The result is reported as it came out: the pass mark was missed.
| Stream | Give Me Some Credit | Home Credit Default Risk 2018 |
|---|---|---|
| No shift | Guarantee 95.1% at a 95% target; 1.1% sent to a person | Guarantee 95.0%; 15.5% sent to a person |
| Moderate shift | At a 50% shift, reviewing the unusual inputs first beat the model's confidence (1.19 times). At 25%, the shift was too mild to count. Not met | Reviewing the unusual inputs first did worse than the model's confidence. Not met |
| Severe shift | Drift alarm in 5 of 5 runs, within about 5 to 7 decisions. The guarantee falls to 81% and 77%. Met | Drift alarm in 3 of 5 runs at a 75% shift and 4 of 5 at 100%, after about 1,400 decisions. Not met |
| Unit error, 5% and 10% | 95% and 82% of the corrupted decisions caught (confidence: 5%). Met | 100% and 95% caught (confidence: 3% to 4%). Met |
On Home Credit the shifted applicants are defined by credit-bureau data the model never sees; in the application fields they look ordinary, so nothing that reads the inputs can catch that shift. The outcome check does, once defaults land. The full test, with intervals, is in section 5 of the full version.
2.3At the lender's cut-off exploratory
On a model built the way a credit team would build it, with M = 600 and L = 5,000 (PD* 10.7%). Mean of five runs on held-out test rows. This was designed after section 2.2 was seen, so it is exploratory.
| Taiwan | Give Me Some Credit | Home Credit Default Risk 2018 | |
|---|---|---|---|
| Approved / sent to a person / refused | 13% / 39% / 48% | 80% / 6% / 14% | 64% / 19% / 18% |
| Default rate among automatic approvals: lender alone / with Reliax | 7.0% / 5.6% | 2.7% / 2.4% | 4.6% / 3.7% |
| Test runs where approvals came out above PD*, or refusals below it | 0 of 100 | 0 of 100 | 0 of 100 |
The cases sent to a person default at about PD*, 10%: exactly the ones where lending is a coin toss for the lender. How many there are depends on the model and on the calibration data, not on Reliax: 6% on Give Me Some Credit, where the model separates risk well, and 39% on Taiwan, where it is weaker.
3Where the value stops
- Not a better filter on stable data. Ranking decisions by the model's confidence catches as many mistakes at the same review rate. What Reliax adds is the guarantee behind the threshold, the alarm and the record.
- The calibration data is a requirement. The guarantees need a few thousand past decisions with known outcomes, drawn from the population the model serves and kept out of its training. In credit, outcomes arrive late, so the deployment starts on 30, 60 and 90-day delinquency and recalibrates as defaults land.
- The share sent to a person is the model's number. A better model sends fewer decisions to a person.
- The guarantee is over many decisions, not for one person. Reliax never prints a percentage beside a single decision.
- A shift that does not show in the inputs is caught late. The outcome check and recalibration catch it, once outcomes land.
- These are our own measurements. The code and data are public so that anyone can rerun them.
4Code and references
The package is public at github.com/reliax-io/reliax-evaluation, with the code for every experiment. Give Me Some Credit and Home Credit are fetched from Kaggle by script and never redistributed. The full version has every table, the figures and the annexes.
- Angelopoulos, A. N., Bates, S. (2023). Conformal prediction: a gentle introduction. Foundations and Trends in Machine Learning, 16(4).
- Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., Lei, L. (2021). Learn then test: calibrating predictive algorithms to achieve risk control. arXiv:2110.01052.
- Clopper, C. J., Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4).
- Ville, J. (1939). Étude critique de la notion de collectif. Gauthier-Villars.
- Vovk, V., Gammerman, A., Shafer, G. (2005). Algorithmic Learning in a Random World. Springer.
- Yeh, I-C., Lien, C-h. (2009). The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2).
This is a first evaluation on public data. The next step is to replay it on your own past decisions, with success criteria agreed in advance.
Email: contact@reliax.io