What it takes for "90% sure" to mean 90%

TL;DR

• TypeSafe's Jev returns probabilities over typed options instead of text. The company was valued at $7.5B weeks after Jev launched on September 15.
• Guo et al. showed that modern deep nets are overconfident, measured this with ECE, and fixed it by dividing the logits by one learned temperature.
• Jev's pitch rests on calibration. This paper gives you the tools to test that claim, which TypeSafe has not yet backed with published evidence.

What happened

On October 9, TechCrunch reported that TypeSafe AI raised $870 million at a $7.5 billion valuation. Andreessen Horowitz led the round, and existing investors Sequoia and DCVC took part. TypeSafe was founded in 2024 by Diogo Almeida (formerly of OpenAI), Sasha Sheng (formerly of Meta) and Erik Gafni. Its model, Jev, launched on September 15. The company says Jev is built on a transformer but is not an LLM. It doesn't write text. Instead it produces "probabilities, or what the company calls 'calibrated decisions.'" Almeida's pitch: "computers speak a different language." According to TechCrunch, TypeSafe also claims that Jev is much faster than LLMs, uses far fewer tokens, and is already used by a third of Fortune 500 companies. Let's Data Science reports that the training method is called Reinforcement Learning for Calibrated Decisions (RLCD), and that end-to-end latency is 70 to 500 ms. Both are vendor statements, and no technical details have been released.

Very little has been checked independently. TechCrunch cites no benchmarks. One outside test, the jev-sec-bench repository on GitHub, ran Jev on 662 labelled messages from the deepset prompt-injection dataset. It reported 96.5% accuracy and an expected calibration error (ECE) of 0.0588. On a second task in the same repo, flagging vulnerable code, ECE was 0.1868, although the author warns that this synthetic corpus has "demonstrably noisy" labels. Victor Dibia's explainer proposes a plausible way Jev could work: score each candidate option with a language model, then normalize the scores. He notes that "TypeSafe has not disclosed how Jev is implemented."

The paper in brief

Guo and colleagues asked a simple question: when a classifier says it is 80% confident, is it right 80% of the time? For a 5-layer LeNet on CIFAR-100, roughly yes. For a 110-layer ResNet, no. It is more accurate, but its average confidence is well above its accuracy. They traced this miscalibration to trends that improved accuracy, namely more depth and width, Batch Normalization and less weight decay. They then compared post-hoc fixes and found that the simplest, temperature scaling, was usually the best. On CIFAR-100 it cut the ResNet-110's calibration error from 16.53% to 1.26% without changing a single prediction. The paper became the standard reference on deep-net calibration because it made the problem measurable and the fix a single line of code.

The math

The setup

A network $h$ reads an input $X$ and returns a predicted class $\hat{Y}$ from $K$ options, plus a confidence $\hat{P}$. The confidence is the top softmax probability, $\hat{p}_i = \max_k \sigma_{\text{SM}}(\mathbf{z}_i)^{(k)}$, where $\mathbf{z}_i$ is the vector of logits. The paper never touches accuracy, only whether $\hat{P}$ can be trusted.

Calibration, defined

$$\mathbb{P}\left(\hat{Y}=Y \mid \hat{P}=p\right)=p, \quad \forall p\in[0,1]$$

In words: among all predictions made with confidence $p$, a fraction $p$ should be correct. $Y$ is the true label, and the probability is taken over the data distribution. This is a statement about conditional probability, not about any single prediction. A model that always outputs its overall accuracy as its confidence is perfectly calibrated and useless for ranking. Calibration tells you whether the confidence numbers can be trusted. It does not tell you whether the model can separate easy inputs from hard ones.

Measuring it: reliability diagrams and ECE

$\hat{P}$ is continuous, so we bin it. Split $[0,1]$ into $M$ equal intervals and let $B_m$ be the set of predictions whose confidence falls in bin $m$. Then $\text{acc}(B_m)$ is the fraction of those predictions that are correct, and $\text{conf}(B_m)$ is their average confidence. A reliability diagram plots $\text{acc}(B_m)$ against the bins. Perfect calibration is the diagonal. The Expected Calibration Error collapses that plot into one number:

$$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\text{acc}(B_m)-\text{conf}(B_m)\right|$$

Here $n$ is the total number of predictions. ECE is a weighted average of the gaps between bars and diagonal, where each bin's weight is its share of the data. The Maximum Calibration Error replaces the weighted sum with $\max_m \left|\text{acc}(B_m)-\text{conf}(B_m)\right|$. That is the worst bin, and it matters when one bad region is unacceptable. The paper uses $M=15$.

Worked example. Take 100 predictions, sorted into three bins:

Bin	Count	Avg confidence	Accuracy
(0.9, 1.0]	50	0.97	0.80
(0.7, 0.8]	30	0.75	0.70
(0.5, 0.6]	20	0.55	0.55

ECE = 0.5 × 0.17 + 0.3 × 0.05 + 0.2 × 0 = 0.10, and MCE = 0.17. The bin carrying most of the weight is also the most overconfident one, which is the typical pattern for modern nets.

Why deep nets drift overconfident

Training minimizes negative log-likelihood (cross-entropy), $-\sum_i \log \hat{\pi}(y_i \mid \mathbf{x}_i)$. Suppose the network already classifies a training example correctly. If you multiply all its logits by some $\alpha>1$, the argmax stays the same, but the probability of the correct class rises and the loss falls. So once training accuracy is near 100%, the cheapest way to keep lowering the loss is to make the logits bigger, which means more confidence. The paper shows this on CIFAR-100: test NLL starts climbing (overfitting) while test error keeps falling from 29% to 27%. In the authors' words, overfitting shows up "in probabilistic error rather than classification error." One intuitive reading of their weight-decay result is that an L2 penalty caps how large the weights, and therefore the logits, can grow.

The fix: temperature scaling

Platt scaling, a classic method for binary classifiers, fits $\hat{q}_i=\sigma(a z_i + b)$ on held-out data. Temperature scaling is the multiclass version with a single parameter $T>0$:

$$\hat{q}_i=\max_k \, \sigma_{\text{SM}}\!\left(\mathbf{z}_i / T\right)^{(k)}$$

When $T>1$ the softmax gets flatter (higher entropy). $T=1$ gives back the original model. As $T\to\infty$ every class gets $1/K$. Dividing every logit by the same positive number never changes which one is largest, so predictions and accuracy are unchanged. Only the confidence moves. This is the same softmax temperature used in knowledge distillation and in LLM sampling, used here for a different job.

Take logits $(4, 2, 0)$. At $T=1$ the top probability is $e^4/(e^4+e^2+1)\approx 0.87$. At $T=2$ it becomes $e^2/(e^2+e+1)\approx 0.67$. The class is the same, but the model now claims less certainty.

One temperature straightens a reliability curve

interactive

• perfect calibration
• accuracy at this confidence

Hover or drag across the chart to read values.

hidden overconfidence k2.5temperature T1

accuracy when it says 90%:

70.7%

accuracy when it says 99%:

86.3%

calibrated?:

no

Real networks are not exactly this toy, but the paper's central finding is that they come surprisingly close. $T$ is fit by minimizing NLL on a held-out validation set, with the network frozen:

$$T^{*}=\arg\min_{T>0}\; -\sum_{i=1}^{n}\log \sigma_{\text{SM}}\!\left(\mathbf{z}_i/T\right)^{(y_i)}$$

This is maximum likelihood estimation of one parameter. Written in terms of $1/T$, the objective is a linear term plus a log-sum-exp, so it is convex. The authors report that a conjugate-gradient solver finds the optimum in about 10 iterations.

The alternatives, briefly

Histogram binning replaces each confidence with the empirical accuracy of its bin. Isotonic regression learns a monotone step function, choosing both the bin edges and the values. BBQ averages over many binning schemes in a Bayesian way. Vector and matrix scaling apply a diagonal or full linear map $\mathbf{W}\mathbf{z}+\mathbf{b}$ before the softmax. All of these are more flexible than temperature scaling, and that flexibility is where they get into trouble.

What the results say

• CIFAR-100, ResNet-110: ECE 16.53% before calibration. After calibration: 1.26% with temperature scaling, 2.66% with histogram binning, 4.99% with isotonic regression, 5.46% with BBQ, and 25.49% with matrix scaling, which made things worse.
• ImageNet, ResNet-152: 5.48% to 1.86% with temperature scaling. Matrix scaling failed to converge with 1,000 classes.
• Low-dimensional miscalibration: vector scaling learned nearly constant entries, so it ended up doing temperature scaling anyway. The authors conclude that "network miscalibration is intrinsically low dimensional."
• Speed: binning methods take roughly an order of magnitude longer than temperature scaling, and BBQ roughly three orders of magnitude longer.

Limitations. The paper says it "cannot claim causality" for the depth, BatchNorm and weight-decay trends. On Reuters, which starts out well calibrated, temperature scaling made calibration slightly worse (0.85% to 0.91%). The text says temperature scaling beats every other method on the vision tasks, but Table 1 has exceptions: histogram binning wins on Cars (1.74% vs 2.35%) and on CIFAR-10 ResNet-110 (0.58% vs 0.83%). ECE depends on the binning scheme. Most importantly, every method assumes that validation and test data come from the same distribution. A single $T$ can only fix miscalibration that looks like uniform logit inflation.

Connecting the dots

Guo et al. is not the paper behind Jev. TypeSafe hasn't said how RLCD works. Still, Jev's interface (choose among $K$ typed options and return normalized probabilities) is exactly the setting this paper studies. That makes reliability diagrams and ECE the obvious evidence to ask for. The outside ECE of 0.0588 on prompt injection is one measurement on one dataset, and the repo doesn't say what binning it used. It is encouraging, but it is not an established result. The 0.1868 on the noisy code task shows how much the number depends on the data.

Dibia's experiments, which used open Qwen2.5 models rather than Jev, echo the paper's story in an LLM setting that Guo et al. never studied. His 7B instruct model, on a 10-option Banking77 intent task, put 1,152 of 1,200 answers above 90% confidence, and only 90.6% of those were right. ECE was 0.10, against 0.02 for the base model. After temperature scaling, ECE fell to 0.03, and the 872 answers still above 90% were 97.4% correct.

The news goes beyond the paper in one important way. RLCD is presented as a training objective, while Guo et al. calibrate after training on held-out data from the same distribution. A general-purpose decision model will be used on each customer's own tasks, and the paper offers no guarantee that calibration measured on one distribution carries over to another. Dibia makes the practical point directly: temperature scaling needs "labelled examples from your own task."

Why it matters

Automation decisions depend on thresholds, such as "act automatically above 95%, otherwise escalate to a human." That rule only works if 95% means 95%. For practitioners, the paper's lesson is cheap and actionable. Hold out some labelled data, draw a reliability diagram, and fit $T$ if the model is overconfident. For research, it raises the question of whether calibration can be built into training rather than patched on afterwards, which is the claim RLCD makes. For buyers and regulators, "calibrated" is a testable claim, and the test is a reliability diagram on your own data, not a press release.

Things to watch: whether TypeSafe publishes RLCD details or per-task reliability diagrams, and whether independent ECE measurements across several tasks, especially under distribution shift, match the single prompt-injection result.

Go deeper

• On Calibration of Modern Neural Networks (arXiv:1706.04599), the paper, with a PyTorch implementation of temperature scaling from co-author Geoff Pleiss
• TechCrunch: The maker of non-text AI model Jev valued at $7.5B, the funding news and TypeSafe's claims
• Victor Dibia: How Jev works, a hands-on explainer that measures scoring versus generation and applies temperature scaling to open models
• jev-sec-bench, the outside benchmark behind the 0.0588 ECE figure, with its caveats
• Let's Data Science: TypeSafe AI launches Jev, on RLCD and the vendor-reported latency