The number every analyst checks first.

Open almost any initial calibration (ICAL) report and the first value a reviewer looks for is the correlation coefficient. Many methods and SOPs still set acceptance at r ≥ 0.995, which is the same requirement as r² ≥ 0.990. If the curve clears that line, it is accepted, and every sample in the batch is quantitated against it.

The habit is understandable. r² is a single number, every chromatography data system reports it, and a value of 0.999 looks like proof of a good fit. The problem is that r² answers a different question from the one the lab needs answered. The lab needs to know how accurately the curve will report a concentration, especially near the reporting limit. r² describes how well the points line up across the whole range.

What r² actually measures.

The coefficient of determination is the fraction of the variance in instrument response that the regression line explains. An unweighted least-squares fit places the line to minimize the sum of squared absolute residuals. Detector response grows with concentration, so the absolute residuals at the top of the range are far larger than those at the bottom, and they dominate both the fit and the r² value.

Take a residual of 200 area counts. At a 100 ppb standard with a response near 100,000, that is a 0.2% miss. At a 0.5 ppb standard with a response near 500, the same 200 counts is an error of roughly 40%. The regression treats those two misses as equal. The analyst reporting a result near the low standard does not.

This is why the International Union of Pure and Applied Chemistry has described the correlation coefficient as having no meaning in calibration, a position Burrows and Parr cite in their review of calibration acceptance practice [1].

A worked example.

The table below is an eight-level calibration from 0.5 to 100 ppb, with the kind of proportional scatter typical of chromatographic detectors. The same data were fit three ways: unweighted linear regression, linear regression weighted 1/x, and linear regression weighted 1/x². Each standard was then back-calculated against its own curve and compared with its true concentration.

Illustrative data set constructed for this article. RE is the relative error of each back-calculated standard. Every derived value is computed directly from the responses shown.

The unweighted curve has the highest r² of the three, and it is the only one that would bias a low-level result by more than a third. A sample that truly contains 0.5 ppb would be reported near 0.32 ppb. If the reporting limit sat at 0.5 ppb, that sample would be reported as not detected. The weighted curves have a marginally lower r², and they hold every standard within 3.5% of its true value.

Ranking these three fits by r² puts them in exactly the wrong order. Burrows and Parr report the same pattern in real environmental data: of the curves they compared, the unweighted regression was the only one that passed an r = 0.995 criterion, and it carried the largest error at the low end [1].

What a reviewer should look at instead.

None of the alternatives is exotic. Each one measures error in the units the lab reports, as concentration relative to the true value.

Why this matters in an audit.

Accreditation requirements have already moved. The TNI 2016 standard for environmental laboratories requires calibration curves generated by regression to be evaluated by RSE or by the relative error of the low and mid calibration points [2]. Recent revisions of EPA SW-846 methods, ASTM methods, and the wastewater methods under 40 CFR Part 136 include RSE, relative error, or both as acceptance options [1][3].

The practical consequence is that an auditor can ask two questions a correlation coefficient cannot answer. First, what error does this curve introduce at the reporting limit? Second, why was this model and this weighting chosen over the alternatives? A lab that accepted the curve on r² alone has no documented answer to either. In a data integrity review, an undocumented model choice is a weakness even when the reported results happen to be correct.

A review checklist for every ICAL.

1. Back-calculate every standard. Record the relative error at each calibration level against the curve it belongs to.
2. Report the right pooled statistic. RSE for regression curves, RSD for average response factor curves, against the limit in your method or SOP.
3. Check the low and mid standards on their own. A good pooled statistic can hide one bad level.
4. Plot relative residuals against concentration. Look for a trend, curvature, or a fan shape.
5. Compare every model your method permits. Average RF, linear and quadratic, each unweighted and weighted, judged on the same error metrics.
6. Document the choice. Record why the selected model and weighting were chosen, so the rationale is in the record before an auditor asks.
7. Treat r² as a supporting value. It can flag a gross problem. It cannot confirm a good curve.

Where WLTR fits.

Step five is where manual review usually stops short. Fitting and comparing every permitted model for every analyte is slow work in a spreadsheet, so most labs evaluate only the model their chromatography data system applies by default.

WLTR is calibration data integrity infrastructure for regulated analytical laboratories. For each analyte, it derives nine calibration regression models (average response factor, plus linear and quadratic fits across four weightings each) and evaluates them against fifteen ICAL parameters, including RSE, computed orthogonally to the CDS. The methodology is covered by U.S. Patent 12,038,448 B2 [4].