Quality & InspectionFree calculatorPDF, Excel, Word exportDiscrete Manufacturing

Gage Resolution and Number of Distinct Categories Calculator

Two quick checks catch many weak measurement systems: is the gage resolution fine enough for the tolerance, and can the system tell parts apart. This calculator does both from numbers you already have.

Your numbers

Total tolerance (USL - LSL)

Total width of the tolerance band.

Gage resolution (smallest increment)

Display increment or least count.

Part variation (PV)

From your gage R&R study.

Gage R&R standard deviation (GRR)

From your gage R&R study.

Your results

Tolerance to resolution ratio

A ratio of 10 or more is the common rule of thumb.

Number of distinct categories

At least 5 is generally expected.

Resolution needed for 10 to 1

0.02

Resolution as percent of tolerance

Passes 10 to 1 rule (1 = yes)

Rules of thumb from common MSA guidance. Your customer or standard may set different limits. Use the same units for all inputs.

Save results:

Review your measurement system strategy

Optional conversation about connecting gage and inspection data to your ERP.

Optional. The tool and every export work without this. No spam.

How to use it

Enter the total tolerance and the smallest increment your gage can display. The calculator shows the ratio of tolerance to resolution and the resolution that would satisfy a 10 to 1 rule. For the second check, enter the part variation and the gage R&R standard deviation from your study, and read the number of distinct categories.

Use the same units everywhere
Resolution means the display increment, not the accuracy
Use standard deviations, not study variation multiples, for PV and GRR

The formulas

The resolution check is tolerance divided by resolution, with 10 or more considered adequate. The number of distinct categories is 1.41 times PV divided by GRR, rounded down, following common AIAG MSA practice. A value of 5 or more suggests the system can distinguish parts well enough for process control, while below 2 means it is little better than a pass-fail call.

Reading the result

If the ratio is below 10, consider a gage with finer resolution, but note that a finer display does not fix poor repeatability. If ndc is low, look at the sources of variation in the gage R&R study, such as appraiser technique, fixturing or part form, before buying new equipment. Document the decision in your MSA plan.

Frequently Asked Questions

Why 10 to 1?

It is a long-standing rule of thumb that the gage increment should divide the tolerance into at least ten steps. It limits rounding effects and helps the measurement system see variation. Some applications accept less with justification.

Is resolution the same as accuracy?

No. Resolution is the smallest increment shown. Accuracy, bias and repeatability describe how close and consistent readings are. A gage can display many digits and still be inaccurate.

What if ndc is under 5?

The system may be too noisy for process control. Investigate repeatability and reproducibility, improve fixturing or training, or choose a better instrument. Re-run the study after changes.

Which standard deviations do I use?

Use the part variation and the gage R&R standard deviations from your study output, not the multiplied study variation figures. If you only have the percent columns, calculate from the raw figures if possible.

Netray can help bring gage study results from your ERP and quality system into one view, privately hosted.

Talk to an Expert Explore More Free Tools

Related Tools

Quality & Inspection

More Quality & Inspection tools

See all

Back to All Tools