Quality & InspectionFree worksheetPDF, Excel, Word export

Attribute Agreement Analysis Worksheet

Not every measurement is a number. Visual inspection, go/no-go gaging and cosmetic checks produce pass or fail decisions, and those decisions need their own study. Attribute agreement analysis measures how consistent inspectors are with themselves and with the truth.

Download:
Sample no.Reference (1 = pass, 0 = fail)Appraiser A, trial 1Appraiser A, trial 2Appraiser B, trial 1Appraiser B, trial 2A repeats (1 = yes)B repeats (1 = yes)A matches referenceB matches referenceAll four match reference
1
1
1
1
1
1
1
1
1
1
1
0
1
0
0
0
1
0
1
0
1
1
1
1
1
1
1
0
1
0
1
1
1
1
1
1
1
1
1
1
Samples
8
A repeats
7
B repeats
7
A matches reference
6
B matches reference
7
All match
5
A within-appraiser agreement
87.5%
B within-appraiser agreement
87.5%
A versus reference
75%
B versus reference
87.5%
All appraisers versus reference
62.5%

Enter 1 for pass and 0 for fail on every call. Use at least 30 samples in a real study, mixing clearly good, clearly bad and borderline units, and randomize the order. Kappa statistics need a statistics package; this sheet gives the percent agreement figures.

Analyse inspection decisions from your ERP data

We can help look at inspector results already captured in your ERP or MES and highlight inconsistency.

Optional. The tool and every export work without this. No spam.

How to use it

Select samples that include clear passes, clear fails and borderline cases, and have an expert establish the reference decision for each. Give them to each appraiser in random order, at least twice, without letting anyone see previous calls. Enter 1 for pass and 0 for fail in the sample rows. The sheet marks whether each appraiser repeated their own call, whether they matched the reference, and totals the percent agreement at the bottom. Add rows until you have at least 30 samples.

Reading the results

Within-appraiser agreement shows repeatability: does the same inspector give the same answer twice. Appraiser versus reference shows accuracy. The all-match figure is the toughest: every call on a sample agrees with the standard. AIAG guidance commonly treats kappa of 0.9 or higher as good and below 0.7 as needing improvement. Percent agreement alone does not remove chance agreement, so use kappa from a statistics package when the result drives a decision. Separately look at miss rate (failing parts passed) and false alarm rate.

  • Within-appraiser: repeatability
  • Versus reference: accuracy
  • Miss rate matters more than false alarms for safety
  • Kappa corrects for chance

Common mistakes

Using only obvious good and bad samples makes everyone look perfect. Using too few samples gives wide error bars, so a 100% result on 10 parts proves little. Letting appraisers discuss calls ruins independence. Neglecting the reference decision is the biggest gap: without a trusted standard you can measure consistency, but not correctness. If results are poor, improve the work instruction, add boundary samples or limit samples to retrain, then repeat the study.

Frequently Asked Questions

How many samples do I need?

A common recommendation is at least 30 samples, with roughly equal numbers of good, bad and borderline units, and 2 or 3 trials per appraiser. Larger studies narrow the confidence interval on the agreement figures.

What is a good agreement percentage?

Targets differ by industry. As a rule, kappa of 0.9 or above is considered good and below 0.7 unacceptable under AIAG guidance. Percent agreement should be as close to 100% as practical, with extra attention to missed defects.

What is the difference between this and gage R&R?

Gage R&R in the variable sense measures continuous readings and splits variation into repeatability and reproducibility. Attribute agreement analysis handles discrete decisions such as pass or fail and uses agreement rates and kappa.

Do I need a reference standard?

For accuracy, yes. Without a known reference you can still test whether appraisers agree with themselves and each other, but not whether they are right. Have an experienced expert or a measuring system establish the reference.

If inspection pass or fail results are already captured in your ERP or MES, Netray can help analyse that data for consistency issues.

Talk to an ExpertExplore More Free Tools

Related Tools

Quality & Inspection

Gage R&R Average and Range Method Calculator

Turn the averages from a variable gage R&R study into EV, AV, GRR, part variation, %GRR and number of distinct categories.

Quality & Inspection

Measurement Uncertainty Budget Worksheet

List each uncertainty source, convert it to a standard uncertainty and combine them into expanded uncertainty.

Quality & Inspection

Gage Master List Worksheet

An editable inventory of every measuring device with identification, location, status and calibration interval.

Quality & Inspection

Calibration Log Template

An editable calibration record log with as-found results and a running out-of-tolerance rate.

More Quality & Inspection tools

See all
Back to All Tools