Attribute Agreement Analysis Worksheet
Not every measurement is a number. Visual inspection, go/no-go gaging and cosmetic checks produce pass or fail decisions, and those decisions need their own study. Attribute agreement analysis measures how consistent inspectors are with themselves and with the truth.
| Sample no. | Reference (1 = pass, 0 = fail) | Appraiser A, trial 1 | Appraiser A, trial 2 | Appraiser B, trial 1 | Appraiser B, trial 2 | A repeats (1 = yes) | B repeats (1 = yes) | A matches reference | B matches reference | All four match reference | |
|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 1 | 1 | 1 | 1 | |||||||
1 | 1 | 1 | 1 | 1 | |||||||
1 | 0 | 1 | 0 | 0 | |||||||
0 | 1 | 0 | 1 | 0 | |||||||
1 | 1 | 1 | 1 | 1 | |||||||
1 | 1 | 0 | 1 | 0 | |||||||
1 | 1 | 1 | 1 | 1 | |||||||
1 | 1 | 1 | 1 | 1 |
Enter 1 for pass and 0 for fail on every call. Use at least 30 samples in a real study, mixing clearly good, clearly bad and borderline units, and randomize the order. Kappa statistics need a statistics package; this sheet gives the percent agreement figures.
Analyse inspection decisions from your ERP data
We can help look at inspector results already captured in your ERP or MES and highlight inconsistency.
Optional. The tool and every export work without this. No spam.
How to use it
Select samples that include clear passes, clear fails and borderline cases, and have an expert establish the reference decision for each. Give them to each appraiser in random order, at least twice, without letting anyone see previous calls. Enter 1 for pass and 0 for fail in the sample rows. The sheet marks whether each appraiser repeated their own call, whether they matched the reference, and totals the percent agreement at the bottom. Add rows until you have at least 30 samples.
Reading the results
Within-appraiser agreement shows repeatability: does the same inspector give the same answer twice. Appraiser versus reference shows accuracy. The all-match figure is the toughest: every call on a sample agrees with the standard. AIAG guidance commonly treats kappa of 0.9 or higher as good and below 0.7 as needing improvement. Percent agreement alone does not remove chance agreement, so use kappa from a statistics package when the result drives a decision. Separately look at miss rate (failing parts passed) and false alarm rate.
- Within-appraiser: repeatability
- Versus reference: accuracy
- Miss rate matters more than false alarms for safety
- Kappa corrects for chance
Common mistakes
Using only obvious good and bad samples makes everyone look perfect. Using too few samples gives wide error bars, so a 100% result on 10 parts proves little. Letting appraisers discuss calls ruins independence. Neglecting the reference decision is the biggest gap: without a trusted standard you can measure consistency, but not correctness. If results are poor, improve the work instruction, add boundary samples or limit samples to retrain, then repeat the study.
Frequently Asked Questions
How many samples do I need?
A common recommendation is at least 30 samples, with roughly equal numbers of good, bad and borderline units, and 2 or 3 trials per appraiser. Larger studies narrow the confidence interval on the agreement figures.
What is a good agreement percentage?
Targets differ by industry. As a rule, kappa of 0.9 or above is considered good and below 0.7 unacceptable under AIAG guidance. Percent agreement should be as close to 100% as practical, with extra attention to missed defects.
What is the difference between this and gage R&R?
Gage R&R in the variable sense measures continuous readings and splits variation into repeatability and reproducibility. Attribute agreement analysis handles discrete decisions such as pass or fail and uses agreement rates and kappa.
Do I need a reference standard?
For accuracy, yes. Without a known reference you can still test whether appraisers agree with themselves and each other, but not whether they are right. Have an experienced expert or a measuring system establish the reference.
If inspection pass or fail results are already captured in your ERP or MES, Netray can help analyse that data for consistency issues.
Related Tools
Gage R&R Average and Range Method Calculator
Turn the averages from a variable gage R&R study into EV, AV, GRR, part variation, %GRR and number of distinct categories.
Quality & InspectionMeasurement Uncertainty Budget Worksheet
List each uncertainty source, convert it to a standard uncertainty and combine them into expanded uncertainty.
Quality & InspectionGage Master List Worksheet
An editable inventory of every measuring device with identification, location, status and calibration interval.
Quality & InspectionCalibration Log Template
An editable calibration record log with as-found results and a running out-of-tolerance rate.
More Quality & Inspection tools
See all- SAP QM Process Capability Cp Cpk Calculator
- SAP QM Lot Rejection Rate, PPM and Sigma Level Calculator
- SAP QM Gauge R&R Study Calculator
- SAP QM Cp and Cpk Capability Calculator
- Job Shop Material Certification Log Worksheet
- Job Shop First Piece Inspection Sign-Off Template
- Job Shop Scrap and Rework Cost Calculator
- Master Data Audit Sample Size Calculator
- Data Quality Sigma Level Calculator
Keep exploring
Free tools
- Supplier Quality Agreement TemplateA fill-in supplier quality agreement outline covering quality system, change notification, nonconformance, traceability and records. Export to PDF or Word.
- Oracle EBS Item Attribute Template WorksheetPlan item master attribute values for a new item template, with the underlying MTL_SYSTEM_ITEMS_B column for each attribute.
- SAP Scheduling Agreement Delivery Schedule WorksheetLay out schedule lines for a scheduling agreement, track cumulative quantity against the target and see open balance.
ERP fixes and how-tos
- Oracle Workflow error: Attribute does not exist for item typeFix Oracle Workflow's Attribute does not exist for item type error in EBS by checking the item type's attribute definitions in Workflow Builder against every activity and message referencing it, including custom customizations.
- ERP Data Sovereignty for AI: What CIOs Need to CheckHow to add AI to an ERP without violating data sovereignty or residency requirements: what actually leaves your infrastructure, hosting options, and a due-diligence checklist.
- How to Search and Upload Documents with the Infor IDM REST APIUse the IDM REST API through ION API to search by entity attributes, fetch a document resource and upload a new item with attributes. Request examples included.
Guides
- Service Level Agreement Tracking with ERP IntegrationMonitor and enforce service level agreements with integrated field service and ERP data for response time tracking, penalty calculation, and compliance reporting.
- AI Review of Medical Device RFQs and Supplier Quality AgreementsUse private AI to read medical device RFQ packages and supplier quality agreements: extract validation, traceability, labeling and change control obligations.
- Setting Up Cost Variance Analysis in SyteLineConfigure cost variance analysis in Infor SyteLine for material, labor, and overhead variances. Set up variance reporting for standard cost manufacturing environments.
AI on your ERP
- AI for Salesforce Manufacturing Cloud: Sales Agreements Reconciled with ERP ActualsConnect Salesforce Manufacturing Cloud sales agreements and forecasts to ERP actuals with private AI: run-rate variance, rebate checks and account planning.
- IFS Field Service Management AI Integration: Private AI on IFS Cloud Service DataAI for IFS Field Service Management: service requests, work tasks, agreements and technician data via IFS Cloud APIs, joined to parts and finance. On-prem.