Legion Diagnostics

Troubleshooting / Common Issues

Low Positive Percent Agreement (high false-negative rate)

Possible Causes

  • Test method has a higher positivity threshold (cutoff) than the reference method
  • Matrix differences between the reference and test method platforms causing suppression of signal in positive specimens
  • Near-cutoff specimens dominating the positive specimen set — agreement naturally decreases when many positives are clustered near the decision point
  • Insufficient positive specimens in the study — CLSI EP12 requires at least 20 total specimens with approximately equal positive and negative distribution; fewer positive specimens yields very wide PPA confidence intervals

Recommended Action

Review the confusion matrix to confirm which specimens are being mis-classified. Examine whether the false-negative specimens are near the cutoff (index values close to the threshold). If the test method uses a numeric cutoff, verify that the configured cutoff matches the manufacturer's specification and review the raw numeric values for the discordant specimens.

Significant McNemar test result (p ≤ 0.05)

Possible Causes

  • The test method systematically over-calls positive results relative to the reference method (more Test Positive / Reference Negative discordances than the reverse)
  • The test method systematically under-calls positive results (more Test Negative / Reference Positive discordances than the reverse)
  • The study specimen set has very few discordant pairs — small absolute discordance counts can produce significant p-values by chance when the distribution is lopsided

Recommended Action

Examine the directional discordance counts (Test > Reference and Test < Reference). If one direction strongly dominates, investigate whether the test method's cutoff or signal gain is calibrated differently from the reference method. A significant McNemar result with comparable PPA and NPA may indicate an acceptable, clinically unimportant imbalance — apply clinical judgment.

Wide confidence interval for Cohen's Kappa

Possible Causes

  • Insufficient total specimen count — fewer than 20 specimens yields imprecise Kappa estimates
  • Imbalanced specimen prevalence — a very high or very low proportion of positives reduces the stability of the Kappa estimate

Recommended Action

Increase the total number of specimens. CLSI EP12 requires at least 20 specimens with an approximately equal distribution of positive and negative results. Studies with 50 or more well-distributed specimens yield substantially narrower confidence intervals for PPA, NPA, and Kappa, making pass/fail determinations more defensible.

Red-bubble discordances (2+ level disagreements) in the bubble chart

Possible Causes

  • Specimens with true discordant results between the methods — e.g., one method classifies as Positive while the other classifies as Negative
  • Incorrect cutoff configuration resulting in wrong categorical assignment for numeric results
  • Data entry or transcription errors pairing results from different specimens

Recommended Action

Investigate each red-bubble specimen individually. Retrieve the original result records for both methods and confirm the values were correctly entered. For semi-quantitative data, verify that the configured cutoff matches your instrument's documentation. If a specimen is confirmed discordant, document the clinical context. Do not exclude specimens solely because they produce a large discordance — investigate the reason before any exclusion decision is made.

Agreement statistics appear perfect (100%) but the study is small

Possible Causes

  • Fewer than 20 total specimens — with small sample sizes, perfect agreement may occur by chance and confidence intervals will be very wide
  • Specimens may not represent the full range of clinical presentations, particularly near-cutoff specimens

Recommended Action

Review the 95% confidence intervals for PPA and NPA. A point estimate of 100% agreement with a lower CI bound below 72% (as seen with N=20, per CLSI EP12 Table 12) indicates insufficient precision. Expand the specimen set to include near-cutoff specimens and a wider range of clinical presentations. CLSI EP12 requires a minimum of 20 specimens with approximately equal positive and negative distribution for a defensible verification study.