summaryrefslogtreecommitdiff
path: root/DomainDigUITests/AccessibilityAuditHarness.swift
diff options
context:
space:
mode:
authorChristian Cleberg <[email protected]>2026-07-20 17:38:46 -0500
committerChristian Cleberg <[email protected]>2026-07-20 17:55:24 -0500
commitea57052e998007dc914257f84f383c63f945cf08 (patch)
tree0863b3a7180b2807d9709193ca0b382a8b97281c /DomainDigUITests/AccessibilityAuditHarness.swift
parent38723030eb0db58a9d62a45c7725861901309a05 (diff)
downloaddomain-dig-ea57052e998007dc914257f84f383c63f945cf08.tar.gz
domain-dig-ea57052e998007dc914257f84f383c63f945cf08.tar.bz2
domain-dig-ea57052e998007dc914257f84f383c63f945cf08.zip
fix(a11y): survive audit timeouts, and stop overclaiming CI floor coverage
Two problems the first CI run exposed. Audit timeouts. Three tests failed with "Audit failed to complete in time" (code -56) on the GitHub runner. That is the audit's own internal deadline on a slower machine, not an app defect, and the harness had no resilience to it. Audits now retry up to three times, and a screen that still cannot be audited is reported via XCTSkip rather than passing. Skips are distinct from passes in CI, so an unaudited screen stays visible instead of being silently counted as clean. The Dynamic Type sweep attempts every screen before skipping, so one slow screen cannot drop the other four. Overclaimed floor coverage. The two-simulator matrix was justified on covering the oldest supported OS, but the macos-26 image ships only iOS 26.x runtimes, so "floor" resolved to 26.2 and "current" to 26.5 — the run compared two 26.x images and never touched an 18.x one. The measured non-nested coverage that motivated the matrix (18.6 vs 27.0) reproduces locally but not on this runner. The workflow comment now states this plainly, and the selection step emits a warning annotation when the resolved floor sits a major version or more above the deployment target, so the gap is visible in the CI UI rather than assumed away. Installing an older runtime in CI is possible via xcodebuild -downloadPlatform but costs several GB and minutes per job; left out pending a call on whether that trade is worth it.
Diffstat (limited to 'DomainDigUITests/AccessibilityAuditHarness.swift')
-rw-r--r--DomainDigUITests/AccessibilityAuditHarness.swift61
1 files changed, 54 insertions, 7 deletions
diff --git a/DomainDigUITests/AccessibilityAuditHarness.swift b/DomainDigUITests/AccessibilityAuditHarness.swift
index 80257e1..9767d95 100644
--- a/DomainDigUITests/AccessibilityAuditHarness.swift
+++ b/DomainDigUITests/AccessibilityAuditHarness.swift
@@ -40,6 +40,9 @@ enum AccessibilityAuditHarness {
/// `.sufficientElementDescription`, `.trait`
static let enforcedAuditTypes: XCUIAccessibilityAuditType = []
+ /// How many times to retry an audit that misses its internal deadline.
+ private static let auditAttempts = 3
+
/// Launches the app with feature gating lifted, optionally at a specific
/// content size category.
static func launch(contentSizeCategory: String? = nil) -> XCUIApplication {
@@ -56,19 +59,53 @@ enum AccessibilityAuditHarness {
///
/// Findings are logged and attached to the result bundle so a CI run
/// produces the burndown list as an artifact rather than only a pass/fail.
+ ///
+ /// Returns `false` if the audit could not complete, leaving the screen
+ /// unaudited. Callers turn that into an `XCTSkip` — reporting a pass would
+ /// claim coverage that did not happen.
+ @discardableResult
static func audit(
_ app: XCUIApplication,
screen: String,
test: XCTestCase
- ) throws {
+ ) throws -> Bool {
var findings: [String] = []
+ var timeout: Error?
+
+ // The audit traverses the whole element tree and has its own internal
+ // deadline, which slower CI runners miss on the denser screens. That is a
+ // tooling timeout, not an app defect, so retry before giving up.
+ //
+ // Only the timeout is retried. If a category is enforced and the audit
+ // reports findings before timing out, those failures are already recorded
+ // and a retry would duplicate them — accepted, because the alternative is
+ // losing the run to an infrastructure hiccup.
+ for attempt in 1...auditAttempts {
+ findings.removeAll()
+ timeout = nil
+ do {
+ try app.performAccessibilityAudit { issue in
+ let isEnforced = !enforcedAuditTypes.intersection(issue.auditType).isEmpty
+ let marker = isEnforced ? "FAIL" : "report"
+ findings.append("[\(marker)][\(name(for: issue.auditType))] \(issue.compactDescription)")
+ // true suppresses the finding, false reports it as a test failure.
+ return !isEnforced
+ }
+ break
+ } catch let error as NSError where error.isAccessibilityAuditTimeout {
+ timeout = error
+ print("\(screen): audit timed out (attempt \(attempt) of \(auditAttempts))")
+ }
+ }
- try app.performAccessibilityAudit { issue in
- let isEnforced = !enforcedAuditTypes.intersection(issue.auditType).isEmpty
- let marker = isEnforced ? "FAIL" : "report"
- findings.append("[\(marker)][\(name(for: issue.auditType))] \(issue.compactDescription)")
- // true suppresses the finding, false reports it as a test failure.
- return !isEnforced
+ if timeout != nil {
+ let message = "\(screen): audit did not complete in time after \(auditAttempts) attempts — screen NOT audited"
+ print(message)
+ let attachment = XCTAttachment(string: message)
+ attachment.name = "a11y-audit-\(screen)-timeout"
+ attachment.lifetime = .keepAlways
+ test.add(attachment)
+ return false
}
let summary = findings.isEmpty
@@ -81,6 +118,8 @@ enum AccessibilityAuditHarness {
attachment.name = "a11y-audit-\(screen)"
attachment.lifetime = .keepAlways
test.add(attachment)
+
+ return true
}
/// `XCUIAccessibilityAuditType` is an option set whose description is just a
@@ -102,6 +141,14 @@ enum AccessibilityAuditHarness {
}
}
+private extension NSError {
+ /// `Audit failed to complete in time` — the audit's own deadline, raised by
+ /// XCTest rather than by anything wrong with the app.
+ var isAccessibilityAuditTimeout: Bool {
+ domain == "com.apple.xcode.xctest.accessibilityAudit" && code == -56
+ }
+}
+
extension XCUIApplication {
/// Taps a root tab by its visible label.
///