diff options
| -rw-r--r-- | Docs/data-migration.md | 94 | ||||
| -rw-r--r-- | DomainDataPortabilityService.swift | 63 | ||||
| -rw-r--r-- | DomainDig.xcodeproj/project.pbxproj | 4 | ||||
| -rw-r--r-- | DomainDigTests/DataMigrationServiceTests.swift | 109 | ||||
| -rw-r--r-- | README.md | 2 |
5 files changed, 267 insertions, 5 deletions
diff --git a/Docs/data-migration.md b/Docs/data-migration.md new file mode 100644 index 0000000..1ae4030 --- /dev/null +++ b/Docs/data-migration.md @@ -0,0 +1,94 @@ +# DomainDig Data Migration Policy + +How DomainDig's persisted data evolves across app versions without losing or +corrupting a user's on-device store. + +## What is persisted + +The store is a set of independent JSON blobs in `UserDefaults`, each under a +stable key (see `DomainDataPortabilityService.StorageKey`): + +| Data | Key | +|------|-----| +| Tracked domains | `trackedDomains` (legacy: `watchedDomains`) | +| Lookup history (snapshots) | `lookupHistory` | +| Audit sessions | `domainAudits` | +| Workflows | `domainWorkflows` | +| Monitoring settings / logs | `monitoring.settings`, `monitoring.logs` | +| App settings | `recentSearches`, `savedDomains`, resolver URL, density | +| Feature metadata | `purchase.cachedEntitlement`, `usageCredits.ledger` | + +A **backup export** (`DomainDigBackup`) is a separate, self-describing file that +bundles all of the above with its own `schemaVersion`. + +## Two version lines + +- **Store schema version** — `DataMigrationService.currentStoreSchemaVersion`, + persisted under `data.storeSchemaVersion`. Describes the shape of the + *on-device* `UserDefaults` store. Advanced by the migration runner. +- **Backup schema version** — `DomainDigBackup.currentSchemaVersion`, written + into every exported file. Describes the shape of an *export*. Checked on import + by `DataValidationService`. + +They advance independently: a store migration that doesn't change the export +shape need not bump the backup version, and vice versa. + +## How models evolve + +Prefer **additive, lenient decoding** — it needs no migration: + +- New optional field → add it with `decodeIfPresent(...) ?? default` in the + model's `init(from:)`. Old data simply lacks the key and falls back. +- New value in a `String`-backed enum → decode unknown values to a safe default + rather than throwing. + +Reach for a **migration step** only when lenient decoding can't express the +change: + +- Renaming or removing a storage key (e.g. `watchedDomains` → `trackedDomains`). +- Re-normalizing existing rows (dedup, canonicalizing domain casing). +- Reshaping a blob in a way old readers would misread. + +## The migration runner + +`DataMigrationService.migrateIfNeeded(defaults:)` runs at launch (and before any +backup export/import). Its contract: + +1. **Forward-only.** It reads the stored version and runs each step with a target + greater than it, in ascending order, up to `currentStoreSchemaVersion`, + stamping the new version after each step. +2. **Never downgrades.** A store stamped at a version *higher* than this build + understands (a user who ran a newer build first) is left untouched — no + rewrite, no data loss. +3. **Idempotent & safe on any state.** Every step must be safe to run on an empty + store and to re-run, because a downgrade-then-upgrade or a partial run can + replay it. v1 (the `watchedDomains` drop + dedup normalization) satisfies this + by loading through the deduplicating loaders and writing back. +4. **Pre-versioning installs.** Before this framework, a boolean marker + (`data.migrations.v3_4_0`) recorded that the v1 normalization had run. A set + marker is read as "already at version 1," so v1 never re-runs for those users. + +## Adding a migration + +1. Add a `case N:` to `DataMigrationService.runMigration(to:defaults:)` and a + private helper that performs the change. +2. Bump `currentStoreSchemaVersion` to `N`. +3. Make the helper idempotent and safe on an empty/older store. +4. Add a `DataMigrationServiceTests` case that seeds a pre-`N` fixture, runs + `migrateIfNeeded`, and asserts the upgrade plus the version stamp. +5. If the change also alters the export shape, bump + `DomainDigBackup.currentSchemaVersion` and update `Docs/local-api.md` / + backup validation as needed. + +## Backup import compatibility + +On import, `DataValidationService.validate(backup:)` compares the file's +`schemaVersion` to the current one: + +- **Newer** than this build → surfaced as an error (the build can't safely read + it). +- **Older** → imported under the same lenient decoders and merge/dedup rules that + govern the live store; a note is surfaced, not an error. + +Imported data flows through `migrateIfNeeded` and the same `save*` deduplication +as everything else, so an old backup lands in the store already normalized. diff --git a/DomainDataPortabilityService.swift b/DomainDataPortabilityService.swift index b6f9eee..bfca546 100644 --- a/DomainDataPortabilityService.swift +++ b/DomainDataPortabilityService.swift @@ -342,12 +342,69 @@ enum DataPortabilityCSV { } } +/// Versioned migration runner for the on-device persisted store. +/// +/// The store is a set of independent JSON blobs in `UserDefaults` (tracked +/// domains, history, audits, workflows, monitoring settings/logs, app settings). +/// Most model evolution is handled additively by the models' own lenient +/// decoders (`decodeIfPresent` with defaults), which need no migration at all. +/// This runner exists only for changes lenient decoding can't express: dropping +/// a renamed storage key, re-normalizing existing rows, or reshaping a blob. +/// +/// `currentStoreSchemaVersion` is bumped whenever such a step is added. Each step +/// runs exactly once, in ascending order, and must be safe to run on any prior +/// state — including an empty store. A store written by a newer build (a higher +/// version than this build knows) is left untouched; migrations never downgrade. +/// The policy is documented in `Docs/data-migration.md`. enum DataMigrationService { - private static let migrationMarkerKey = "data.migrations.v3_4_0" + /// The schema version this build expects the on-device store to be at. + static let currentStoreSchemaVersion = 1 + + /// UserDefaults key holding the store's current schema version. + static let storeSchemaVersionKey = "data.storeSchemaVersion" + + /// Pre-versioning installs recorded that the one-shot v1 normalization had + /// run using this boolean marker; `true` means the store is already at v1. + private static let legacyNormalizationMarkerKey = "data.migrations.v3_4_0" + + /// The store's current schema version. Absent on pre-versioning installs: a + /// set legacy marker means v1 already ran, otherwise the store is fresh or + /// never-migrated at v0. + static func storeSchemaVersion(defaults: UserDefaults = .standard) -> Int { + if let version = defaults.object(forKey: storeSchemaVersionKey) as? Int { + return version + } + return defaults.bool(forKey: legacyNormalizationMarkerKey) ? 1 : 0 + } static func migrateIfNeeded(defaults: UserDefaults = .standard) { - guard !defaults.bool(forKey: migrationMarkerKey) else { return } + // At or ahead of this build's version: nothing to do, and never rewrite + // a store a newer build may have reshaped (forward compatibility). + guard storeSchemaVersion(defaults: defaults) < currentStoreSchemaVersion else { return } + + var version = storeSchemaVersion(defaults: defaults) + while version < currentStoreSchemaVersion { + let target = version + 1 + runMigration(to: target, defaults: defaults) + defaults.set(target, forKey: storeSchemaVersionKey) + version = target + } + } + + private static func runMigration(to version: Int, defaults: UserDefaults) { + switch version { + case 1: + normalizeAllStores(defaults: defaults) + default: + break + } + } + /// v1: load every store through its deduplicating loader and write it back. + /// This consolidates duplicate rows, drops the legacy `watchedDomains` key + /// (via `saveTrackedDomains`), and sanitizes monitoring settings against the + /// surviving tracked domains. Safe on an empty store (every step is a no-op). + private static func normalizeAllStores(defaults: UserDefaults) { let trackedDomains = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults) DomainDataPortabilityService.saveTrackedDomains(trackedDomains, defaults: defaults) @@ -366,8 +423,6 @@ enum DataMigrationService { let monitoringLogs = DomainDataPortabilityService.loadMonitoringLogs(defaults: defaults) DomainDataPortabilityService.saveMonitoringLogs(monitoringLogs, defaults: defaults) - - defaults.set(true, forKey: migrationMarkerKey) } } diff --git a/DomainDig.xcodeproj/project.pbxproj b/DomainDig.xcodeproj/project.pbxproj index e65c63a..4bab46e 100644 --- a/DomainDig.xcodeproj/project.pbxproj +++ b/DomainDig.xcodeproj/project.pbxproj @@ -10,6 +10,7 @@ 05F775C7E0743AF727B008EE /* LocalAPIContract.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7572DA0A838E0A044F045260 /* LocalAPIContract.swift */; }; 38316D90539394C2CC7C12BE /* Foundation.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = DE8B269A01CC5E593DA3DFC2 /* Foundation.framework */; }; 47CD3BB1AE143733A73E0E5B /* DomainReportExporterTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */; }; + 4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */; }; 81359F63C7A23454B8FA0141 /* DomainReportBuilderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E82320955416797CFE59464A /* DomainReportBuilderTests.swift */; }; 8BBFEF092F9874AE00E8E144 /* DomainInspectionService.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8BBFEF032F9874AE00E8E144 /* DomainInspectionService.swift */; }; 8BBFEF0A2F9874AE00E8E144 /* DomainReportBuilder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8BBFEF042F9874AE00E8E144 /* DomainReportBuilder.swift */; }; @@ -75,6 +76,7 @@ /* Begin PBXFileReference section */ 0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContractTests.swift; sourceTree = "<group>"; }; 15A25DF2B8BB52589D49986B /* DiffServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DiffServiceTests.swift; sourceTree = "<group>"; }; + 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DataMigrationServiceTests.swift; sourceTree = "<group>"; }; 54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DomainReportExporterTests.swift; sourceTree = "<group>"; }; 5CA789D3607B55E3612E6FFA /* SnapshotFixture.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = SnapshotFixture.swift; sourceTree = "<group>"; }; 7572DA0A838E0A044F045260 /* LocalAPIContract.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContract.swift; sourceTree = "<group>"; }; @@ -229,6 +231,7 @@ 54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */, CC948A02EC0184228BC4630E /* DomainDataPortabilityServiceTests.swift */, 0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */, + 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */, ); name = DomainDigTests; path = DomainDigTests; @@ -487,6 +490,7 @@ C7CA9E02B0DC2708DE7A8563 /* DomainDataPortabilityServiceTests.swift in Sources */, A5AF921BC1C2E6E940CC05DC /* SnapshotFixture.swift in Sources */, EA12012CDB4C21ABB4217D86 /* LocalAPIContractTests.swift in Sources */, + 4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources */, ); runOnlyForDeploymentPostprocessing = 0; }; diff --git a/DomainDigTests/DataMigrationServiceTests.swift b/DomainDigTests/DataMigrationServiceTests.swift new file mode 100644 index 0000000..3a6cdd9 --- /dev/null +++ b/DomainDigTests/DataMigrationServiceTests.swift @@ -0,0 +1,109 @@ +import XCTest +@testable import DomainDig + +/// Characterization tests for the versioned store-migration runner. They drive +/// `migrateIfNeeded` against legacy on-disk fixtures in an ephemeral +/// `UserDefaults` suite and pin the policy: forward-only, idempotent, and +/// non-destructive to a store written by a newer build. +/// +/// The raw storage-key strings ("trackedDomains", "watchedDomains", the legacy +/// marker) are duplicated here on purpose — they are the on-disk contract, and +/// hard-coding them means an accidental rename shows up as a failing migration. +final class DataMigrationServiceTests: XCTestCase { + private let suiteName = "DomainDigTests.migration" + private var defaults: UserDefaults! + private let base = Date(timeIntervalSince1970: 1_700_000_000) + + override func setUp() { + super.setUp() + defaults = UserDefaults(suiteName: suiteName) + defaults.removePersistentDomain(forName: suiteName) + } + + override func tearDown() { + defaults.removePersistentDomain(forName: suiteName) + defaults = nil + super.tearDown() + } + + func testFreshStoreIsStampedAtCurrentVersion() { + XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 0) + + DataMigrationService.migrateIfNeeded(defaults: defaults) + + XCTAssertEqual( + DataMigrationService.storeSchemaVersion(defaults: defaults), + DataMigrationService.currentStoreSchemaVersion + ) + } + + func testLegacyWatchedDomainsAreMigratedAndTheOldKeyIsDropped() throws { + let legacy = [WatchedDomain(domain: "legacy.example", createdAt: base, lastKnownAvailability: .registered)] + defaults.set(try JSONEncoder().encode(legacy), forKey: "watchedDomains") + + DataMigrationService.migrateIfNeeded(defaults: defaults) + + let tracked = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults) + XCTAssertEqual(tracked.map(\.domain), ["legacy.example"]) + XCTAssertNil(defaults.data(forKey: "watchedDomains"), "legacy key is dropped after migration") + XCTAssertNotNil(defaults.data(forKey: "trackedDomains"), "data is rewritten under the current key") + XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1) + } + + func testMigrationDeduplicatesTheStoredBlobInPlace() throws { + let dupes = [ + TrackedDomain(domain: "dupe.example", updatedAt: base.addingTimeInterval(-10)), + TrackedDomain(domain: "DUPE.example", updatedAt: base) + ] + defaults.set(try JSONEncoder().encode(dupes), forKey: "trackedDomains") + + DataMigrationService.migrateIfNeeded(defaults: defaults) + + let data = try XCTUnwrap(defaults.data(forKey: "trackedDomains")) + let stored = try JSONDecoder().decode([TrackedDomain].self, from: data) + XCTAssertEqual(stored.count, 1, "the persisted blob is deduplicated, not just the load result") + } + + func testMigrationIsIdempotent() throws { + defaults.set( + try JSONEncoder().encode([TrackedDomain(domain: "a.example", updatedAt: base)]), + forKey: "trackedDomains" + ) + + DataMigrationService.migrateIfNeeded(defaults: defaults) + let afterFirst = defaults.data(forKey: "trackedDomains") + + DataMigrationService.migrateIfNeeded(defaults: defaults) + let afterSecond = defaults.data(forKey: "trackedDomains") + + XCTAssertEqual(afterFirst, afterSecond, "a second run makes no further changes") + XCTAssertEqual( + DataMigrationService.storeSchemaVersion(defaults: defaults), + DataMigrationService.currentStoreSchemaVersion + ) + } + + func testLegacyBooleanMarkerCountsAsVersionOne() { + defaults.set(true, forKey: "data.migrations.v3_4_0") + + XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1) + + // Already at v1, so the v1 step must not re-run: a leftover legacy blob + // is left exactly as found. + defaults.set(Data("x".utf8), forKey: "watchedDomains") + DataMigrationService.migrateIfNeeded(defaults: defaults) + XCTAssertNotNil(defaults.data(forKey: "watchedDomains"), "v1 is treated as already applied; no re-run") + } + + func testNewerStoreVersionIsNeverDowngradedOrRewritten() throws { + let future = DataMigrationService.currentStoreSchemaVersion + 1 + defaults.set(future, forKey: DataMigrationService.storeSchemaVersionKey) + let blob = try JSONEncoder().encode([TrackedDomain(domain: "keep.example", updatedAt: base)]) + defaults.set(blob, forKey: "trackedDomains") + + DataMigrationService.migrateIfNeeded(defaults: defaults) + + XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), future, "must never downgrade") + XCTAssertEqual(defaults.data(forKey: "trackedDomains"), blob, "future-version data is left byte-for-byte") + } +} @@ -51,7 +51,7 @@ Network inspection requests are made only to perform the requested domain checks xcodebuild -project DomainDig.xcodeproj -scheme DomainDig -destination 'platform=iOS Simulator,name=iPhone 16' build ``` -The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md). +The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md). How the on-device store evolves across app versions is documented in [Docs/data-migration.md](Docs/data-migration.md). ### Accessibility Audit |