aboutsummaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorChristian Cleberg <[email protected]>2026-07-24 23:35:18 -0500
committerChristian Cleberg <[email protected]>2026-07-24 23:37:00 -0500
commit99e4623af1b08f36120a01b67cbe60df99668651 (patch)
tree07c8365f0c0054837f2b63611d71d1fd9c2f4e6c
parenta52dee116d4066d1b59bd90b4ebc4def4e1597d6 (diff)
downloaddomain-dig-99e4623af1b08f36120a01b67cbe60df99668651.tar.gz
domain-dig-99e4623af1b08f36120a01b67cbe60df99668651.tar.bz2
domain-dig-99e4623af1b08f36120a01b67cbe60df99668651.zip
feat: versioned store-migration policy for persisted data (v5 step 2)
Third v5.0.0 roadmap item: define and implement a migration policy for the on-device persisted store (tracked domains, history/snapshots, audits, workflows, monitoring, settings), so data upgrades cleanly across app versions instead of relying on a one-shot marker. - DataMigrationService is reworked from a single boolean marker (`data.migrations.v3_4_0`) into a versioned runner keyed by an integer store schema version (`data.storeSchemaVersion`). It runs each step once in ascending order up to `currentStoreSchemaVersion`, stamping the version as it goes. Adding a future migration is now a `case N:` plus a version bump. Policy guarantees, all covered by tests: - Forward-only and idempotent; every step must be safe on an empty/older store. - Never downgrades: a store written by a newer build (higher version) is left byte-for-byte untouched. - Pre-versioning installs are handled: a set legacy boolean marker reads as "already at v1", so the v1 normalization never re-runs for them. v1 is the existing normalization pass (dedup + drop the legacy `watchedDomains` key + sanitize monitoring settings), now expressed as migration step 1. - Docs/data-migration.md documents the persisted surface, the two independent version lines (store vs. backup export), when to use lenient decoding vs. a migration step, the runner contract, an "adding a migration" checklist, and backup-import compatibility. Linked from the README. - DataMigrationServiceTests: 6 tests over legacy fixtures — fresh-store stamping, legacy `watchedDomains` migration + key drop, in-place dedup of the stored blob, idempotence, legacy-marker-as-v1, and the no-downgrade guard. Full unit suite: 58 passing.
-rw-r--r--Docs/data-migration.md94
-rw-r--r--DomainDataPortabilityService.swift63
-rw-r--r--DomainDig.xcodeproj/project.pbxproj4
-rw-r--r--DomainDigTests/DataMigrationServiceTests.swift109
-rw-r--r--README.md2
5 files changed, 267 insertions, 5 deletions
diff --git a/Docs/data-migration.md b/Docs/data-migration.md
new file mode 100644
index 0000000..1ae4030
--- /dev/null
+++ b/Docs/data-migration.md
@@ -0,0 +1,94 @@
+# DomainDig Data Migration Policy
+
+How DomainDig's persisted data evolves across app versions without losing or
+corrupting a user's on-device store.
+
+## What is persisted
+
+The store is a set of independent JSON blobs in `UserDefaults`, each under a
+stable key (see `DomainDataPortabilityService.StorageKey`):
+
+| Data | Key |
+|------|-----|
+| Tracked domains | `trackedDomains` (legacy: `watchedDomains`) |
+| Lookup history (snapshots) | `lookupHistory` |
+| Audit sessions | `domainAudits` |
+| Workflows | `domainWorkflows` |
+| Monitoring settings / logs | `monitoring.settings`, `monitoring.logs` |
+| App settings | `recentSearches`, `savedDomains`, resolver URL, density |
+| Feature metadata | `purchase.cachedEntitlement`, `usageCredits.ledger` |
+
+A **backup export** (`DomainDigBackup`) is a separate, self-describing file that
+bundles all of the above with its own `schemaVersion`.
+
+## Two version lines
+
+- **Store schema version** — `DataMigrationService.currentStoreSchemaVersion`,
+ persisted under `data.storeSchemaVersion`. Describes the shape of the
+ *on-device* `UserDefaults` store. Advanced by the migration runner.
+- **Backup schema version** — `DomainDigBackup.currentSchemaVersion`, written
+ into every exported file. Describes the shape of an *export*. Checked on import
+ by `DataValidationService`.
+
+They advance independently: a store migration that doesn't change the export
+shape need not bump the backup version, and vice versa.
+
+## How models evolve
+
+Prefer **additive, lenient decoding** — it needs no migration:
+
+- New optional field → add it with `decodeIfPresent(...) ?? default` in the
+ model's `init(from:)`. Old data simply lacks the key and falls back.
+- New value in a `String`-backed enum → decode unknown values to a safe default
+ rather than throwing.
+
+Reach for a **migration step** only when lenient decoding can't express the
+change:
+
+- Renaming or removing a storage key (e.g. `watchedDomains` → `trackedDomains`).
+- Re-normalizing existing rows (dedup, canonicalizing domain casing).
+- Reshaping a blob in a way old readers would misread.
+
+## The migration runner
+
+`DataMigrationService.migrateIfNeeded(defaults:)` runs at launch (and before any
+backup export/import). Its contract:
+
+1. **Forward-only.** It reads the stored version and runs each step with a target
+ greater than it, in ascending order, up to `currentStoreSchemaVersion`,
+ stamping the new version after each step.
+2. **Never downgrades.** A store stamped at a version *higher* than this build
+ understands (a user who ran a newer build first) is left untouched — no
+ rewrite, no data loss.
+3. **Idempotent & safe on any state.** Every step must be safe to run on an empty
+ store and to re-run, because a downgrade-then-upgrade or a partial run can
+ replay it. v1 (the `watchedDomains` drop + dedup normalization) satisfies this
+ by loading through the deduplicating loaders and writing back.
+4. **Pre-versioning installs.** Before this framework, a boolean marker
+ (`data.migrations.v3_4_0`) recorded that the v1 normalization had run. A set
+ marker is read as "already at version 1," so v1 never re-runs for those users.
+
+## Adding a migration
+
+1. Add a `case N:` to `DataMigrationService.runMigration(to:defaults:)` and a
+ private helper that performs the change.
+2. Bump `currentStoreSchemaVersion` to `N`.
+3. Make the helper idempotent and safe on an empty/older store.
+4. Add a `DataMigrationServiceTests` case that seeds a pre-`N` fixture, runs
+ `migrateIfNeeded`, and asserts the upgrade plus the version stamp.
+5. If the change also alters the export shape, bump
+ `DomainDigBackup.currentSchemaVersion` and update `Docs/local-api.md` /
+ backup validation as needed.
+
+## Backup import compatibility
+
+On import, `DataValidationService.validate(backup:)` compares the file's
+`schemaVersion` to the current one:
+
+- **Newer** than this build → surfaced as an error (the build can't safely read
+ it).
+- **Older** → imported under the same lenient decoders and merge/dedup rules that
+ govern the live store; a note is surfaced, not an error.
+
+Imported data flows through `migrateIfNeeded` and the same `save*` deduplication
+as everything else, so an old backup lands in the store already normalized.
diff --git a/DomainDataPortabilityService.swift b/DomainDataPortabilityService.swift
index b6f9eee..bfca546 100644
--- a/DomainDataPortabilityService.swift
+++ b/DomainDataPortabilityService.swift
@@ -342,12 +342,69 @@ enum DataPortabilityCSV {
}
}
+/// Versioned migration runner for the on-device persisted store.
+///
+/// The store is a set of independent JSON blobs in `UserDefaults` (tracked
+/// domains, history, audits, workflows, monitoring settings/logs, app settings).
+/// Most model evolution is handled additively by the models' own lenient
+/// decoders (`decodeIfPresent` with defaults), which need no migration at all.
+/// This runner exists only for changes lenient decoding can't express: dropping
+/// a renamed storage key, re-normalizing existing rows, or reshaping a blob.
+///
+/// `currentStoreSchemaVersion` is bumped whenever such a step is added. Each step
+/// runs exactly once, in ascending order, and must be safe to run on any prior
+/// state — including an empty store. A store written by a newer build (a higher
+/// version than this build knows) is left untouched; migrations never downgrade.
+/// The policy is documented in `Docs/data-migration.md`.
enum DataMigrationService {
- private static let migrationMarkerKey = "data.migrations.v3_4_0"
+ /// The schema version this build expects the on-device store to be at.
+ static let currentStoreSchemaVersion = 1
+
+ /// UserDefaults key holding the store's current schema version.
+ static let storeSchemaVersionKey = "data.storeSchemaVersion"
+
+ /// Pre-versioning installs recorded that the one-shot v1 normalization had
+ /// run using this boolean marker; `true` means the store is already at v1.
+ private static let legacyNormalizationMarkerKey = "data.migrations.v3_4_0"
+
+ /// The store's current schema version. Absent on pre-versioning installs: a
+ /// set legacy marker means v1 already ran, otherwise the store is fresh or
+ /// never-migrated at v0.
+ static func storeSchemaVersion(defaults: UserDefaults = .standard) -> Int {
+ if let version = defaults.object(forKey: storeSchemaVersionKey) as? Int {
+ return version
+ }
+ return defaults.bool(forKey: legacyNormalizationMarkerKey) ? 1 : 0
+ }
static func migrateIfNeeded(defaults: UserDefaults = .standard) {
- guard !defaults.bool(forKey: migrationMarkerKey) else { return }
+ // At or ahead of this build's version: nothing to do, and never rewrite
+ // a store a newer build may have reshaped (forward compatibility).
+ guard storeSchemaVersion(defaults: defaults) < currentStoreSchemaVersion else { return }
+
+ var version = storeSchemaVersion(defaults: defaults)
+ while version < currentStoreSchemaVersion {
+ let target = version + 1
+ runMigration(to: target, defaults: defaults)
+ defaults.set(target, forKey: storeSchemaVersionKey)
+ version = target
+ }
+ }
+
+ private static func runMigration(to version: Int, defaults: UserDefaults) {
+ switch version {
+ case 1:
+ normalizeAllStores(defaults: defaults)
+ default:
+ break
+ }
+ }
+ /// v1: load every store through its deduplicating loader and write it back.
+ /// This consolidates duplicate rows, drops the legacy `watchedDomains` key
+ /// (via `saveTrackedDomains`), and sanitizes monitoring settings against the
+ /// surviving tracked domains. Safe on an empty store (every step is a no-op).
+ private static func normalizeAllStores(defaults: UserDefaults) {
let trackedDomains = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults)
DomainDataPortabilityService.saveTrackedDomains(trackedDomains, defaults: defaults)
@@ -366,8 +423,6 @@ enum DataMigrationService {
let monitoringLogs = DomainDataPortabilityService.loadMonitoringLogs(defaults: defaults)
DomainDataPortabilityService.saveMonitoringLogs(monitoringLogs, defaults: defaults)
-
- defaults.set(true, forKey: migrationMarkerKey)
}
}
diff --git a/DomainDig.xcodeproj/project.pbxproj b/DomainDig.xcodeproj/project.pbxproj
index e65c63a..4bab46e 100644
--- a/DomainDig.xcodeproj/project.pbxproj
+++ b/DomainDig.xcodeproj/project.pbxproj
@@ -10,6 +10,7 @@
05F775C7E0743AF727B008EE /* LocalAPIContract.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7572DA0A838E0A044F045260 /* LocalAPIContract.swift */; };
38316D90539394C2CC7C12BE /* Foundation.framework in Frameworks */ = {isa = PBXBuildFile; fileRef = DE8B269A01CC5E593DA3DFC2 /* Foundation.framework */; };
47CD3BB1AE143733A73E0E5B /* DomainReportExporterTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */; };
+ 4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */; };
81359F63C7A23454B8FA0141 /* DomainReportBuilderTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = E82320955416797CFE59464A /* DomainReportBuilderTests.swift */; };
8BBFEF092F9874AE00E8E144 /* DomainInspectionService.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8BBFEF032F9874AE00E8E144 /* DomainInspectionService.swift */; };
8BBFEF0A2F9874AE00E8E144 /* DomainReportBuilder.swift in Sources */ = {isa = PBXBuildFile; fileRef = 8BBFEF042F9874AE00E8E144 /* DomainReportBuilder.swift */; };
@@ -75,6 +76,7 @@
/* Begin PBXFileReference section */
0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContractTests.swift; sourceTree = "<group>"; };
15A25DF2B8BB52589D49986B /* DiffServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DiffServiceTests.swift; sourceTree = "<group>"; };
+ 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DataMigrationServiceTests.swift; sourceTree = "<group>"; };
54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DomainReportExporterTests.swift; sourceTree = "<group>"; };
5CA789D3607B55E3612E6FFA /* SnapshotFixture.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = SnapshotFixture.swift; sourceTree = "<group>"; };
7572DA0A838E0A044F045260 /* LocalAPIContract.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContract.swift; sourceTree = "<group>"; };
@@ -229,6 +231,7 @@
54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */,
CC948A02EC0184228BC4630E /* DomainDataPortabilityServiceTests.swift */,
0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */,
+ 435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */,
);
name = DomainDigTests;
path = DomainDigTests;
@@ -487,6 +490,7 @@
C7CA9E02B0DC2708DE7A8563 /* DomainDataPortabilityServiceTests.swift in Sources */,
A5AF921BC1C2E6E940CC05DC /* SnapshotFixture.swift in Sources */,
EA12012CDB4C21ABB4217D86 /* LocalAPIContractTests.swift in Sources */,
+ 4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
};
diff --git a/DomainDigTests/DataMigrationServiceTests.swift b/DomainDigTests/DataMigrationServiceTests.swift
new file mode 100644
index 0000000..3a6cdd9
--- /dev/null
+++ b/DomainDigTests/DataMigrationServiceTests.swift
@@ -0,0 +1,109 @@
+import XCTest
+@testable import DomainDig
+
+/// Characterization tests for the versioned store-migration runner. They drive
+/// `migrateIfNeeded` against legacy on-disk fixtures in an ephemeral
+/// `UserDefaults` suite and pin the policy: forward-only, idempotent, and
+/// non-destructive to a store written by a newer build.
+///
+/// The raw storage-key strings ("trackedDomains", "watchedDomains", the legacy
+/// marker) are duplicated here on purpose — they are the on-disk contract, and
+/// hard-coding them means an accidental rename shows up as a failing migration.
+final class DataMigrationServiceTests: XCTestCase {
+ private let suiteName = "DomainDigTests.migration"
+ private var defaults: UserDefaults!
+ private let base = Date(timeIntervalSince1970: 1_700_000_000)
+
+ override func setUp() {
+ super.setUp()
+ defaults = UserDefaults(suiteName: suiteName)
+ defaults.removePersistentDomain(forName: suiteName)
+ }
+
+ override func tearDown() {
+ defaults.removePersistentDomain(forName: suiteName)
+ defaults = nil
+ super.tearDown()
+ }
+
+ func testFreshStoreIsStampedAtCurrentVersion() {
+ XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 0)
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+
+ XCTAssertEqual(
+ DataMigrationService.storeSchemaVersion(defaults: defaults),
+ DataMigrationService.currentStoreSchemaVersion
+ )
+ }
+
+ func testLegacyWatchedDomainsAreMigratedAndTheOldKeyIsDropped() throws {
+ let legacy = [WatchedDomain(domain: "legacy.example", createdAt: base, lastKnownAvailability: .registered)]
+ defaults.set(try JSONEncoder().encode(legacy), forKey: "watchedDomains")
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+
+ let tracked = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults)
+ XCTAssertEqual(tracked.map(\.domain), ["legacy.example"])
+ XCTAssertNil(defaults.data(forKey: "watchedDomains"), "legacy key is dropped after migration")
+ XCTAssertNotNil(defaults.data(forKey: "trackedDomains"), "data is rewritten under the current key")
+ XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1)
+ }
+
+ func testMigrationDeduplicatesTheStoredBlobInPlace() throws {
+ let dupes = [
+ TrackedDomain(domain: "dupe.example", updatedAt: base.addingTimeInterval(-10)),
+ TrackedDomain(domain: "DUPE.example", updatedAt: base)
+ ]
+ defaults.set(try JSONEncoder().encode(dupes), forKey: "trackedDomains")
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+
+ let data = try XCTUnwrap(defaults.data(forKey: "trackedDomains"))
+ let stored = try JSONDecoder().decode([TrackedDomain].self, from: data)
+ XCTAssertEqual(stored.count, 1, "the persisted blob is deduplicated, not just the load result")
+ }
+
+ func testMigrationIsIdempotent() throws {
+ defaults.set(
+ try JSONEncoder().encode([TrackedDomain(domain: "a.example", updatedAt: base)]),
+ forKey: "trackedDomains"
+ )
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+ let afterFirst = defaults.data(forKey: "trackedDomains")
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+ let afterSecond = defaults.data(forKey: "trackedDomains")
+
+ XCTAssertEqual(afterFirst, afterSecond, "a second run makes no further changes")
+ XCTAssertEqual(
+ DataMigrationService.storeSchemaVersion(defaults: defaults),
+ DataMigrationService.currentStoreSchemaVersion
+ )
+ }
+
+ func testLegacyBooleanMarkerCountsAsVersionOne() {
+ defaults.set(true, forKey: "data.migrations.v3_4_0")
+
+ XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1)
+
+ // Already at v1, so the v1 step must not re-run: a leftover legacy blob
+ // is left exactly as found.
+ defaults.set(Data("x".utf8), forKey: "watchedDomains")
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+ XCTAssertNotNil(defaults.data(forKey: "watchedDomains"), "v1 is treated as already applied; no re-run")
+ }
+
+ func testNewerStoreVersionIsNeverDowngradedOrRewritten() throws {
+ let future = DataMigrationService.currentStoreSchemaVersion + 1
+ defaults.set(future, forKey: DataMigrationService.storeSchemaVersionKey)
+ let blob = try JSONEncoder().encode([TrackedDomain(domain: "keep.example", updatedAt: base)])
+ defaults.set(blob, forKey: "trackedDomains")
+
+ DataMigrationService.migrateIfNeeded(defaults: defaults)
+
+ XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), future, "must never downgrade")
+ XCTAssertEqual(defaults.data(forKey: "trackedDomains"), blob, "future-version data is left byte-for-byte")
+ }
+}
diff --git a/README.md b/README.md
index 3e61467..a3b8636 100644
--- a/README.md
+++ b/README.md
@@ -51,7 +51,7 @@ Network inspection requests are made only to perform the requested domain checks
xcodebuild -project DomainDig.xcodeproj -scheme DomainDig -destination 'platform=iOS Simulator,name=iPhone 16' build
```
-The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md).
+The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md). How the on-device store evolves across app versions is documented in [Docs/data-migration.md](Docs/data-migration.md).
### Accessibility Audit