Sync Algorithm
A robust, offline-first synchronization system designed for unreliable networks and complex relational data. Single-shot, lightweight, parallelized, and fully idempotent.
Architecture Overview
The sync algorithm is purpose-built for educational data collection in environments with intermittent connectivity. It ensures data integrity while minimizing bandwidth usage and network round-trips.
Single-Shot Sync (Not Chatty)
Unlike traditional sync systems that maintain persistent connections or poll frequently, our sync operates as a single atomic operation. This design is critical for unreliable networks where connections may drop mid-sync.
Why Single-Shot?
- •Network resilience: Complete sync in one connection window
- •Battery efficiency: No persistent connections draining power
- •Predictable timing: Users know when sync happens
- •Simple recovery: Failed sync = retry entire operation
Problem with Chatty Protocols
Real-time sync (WebSockets, long-polling) requires stable connections. In field conditions with 2G/3G networks, these connections frequently drop, leaving data in inconsistent states and requiring complex reconciliation.
class SyncService {
bool _isSyncing = false;
Future<void> syncAll() async {
// Mutex check - only one sync at a time
if (_isSyncing) return;
if (!networkService.isConnected) return;
_isSyncing = true;
try {
// Single atomic operation
await _downloadPhase();
await _uploadPhase();
await _processOfflineQueue();
_updateSyncTimestamps();
} finally {
_isSyncing = false;
}
}
}Lightweight (Delta Only)
The sync system only transfers changed records since the last successful sync. This dramatically reduces bandwidth usage—critical when operating on metered mobile connections in the field.
Timestamp-Based Delta Detection
Each entity type tracks its last sync time. On sync, we query only records whereupdatedAt > lastSyncTime.
schools: 2024-01-15T10:30:00Z,
visits: 2024-01-15T10:30:05Z,
assessments: 2024-01-15T10:30:12Z
}
Selective Field Queries
REST API queries with delta parameters avoid over-fetching. Large binary data (photos, audio) synced separately via S3 endpoints.
final lastSync = _lastSyncTimes['schools']
?? DateTime(2020);
// REST endpoint with delta parameter
final url = '/api/schools?updatedAfter='
+ lastSync.toIso8601String();
final response = await dio.get(
url,
options: Options(
headers: {'Authorization': 'Bearer $token'}
),
);
final schools = response.data['items'];
// Only process changed records
for (final school in schools) {
await _upsertSchool(school);
}Bandwidth Savings
A typical deployment with 500 schools only transfers ~10-20 changed records per sync, reducing payload from megabytes to kilobytes.
Parallelization (High Speed)
While entity types sync sequentially (to maintain referential integrity), operations within each phase execute in parallel. This maximizes throughput while ensuring data consistency.
Sync Order (Sequential)
Sequential ordering ensures foreign key references resolve correctly.
Future<List<String>> uploadBatch(
List<String> localPaths
) async {
// Queue all uploads as futures
final uploadFutures = <Future<String>>[];
for (final path in localPaths) {
uploadFutures.add(uploadPhoto(path));
}
// Execute ALL uploads in parallel
return await Future.wait(uploadFutures);
}
// Progress tracking via streams
onSendProgress: (sent, total) {
final progress = UploadProgress(
bytesTransferred: sent,
totalBytes: total,
);
_progressController.add(progress);
}Sequential by entity type
Parallel within batches
Graph-Based Data Model
The sync system handles complex relational data with deep nesting. The entity graph is synced in topological order—parents before children—ensuring referential integrity at every step.
Country ├── Region │ └── District │ └── School ←────────────────────────────────┐ │ ├── Teacher (many) ←──────────────────┐ │ │ └── SchoolVisit (many) │ │ │ ├── TeacherVisit (many) ──────────┘ │ │ │ └── PupilAssessment (many) │ │ │ ├── SectionAnswer (many) │ │ │ └── AssessmentResult │ │ └── MonitoringAnswer (many) │ │ └── ProjectForm (ref) ──────────┘ └── Organisation (tenant root)
Foreign Key Tracking
Every child entity stores its parent reference:
monitorId, teacherVisitId,
pupilAssessmentId
Indexed Fields
Realm indexes enable fast graph traversal:
@Indexed() String schoolId;
@Indexed() String districtId;
Cascade Sync
Nested entities sync with their parents:
teacherVisits: [...],
monitoringAnswers: [...]
}
Idempotency (Safe Retries)
Every sync operation is idempotent—executing it multiple times produces the same result as executing it once. This is essential for reliability when network failures may interrupt operations mid-flight.
Layer 1: UUID Primary Keys
All entities use UUIDs generated client-side. The same entity always has the same ID, regardless of how many times it's synced.
Layer 2: Upsert Semantics
All database writes use upsert: if record exists, update it; if not, create it. No duplicate records ever created.
Layer 3: Timestamp Dedup
Conflict detection compares timestamps. Same or older data is skipped, preventing redundant processing.
skip();
Future<void> save<T extends RealmObject>(T entity) async {
realm.write(() {
// Upsert: create if new, update if exists
// Running this 1x or 100x produces same result
realm.add<T>(entity, update: true);
});
}
// Idempotent queue processing
Future<void> processOfflineQueue() async {
for (final operation in _queue) {
try {
await _executeOperation(operation);
_queue.remove(operation); // Only remove on success
} catch (e) {
if (operation.retryCount >= 3) {
_queue.remove(operation); // Give up after 3 tries
} else {
operation.retryCount++; // Retry is safe (idempotent)
}
}
}
}Arbitrary Graph Support
The sync system uses an adapter pattern that allows any entity type to participate in sync. New entity types can be added without modifying the core sync engine.
abstract class SyncAdapter {
String get adapterName;
/// Get entities needing sync
Future<List<dynamic>> getEntitiesForSync(
String entityType
);
/// Handle response from server
Future<void> handleSyncResponse(
String entityType,
Map<String, dynamic> response
);
}
/// Any entity can implement this
abstract class SyncableEntity {
String get id;
String get entityType;
DateTime get updatedAt;
Map<String, dynamic> toJson();
void updateFromJson(Map<String, dynamic> json);
}Supported Entity Types
Extensibility
Adding a new entity to sync requires only implementing the SyncAdapter interface. The core sync engine handles scheduling, retries, and conflict resolution automatically.
Conflict Resolution
When the same record is modified both locally and remotely, the system detects the conflict and resolves it using last-write-wins semantics with an option for manual override.
Resolution Algorithm
Future<List<Conflict>> _checkForConflicts(
String entityType,
String entityId,
Map<String, dynamic> remoteData
) async {
final conflicts = <Conflict>[];
final localTimestamp = _getLocalTimestamp(
entityType, entityId
);
final remoteTimestamp = DateTime.parse(
remoteData['updatedAt']
);
// Only flag if local is strictly newer
if (localTimestamp != null &&
localTimestamp.isAfter(remoteTimestamp)) {
conflicts.add(Conflict(
entityType: entityType,
entityId: entityId,
localData: _getLocalData(entityType, entityId),
remoteData: remoteData,
localTimestamp: localTimestamp,
remoteTimestamp: remoteTimestamp,
));
}
return conflicts;
}Service Architecture
| Service | Role | Key Responsibilities |
|---|---|---|
| SyncService | Orchestrator | Coordinates full sync cycle, manages mutex, tracks timestamps |
| OfflineQueueService | Queue Manager | Queues operations offline, processes on reconnect, handles retries |
| NetworkService | Connectivity | Monitors network state, triggers sync on reconnect |
| DatabaseService | Persistence | Realm upserts, transactions, schema management |
| PhotoService | Binary Sync | Parallel photo uploads, progress tracking, S3 integration |
| CacheService | Response Cache | TTL-based caching, reduces redundant fetches |
| Repositories | Entity Adapters | Entity-specific sync logic, implement SyncAdapter |
Performance Characteristics
Explore the Platform
See how the sync algorithm integrates with the mobile app and backend systems.