Design File Sync

You save a spreadsheet on your laptop. Before you reach for your phone, the new version is already there, and the copy on your home computer updated itself overnight. If you edit the same file in two places while offline, nothing is silently lost: you get both versions and decide.

Interviewers like this question because it mixes two very different systems. One moves large amounts of bytes and must avoid sending anything twice. The other tracks small metadata records and must be strictly correct, because a wrong file list destroys someone's work. The steps follow A Framework for Any Design.

#Clarify the problem

• Functional: upload, download, update, delete and move files; sync changes to a user's other devices; share folders with other users; keep 30 days of versions; work offline and sync later.
• Non-functional: no acknowledged change is ever lost (durability); a folder's changes are seen in one order by every device (strong consistency per folder); other devices see changes within about 5 seconds when online; upload only what changed.
• Out of scope: real-time co-editing, search inside files, previews.

#Estimate the scale

• Stored data: 10 million users × 2 GB = 2 × 10¹⁶ B = 20 PB before deduplication and before redundancy.
• Changes: 3 million active users × 20 saved changes each = 60 million file changes a day. 60 × 10⁶ ÷ 86,400 ≈ 700 a second on average; assume a working-hours peak of 3x, about 2,100 a second.
• Upload bandwidth: if a change uploads 1 MB on average, 60 million × 1 MB = 60 TB a day, about 700 MB/s on average.
• Connected devices: 10 million users × 3 devices = 30 million devices. If 30% are online at once, 9 million devices wait to hear about changes.

#Define the API

POST /v1/commit
  body { namespace_id, path, base_rev: 7, commit_id, size, chunks: [sha256, ...] }
  200 { rev: 8, journal_seq: 1042 }
  412 { missing: [sha256, ...] }        upload these first, then retry
  409 { current_rev: 8 }                 someone else changed it since rev 7
 
PUT  /v1/chunks/{sha256}     (raw bytes, at most 4 MB)
  201 stored     200 already had it     400 hash does not match bytes
GET  /v1/chunks/{sha256}     -> bytes
 
GET  /v1/changes?namespace_id=N&cursor=1041
  200 { entries: [ { path, rev, chunks, deleted } ], cursor: 1042, has_more }
GET  /v1/notify?cursors=N:1041,M:88       long poll, up to 60 s
  200 { changed: ["N"] }    or    204 nothing changed, poll again

The decision worth arguing is chunks before metadata. The commit only succeeds if every chunk it names is already stored, so a file record can never point at bytes that do not exist. The cost is a second round trip when chunks are missing (the 412). The alternative, writing metadata first and bytes after, leaves other devices downloading a file whose bytes have not arrived, which is worse. Naming chunks by their hash also makes uploads safe to retry: sending the same chunk twice stores it once. The commit carries a client-made commit_id for the same reason: if the reply is lost and the client retries, the server finds the commit already applied and returns it, instead of a false 409 against the device's own edit.

#Model the data

namespaces:  namespace_id PK, owner_id, next_seq BIGINT
files:       (namespace_id, path) PK, file_id, latest_rev INT, deleted BOOL
revisions:   (file_id, rev) PK, size, chunk_hashes BLOB,   -- ordered list
             device_id, created_at                            -- kept 30 days
journal:     (namespace_id, seq) PK, file_id, rev, op         -- the change log
members:     (namespace_id, user_id) PK, role                 -- sharing
 
-- sharded by hash
chunks:      sha256 PK, blob_key, size, ref_count
-- chunk bytes live in object storage under their sha256

The metadata needs transactions: a commit checks the base revision, writes a new revision, bumps the file's latest revision and appends a journal entry, all or nothing. That points to a relational database, sharded by namespace, so every change to one folder lands on one shard and gets one order. Chunk bytes go to object storage, keyed by hash. The chunk index is a simple key-value lookup sharded by hash.

#High-level design

#Deep dives

Interviewers push on how files are cut into chunks, how the metadata stays consistent, what happens on conflicts, and how devices learn about changes.

#Chunking and content hashing

Name every chunk by a cryptographic hash of its bytes, such as SHA-256. Two chunks with the same hash have the same bytes, for every practical purpose, so the server stores each distinct chunk once. That one idea gives three wins: delta sync (upload only chunks whose hash changed), deduplication (the same chunk in two files or two versions is stored once), and safe retries (re-sending a chunk changes nothing).

Content-defined chunking slides a small window, say 48 bytes, over the file and computes a cheap rolling hash at each position. Wherever the low 22 bits of that hash are all zero, it cuts. That happens on average once every 2²² ≈ 4.2 million bytes, so chunks average about 4 MB. Because the cut depends only on the nearby bytes, inserting a paragraph moves the boundaries near it and leaves the rest where they were.

One judgement call worth raising: deduplicating across users means an upload that finishes instantly reveals that someone else already has that exact file. Some services limit deduplication to a user's own files for that reason. I would start per user and say what cross-user deduplication would save and cost.

#The metadata service and consistency

Bytes can be eventually consistent; a chunk either exists or does not, and its content never changes. File lists cannot. If a device applied "rename A to B" before "create A", it would end up with the wrong folder. So every change to a namespace gets the next sequence number in that namespace's journal, inside the same transaction that changes the file record. Devices apply entries in sequence order and remember their cursor.

This gives a precise guarantee: within one namespace, every device sees the same changes in the same order, and never sees revision 8 without the chunks it needs. It does not order changes across two namespaces, and it does not need to. That is why the database is sharded by namespace: one shard, one order, no cross-shard transactions for normal saves. The cost is that a single huge shared folder lands on one shard; split very large namespaces or give them a dedicated shard.

If the metadata database fails over to a replica, it must be one that has every committed transaction (a synchronous replica, or a consensus-based database). An asynchronous replica can be behind: a commit acknowledged to the laptop could vanish, and the phone might already have applied it. See replication for why lag matters here.

#Conflicts

Two devices edit the same file while one is offline. Something has to give.

The mechanism is optimistic concurrency: every commit names its base_rev, and the server accepts it only if that is still the latest revision. The loser gets 409, uploads its version under a conflicted-copy name, and both appear on every device. Deletes conflict too: if one device deletes a file another just edited, keep the edit.

#Notifying other devices

9 million devices online at once need to hear about changes within seconds. Polling every 5 seconds would be 9 million ÷ 5 = 1.8 million requests a second, nearly all answering "nothing new". Instead each device holds one long poll (or a WebSocket) to the notification service, listing its namespaces and cursors. The service answers as soon as an event arrives for one of them, or after 60 seconds with "nothing changed". That is 9 million ÷ 60 = 150,000 empty answers a second, against 1.8 million.

The notification carries no file data, only "namespace N changed". It is a hint, not the source of truth. If one is lost, the next poll catches the device up: the poll carries the device's cursors, and the service answers at once when a cursor is behind the namespace's latest sequence number. The device then asks for changes since its own cursor. Designing it this way lets the notification path be fast and best-effort while the journal stays strict. The realtime communication lesson compares long polling and WebSockets.

#Bottlenecks and failures

At 10x (200 PB, about 21,000 commits a second at peak, 90 million online devices) the chunk store just grows; object storage scales by adding capacity. The metadata database grows by adding shards, which namespace sharding allows, and the hottest single namespaces get their own. The notification service scales by adding connection servers, partitioned by device. Storage cost becomes the main bill, so redundancy moves from three full copies to erasure coding, and old versions past 30 days are garbage-collected by decrementing chunk reference counts.

#What a strong answer sounds like

#Follow-up questions

#How do you restore a file to last week's version?

Revisions are lists of chunk hashes, kept for 30 days. Restoring commits a new revision with the old chunk list; no bytes move, because the chunks are still stored.

#How do you sync a folder with 100,000 files to a new device?

Page through the journal from cursor 0, or start from a snapshot of the current file list plus its cursor, then download chunks in parallel with the smallest or most recent files first.

#How do you handle a 50 GB file?

The same way: 50 GB ÷ 4 MB = 12,500 chunks, uploaded in parallel and resumable chunk by chunk. The revision record holds 12,500 hashes, about 400 KB, which is fine.

#How does sharing work?

A shared folder is its own namespace with a members table. Each member's devices add that namespace and its cursor to their long poll.

#Can you encrypt files so the server cannot read them?

Yes, by encrypting on the device, but then identical files from different users no longer produce identical chunks, so cross-user deduplication is gone, and chunk names must be keyed hashes so they do not reveal content. It is a product choice; say what it costs.