Blob Storage

Keep metadata in the database and bytes in an object store: pre-signed and multipart uploads, and replication vs erasure coding for durability.

17 minLesson 6 of 7
On this page
  1. The idea
  2. Objects, files and blocks
  3. Uploading without touching the bytes
  4. Numbers worth remembering
  5. In the interview
  6. Common mistakes
  7. What to practise next

A photo app saves each upload in a database column. Users add 2 million photos a day at about 3 MB each: 2 × 106 × 3 MB = 6 TB a day, about 2.2 PB a year. Every replica copies every byte, the nightly backup no longer finishes overnight, and each 3 MB upload ties up an app server while it streams through.

The database is good at small rows that change and get queried. Photos are large, written once and read whole. They belong somewhere else.

The idea

An object store (such as Amazon S3 or any of its look-alikes) keeps blobs of bytes under keys like photos/7f3a9c/original.jpg. You PUT an object, GET it, or DELETE it, over HTTP. There are no folders, only key prefixes, and an object is replaced whole, never edited in place.

In exchange it is cheap per gigabyte, scales to petabytes without you thinking about disks, and is built to be extremely durable. The costs: the first byte takes tens of milliseconds, small partial updates are impossible, and listing millions of keys is slow. So the rule is simple: metadata in the database, bytes in the object store. The database row says who owns the photo, its size, its status and its object key. The object store holds the bytes.

Objects, files and blocks

Three kinds of storage
OptionGood atBad atPick it when
Object storage Usual defaultHuge scale, high durability, cheap, reachable over HTTP from anywhere, works with CDNs.Whole-object writes only. Higher latency per request. No rename or append.User uploads, media, backups, logs, data exports. The usual answer.
File storageA shared file system with folders, rename, append and partial writes.Harder to scale to many clients and huge sizes. More expensive per GB.Several servers must share files through ordinary file operations (legacy apps, build caches).
Block storageA raw disk volume for one machine: lowest latency, any file system or database on top.Attached to one machine at a time. You manage capacity.The disk under a database or a single server.

The metadata row is small and ordinary:

photos
id          BIGINT PRIMARY KEY
owner_id    BIGINT
object_key  TEXT        -- "photos/7f3a9c/original.jpg", generated, never the user's filename
size_bytes  BIGINT
status      TEXT        -- pending | ready | deleted
created_at  TIMESTAMP

Uploading without touching the bytes

App servers should not relay uploads: 6 TB a day through them is about 6 × 1012 ÷ 86,400 ≈ 70 MB/s on average, all of it wasted work. Instead the server hands the client a pre-signed URL: a link to one object key, signed with the server's credentials, that allows one action for a short time. The client sends the bytes straight to the store.

A pre-signed upload, then a readUse the arrows or ← → keys
  1. A pre-signed upload, then a read, step 1Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload API3 MB jpegMetadata DBCDNObject storeThumbnail workerask for URLPUT byteson miss

    1The user picks a 3 MB photo. The app asks the API for somewhere to put it, sending only the size and type.

  2. A pre-signed upload, then a read, step 2Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload APIMetadata DBpendingCDNObject storeThumbnail workerask for URLPUT byteson miss

    2The API checks the user's quota and inserts a metadata row with status pending and a generated key.

  3. A pre-signed upload, then a read, step 3Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone apphas URLUpload APIMetadata DBCDNObject storeThumbnail workerask for URLPUT byteson miss

    3It signs a URL that allows a PUT to that one key for 15 minutes, with the declared size and type as conditions, and returns it. No bytes have touched the API.

  4. A pre-signed upload, then a read, step 4Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload APIMetadata DBCDNObject store3 MB storedThumbnail workerask for URLPUT byteson miss

    4The app uploads the 3 MB straight to the object store. The store checks the signature and expiry, then stores the object durably before answering.

  5. A pre-signed upload, then a read, step 5Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload APIMetadata DBreadyCDNObject storeThumbnail workerask for URLPUT byteson miss

    5The store emits an "object created" event. A worker makes thumbnails and flips the row to ready. Only now does the photo appear in feeds.

  6. A pre-signed upload, then a read, step 6Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload APIMetadata DBCDNmiss, then hitsObject storeThumbnail workerask for URLPUT byteson miss

    6A friend opens the photo through the CDN. The first request misses and fetches from the store; the next thousand are served from the edge.

  7. A pre-signed upload, then a read, step 7Phone app to Upload API (ask for URL); Upload API to Metadata DB; Phone app to Object store (PUT bytes); Phone app to CDN; CDN to Object store (on miss); Object store to Thumbnail worker, asynchronous; Thumbnail worker to Metadata DB.Phone appUpload APIMetadata DBstill pendingCDNObject storeThumbnail workerask for URLPUT byteson miss

    7Failure: another user's connection drops at 60%. No object is created and the row stays pending, so nobody ever sees a broken photo. A sweeper later finds the pending row with no object and deletes it.

1 / 7

Multipart upload for big files

A 2 GB video over a 20 Mbps connection takes 2 × 109 × 8 bits ÷ 2 × 107 bits/s = 800 seconds, about 13 minutes. One upload that fails at 90% throws away 12 minutes of work. Multipart upload splits the file into parts, here 2 GB ÷ 100 MB = 20 parts. Each part is a separate request, can go in parallel, and is retried on its own; a failure costs at most one part, 100 MB at 20 Mbps = 40 seconds. A final "complete" call tells the store to stitch the parts into one object. Set a lifecycle rule to clean up uploads that were started and never completed, or their parts sit there costing money.

Durability: copies or erasure coding

Object stores survive disk and rack failures in one of two ways. Replication stores 3 full copies on different machines: 3x the raw storage, simple, and fast to read and rebuild. Erasure coding splits an object into, say, 10 data fragments, computes 4 parity fragments, and spreads all 14 across different disks. Any 10 of the 14 rebuild the object, so it survives 4 lost fragments while using only 14 ÷ 10 = 1.4x the space. The price: rebuilding one lost fragment reads 10 others, and small objects pay overhead. Stores often replicate hot or small objects and erasure-code large, colder ones.

Numbers worth remembering

  • Object store first-byte latency: tens of milliseconds, much more than a database row read. Put a CDN in front of anything popular.
  • Replication: 3x raw space. Erasure coding with 10 + 4 fragments: 1.4x, survives 4 losses.
  • Upload time = size in bits ÷ link speed. 1 GB at 10 Mbps is 8 × 109 ÷ 107 = 800 seconds.
Estimate: Raw disk for a year of photos

How much raw disk does one year of photos use with 10 + 4 erasure coding?

You know
  • 2 million photos a day
  • About 3 MB each
  • 10 data + 4 parity fragments

Show the working
  1. 2 × 106 × 3 × 106 B = 6 × 1012 B = 6 TB a day
  2. 6 TB × 365 ≈ 2.2 PB of photos a year
  3. 2.2 PB × 1.4 ≈ 3.1 PB of raw disk
  4. With 3 full copies it would be 2.2 × 3 ≈ 6.6 PB

Estimate 3 PB (anything from 1 PB to 10 PB is a good answer)

Erasure coding saves about 3.5 PB a year here, which is why large stores use it for big objects.

In the interview

Bring in blob storage as soon as the product has images, video, documents or any file bigger than a few hundred kilobytes. Say where the bytes go, where the metadata goes, and that uploads bypass your servers.

Photo uploads
Interviewer

How do users upload photos?

You

The client asks our API for an upload URL. The API writes a pending metadata row and returns a pre-signed URL for one key, valid 15 minutes. The client uploads straight to the object store, so 70 MB/s of photos never touch our servers. An upload event triggers thumbnailing and marks the row ready.

Interviewer

What if the upload succeeds but the event is lost?

You

The row stays pending. A sweeper checks pending rows older than a few minutes: if the object exists, it finishes processing; if not, it deletes the row. The object store and the database can disagree briefly, and the status column is how we notice.

Follow-ups you should expect:

  • "How do you serve private photos?" Do not make the bucket public. Hand out short-lived pre-signed GET URLs, or signed CDN URLs, after checking permissions.
  • "How do you delete?" Mark the row deleted so it disappears at once, then delete the object and purge the CDN asynchronously.
  • "Can you avoid storing the same file twice?" Use a hash of the content as the key and count references, at the cost of more complex deletes.

Common mistakes

Quick check

An erasure code uses 6 data and 3 parity fragments. How much raw space does it use, and how many lost fragments can it survive?

A 1 GB file goes up in 50 MB parts over a 10 Mbps link, and one part fails. How much upload time does the retry cost?

Why does the metadata row start as pending?

What to practise next

Finding things by their content, not their key, is the job of search systems, next. Uploads, chunking and CDNs come together in Design Video Streaming and Design File Sync.