Semantic Replication: Facebook Video Storage Technique

Semantic replication is a technique described by Facebook for reducing video storage by reducing the number of copies that need to be kept. Rather than relying solely on conventional physical replication of every stored object, it draws on the semantics of video content: the same video already exists in several encodings. According to the published description, the technique enables saving over 30% of video storage, and in practice roughly 35% savings are observed in certain configurations. In another configuration, an HD and an SD encoding are retained with reduced replication.

• Reduces the number of stored copies per video
• Saving of over 30% of video storage
• Around 35% savings in practice in some configurations

Content uploaded to Facebook is stored in Everstore, a storage system that specialises in efficiently storing Binary Large OBjects (BLOBs). The majority of the bytes stored there are video data, which continues to grow rapidly as upload volumes and average video sizes increase. Improved cameras, cheaper compute, faster networks and higher quality video formats are cited as drivers of that growth. Managing video at this scale requires storage systems able to handle exabytes of data.

• Everstore stores BLOBs for uploaded Facebook content
• Video data accounts for the majority of bytes stored
• Goal is less storage per video with minimal degradation in user experience

The approach is built on properties that distinguish video from other content types. Videos are typically larger, ranging from a few megabytes to gigabytes, may be uploaded over a period of time requiring streaming writes and reads, and often need random access to byte ranges for playback. On upload or delivery, videos are re-encoded into many different resolutions and codecs to support various players across platforms. Videos also show temporal affinity: most playbacks occur in the first few days after upload, after which viewing probability falls exponentially with time.

• Large file sizes, from megabytes to gigabytes
• Streaming writes and reads during upload
• Random access to byte ranges for playback
• Multiple encodings per video across resolutions and codecs
• Temporal affinity in playback patterns

Each encoding generated for a video can be large and is split into smaller chunks during encoding. An abstraction layer called OIL hides how chunks are determined and where they are stored, holding them as independent BLOBs from the storage system's point of view. OIL keeps additional metadata mapping each encoding to its individual chunks and offsets so the encoding can be stitched together and served. This allows sub-parts of an encoded file to be stored while it is still being uploaded or processed, approximating streaming writes and reducing the data buffered in memory.

• Encodings split into chunks stored as independent BLOBs
• OIL metadata maps encodings to chunks and offsets
• Supports partial storage during upload and processing

The work was presented as a talk titled "Optimizing video storage via Semantic Replication" during a virtual Systems @Scale event, scheduled at 11am PT on Wednesday, March 3rd, and followed by a live question-and-answer session. Attendees were invited to submit questions in advance by email to systemsatscale@fb.com. The accompanying article is published on the At Scale Conferences site under the 2021 articles section.

• Talk given at a virtual Systems @Scale event
• Live Q&A followed the session
• Questions submitted to systemsatscale@fb.com