Owl: Meta's Large Content Distribution System

Owl addresses the distribution of large, hot content to hosts inside Meta's private cloud. The content types it commonly handles include executables, code artifacts, AI models and search indexes, all of which underpin Meta's software systems. Efficient distribution of this material is described as an increasingly important requirement within the private cloud.

• Executables
• Code artifacts
• AI models
• Search indexes

Three dimensions capture the scope of the task Owl is built for: fanout, size and hotness. Fanout ranges from a handful of clients to millions of processes running in data centres around the globe. Objects to be distributed range from around 1 MB to a few TBs. Hotness varies, in that clients may read an object within a few seconds of each other or their reads may be spread over hours.

Distribution must be fast, because the predictive value of AI models decreases over time and slow executable delivery increases downtime and delays fixes. The stated aim is to provide data at a rate bounded either by the available network bandwidth of the reading host or by the available write bandwidth of its storage media. Efficiency is measured through scalability, network usage in bytes transmitted and communication locality, and resource usage on client machines such as CPU cycles, memory and disk I/O. Reliability is measured as the percentage of download requests satisfied within a latency SLA, with operational ease of management treated as a prerequisite.

• Speed bounded by host network or storage write bandwidth
• Scalability measured by clients served per server
• Locality accounted for, as in-rack transfers cost less than cross-region
• Adjustable resource use for memory- or disk-constrained services
• Reliability measured against a latency SLA

Owl's architecture combines client-side libraries with centralised services. Peer libraries are linked into every binary that uses Owl. Trackers are dedicated Owl services that manage the control plane.

• Peer libraries linked into every binary that uses Owl
• Trackers: dedicated services managing the control plane

Before Owl, Meta had at least three separate systems for large content distribution, which had grown organically to serve the distinct domains of executable distribution, model distribution and index distribution. None of them met all of the stated requirements. Two root causes were identified: no prior system struck the correct balance between decentralisation and centralisation, and no prior system had sufficient flexibility to meet the range of requirements.

• Separate systems for executables, models and indexes
• No correct balance between decentralisation and centralisation
• Insufficient flexibility across requirements

Owl is described in an article hosted on atscaleconference.com, the At Scale Conferences site, which publishes technical articles and talks from Meta engineering teams. The Owl article is filed under the 2022 article year. The same site carries related Meta infrastructure write-ups covering topics such as inference platforms, mobile configuration and AI observability.

• Published on atscaleconference.com
• Article title: Blog for Owl: The Distribution Challenge
• Listed under articles for 2022