Conveyor: Facebook's Continuous Deployment System

Conveyor is the continuous deployment platform used at Facebook to roll out new versions of service code to production as safely as possible. The name refers both to the Conveyor Platform itself and to the team supporting it, while a "Conveyor pipeline" refers to an individual deployment pipeline running on the platform. Its naming draws on industrial conveyor belts, where automation passes bundles of raw materials between stations to create a final product. The system is described as a scalable CD solution capable of over 100k deployments each week across a diverse range of services.

• Unified continuous deployment system at Facebook
• Over 100k deployments each week
• Supports thousands of interconnected services
• Platform and supporting team share the name

Each Conveyor pipeline consists of various "nodes", with each node performing one aspect of the deployment for a single service. A version of a service's code passes through the pipeline as a "bundle" containing a packaged binary. A Conveyor Config defines a pipeline, letting users specify which nodes make it up; each node has a type plus configuration specific to that type. Conveyor is generic, so teams can implement their own nodes.

• Build node: packages an application after Contbuild compiles and unit tests it
• Canary node: temporary deployment to a test environment or production subset, with A/B regression detection
• Push node: incremental deployment of the binary to production
• Common package formats include .rpm and fbpkg, Facebook's BitTorrent-based blob distribution service

Push plans in Conveyor cover a Canary phase, which temporarily deploys a packaged binary to a testing environment or a subset of production, and a Push phase, which incrementally deploys the binary to production. The push process is supported by several components working together. These include a command line tool rolled out to developers, the Conveyor platform's scheduling role, a front-end acting as the entry point to the push process, and an execution layer implemented as a CWS workflow.

• Push Command Line (CLI) tool
• Conveyor platform scheduling
• Push Front-End as entry point
• Push Execution Layer as a CWS workflow

To push diverse services, Conveyor models each service through a set of shared concepts. Configuration captures what each type of service needs, while units describe the logical components of a service, which may in turn be broken into subunits. Each service type has its own way of applying updates, units and optional subunits run some version of the code, and healthchecks assess whether a unit is healthy.

• Configuration
• Units
• Subunits
• Updates
• Versions
• Healthchecks

Facebook's services are in a constant state of improvement, with optimisations, bug fixes, security patches and new features landing regularly. Teams must roll out patches quickly when a security exploit is identified by an internal team or an external researcher. Infrequent deployments cause many changes to land at once, making breakages harder to debug. Because many developers work across a single monolithic codebase, tooling is expected to catch bugs early and stop them reaching production.

Conveyor is documented in the article "Conveyor: Continuous Deployment at Facebook", published on the At Scale Conferences website under its Articles section for 2021. The piece is credited to the @scale team and includes sections on pushing diverse services and flexible push plans. It is the source for the platform description on this page.

• Published on atscaleconference.com
• Article type, 2021
• Credited to the @scale team