Streaming provider Netflix stores its videos for optimal delivery at different internet speeds and playback devices in different Resolutions, quality levels and formats. Developed was coding platform "Reloaded" for about seven years. In the meantime, however, Netflix's load has grown tenfold thanks to an expansion into more markets and significantly more content. According to the Netflix engineering blog, this has caused problems. The developers responsible have therefore developed a new encoding platform called "Cosmos" created.
As Frank San Miguel explains on behalf of the Cosmos team, Reloaded's old architecture was designed exclusively to scale audio and video content as well as possible. However, the monolithic architecture made it much more difficult to introduce new features. In addition, the team responsible for coding has grown significantly over the past few years, which meant that new members in particular had virtually no overview of the outdated Reloaded architecture. Responsible for this was above all that of the application logic with the infrastructure code.
Microservices architectures as a role model
Due to the major problems, the team decided on a completely new development instead of a revision of the existing architecture. The modern microservices architecture was chosen as a model. In addition to the actual encoding tasks, this enables the modular addition of further services, including a tool for monitoring and tools for the continuous delivery pipeline. This should enable new code to be developed directly in the production environment.Contributions can be imported without additional interventions.
Despite the similarities, Cosmos services are not microservices. Application data and its dependencies are also separated from other systems in the Netflix coding tool, as in microservices, but the services use clearly defined APIs to exchange data.
The highly compute-intensive work that occurs alongside encoding loads its dependencies as Docker containers. Depending on the size of the task, these are several thousand different containers at Netflix, which are stored in a cloud computing environment. A run often takes several days for particularly large files, according to the developers.
Development of Cosmos began in 2018, with initial tests in the product environment starting in 2019. The actual rollout of the system, which will then completely replace the Reloaded coding platform, will take place in the current year 2021.
