Cloud Foundry Summit Silicon Valley 2017

Cloud Foundry makes extensive use of Consul as a distributed key/value store in order to achieve active-passive high availability for components that are harder to operate in parallel. This is done through implementing a distributed locking mechanism that allows the active component to claim ownership of the lock and take responsibility on execution.

However, Consul’s underlying RAFT algorithm becomes fragile when the entire environment undergoes multiple BOSH re-deployments. This results in an array of different failures, e.g., Consul not being able to elect a leader or mistakenly choosing multiple nodes as leaders. These recurring inconsistencies in Consul can potentially reduce reliability in a distributed system like Cloud Foundry.

In Diego, as one of the primary subsystems of Cloud Foundry, we have taken advantage of Consul’s distributed locking mechanism in several components such as the Diego database (BBS), the auctioneer, and the route emitters. As such, we have spent extensive amount of time to understand how these distributed locks can potentially be avoided or redesigned in order to reduce dependencies to Consul. To this point, two of the Diego components, i.e., the route emitters and the database have adopted alternative solutions to achieve high availability.

In this talk, we will go over lessons learned from using Consul in Cloud Foundry, and how we are trying to find alternative solutions that can help with improving robustness and reliability of Diego components without having to use a complex system like Consul.