When and How CC Updates the Runtime

Introduction

There are multiple paths through which CC updates the runtime (Diego or Eirini). There are multiple paths and they can be hidden and circuitous. As a result, it can be difficult to reason about whether an API interaction will update the runtime.

Overview

The two main methods for updating the runtime are the ProcessObserver class and the Diego::Sync job. The ProcessObserver triggers when certain fields are updated on processes and immediately update the runtime. The Diego::Sync job runs periodically and will update the runtime if certain fields have changed on the process.

Details

Process Version

To understand if a change to a process will automatically propagate to the runtime, one must first understand process versioning. When one or more of a specific set of fields are updated, then the process's version will be updated to a new random guid. Changing a process's version will then result in the process's LRP changing as will be demonstrated in coming sections.

Process Observer

ProcessObserver triggers when changes to processes are committed to to the database. It is called using after_commit hooks in the after_save and after_destroy model hooks.

Note: Because most of Cloud Controller unit tests depend on database transactions, these after_commit hooks don't trigger in tests, unless you switch the test's database isolation to :truncation.

If the process's state, diego, enable_ssh, or ports fields are updated the the ProcessObserver will start or updated the process. Note that this method name is a bit confusing, because, as we will see, the Runner.start method will also update running processes.

Focusing on the Diego case, Diego::Runner#start directly forwards on to Diego::Messenger#send_desire_request, which then forwards on to the Diego::DesireAppHandler.create_or_update_app, where it is finally revealed that we will also be updated an existing process, not just creating new processes.

Here is where the process version comes into play. The Diego::DesireAppHandler checks to see if the process has a matching LRP. To do this, it uses Diego::BbsAppsClient#get_app, which in turn uses Diego::ProcessGuid. Here we see that the LRP is identified using a combination of the process's guid and it's version. This means if the process's version has changed since Diego was last updated, Diego::BbsAppsClient will not find a matching LRP for the process.

If a matching LRP is found (e.g. the process's version hasn't changed), then Diego::BbsAppsClient#update_app is called. Note that only instances, updated_at, and routes can be updated on an LRP. Changing other fields will require creating a new LRP.

If a matching LRP is NOT found (e.g. the process is new or the version has changed), then Diego::BbsAppsClient#desire_app to create a new LRP for the process.

Open Question: What happens to the old LRP? Do we wait for the sync job to sweep it up?

Sync Job

The Diego sync job runs every 30 seconds (by default) on the Cloud Controller Clock. When the sync job runs, it checks to make sure that CC's processes match the LRPs in Diego using Diego::ProcessesSync.

Diego::ProcessesSync loops over all the processes in the ccdb and checks if there is a corresponding LRP in Diego. To do this comparison, it again uses Diego::ProcessGuid. Remember that this class identifies processes using a combination of the process's guid and it's version.

At this point, the sync job behaves very similarly to the ProcessObserver. If a matching LRP is found and the process's updated_at is different than the LRP's, then it calls Diego::BbsAppsClient#update_app. If a matching LRP is NOT found, then it calls Diego::BbsAppsClient#desire_app.

Finally, the sync job deletes all remaining LRPs that haven't been matched to a process.

Manifests

Restart Action

Tasks

Deployments