PROFESSIONAL-CLOUD-DEVOPS-ENGINEER · Question #153
You are running an experiment to see whether your users like a new feature of a web application. Shortly after deploying the feature as a canary release, you receive a spike in the number of 500 error
The correct answer is A. Roll back the experimental canary release.. Upon observing a spike in errors and increased latency after a canary release, the immediate action to minimize negative user impact is to roll back the problematic release.
Question
Options
- ARoll back the experimental canary release.
- BStart monitoring latency, traffic, errors, and saturation.
- CRecord data for the postmortem document of the incident.
- DTrace the origin of 500 errors and the root cause of increased latency.
How the community answered
(24 responses)- A83% (20)
- B4% (1)
- C4% (1)
- D8% (2)
Why each option
Upon observing a spike in errors and increased latency after a canary release, the immediate action to minimize negative user impact is to roll back the problematic release.
A canary release is designed to test new features with a small subset of users and provides a quick rollback mechanism if issues arise. Rolling back the experimental canary release immediately removes the faulty code from production, stopping the negative impact (500 errors, increased latency) on users and preventing further degradation of service.
Monitoring these metrics should already be in place as part of standard operations and certainly before/during a canary release; starting to monitor them now is too late to mitigate an active incident.
Recording data for a postmortem is an important step *after* the incident is stabilized or resolved, not the first action to take when actively minimizing user impact.
Tracing the root cause is crucial for a permanent fix and preventing recurrence, but it's a diagnostic step that comes after immediate mitigation to restore service.
Concept tested: Incident response and canary release rollback
Source: https://cloud.google.com/architecture/application-deployment-and-testing-strategies#canary_deployment
Topics
Community Discussion
No community discussion yet for this question.