Root Cause:
The deployment resulted in downtime due to a database migration step that took significantly longer than originally anticipated. As the deployment pipeline was configured to execute migrations synchronously prior to launching the new version, the prolonged migration caused the older version to terminate before the new version was fully ready to serve traffic, resulting in temporary unavailability.
Resolution:
To avoid recurrence, future deployments will separate database migrations from application releases, using incremental migration strategies or pre-migration scripts to ensure database readiness. Additionally, we've enhanced our monitoring to better estimate migration durations and introduced procedures to roll back or pause deployment if a migration exceeds expected durations.
Follow-up Actions:
Implemented separate pre-deployment migration jobs with clearly defined thresholds for maximum allowable duration. Improved monitoring and alerting on migration performance and durations. Updated deployment procedures to allow manual approval for potentially long-running migrations.
Posted Mar 16, 2025 - 10:15 UTC
Resolved
This incident has been resolved.
Posted Feb 26, 2025 - 10:01 UTC
Investigating
We are investigating reports of degraded performance in our app. Our team is actively investigating and seeking a resolution, and we'll continue to update here.
Posted Feb 26, 2025 - 09:51 UTC
This incident affected: Entitle Portal, API, Access Change Requests, and Auditlogs Webhooks.