Summary:
Our engineering team implemented a hotfix to address functionality in one of our recently released features. During the implementation process, some users experienced a period where data in the portal was not refreshing properly and login attempts were unsuccessful.
Impact:
Affected Systems: Customer portal authentication and data services.
Root Cause:
API Connection Issue: The API service did not automatically reconnect after pushing the mentioned update, requiring a manual restart of API.
Resolution:
Restarted the API pods to establish a new connection.
Prevention & Lessons Learned:
Automated API Recovery: We've added an API automation that restarts the services to prevent manual intervention.
Improved Deployment Coordination: Ensure critical dependencies (e.g., DB migrations) are included in deployment plans before syncing updates.