API
Payment Page
SDK
Production Dashboard
Sandbox
Nov 2022
Sandbox is now operational! This update was created by an automated monitoring service.
Sandbox cannot be accessed at the moment. This incident was created by an automated monitoring service.
Oct 2022
Timeline of the Issue - 19:10 to 19:14
We just resolved the issue!
Sep 2022
The issue is resolved now
We are closely monitoring the key metrics
We are currently investigating this incident.
Previous fix only had partial recovery. Migrating to a new cluster with higher network bandwidth allowance and shards.
We implemented a fix by stopping non critical services consuming Redis network bandwidth
We are seeing issues with Redis Cluster. Working with AWS support to mitigate the issue.
We are seeing timeouts for multiple APIs and investigating the root cause.
We scaled down some post transaction batch jobs (status sync consumers) which freed up a chunk of DB connections to normal, reduced contention for the tables. The dependency of legacy EC service was turned off to allow EC to prevent it from holding up further connections due to upstream latencies.
We have initiated restart of all application pods across major services which were affected the most (txn, customer, wallets and Saved Payment methods). We are continuously monitoring the key metrics.
We are seeing an increase in 504s across all the production k8s clusters.
We are seeing outages for multiple APIs and investigating the root cause.
Sep 2022 to Nov 2022