History lag and degraded performance on certain Horizon endpoints
Incident Report for Stellar.org
Resolved
Service has remained stable over the past week and thus the incident is resolved. Note that rate limits may dynamically adjust to preserve service health in times of high network activity or volatility.
Posted Mar 18, 2024 - 23:01 UTC
Update
Fix appears to have resolved performance issues, but reduced rate limits are still in effect as we continue to monitor.
Posted Mar 12, 2024 - 18:53 UTC
Update
A fix has been deployed. We are monitoring to ensure it is stable.
Posted Mar 11, 2024 - 23:15 UTC
Update
Additional reduction of rate limits.
Posted Mar 11, 2024 - 21:00 UTC
Update
Performance degradation observed again.
Posted Mar 11, 2024 - 20:00 UTC
Update
We flipped the standby and active endpoint. This temporarily fixed the problem.
Posted Mar 11, 2024 - 15:15 UTC
Monitoring
Reports of degraded performance in Discord.
Posted Mar 11, 2024 - 15:00 UTC
Update
We reduced global rate limits by half for Horizon.
Posted Mar 11, 2024 - 14:30 UTC
Investigating
We've identified a high CPU utilization on SDF Horizon nodes due to increased request load that saturated DB connections.
Posted Mar 11, 2024 - 14:00 UTC
This incident affected: SDF Public Network Horizon.