Horizon Ingestion Outage

Incident Report for Stellar.org

Resolved

We experienced an issue with Horizon nodes processing ledger 47380844 which affected both private instances as well as horizon.stellar.org, and ran from approximately 02:25am (UTC) to 03:51am (UTC). The root cause is still to be determined, but is related to the handling of overflows in liquidity pool calculations.

This issue has been patched and released in version 2.26.1, which is currently available for download through apt-get, and on Github at https://github.com/stellar/go/releases/tag/horizon-v2.26.1

Note that during this time there was nothing wrong with the underlying network, as core nodes continued to produce and close ledgers.

Providers of Horizon should immediately download and upgrade to version 2.26.1 of Horizon to resolve issues on their instances. Upon upgrade, Horizon will catch up to the latest ledger automatically.

Posted Jul 27, 2023 - 04:00 UTC