On 3/09 from 18:00 UTC to 20:35 UTC and 3/10/2024 from 15:15 UTC to 17:08 UTC, the Advanced Billing system encountered a system-wide performance degradation that primarily affected signups, renewals, product migrations and reactivations. Component allocations and refunds may also have failed during these periods. This degradation was the result of a system background process that encountered a rare data condition that triggered an unindexed query to run. This very slow query caused the background process to consume system resources to the level that it adversely impacted other system processes, resulting in the system-wide performance degradation that occurred.

During these periods of system degradation, subscription creation via Public Signup Pages, API, and UI channels may have failed. Product migrations attempted during this timeframe also may have encountered failures, and many subscription renewals encountered processing errors, pausing their renewal for review and remediation by our team. We recommend checking any internal logs for systems integrated with our platform via API for failed subscription creation and migration or other failed events in order to reattempt where appropriate. Our team is happy to enable audit log exports for your account to help pinpoint events affected by these incidents as well.

Our engineers are currently working to remediate any processing errors that occurred for subscriptions as a result of these incidents. Once the processing errors are cleared, the subscriptions will resume processing as normal and bill for their renewal. No action is necessary to resolve these errors on your end unless our Sr. Product Support Consultant team reaches out directly requesting further insight on specific subscriptions.

We are happy to report that the Advanced Billing system is now fully operational, and the underlying cause has been identified and addressed. Proper steps including improved code, testing, and monitoring are being taken by our developers to ensure this does not occur again.

Thank you for your patience as our developers worked to identify and resolve this matter, and our sincere apologies for the disruption caused by these incidents.