Yesterday, Wise-Sync experienced an outage of our database infrastructure. As a result of this outage, our partners experienced 1 hour 40 minutes of downtime of services as we worked to restore full service. We would like apologize to our partners for the impact to your daily operations as a result of this outage. Unplanned downtime of any length is unacceptable to us.
Further investigation revealed:
- Throughout the day our monitoring systems alerted us to there being increased delays in database transactions, which were self-resolved normally within a few minutes.
- We were also notified that the frequency was increasing, which was attributed to the sync activity being month end.
- There was also significant latency in our API calls to both Xero and QBO which is likely due to server / API loads at this period.
- Around 4:30pm it was noted that database transactions were timing out when using Wise-Pay indicating that there was an underlying infrastructure issue.
- At 4:45pm further investigation was completed on the database and it was identified at this time that the index tables were corrupting at an increasing rate, and the database required unscheduled reindex.
- A reindex was initiated but failed due to what appears to be a database error.
- To ensure the data integrity was maintained, the applications were taken offline to allow us to further identify the issue and also apply remediation steps to being the indexes back to a usable state.
- The Full Re-Index was completed at 5:30pm.
- The database had been operational for 480 days, and as a result of the un-scheduled outage the opportunity to upgrade the instance types of the database servers to allow greater overhead as the database continues to grow. This process took longer than expected with the system shutdown taking over 15 mins. Once the servers were shut-down and upgraded, they were brought online within minutes.
- Several tests were completed to make sure that the sync functions were operational, and the services were restored at 6:10pm.
We realize the impact this had on our partners' operations and again, would like to take a moment to apologize for the outage.
We strive to provide the best possible service and will continue working diligently to provide our partners with a stable and efficient syncing and payment tool.
Thank you for your patience and continued support of Wise-Sync and Wise-Pay. If you have any questions or need assistance, please reach out to our team on support@wise-sync.com or support@wise-pay.com.
*Please clear your cache (CTRL + F5) to access Wise-Sync and Wise-Pay in your browser.
