On 21 February 2018 at 8:35 am (AEDT), ConnectWise Cloud API experienced an outage. As a result of this outage, our partners experienced 23 hours 20 minutes of downtime of services as we worked with ConnectWise to restore full service. We would like apologize to our partners for the impact to your daily operations as a result of this outage.
Here's what we know:
- On 21 February, at 8:45 am (AEDT), the first error, "500 - Internal Server Error", was reported to Wise-Sync support, followed by multiple partners reporting the same error.
- Tickets were immediately escalated to our Senior Product team, who were able to investigate and identify that the issue was only affecting ConnectWise Cloud users.
- At 9:07 am (AEDT) we logged a ConnectWise Platform Ticket for the error, 'Increasing 500 Internal Server Errors'.
- Two hours later, we still had not received an update from the ConnectWise Platform team.
- At 11:44am (AEDT) a further ticket was logged with ConnectWise Support requesting emergency escalation.
- Throughout the day (evening and over night), the Wise-Sync team continued to follow up with ConnectWise requesting an update, for which we were only advised 'The Cloud On-Call Team are working on it'.
- On 22 February, at 6:27 am (AEDT) ConnectWise confirmed that a fix was deployed to AU, EU and Staging Cloud platforms. NA Cloud was scheduled to be released at 1 pm (AEDT) which was 9pm ET.
- An interim work-around was provided for NA partners where a specific Endpoint could be used, which was deployed to all NA cloud partners.
- Shortly after, Wise-Sync resolved critical overrun of logging server affecting all user login as a result of log bloat due to extended outage.
- At 7:00 am (AEDT) on 23 February, ConnectWise Support confirmed that the patches were now fully deployed and the issue now considered resolved.
A full detailed report can be viewed here.
We have raised our concerns with ConnectWise and are working with them to achieve better communication channels and monitoring of progress during such outages; we recognise the importance of services such as ours and how they can be impacted by upstream providers. Ensuing clarity of communication throughout outages is for us, paramount. At Wise-Sync we strive to provide the best possible service and will continue working diligently with our integration partners to provide our partners with a stable and efficient syncing and payment tool.
To all those thank you for your patience and continued support of Wise-Sync and Wise-Pay. If you have any questions or need assistance, please reach out to our team on support@wise-sync.com.
