Issue Resolved:
ConnectWise have confirmed that the issue affecting the API for all cloud platforms has been resolved, with the patch applied to NA Servers late evening on the February 22.
The issue as reported related to a component on the ConnectWise cloud, known as the Multi-Version Router. This component is responsible for brokering connections to the various versions of the API. On February 21, this component started reporting an error (Internal Server Error) indicating that the API could not be reached.
A Platform ticket was logged with ConnectWise at 5:07pm ET on the 21st, and was not actioned until 11pm after repeated requests for escalation. First confirmation was provided by the Platform Team at 8:45 ET, with notification at 3:45pm ET that AU and EU clouds had been updated, and NA cloud was due to for update after 9pm ET. An interim work-around was provided for NA partners where a specific Endpoint could be used. At 4:30 ET the endpoint change was confirmed as working, and rolled out to all affected accounts at 4:45pm ET.
On 23 February 3pm ET, ConnectWise Support confirmed that the patches were now fully deployed and the issue now considered resolved.
Background
An error was reported 21st February at 8:35am AEST, with some companies reporting Internal Server Errors when trying to sync their accounting data. A ticket was logged with the ConnectWise Platform Team at 09:07am AEST advising of elevated errors being reported.
A ticket was confirmed by the Platform Team, ticket number provided #9831142.
The current situation is that no partners are able to sync due to sync to the Accounting API failing with the internal server error.
Issue Updates (all times in Australian Eastern Daylight Time)
Issue Updates (all times in Australian Eastern Daylight Time)
- 08:05 on 22nd of February, Wise-Sync confirmed work-around successful, confirming with all NA Partner Accounts.
- 07:01 on 22nd of February, Wise-Sync confirmed operation of AU Cloud. NA Cloud still affected, work-around is being implemented at direction of ConnectWise.
- 06:59 on 22nd of February, Wise-Sync resolved critical overrun of logging server affecting all user login as a result of log bloat due to extended outage.
- 06:27 on 22nd of February, ConnectWise have confirmed fix is deployed to AU, EU and Staging Cloud platforms. NA Cloud is scheduled for release at 9pm EST.
- 04:41 on 22nd of February, Contacted ConnectWise support for update. Nothing has been resolved as of right now.
- 03:24 on 22nd of February, Contacted ConnectWise support for update. Nothing has been resolved as of right now.
- 02:12 on 22nd of February, Contacted ConnectWise support for update. Issue has not been resolved yet.
- 01:12 on 22nd of February, Contacted ConnectWise support for update. No new update.
- 00:12 on 22nd of February, Contacted ConnectWise support for update. Currently no update.
- 22:40 on 21st of February, Contacted ConnectWise support for update. Issue still ongoing.
- 21:30 on 21st of February, Contacted ConnectWise support for update. No new update.
- 20:19 on 21st of February, Contacted ConnectWise support for update. Issue has not been resolved yet.
- 18:51 on 21st of February, Contacted ConnectWise support for update. Currently no update
- 18:35 on 21st of February, Emailed to Partner Support, Platform Manager to request update as no further information has been provided.
- 14:08 on 21st of February, Initiated ConnectWise Chat to determine status of ticket after several non-responses. Issue escalated to Cloud Support team as Emergency.
- 13:10 on 21st of February, Direct request to Platform Manager for Update, due to time being out of normal support hours.
- 11:44 on 21st of February, Request for update sent to ticket, due to no response.
- 09:20 on 21st of February, ConnectWise confirms they have ticket, ticket number provided #9831142
- 09:07 on 21st of February, ConnectWise Platform Ticket logged, 'Increasing 500 Internal Server Errors'
- 09:05 on 21st of February, Wise-Sync Product Associates escalate error to senior product team. Issue confirmed as only affecting ConnectWise Cloud
- 08:55 on 21st of February, Multiple errors reported by ConnectWise API, and logged through Wise-Sync Support.
- 08:45 on 21st of February, First partners report Internal Server Error