tag:status.corepractice.com.au,2005:/historyCore Practice Status - Incident History2024-03-29T11:06:35+11:00Core Practicetag:status.corepractice.com.au,2005:Incident/190507912023-11-09T09:57:12+11:002023-11-09T09:57:12+11:00Optus Outage affecting some users logging into Core<p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>09:57</var> AEDT</small><br><strong>Resolved</strong> - Optus network has now been restored and we are expecting all Core services to be operational again. We again thank you for your patience.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:44</var> AEDT</small><br><strong>Investigating</strong> - We want to inform you of a service disruption affecting SMS delivery and two-factor authentication capabilities, due a nationwide Optus outage. We are currently investigating the full impact on our services.<br /> <br />If you are experiencing issues logging in, please reach out to a user who is already logged in to temporarily disable your two-factor authentication.<br /> <br />We apologize for the inconvenience and appreciate your patience as we work to resolve this matter.<br /> <br />Stay tuned for updates.</p>tag:status.corepractice.com.au,2005:Incident/189184442023-10-25T16:38:10+11:002023-10-25T16:38:10+11:00System Outage<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>16:38</var> AEDT</small><br><strong>Resolved</strong> - Summary of Impact:<br />On 25th October 2023, between 11:00AM to 12:30PM (AEDT), a server outage was identified. A significant number of users encountered an "SSL Certificate Expired" warning from their browser, which inhibited their ability to access the Core Practice system.<br /><br />Root Cause:<br />Upon investigation, the expiration of the SSL Certificate was traced back to an unsuccessful automatic renewal. The failure in automatic renewal was attributed to modifications in the validation process of approved domains by our Certificate Authority provider. While we aimed to reduce the outage duration, this incident was exacerbated when it was discovered that the certificate had not been successfully installed across multiple servers. Fortunately, we managed to utilize redundant servers on standby to meet the user demands.<br /><br />Mitigation:<br />The Core Practice team is dedicated to improving our infrastructure to prevent such issues from recurring. We are currently incorporating Application Gateway technology to mitigate the risks tied to certificate management.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>13:11</var> AEDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>13:06</var> AEDT</small><br><strong>Monitoring</strong> - Our system is working expected now and we will continue to monitor the issue. We sincerely apologize for the impact to affected customers.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:34</var> AEDT</small><br><strong>Investigating</strong> - We are currently experiencing a system outage. Our teams are actively investigating and working on a resolution. We understand how critical our services are to your operations, and we apologize for any inconvenience this may be causing.</p>tag:status.corepractice.com.au,2005:Incident/181798192023-08-18T12:58:32+10:002023-08-18T12:58:32+10:00Issue with CP users not able to send SMS/receive SMS through Core<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>12:58</var> AEST</small><br><strong>Resolved</strong> - Summary of impact: On 18 August 2023 between 10AM-12:30PM, we have identified an issue where some users may not be able to send SMS massages or receive SMS messages through Core with error message "service unavailable. Please try again". This issue prevents two-way SMS from functioning.<br /><br />Root Cause: We have identified the SMS service was not available in Core for some of our users due to our SMS provider is experiencing a technical issue since 10:00 am AEST time today, impacting multiple services including Messaging.<br /><br />Mitigation: Our SMS provider's engineering team has been actively working to resolve the issue since it was reported this morning. According to the update provided as well as testing internally by our team, we confirmed the issue has been resolved and our SMS service is available for our users again.<br /> <br />Next Steps: Core Practice and our SMS provider will continue monitoring to ensure a full recovery of our SMS services. We appreciate your understanding and patience during this time.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>11:47</var> AEST</small><br><strong>Identified</strong> - We have identified the SMS service was not available in Core for some of our users due to our SMS provider is experiencing a technical issue with the Rest API since 10:00 am AEST time today, impacting multiple services including Messaging. Their engineering team is actively working to resolve the issue and providing updates.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>11:40</var> AEST</small><br><strong>Investigating</strong> - We have identified some users may not be able to send SMS massages or receive SMS messages through Core with error message "service unavailable. Please try again." <br /><br />Our team is currently investigating the issue working towards a resolution. <br /><br />We appreciate your patience and understanding.</p>tag:status.corepractice.com.au,2005:Incident/175880302023-06-16T15:24:48+10:002023-06-16T15:24:48+10:00Issue with patients not able to reply to SMS sent from Core<p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>15:24</var> AEST</small><br><strong>Resolved</strong> - Summary of impact: On 16 June 2023 between 9AM-3PM, we have identified an issue where some patients are unable to reply to SMS messages sent through Core. This issue prevents two-way SMS from functioning as expected, affecting SMS chat and reply confirmations for appointments.<br /> <br />Root Cause: Our SMS provider made some updates this morning in response to compliance with regulatory changes that require the registration of Alphanumeric Sender IDs across the entire industry. Unfortunately, these changes had unforeseeable consequences on standard SMS messages sent from regular numbers. Core Practice has been working closely with our supplier to reverse these changes.<br /><br />Mitigation: Our SMS provider has successfully completed a rollback of the settings. All SMS messages should now be sent from standard numbers, and SMS reply functionality should be working as expected.<br /><br />Next Steps: Core Practice will continue working closely with our supplier and will apply necessary hotfixes in the coming days. We appreciate your understanding and patience during this time.</p><p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>09:57</var> AEST</small><br><strong>Investigating</strong> - We have identified patients not able to reply to SMS massages sent through Core for some users.<br /><br />Our team is currently investigating the issue working towards a resolution. <br /><br />We appreciate your patience and understanding.</p>tag:status.corepractice.com.au,2005:Incident/175539602023-06-13T11:25:32+10:002023-06-13T11:25:32+10:00Issue with Application Icons<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>11:25</var> AEST</small><br><strong>Resolved</strong> - Summary of impact: On 13 June 2023 between 10AM-11AM, application icons in Core Practice did not display correctly for some users.<br /><br />Root Cause: After investigation we have identified that this incident is related to some unexpected downtime from our hosting provider for icon package.<br /><br />Mitigation: Currently the issue has been resolved and icons should appear correctly in Core Practice. Please refresh your browser cache.<br /><br />Next steps: Our engineering team is working on a solution with our service provider to mediate this issue for future events.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>10:53</var> AEST</small><br><strong>Investigating</strong> - We have identified application icons in Core Practice may not display correctly for some users.<br /><br />Our team is currently investigating the issue working towards a resolution. <br /><br />We appreciate your patience and understanding.</p>tag:status.corepractice.com.au,2005:Incident/163184702023-03-01T21:00:00+11:002023-03-02T09:01:25+11:00Planned maintenance - Completed<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>21:00</var> AEDT</small><br><strong>Resolved</strong> - Planned maintenance to improve system performance has been successfully completed overnight. The maintenance work involved upgrading our servers and infrastructure, and implementing various optimizations to enhance the overall stability and responsiveness of our system. <br /><br />Our engineers have thoroughly tested the system and we expect everything to be working as expected.</p>tag:status.corepractice.com.au,2005:Incident/159291972023-01-25T21:02:17+11:002023-01-25T21:02:17+11:00System outage<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>21:02</var> AEDT</small><br><strong>Resolved</strong> - We sincerely apologize for the impact to affected customers. Investigations have assessed the best ways to improve architectural resiliency and mitigate the issues that contributed to this incident and its widespread impact.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>20:53</var> AEDT</small><br><strong>Monitoring</strong> - Core Practice seem to be working as expected, we will continue to monitor server performance.<br /><br />Microsoft Azure have determined the network connectivity issue is occurring with devices across the Microsoft Wide Area Network (WAN). This impacts connectivity between clients on the internet to Azure, as well as connectivity between services in datacenters, as well as ExpressRoute connections. The issue is causing impact in waves, peaking approximately every 30 minutes. <br /><br />For more info: https://status.azure.com/en-us/status</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>19:25</var> AEDT</small><br><strong>Investigating</strong> - Starting at 06:30PM SYDNEY Time, we're aware of a connectivity issue relating to Service Provider Microsoft Azure. <br /><br />We have temporarily restored some services in Core Practice but some users may still experience degraded performance.<br /><br />We are actively investigating and will share updates as soon as more is known.</p>tag:status.corepractice.com.au,2005:Incident/111352122022-09-15T13:08:35+10:002022-09-15T13:08:35+10:00Elevated level of application errors<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>13:08</var> AEST</small><br><strong>Resolved</strong> - Between 9:00AM and 10:00AM AEDT (approximately) on 15 Sept 2022, Core Practice monitoring systems have detected high levels of application errors that triggered automatic alerts.<br /><br />Engineers have investigated the root cause and have identified an isolated customer with incorrect settings. Settings was corrected and application errors is now at a normal level.<br /><br />System is now working as expected and no users were affected during this time.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>11:16</var> AEST</small><br><strong>Identified</strong> - Services are restored. We are continuing to investigate the issue to find the root cause.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>09:08</var> AEST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of application errors and are currently looking into the issue.</p>tag:status.corepractice.com.au,2005:Incident/103559892022-06-20T12:27:48+10:002022-06-21T08:57:33+10:00Elevated Application Errors when sending Online Form<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>12:27</var> AEST</small><br><strong>Resolved</strong> - Performance should be restored to normal service, sending Online Form email should be working as expected. <br /><br />We apologize for the impact to affected customers. We are continuously taking steps to improve Core Practice platform.<br /><br />We are monitoring system stability during this time.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>11:13</var> AEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>11:03</var> AEST</small><br><strong>Investigating</strong> - Summary of impact: From 08:00AM to 12:00PM AEDT (approx) on 22 June 2022, we're experiencing an elevated level of application errors when trying to send Online Form using email or sms. During this time some users may not be able to send Online Form.<br /><br />Root Cause: We are investigating the root cause may be related to third party integration related to short link service.<br /><br />Mitigation: As a workaround we are recommending users to use Online Form Cast feature to send to iPad during this time.<br /><br />Next steps: We are releasing an emergency hotfix, will provide further update once this is released.</p>tag:status.corepractice.com.au,2005:Incident/101037382022-05-30T21:26:53+10:002022-05-30T21:26:53+10:00Planned maintenance for HICAPS and Media Bridge Server<p><small>May <var data-var='date'>30</var>, <var data-var='time'>21:26</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>21:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>17:13</var> AEST</small><br><strong>Scheduled</strong> - Planned maintenance has been scheduled to improve performance and stability for HICAPS and Media Bridge Server. <br /><br />We expect to complete the maintenance within 1 hours and will post frequent updates to indicate maintenance progress.<br /><br />Core Practice platform on Monday 30th May 2022 from 9:00 pm AEST.<br /><br />During this time, HICAPS and Media Bridge functionality or won’t be available.<br /><br />For details on this work and to subscribe to future system maintenance updates, go to https://status.corepractice.com.au</p>tag:status.corepractice.com.au,2005:Incident/98995532022-05-03T16:29:44+10:002022-05-03T16:29:44+10:00Core Practice Bridge connectivity<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>16:29</var> AEST</small><br><strong>Resolved</strong> - We apologize for the impact to affected customers. We are continuously taking steps to improve Core Practice platform.<br /><br />To help ensure such incidents do not occur in the future we have added dedicated Server and new end point to handle Bridge connections.<br /><br />We will monitor stability of new infrastructure before migrating users.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>10:12</var> AEST</small><br><strong>Investigating</strong> - Between 8:00AM and 10:00AM AEDT (approx) on 03 May 2022, we have identified one of our server may have intermittently experienced connectivity issue, during this time your Core Practice Bridge Software for HICAPS or Media Bridge may not work as expected.</p>tag:status.corepractice.com.au,2005:Incident/88491312021-12-15T12:40:46+11:002023-06-13T10:54:21+10:00Issue with Application Icon<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>12:40</var> AEDT</small><br><strong>Resolved</strong> - Service provider have resolved the issue. Icon should appear correctly in Core Practice.<br /><br />Please refresh browser cache.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>12:18</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>12:17</var> AEDT</small><br><strong>Update</strong> - We have identified Icon Service Provider is currently experiencing an issue which is affecting our ability to provide Icon Service to Core Practice.<br /><br />We will resolve this incident once we have received confirmation the issue has been resolved from our upstream provider.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>12:10</var> AEDT</small><br><strong>Investigating</strong> - We have identified application icons in Core Practice may not display correctly for some users.</p>tag:status.corepractice.com.au,2005:Incident/86681592021-11-23T22:31:26+11:002021-11-23T22:31:26+11:00New message status appear for all Appointments<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>22:31</var> AEDT</small><br><strong>Resolved</strong> - Issues has been resolved, service should return to normal operation. We sincerely apologize for the impact to affected customers for the inconvenience.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>15:39</var> AEDT</small><br><strong>Identified</strong> - We have identified the root cause and have created a fix, we will be updating our system and refresh the cache after 6PM</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>15:36</var> AEDT</small><br><strong>Investigating</strong> - Some users may be experiencing New message status appear for all Appointments. Our Development team is currently investigating issues related to recent update which create incorrect appointment cache.</p>tag:status.corepractice.com.au,2005:Incident/78099902021-08-21T14:55:40+10:002021-08-21T14:55:40+10:00Elevated Application Errors<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>14:55</var> AEST</small><br><strong>Resolved</strong> - Performance should be restored to normal service.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>09:34</var> AEST</small><br><strong>Monitoring</strong> - This issue have been resolved, users should be able to login using two-step authentication and SMS should be working as expected. We will continue to monitor.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>08:49</var> AEST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of application errors When trying to login using two-step authentication or sending SMS. We are currently looking into the issue with SMS provider.</p>tag:status.corepractice.com.au,2005:Incident/73345732021-06-27T21:46:17+10:002021-06-27T21:46:17+10:00Planned database maintenance for some customers<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>21:46</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>21:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>11:18</var> AEST</small><br><strong>Scheduled</strong> - Planned database maintenance has been scheduled to improve performance and stability. Intermittent service disruptions should be expected while working with the indicated Core Practice features. We expect to complete the maintenance within 1 hours and will post frequent updates to indicate maintenance progress.<br /><br />Core Practice platform on Sunday 27th June 2021 from 9:00 pm AEST.<br /><br />During this time, our services will have limited functionality or won’t be available.<br /><br />For details on this work and to subscribe to future system maintenance updates, go to https://status.corepractice.com.au</p>tag:status.corepractice.com.au,2005:Incident/65393942021-03-16T13:20:18+11:002021-03-23T15:00:55+11:00Elevated Application Errors<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:20</var> AEDT</small><br><strong>Resolved</strong> - Performance should be restored to normal service. Please let us know if you still experience issue.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:00</var> AEDT</small><br><strong>Monitoring</strong> - Affected server was taken out of the pool, services should be restored. Some users may still experience slower than normal performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:00</var> AEDT</small><br><strong>Identified</strong> - Affected server was taken out of the pool, services should be restored. Some users may still experience slower than normal performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:00</var> AEDT</small><br><strong>Update</strong> - Affected server was taken out of the pool, services should be restored. Some users may still experience slower than normal performance.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:43</var> AEDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:39</var> AEDT</small><br><strong>Investigating</strong> - Starting 6AM on 15th March 2021, Core Practice rely on Microsoft Azure Active Directory to authenticate with Microsoft services, including Storage Service to upload Media, Attachment and to encrypt Form Data. We're experiencing an elevated level of application errors and are currently looking into the issue.</p>tag:status.corepractice.com.au,2005:Incident/45004812020-07-01T12:37:24+10:002020-07-01T12:37:24+10:00Degraded Performance for some users<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>12:37</var> AEST</small><br><strong>Resolved</strong> - We sincerely apologize for the impact to affected customers. Investigations have assessed the best ways to improve architectural resiliency and mitigate the issues that contributed to this incident and its widespread impact.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>10:54</var> AEST</small><br><strong>Investigating</strong> - Summary of impact: On 1 July 2020 between 7AM-8AM, one of our server seem to experienced degraded performance or connectivity issue, during this time some users may not be able to login. <br /><br />Root Cause: We are investigating the root cause may be related to recent software update.<br /><br />Mitigation: Affected server was taken out of the pool, this will allow users to access Core Practice.<br /><br />Next steps: We are investigating for potential software updates to recover the affected server. We are also adding new server in case affected server can not be recovered. During this time you may experience slow performance than usual during peak period.</p>tag:status.corepractice.com.au,2005:Incident/32264632019-11-18T09:15:40+11:002019-11-18T13:04:59+11:00Core Practice is down for some users<p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>09:15</var> AEDT</small><br><strong>Resolved</strong> - Service monitor detected the build-up of unprocessed requests at 9:00AM AEDT. Engineer initiated secondary backup memory server and began reloading the data into memory server by 9:15AM AEDT. The cache refresh process on memory server took 15 minutes to complete and at 9:30AM AEDT most service was restored. The impacted network path was pinpointed and traffic was reconfigured to bypass the failure path. <br /><br />Next steps: We sincerely apologize for the impact to affected customers. Investigations have assessed the best ways to improve architectural resiliency and mitigate the issues that contributed to this incident and its widespread impact. <br /><br />Please perform the following:<br /> 1. Reconnect Hicaps and Media Bridge<br /> 2.Refresh browser</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>09:00</var> AEDT</small><br><strong>Identified</strong> - Lost of network connectivity on our memory server was determined as the underlying root cause, it also coincided with morning peak traffic. This combination of events caused build-up of unprocessed request and longer recovery time. Engineers have routed the traffic to minimize any customer impact.</p><p><small>Nov <var data-var='date'>18</var>, <var data-var='time'>08:45</var> AEDT</small><br><strong>Investigating</strong> - Between 8:45AM and 9:15AM AEDT (approx) on 18 Nov 2019, we have identified one of our server may have intermittently experienced degraded performance, during this time our users may experience limited functionality or was not available.</p>