tag:status.unific.com,2005:/historyUnific's Status - Incident History2024-03-29T01:22:51-04:00Unific'stag:status.unific.com,2005:Incident/203012032024-03-12T12:30:00-04:002024-03-19T09:28:03-04:00Hubspot OAuth tokens not refreshing due to Hubspot outage.<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>12:30</var> EDT</small><br><strong>Resolved</strong> - Hubspot had an outage that caused OAuth tokens to not be refreshed. A utility has been run to restore Hubspot OAuth tokens after the outage.<br /><br />https://status.hubspot.com/incidents/590gy6mpxjhk</p>tag:status.unific.com,2005:Incident/199685282024-02-13T09:12:42-05:002024-02-13T09:12:42-05:00Clients have reported that Unific syncs are slow or failing.<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:12</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>00:20</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>21:15</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:06</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:05</var> EST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.unific.com,2005:Incident/169269102023-04-21T14:12:50-04:002023-04-21T14:12:50-04:00Hubspot OAuth tokens are failing to refresh again.<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>14:12</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>09:50</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>23:46</var> EDT</small><br><strong>Identified</strong> - We are working with Hubspot to address this recurring issue.</p>tag:status.unific.com,2005:Incident/169246492023-04-20T18:38:26-04:002023-04-20T18:38:26-04:00Hubspot OAuth tokens are failing to refresh.<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>18:38</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>17:06</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.unific.com,2005:Incident/169222622023-04-20T14:02:51-04:002023-04-20T14:02:51-04:00Hubspot OAuth tokens are failing to refresh.<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>14:02</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>13:55</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:52</var> EDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>11:52</var> EDT</small><br><strong>Identified</strong> - Hubspot is returning "Forbidden (403) - error code: 1006" while trying to refresh OAuth tokens. This error is happening for Hubspot API calls as well. We are in communication with Hubspot to get this issue resolved as quickly as possible.</p>tag:status.unific.com,2005:Incident/127445822022-11-07T11:52:57-05:002022-11-07T11:52:57-05:00Multiple Unific services are failing to start.<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:52</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:22</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:19</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>11:10</var> EST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.unific.com,2005:Incident/126831342022-10-31T17:30:00-04:002022-11-01T11:15:50-04:00Webhooks are failing due to expired AWS token<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>17:30</var> EDT</small><br><strong>Resolved</strong> - Webhooks are failing to due to expired AWS token. This issue has been resolved.</p>tag:status.unific.com,2005:Incident/99040252022-05-02T22:30:00-04:002022-05-03T10:18:29-04:00Dynamic coupons not generating.<p><small>May <var data-var='date'> 2</var>, <var data-var='time'>22:30</var> EDT</small><br><strong>Resolved</strong> - Dynamic coupons are not generating. A Unific release to upgrade the Hubspot signature validator to version 3 is preventing dynamic coupon webhooks from Hubspot from validating the signature correctly. This change was rolled back for Unific Dynamic Coupon apps and coupons should be generating now.</p>tag:status.unific.com,2005:Incident/98817582022-04-29T13:30:00-04:002022-04-29T14:51:15-04:00503 Service Temporarily Available when trying to login to Unific<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>13:30</var> EDT</small><br><strong>Resolved</strong> - Multiple users have reported 503 Service Temporarily Available when trying to login to Unific. This issue has been identified and resolved.</p>tag:status.unific.com,2005:Incident/95386312022-03-13T17:30:00-04:002022-03-14T10:40:36-04:00Sync, dynamic coupons not working.<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>17:30</var> EDT</small><br><strong>Resolved</strong> - Tasks were failing to start.</p>tag:status.unific.com,2005:Incident/94903262022-03-08T12:20:59-05:002022-03-08T12:20:59-05:00Unific login failing.<p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>12:20</var> EST</small><br><strong>Resolved</strong> - This was related to a DNS issue and has been resolved.</p><p><small>Mar <var data-var='date'> 8</var>, <var data-var='time'>11:06</var> EST</small><br><strong>Investigating</strong> - Unific logins are failing with error: {"crossDomain":true,"method":"POST","url":"https://auth.unific.com/co/authenticate"}. We are investigating the issue.</p>tag:status.unific.com,2005:Incident/95019152022-03-05T12:00:00-05:002022-03-09T16:36:22-05:00Coupons are slow to create.<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>12:00</var> EST</small><br><strong>Resolved</strong> - Multiple customers have reported that dynamic coupons are taking a long time to create.</p>tag:status.unific.com,2005:Incident/88681792021-12-16T18:38:00-05:002021-12-16T18:38:00-05:00Hubspot OAuth tokens are not refreshing.<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>18:38</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved. Most clients weren't affected by this issue, but please verify your Hubspot integration is still active. If it is not active, you will need to reconnect your Hubspot integration. Please reach out to Unific support if you need any help.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>18:18</var> EST</small><br><strong>Monitoring</strong> - Unific restored a previous version of integrations to try to mitigate this issue. Monitoring.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>16:14</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue with Hubspot.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>15:28</var> EST</small><br><strong>Update</strong> - We are in contact with Hubspot about this issue.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>15:01</var> EST</small><br><strong>Investigating</strong> - At around 11:23am EST, Hubspot OAuth tokens started returning invalid characters which prevent the token from refreshing. This is preventing syncs to Hubspot. We are actively investigating this issue.</p>tag:status.unific.com,2005:Incident/87817812021-12-07T16:25:27-05:002021-12-07T16:25:27-05:00Sync slowness<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:25</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:56</var> EST</small><br><strong>Update</strong> - Unific sync is catching up and is now 2 hours behind.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:03</var> EST</small><br><strong>Update</strong> - Unific sync is catching up and is now 5 hours behind.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:50</var> EST</small><br><strong>Monitoring</strong> - Unific has scaled up its infrastructure to work through the backlog of webhooks.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:32</var> EST</small><br><strong>Identified</strong> - There has been a backup of inbound webhooks beginning around 1:15am EST. Some webhooks are behind by up to 12 hours.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:12</var> EST</small><br><strong>Investigating</strong> - Sync slowness has been reported and we are investigating the issue.</p>tag:status.unific.com,2005:Incident/86741472021-11-23T16:40:41-05:002021-11-23T16:40:41-05:00Sync slowness<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>16:40</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>15:36</var> EST</small><br><strong>Update</strong> - Unific is catching up and is about ~1.5 hours behind processing webhooks.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>14:53</var> EST</small><br><strong>Monitoring</strong> - Unific is experiencing high volumes since this morning and has scaled up to process syncs more quickly. There's a 2.5 hour delay with some syncs. We are monitoring the situation.</p>tag:status.unific.com,2005:Incident/84483462021-11-10T19:36:38-05:002021-11-10T19:36:38-05:00Issues refreshing OAuth tokens.<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>19:36</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:57</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>18:47</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>16:39</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue as a top priority. We've made some adjustments that allow tokens to refresh more often, however, we are experiencing timeouts when the tokens refresh. If a token is not refreshed in the 30-minute window, Unific will not be able to sync data to HubSpot until the token is refreshed. Thus you may see delays in the data sync and coupon generations.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>10:03</var> EST</small><br><strong>Investigating</strong> - On Monday November 8th, Hubspot changed the length of time an OAuth token expires from 6 hours to 30 minutes. Many OAuth tokens are not refreshing within that 30 minute window. We are actively investigating the issue and working on a fix.</p>tag:status.unific.com,2005:Incident/76930182021-08-06T08:57:50-04:002021-08-06T08:57:50-04:00Order sync slowness.<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>08:57</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>17:26</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>15:34</var> EDT</small><br><strong>Identified</strong> - A large historical is impacting the speed of live order syncs. We are actively working on resolving the issue.</p>tag:status.unific.com,2005:Incident/70953142021-05-26T23:53:09-04:002021-05-26T23:53:09-04:00Order sync slowness.<p><small>May <var data-var='date'>26</var>, <var data-var='time'>23:53</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>23:24</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:16</var> EDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:16</var> EDT</small><br><strong>Identified</strong> - We have identified an issue and are working on resolving it.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>20:13</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating order sync slowness that began around 12pm EST today.</p>tag:status.unific.com,2005:Incident/67968642021-04-20T16:11:23-04:002021-04-20T16:11:23-04:00Trouble logging in to Unific.<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>16:11</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:24</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>12:24</var> EDT</small><br><strong>Investigating</strong> - Logins to Unific are failing due to a service provider being down. Currently investigating.</p>tag:status.unific.com,2005:Incident/60703742021-01-19T17:00:00-05:002021-01-20T12:08:06-05:00Contact and orders syncs stopped for certain customers.<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>17:00</var> EST</small><br><strong>Resolved</strong> - After Unific's production release today, certain customers Hubspot queues were failing to process and sync to Hubspot. The issue has been identified and fixed and those customers queues who were affected should be processing again.</p>tag:status.unific.com,2005:Incident/50114902020-09-03T13:39:00-04:002020-09-03T13:41:44-04:00Delay in data sync<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>13:39</var> EDT</small><br><strong>Resolved</strong> - There was a delay of about 4 hours for order, customer, and coupon syncs. The issue has been resolved and the sync is back to normal.</p>tag:status.unific.com,2005:Incident/44316582020-06-22T12:29:19-04:002020-06-22T12:29:19-04:00Audit & Recovery feature is currently not working.<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>12:29</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>11:14</var> EDT</small><br><strong>Identified</strong> - After Unific's release on Friday, June 19th, the audit and recovery feature that verifies syncs as well as associations are successful to HubSpot stopped working due to an index creation on a table.<br /><br />Some HubSpot portals might experience delays with associating contacts to deals and products to deal line items in HubSpot.</p>tag:status.unific.com,2005:Incident/42505092020-05-31T07:30:00-04:002020-06-04T12:24:22-04:00Contact and order sync metrics stopped reporting<p><small>May <var data-var='date'>31</var>, <var data-var='time'>07:30</var> EDT</small><br><strong>Resolved</strong> - Metrics stopped reporting to our provider, DataDog, which then triggered monitors to notify us that contact and order syncs stopped. However, since this was just a failure of reporting metrics to DataDog, syncs continued during this period of time without issue.<br /><br />https://docs.datadoghq.com/agent/faq/certificate_verify_failed-error/</p>tag:status.unific.com,2005:Incident/42505312020-05-15T12:00:00-04:002020-06-01T12:58:57-04:00Test<p><small>May <var data-var='date'>15</var>, <var data-var='time'>12:00</var> EDT</small><br><strong>Resolved</strong> - No outage, just testing the alert triggers properly. It does!</p>tag:status.unific.com,2005:Incident/27618672019-07-24T09:46:35-04:002020-05-15T15:10:26-04:00This is an example incident<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>09:46</var> EDT</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>09:36</var> EDT</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>09:42</var> EDT</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>10:12</var> EDT</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>