tag:status.geni.us,2005:/historyGeniuslink Status - Incident History2024-03-27T19:50:10-07:00Geniuslinktag:status.geni.us,2005:Incident/194288342023-12-13T13:30:12-08:002023-12-13T13:30:12-08:00Click reports are currently running behind. We are investigating the issue.<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>13:30</var> PST</small><br><strong>Resolved</strong> - Issue with delayed click reports has been resolved, and all backed up clicks should now be reflected in the dashboard.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>09:48</var> PST</small><br><strong>Monitoring</strong> - The issue with delayed reporting has been found and reporting metrics should be flowing into the dashboard again. Note that this may take a bit for reports to fully catch back up, so we will continue monitoring and update once things are caught up.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>07:36</var> PST</small><br><strong>Investigating</strong> - Click reports in the dashboard appear to be lagging behind this morning due to a reporting backup. Links are functioning normally and we expect reports to be updated with correct and current values when this issue is resolved.</p>tag:status.geni.us,2005:Incident/187441492023-10-11T17:27:23-07:002023-10-14T08:51:36-07:00Intermittent issue preventing some links from resolving<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>17:27</var> PDT</small><br><strong>Resolved</strong> - Our team has monitored this issue for the past ~32 hours since 10:30AM PST yesterday to ensure that everything remained operational and that the improvements to infrastructure were able to handle the increased capacity during Prime day. Everything remains operational and there was no further issues throughout this time.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>11:34</var> PDT</small><br><strong>Monitoring</strong> - Engineers have re-tuned infrastructure significantly to handle the increased traffic. They’ve added increased server capacity and made other improvements to be able to handle increased requests. Link resolution as well as dashboard functionality has been fully operational since 10:30AM PST (it was intermittent starting at 7:30AM PST), and we are continuing to monitor.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>09:59</var> PDT</small><br><strong>Investigating</strong> - Dashboard and Links are still intermittently timing out, but our engineers are all hands on deck to get things resolved.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>08:40</var> PDT</small><br><strong>Monitoring</strong> - Server slowdown has been stabilized as of 8:13PST, and Links/Dashboard should be resolving/loading properly. Our engineers are still monitoring closely to ensure that everything remains operational.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>07:41</var> PDT</small><br><strong>Investigating</strong> - Today at around 7:18PST Links and the dashboard became slow or unresponsive. We are investigating the issue and hope to resolve it shortly.</p>tag:status.geni.us,2005:Incident/184940212023-09-13T15:55:35-07:002023-09-13T15:55:35-07:00Partial click processing outage.<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>15:55</var> PDT</small><br><strong>Resolved</strong> - The issues with the upstream provider have been resolved by their team and everything is fully operational again for Geniuslink as of 3:51PM PST.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>13:48</var> PDT</small><br><strong>Investigating</strong> - There appears to be an issue with one of our hosting providers in San Francisco that is causing intermittent click processing issues for some customers. Our engineers are investigating and we will work to resolve things as soon as possible.</p>tag:status.geni.us,2005:Incident/182273242023-08-23T07:30:00-07:002023-08-23T08:31:28-07:00Delayed click reporting for EU clicks<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>07:30</var> PDT</small><br><strong>Resolved</strong> - Click reporting is all caught up as of 7:35AM PST, and dashboard reports should all reflect the previously backed up clicks.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>17:29</var> PDT</small><br><strong>Monitoring</strong> - Click reporting for clicks coming from the EU was backed up starting on Aug 17th, which meant that these clicks may not have been properly reported inside of our dashboard. This had no impact for customers clicking these links, but simply meant that there may have been a temporary delay in the reporting inside of the Geniuslink Dashboard. This has since been fixed, and these clicks should be flowing into the dashboard over the next few hours.</p>tag:status.geni.us,2005:Incident/180430812023-08-03T10:37:56-07:002023-08-03T14:31:31-07:00Visual Bug in Global Visualizer<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>10:37</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>10:37</var> PDT</small><br><strong>Update</strong> - Issue has been resolved as of 10:30AM PST.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>09:52</var> PDT</small><br><strong>Investigating</strong> - There appears to be a visual bug in our Global Visualizer tool that is showing "no local match" for all countries. Rest assured though, all automatic localization and affiliation for links is still working as it should. This is only a visual bug in the dashboard. Our engineers are investigating this and will get things back up and running as soon as possible.</p>tag:status.geni.us,2005:Incident/180450532023-08-03T09:30:00-07:002023-08-03T14:59:54-07:00Proxy link processing issues.<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>09:30</var> PDT</small><br><strong>Resolved</strong> - At 10AM PST Aug 3, 2023, we discovered an issue with our Proxy link format, which impacted Amazon Link Engine Plugin, JS Snippet, and Quickbuild links run through our service. This issue caused these links to fail to be processed by our service, defaulting back to the original url. The deployment that caused this issue was rolled back, and all links were back to functional as of 10:30AM PST Aug 3, 2023.</p>tag:status.geni.us,2005:Incident/175758972023-06-14T14:15:05-07:002023-06-14T14:15:05-07:00Scheduled Infrastructure Maintenance<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>14:15</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>14:09</var> PDT</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:52</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:51</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.geni.us,2005:Incident/175757982023-06-14T13:50:18-07:002023-06-14T13:50:18-07:00Scheduled Infrastructure Maintenance<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:50</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:40</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:40</var> PDT</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>13:39</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.geni.us,2005:Incident/175657312023-06-13T15:18:16-07:002023-06-13T15:18:16-07:00Scheduled Dashboard Maintenance<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:18</var> PDT</small><br><strong>Completed</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:15</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:14</var> PDT</small><br><strong>Scheduled</strong> - We are undergoing scheduled dashboard maintenance which may impact account login.</p>tag:status.geni.us,2005:Incident/165179142023-03-15T16:47:23-07:002023-03-15T16:47:23-07:00Scheduled MySQL Database Maintenance<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:47</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:30</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:15</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing a short period of maintenance on our MySQL database from 4:30PM-4:55PM PDT. This will only impact link creation and Account creation for a few minutes as we complete the scheduled updates.</p>tag:status.geni.us,2005:Incident/165144022023-03-15T11:15:40-07:002023-03-15T11:15:40-07:00Account / Link Creation Issues.<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:15</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved, and account / link creation is all working again. The issue was a result of a MySQL database interruption, that has since been resolved, and only impacted new link creation / account creation. All existing links were unaffected by this issue.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>10:29</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>10:28</var> PDT</small><br><strong>Identified</strong> - There is currently a dashboard slowdown that appears to only be affecting new signups and new link creation. We are investigating this issue now, and will update when we have further information.</p>tag:status.geni.us,2005:Incident/157656572023-01-05T09:41:33-08:002023-01-05T09:41:33-08:00Issue with Credit Card Processing<p><small>Jan <var data-var='date'> 5</var>, <var data-var='time'>09:41</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. All Invoices for usage in December 2022 have been generated, and will be charged on Jan 25th 2023.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>10:54</var> PST</small><br><strong>Investigating</strong> - We have been alerted to an issue in our regularly scheduled billing cycle that has caused some payments to fail even though a valid card is on file. Please ignore any failed Payment emails that you may have received. We are currently investigating and will update as soon as we learn more.</p>tag:status.geni.us,2005:Incident/140156752022-11-30T10:16:33-08:002022-11-30T10:20:29-08:00Scheduled Infrastructure Maintenance<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:16</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>08:57</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance at this time. Dashboard may be temporarily unavailable, but link resolution and reporting will not be effected.</p>tag:status.geni.us,2005:Incident/139691972022-11-29T17:15:44-08:002022-11-29T17:26:47-08:00Scheduled Infrastructure Maintenance<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:26</var> PST</small><br><strong>Update</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:15</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>17:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>16:40</var> PST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>16:39</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. Dashboard may be temporarily unavailable, but link resolution will be unaffected.</p>tag:status.geni.us,2005:Incident/139643872022-11-29T15:17:08-08:002022-11-29T15:17:08-08:00Scheduled Infrastructure Maintenance<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:17</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:03</var> PST</small><br><strong>Update</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:03</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:02</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance at this time. Dashboard may be temporarily unavailable, but link resolution will not be impacted.</p>tag:status.geni.us,2005:Incident/113020942022-09-28T12:07:35-07:002022-09-28T12:07:35-07:00Scheduled Infrastructure Maintenance<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>12:07</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>10:07</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>10:06</var> PDT</small><br><strong>Scheduled</strong> - We will be doing scheduled infrastructure maintenance from 10am - 12pm PDT 9/28/22. This will impact signup as well as link and group creation/editing, but will have no impact on click processing. All existing links will continue to work as expected during this time.</p>tag:status.geni.us,2005:Incident/112891142022-09-28T10:00:43-07:002022-09-28T10:00:43-07:00Scheduled Infrastructure Maintenance<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>10:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>11:20</var> PDT</small><br><strong>Scheduled</strong> - We will be doing scheduled infrastructure maintenance from 9am - 10am PDT 9/28/22. This will impact signup as well as link and group creation/editing, but will have no impact on click processing. All existing links will continue to work as expected during this time.</p>tag:status.geni.us,2005:Incident/112210262022-09-23T09:00:46-07:002022-09-23T08:07:04-07:00Scheduled Infrastructure Maintenance<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>07:38</var> PDT</small><br><strong>Update</strong> - Our engineers are just getting started with this migration, and are a few minutes behind schedule. We will update this status as soon as everything has been completed.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>07:01</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>07:06</var> PDT</small><br><strong>Scheduled</strong> - We will be doing scheduled infrastructure maintenance from 7am - 8am PDT 9/23/22. This will impact signup as well as link and group creation/editing, but will have no impact on click processing. All existing links will continue to work as expected during this time.</p>tag:status.geni.us,2005:Incident/112356112022-09-23T08:45:23-07:002022-09-23T08:45:23-07:00Scheduled maintenance still in progress.<p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>08:45</var> PDT</small><br><strong>Completed</strong> - Initial round of maintenance has been completed, and all dashboard functionality should be fully operational again. We will be scheduling a followup maintenance window in the future to complete the work.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>08:13</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>23</var>, <var data-var='time'>08:12</var> PDT</small><br><strong>Scheduled</strong> - Scheduled maintenance is still in progress. This will impact signup as well as link and group creation/editing, but will have no impact on click processing. All existing links will continue to work as expected during this time.</p>tag:status.geni.us,2005:Incident/111993962022-09-21T09:22:49-07:002022-09-21T09:22:49-07:00Temporary Reporting Lag<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>09:22</var> PDT</small><br><strong>Resolved</strong> - Reporting has all been caught up as of 11:30PM PST.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>16:46</var> PDT</small><br><strong>Investigating</strong> - There is currently a slight reporting lag due to infrastructure deployments. We will update once reports have been caught back up.</p>tag:status.geni.us,2005:Incident/110116112022-09-01T17:34:43-07:002022-09-01T17:34:43-07:00Interruption Preventing Choice page Creation/Edit in Link Creator<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>17:34</var> PDT</small><br><strong>Resolved</strong> - We've identified the source of the conflict causing our Choice Page image uploader to error and have deployed a hotfix - this issue should now be resolved, but we are continuing to monitor and will deploy additional fixes if needed early next week.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Identified</strong> - An issue with our image hosting widget used for handling image uploads in Choice Pages is currently causing errors when creating or editing Choice Pages. We have identified the specific issue and are working with our image host to deploy a fix to correct this.</p>tag:status.geni.us,2005:Incident/108787602022-08-22T13:36:06-07:002022-08-22T13:36:06-07:00Interruption Preventing Choice page Creation/Edit in Link Creator<p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>13:36</var> PDT</small><br><strong>Resolved</strong> - Choice page creation and editing functionality has been resolved. Custom image upload for choice pages may still be intermittently impacted due to the 3rd party updates.</p><p><small>Aug <var data-var='date'>22</var>, <var data-var='time'>07:58</var> PDT</small><br><strong>Identified</strong> - An update in to a third party service provider seems to be the cause of this incident. Our engineers are currently looking for a potential fix for this.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>17:27</var> PDT</small><br><strong>Investigating</strong> - Choice page and A/B Split link creation/editing is temporarily broken due to a UI bug. Existing choice pages/links are not impacted by this issue and will continue to work as expected. We are investigating and will update once we have more information.</p>tag:status.geni.us,2005:Incident/95532312022-03-16T09:01:18-07:002022-03-16T09:01:18-07:00Reporting data may be lagging behind.<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:01</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>20:24</var> PDT</small><br><strong>Monitoring</strong> - Reporting data may be lagging behind a bit, as database is refreshed. Clicks are still being recorded properly, but display data in the dashboard may temporarily not appear in real-time.</p>tag:status.geni.us,2005:Incident/95406982022-03-14T16:36:23-07:002022-03-14T16:36:23-07:00Delayed Click Reporting.<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>16:36</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. Delayed click summaries have completed syncing and reports should now show the most recent click data.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>11:51</var> PDT</small><br><strong>Monitoring</strong> - Click reporting is currently delayed, but servers have been refreshed and data should be filtering back in. We will update when resend is completed.</p>tag:status.geni.us,2005:Incident/82902082021-10-22T09:44:31-07:002021-10-22T09:44:31-07:00We currently investigating an issue preventing login to the dashboard<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>09:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>05:57</var> PDT</small><br><strong>Monitoring</strong> - We have applied a fix and logins appear to be functioning normally.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>03:15</var> PDT</small><br><strong>Investigating</strong> - Today beginning around 3:15am Pacific, users have reported that they are intermittently unable to log into the dashboard and/or are being unexpectedly logged out. We are investigating. Note that this does not affect link service or performance.</p>