On October 10, 2023 at approximately 7:18AM PDT our systems experienced a large spike in traffic which caused intermittent issues in click processing as well as limited Dashboard access. Our engineers acted quickly to increase our infrastructure capacity to be able to handle the spike in traffic, but there were still intermittent issues while they worked to resolve things that lasted until 10:30AM PDT.
While all servers remained online during this time and the majority of clicks were still being processed, there were still some clicks that failed to be processed and resulted in timeout errors, with the majority of the impact being limited to the Western part of the United States. In total there was approximately two hours of partial downtime / intermittent service impact, before our engineers were able to get everything running smoothly again.
In order to prevent this moving forward we have significantly tuned our infrastructure (increased timeouts, better request processing distribution, improved caching, better monitoring, etc) as well as increased our overall infrastructure capacity in order to be able to better handle large spikes in traffic moving forward.