Application Server 
Our main application that you probably want to be up and running all the time
Time | Status | Message |
---|---|---|
1 week, 2 days ago | ![]() |
All clear, PDFs are loading normally now. |
1 week, 2 days ago | ![]() |
We're seeing issues with PDF generation and the team is investigating. |
1 week, 4 days ago | ![]() |
The upgrade is complete and everything is starting back up! |
1 week, 4 days ago | ![]() |
We are down for scheduled maintenance. We are upgrading the primary DB server to a machine with 500GB of RAM. #bigserver #nomoreoutages |
1 week, 5 days ago | ![]() |
Performance has returned to normal. The development team will continue to monitor, but everything appears fine now. |
1 week, 5 days ago | ![]() |
We're seeing issues adding ticket comments and other performance issues. The team is investigating. |
3 weeks, 1 day ago | ![]() |
System should be back to normal now. |
3 weeks, 1 day ago | ![]() |
We're seeing intermittent issues with page loads and the development team is working on it. |
3 weeks, 5 days ago | ![]() |
We are observing normal traffic. |
3 weeks, 6 days ago | ![]() |
We are starting to see page loads go back to normal, but are still closely watching traffic. |
3 weeks, 6 days ago | ![]() |
The team is continuing to work on the load times. |
3 weeks, 6 days ago | ![]() |
Traffic is returning to normal and the team is carefully monitoring traffic on the site. |
3 weeks, 6 days ago | ![]() |
We experienced a brief outage caused by high latency. The team is bringing services back to normal. |
3 weeks, 6 days ago | ![]() |
We are experiencing high latency on the app. The team is actively working on bringing performance back in line. Will update as we have more information. |
1 month ago | ![]() |
Latency has returned to normal. The devs are continuing to monitor traffic closely and further refine search performance. |
1 month, 1 week ago | ![]() |
Ticket Comment and invoice line item searching are causing search performance degradation. We have temporarily disabled search for those and will bring them back online as soon as possible (you can still add line items to invoices normally). |
1 month, 1 week ago | ![]() |
We are seeing some high latency causing slow performance on the site. Some search query performance is effected. The devs are working on getting this resolved as fast as possible. |
1 month, 1 week ago | ![]() |
Performance has returned to normal. |
1 month, 1 week ago | ![]() |
We're seeing slow page loads and the development team is investigating. |
1 month, 2 weeks ago | ![]() |
It appears page load times are back to normal but we will have a team monitoring closely. |
1 month, 2 weeks ago | ![]() |
We had a production database issue that caused everything to timeout and show the Oops error page. Developers are working on it now and it should be running but more slowly while it recovers. |
1 month, 2 weeks ago | ![]() |
We're seeing slow page loads and the development team is investigating. |
1 month, 2 weeks ago | ![]() |
We are not down but getting lots of reports that we are down. This is probably due to some broader 'internet weather' type issues and we'll keep an eye on it. Try other internet connections if you can, like tethering to a cell phone/etc |
1 month, 3 weeks ago | ![]() |
All clear, back to normal. |
1 month, 3 weeks ago | ![]() |
We're having issues but the development team is working on it. |
1 month, 3 weeks ago | ![]() |
Performance is much closer to normal now. The site should be totally usable now. |
1 month, 3 weeks ago | ![]() |
We're seeing performance issues and the development team is investigating. |
2 months, 2 weeks ago | ![]() |
We are proceeding with replacing the search cluster and the underlying tech. Please report any search issues you find so we can make sure we are tracking any open issues thoroughly. |
2 months, 2 weeks ago | ![]() |
The search database is functioning in a limited state and we are planning to swap out the underlying infrastructure as soon as possible (probably tomorrow). This is urgent and we will do everything we can to do this with zero downtime. |
2 months, 2 weeks ago | ![]() |
We are seeing better stability currently. In case you missed our earlier message: "We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster. |
2 months, 2 weeks ago | ![]() |
New Update for everyone. We are experiencing another search index issue with The Search All The Things top field. Search will work in a reduced state during this time. Work continues to mitigate this and we apologize for the inconvenience. More updates to come as we have more information. |
2 months, 2 weeks ago | ![]() |
We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster. |
2 months, 2 weeks ago | ![]() |
We are experiencing another search index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com. |
2 months, 2 weeks ago | ![]() |
We're seeing issues with the search again and the development team is working on it. |
2 months, 2 weeks ago | ![]() |
We have re-enabled the main search system so everything should be back to normal. We will continue to monitor. We can't rule out new issues coming up tomorrow during peak load times based on what we saw so we're cautiously calling this incident closed and we'll be working on more safeguards into tomorrow. Expect a full writeup of the incident in the coming days as we take time to post-mortem. |
2 months, 2 weeks ago | ![]() |
Page performance is returning to normal. The Search remains working but with degraded performance. You can search for most things in their respective tabs until we get Search back up. We apologize and recognize the inconvenience this is causing. |
2 months, 2 weeks ago | ![]() |
We are seeing some slower load times and are actively working on it. We will update as we have more information. |
2 months, 2 weeks ago | ![]() |
We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue. |
2 months, 2 weeks ago | ![]() |
We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue. |
2 months, 2 weeks ago | ![]() |
We are experiencing a Search Index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com. |
2 months, 2 weeks ago | ![]() |
We are seeing site performance return to normal. We will continue to monitor everything as the day continues. |
2 months, 2 weeks ago | ![]() |
We are working on performance issues related to the search system and will update shortly. |
3 months, 2 weeks ago | ![]() |
It might be too soon to call it, but it appears everything is back up and processing again. We will keep an eye on it. |
3 months, 2 weeks ago | ![]() |
Our hosting provider (Heroku) is having issues that are impacting our availability. We are working with them to mitigate and restore access as fast as possible. You can follow this status page for information or their status page (https://status.heroku.com/) More updates to come. |
3 months, 2 weeks ago | ![]() |
We are experiencing an Up Stream issue and are actively investigating. |
6 months ago | ![]() |
All services appear to be functioning like normal again. We will continue monitoring the situation. |
6 months ago | ![]() |
It appears that an Amazon AWS global issue is impacting our connectivity to services that our software depends on, affecting performance. Our team is monitoring the situation closely. |
6 months ago | ![]() |
We're seeing general performance issues now. The development team is investigating. |
6 months ago | ![]() |
Our search database is showing errors and we are temporarily disabling it. The development team is investigating. |
8 months, 1 week ago | ![]() |
Everything is back to normal now! |
8 months, 1 week ago | ![]() |
We're seeing issues with Calendar entries loading and the development team is investigating. You should still be able to see a list of your appointments at /appointments. |
9 months, 1 week ago | ![]() |
The issue has been resolved. |
9 months, 1 week ago | ![]() |
We are currently experiencing issues and the development team is investigating. |
10 months, 2 weeks ago | ![]() |
We have isolated the issue and pushed a fix, and Ticket Charges are now working properly. |
10 months, 2 weeks ago | ![]() |
We've received reports of Ticket Charges not working for some users and we are currently investigating the cause. |
12 months ago | ![]() |
We’ve reverted an update to the site and standard functionality on the invoices page has returned. |
12 months ago | ![]() |
We are seeing a high error rate and are investigating, we'll provide updates here every few minutes. |
1 year ago | ![]() |
Maintenance has completed and the site is back online. Thanks! |
1 year ago | ![]() |
We are down for maintenance - we anticipate the downtime being less than 30 minutes. We will update every 15 minutes here. |
1 year, 1 month ago | ![]() |
We have brought the search database back up and are continuing to watch for any long load times or search issues. |
1 year, 1 month ago | ![]() |
We had to bring down our search data base temporarily after observing longer than normal load times. We will continue to update as we have more info. |
1 year, 1 month ago | ![]() |
Our search database is going to remain disabled for a few more minutes but the site has recovered and is usable. |
1 year, 1 month ago | ![]() |
Our new search database is not behaving well, we are temporarily disabling it. Engineers are working on it. |
1 year, 1 month ago | ![]() |
We are going to mark the application server as up and continue adding notes to the background processor here on the status site. |
1 year, 1 month ago | ![]() |
We have a mitigation idea we are testing out that might have background processes working shortly. S3 is still very broken. |
1 year, 1 month ago | ![]() |
The search database is going to be outdated until the background processes start up again, except for Tickets - which should be updating in the search db properly. |
1 year, 1 month ago | ![]() |
Although the application servers are working, due to an Amazon S3 outage our background processes are blocked. Our platform provider isn't allowing us to push updates while S3 is broken. |
1 year, 1 month ago | ![]() |
We have returned to normal activity and will continue to monitor. |
1 year, 1 month ago | ![]() |
We’re investigating issues with making invoices |
1 year, 1 month ago | ![]() |
We have returned to normal activity. |
1 year, 1 month ago | ![]() |
We are investigating a issue with print job services not sending. We will update as we have more information. |
1 year, 2 months ago | ![]() |
Returned to normal activity. We will continue to monitor. |
1 year, 2 months ago | ![]() |
Our new search database we installed a few days ago is not performing well, we are mitigating. |
1 year, 2 months ago | ![]() |
Returned to normal activity |
1 year, 2 months ago | ![]() |
We're seeing intermittent issues with pages timing out. The team is investigating. |
1 year, 2 months ago | ![]() |
resolved |
1 year, 2 months ago | ![]() |
We are aware of an SSL issue causing some PDFs to not load, we are working on it and it should be fixed in a few minutes. |
1 year, 3 months ago | ![]() |
One of our 8 web servers was serving error pages for a couple minutes causing a high error rate, things seem to be back to normal |
1 year, 4 months ago | ![]() |
Back to normal. |
1 year, 4 months ago | ![]() |
We're seeing a delay in search indexing at the moment and the development team is investigating. |
1 year, 6 months ago | ![]() |
One webserver in our fleet crashed causing a small percentage of total traffic to serve error pages for a few minutes while we found the issue and put up replacement webservers. All clear now. |
1 year, 6 months ago | ![]() |
We're currently seeing an issue and the team is investigating. |
1 year, 7 months ago | ![]() |
Engineers have deployed the fix and the service is stable, including Tickets and Invoices. |
1 year, 7 months ago | ![]() |
Engineers are aware of the issue and are working on applying the fix now. |
1 year, 7 months ago | ![]() |
Due to the HTTP 500 errors, a subset of users might experience issues creating Tickets and Invoices. |
1 year, 7 months ago | ![]() |
We are currently experiencing HTTP 500's on our end due to some notifications issues. |
1 year, 10 months ago | ![]() |
All clear! |
1 year, 10 months ago | ![]() |
Part of our hosting infrastructure is experiencing a platform issue, we are working to replace that service with a backup now |
1 year, 10 months ago | ![]() |
We're currently experiencing intermittent slowness connecting with an external server. |
2 years, 1 month ago | ![]() |
Note about what happened - we were seeing high error rates but mostly the app was working. The only symptom was high CPU load on all app servers, it seems to be a miss-behaving DoS prevention tool, restarting the app servers brought performance back to normal. We will be patching that tool today. |
2 years, 1 month ago | ![]() |
All clear now! |
2 years, 1 month ago | ![]() |
We're investigating issues with pages not loading. |
2 years, 1 month ago | ![]() |
All clear!! |
2 years, 1 month ago | ![]() |
We’re investigating issues with print jobs and emails. |
2 years, 2 months ago | ![]() |
All clear |
2 years, 2 months ago | ![]() |
Expect delays on print jobs and such for the next few minutes. |
2 years, 2 months ago | ![]() |
Search performance returned to normal. |
2 years, 2 months ago | ![]() |
We are re-enabling search to see how it performs, expect another update in 20 minutes |
2 years, 2 months ago | ![]() |
We're investigating issues with our search provider |
2 years, 4 months ago | ![]() |
Maintenance was a few minutes slower than anticipated but everything is back to normal |
2 years, 4 months ago | ![]() |
We will be performing scheduled maintenance in a few minutes, we expect the app to be unavailable for about 15 minutes. We'll post here every 10 minutes if we have to go over that. |
2 years, 5 months ago | ![]() |
A stuck database transaction at the same time we were trying to modify the database caused all access to be delayed longer than anticipated, causing a brief outage. |
2 years, 5 months ago | ![]() |
A subset of users may have issues accessing the RepairShopr website. We'll be back up soon. |
2 years, 5 months ago | ![]() |
We determined a stuck database transaction prevented some invoices from being updated. This has been resolved |
2 years, 5 months ago | ![]() |
We're seeing issues and the team is investigating. |
2 years, 6 months ago | ![]() |
We're back online and the development team is closely monitoring performance. |
2 years, 6 months ago | ![]() |
We're seeing issues and the development team is investigating. |
2 years, 7 months ago | ![]() |
We're seeing a return to normal performance across our upstream service providers and RepairShopr appears to be running as normal again as well. We'll continue to monitor for any issues. This affected many major sites across the internet and others are also seeing a return to expected performance (http://bit.ly/1Pgvfim). |
2 years, 7 months ago | ![]() |
We're continuing to see the effects of the Amazon outage affecting many major sites across the web (http://bit.ly/1Pgvfim). We're receiving reports that services are coming back online, but page loads remain extremely slow on a number of requests. |
2 years, 7 months ago | ![]() |
There are multiple reports across the internet of an Amazon S3 issue which we're seeing affect page load times in RepairShopr. We're monitoring the issue for resolution and will post updates here. |
2 years, 8 months ago | ![]() |
System performance returned to normal. |
2 years, 8 months ago | ![]() |
We had an issue with our search database, and it’s disabled for now while we troubleshoot |
2 years, 8 months ago | ![]() |
We're seeing an issue with server response times and are looking into the issue. |
2 years, 10 months ago | ![]() |
We saw a spike in errors for 30 seconds or so while a process was restarting. System has returned to normal performance. |
2 years, 10 months ago | ![]() |
We're seeing an uptime issue and researching the cause. We expect to be back up shortly. |
2 years, 10 months ago | ![]() |
System performance has returned to normal. |
2 years, 11 months ago | ![]() |
We are seeing a higher than normal error rate due to server performance, we are investigating |
2 years, 11 months ago | ![]() |
We have implemented a few performance improvements that we think will drive to the root of the intermittent issues, we will continue to monitor. |
2 years, 11 months ago | ![]() |
We are seeing intermittent DB issues and should have it resolved shortly |
3 years ago | ![]() |
We were seeing errors coming from our connection to our database, we adjusted a setting and restarted a few services and the app seems to be running well again. We'll continue to investigate. |
3 years ago | ![]() |
We are currently down and investigating |
3 years ago | ![]() |
Maintenance is complete and everything checks out! Let us know if you see anything that seems out of the ordinary. |
3 years ago | ![]() |
One of the processes is going a little slower than it did in our tests, it's 9:34PM now and we anticipate we'll be online before 9:45PM. |
3 years ago | ![]() |
Looks like our time estimate was really close, a few more steps to run and we'll be back online. Hopefully just about 10 more minutes. |
3 years ago | ![]() |
We are performing a major upgrade to our software and will require some downtime. We anticipate this maintenance taking about 30 minutes. We will provide updates here if that changes. |
3 years, 1 month ago | ![]() |
Page load times have returned to normal. |
3 years, 1 month ago | ![]() |
We're seeing longer than normal response times from some pages and are currently investigating the issue. |
3 years, 1 month ago | ![]() |
System performance has returned to normal. |
3 years, 1 month ago | ![]() |
We're seeing higher than normal error rates in the app and are researching the issue. |
3 years, 2 months ago | ![]() |
One of our network infrastructure providers experienced an issue. We will investigate with them on how to prevent this in the future. |
3 years, 2 months ago | ![]() |
We are currently experiencing an issue. We are investigating and will keep you updated. |
3 years, 2 months ago | ![]() |
We just had an issue with our rate limiting system, we are investigating - it seems to have been resolved already. |
3 years, 2 months ago | ![]() |
The issue is resolved and we will continue to monitor |
3 years, 2 months ago | ![]() |
Error levels have returned to normal. We are continuing to monitor. |
3 years, 2 months ago | ![]() |
A service we rely on for caching is down. We will post an update soon. |
3 years, 2 months ago | ![]() |
The issue has been resolved |
3 years, 2 months ago | ![]() |
We're seeing a high rate of errors and investigating |
3 years, 3 months ago | ![]() |
We resolved the Payment issue |
3 years, 3 months ago | ![]() |
We know about an issue with Payments and a patch should be live in the next 10 minutes |
3 years, 4 months ago | ![]() |
A request spike caused server instability. We rebooted the server immediately and resolved the issue. System performance has returned to normal. |
3 years, 4 months ago | ![]() |
We are currently seeing abnormal errors and are working on it. We will keep you updated. |
3 years, 5 months ago | ![]() |
We resolved the issue and are seeing error rates back to normal. The site is stable again. |
3 years, 5 months ago | ![]() |
We identified the cause as a miss-behaving application server and are expecting it to be replaced in the next couple minutes. More updates to come. |
3 years, 5 months ago | ![]() |
We are currently seeing abnormal errors and are working on it. We will keep you updated. |
3 years, 5 months ago | ![]() |
We rolled back the change and will attempt it again after hours. |
3 years, 5 months ago | ![]() |
We are having a problem with a new configuration and it's causing a high error rate, we are working on a fix now. |
3 years, 5 months ago | ![]() |
Restarting of the app servers had made it stable again, we'll continue to monitor. |
3 years, 5 months ago | ![]() |
We are seeing some of our app servers reporting a high level of errors, we are investigating. |
3 years, 6 months ago | ![]() |
Error rates have returned to normal. |
3 years, 6 months ago | ![]() |
We're seeing occasional 500 errors on some pages. We're looking into the issue and with post an update. |
3 years, 6 months ago | ![]() |
PDF generation and printing were delayed for about 20 minutes. Everything should be caught up and running smoothly again. |
3 years, 6 months ago | ![]() |
We're seeing an issue with printing and with some PDF generation that we're looking into and will provide an update shortly. |
3 years, 6 months ago | ![]() |
This is a confirmed issue on Authorize.net's end. You can find updates here: https://twitter.com/AuthorizeNet. |
3 years, 6 months ago | ![]() |
We're having reports of issues with Authorize.net and are looking into the issue. |
3 years, 6 months ago | ![]() |
Error rates have returned to normal and the issue appears to be resolved |
3 years, 6 months ago | ![]() |
We're seeing an elevated number of errors and are restarting a couple servers. You may see some failed page loads which should resolve if you refresh. We expect to have this resolved shortly. |
3 years, 6 months ago | ![]() |
We had an 8 minute downtime a couple hours ago, and our host has not been specific yet about what happened. Here is a link to their status site, and we will update this in a few hours if there is more information. https://status.heroku.com/ |
3 years, 6 months ago | ![]() |
The payments issue should now be resolved. |
3 years, 6 months ago | ![]() |
We're seeing an issue with payments processing. We should have a fix live in 10 minutes. |
3 years, 7 months ago | ![]() |
Error rates have returned to normal and the issue appears to be resolved |
3 years, 7 months ago | ![]() |
Our hosting platform is experiencing performance issues, we are looking into it. |
3 years, 7 months ago | ![]() |
The elevated errors we were seeing previously have returned to normal |
3 years, 7 months ago | ![]() |
We've seen some elevated error rates in the app and are looking into the issue |
3 years, 7 months ago | ![]() |
Increasing the servers did clear out the backlog, we are seeing error rates back to normal. (near 0) |
3 years, 7 months ago | ![]() |
We had a denial of service of sorts that clogged our load balancer a bit, and we are seeing some small percentage of traffic still getting timeout errors. We are increasing the number of application servers and think this will be cleared up shortly. |
3 years, 7 months ago | ![]() |
The database upgrade is complete and we are back online. |
3 years, 7 months ago | ![]() |
The process is going as planned, we'll provide an update in 15 minutes |
3 years, 7 months ago | ![]() |
We are taking the site down for approximately 30 minutes for planned maintenance. |
3 years, 8 months ago | ![]() |
After adding some new servers to the cluster and restarting the affected ones, we have seen the error rate go down and it appears all traffic is processing normally again. We will be keeping an eye on it, and following up with our platform provider. |
3 years, 8 months ago | ![]() |
We are still investigating, it seems 95% of the traffic is being processed properly, although some portion of the platform is currently stuck rebooting. We will have another update from our platform provider shortly. |
3 years, 8 months ago | ![]() |
We are seeing a high error rate and are investigating now, we'll update here every few minutes as we learn more. |
3 years, 9 months ago | ![]() |
We are just starting this new status dashboard, we are looking in to a way to backfill historical outages - at this time, the data will be accurate going forward. |
Status Legend
-
Scheduled maintenance
-
The service is currently down
-
The service is up
-
The service is experiencing intermittent problems
-
Delay in processing