RepairShopr Status

Application Server

Our main application that you probably want to be up and running all the time

Time	Status	Message
2 months, 1 week ago		All clear, PDFs are loading normally now.
2 months, 1 week ago		We're seeing issues with PDF generation and the team is investigating.
2 months, 1 week ago		The upgrade is complete and everything is starting back up!
2 months, 1 week ago		We are down for scheduled maintenance. We are upgrading the primary DB server to a machine with 500GB of RAM. #bigserver #nomoreoutages
2 months, 2 weeks ago		Performance has returned to normal. The development team will continue to monitor, but everything appears fine now.
2 months, 2 weeks ago		We're seeing issues adding ticket comments and other performance issues. The team is investigating.
2 months, 3 weeks ago		System should be back to normal now.
2 months, 3 weeks ago		We're seeing intermittent issues with page loads and the development team is working on it.
2 months, 4 weeks ago		We are observing normal traffic.
2 months, 4 weeks ago		We are starting to see page loads go back to normal, but are still closely watching traffic.
2 months, 4 weeks ago		The team is continuing to work on the load times.
2 months, 4 weeks ago		Traffic is returning to normal and the team is carefully monitoring traffic on the site.
2 months, 4 weeks ago		We experienced a brief outage caused by high latency. The team is bringing services back to normal.
2 months, 4 weeks ago		We are experiencing high latency on the app. The team is actively working on bringing performance back in line. Will update as we have more information.
3 months, 1 week ago		Latency has returned to normal. The devs are continuing to monitor traffic closely and further refine search performance.
3 months, 1 week ago		Ticket Comment and invoice line item searching are causing search performance degradation. We have temporarily disabled search for those and will bring them back online as soon as possible (you can still add line items to invoices normally).
3 months, 1 week ago		We are seeing some high latency causing slow performance on the site. Some search query performance is effected. The devs are working on getting this resolved as fast as possible.
3 months, 1 week ago		Performance has returned to normal.
3 months, 1 week ago		We're seeing slow page loads and the development team is investigating.
3 months, 2 weeks ago		It appears page load times are back to normal but we will have a team monitoring closely.
3 months, 2 weeks ago		We had a production database issue that caused everything to timeout and show the Oops error page. Developers are working on it now and it should be running but more slowly while it recovers.
3 months, 2 weeks ago		We're seeing slow page loads and the development team is investigating.
3 months, 2 weeks ago		We are not down but getting lots of reports that we are down. This is probably due to some broader 'internet weather' type issues and we'll keep an eye on it. Try other internet connections if you can, like tethering to a cell phone/etc
3 months, 3 weeks ago		All clear, back to normal.
3 months, 3 weeks ago		We're having issues but the development team is working on it.
3 months, 4 weeks ago		Performance is much closer to normal now. The site should be totally usable now.
3 months, 4 weeks ago		We're seeing performance issues and the development team is investigating.
4 months, 2 weeks ago		We are proceeding with replacing the search cluster and the underlying tech. Please report any search issues you find so we can make sure we are tracking any open issues thoroughly.
4 months, 2 weeks ago		The search database is functioning in a limited state and we are planning to swap out the underlying infrastructure as soon as possible (probably tomorrow). This is urgent and we will do everything we can to do this with zero downtime.
4 months, 2 weeks ago		We are seeing better stability currently. In case you missed our earlier message: "We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster.
4 months, 2 weeks ago		New Update for everyone. We are experiencing another search index issue with The Search All The Things top field. Search will work in a reduced state during this time. Work continues to mitigate this and we apologize for the inconvenience. More updates to come as we have more information.
4 months, 2 weeks ago		We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster.
4 months, 2 weeks ago		We are experiencing another search index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com.
4 months, 2 weeks ago		We're seeing issues with the search again and the development team is working on it.
4 months, 2 weeks ago		We have re-enabled the main search system so everything should be back to normal. We will continue to monitor. We can't rule out new issues coming up tomorrow during peak load times based on what we saw so we're cautiously calling this incident closed and we'll be working on more safeguards into tomorrow. Expect a full writeup of the incident in the coming days as we take time to post-mortem.
4 months, 2 weeks ago		Page performance is returning to normal. The Search remains working but with degraded performance. You can search for most things in their respective tabs until we get Search back up. We apologize and recognize the inconvenience this is causing.
4 months, 2 weeks ago		We are seeing some slower load times and are actively working on it. We will update as we have more information.
4 months, 2 weeks ago		We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue.
4 months, 2 weeks ago		We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue.
4 months, 2 weeks ago		We are experiencing a Search Index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com.
4 months, 2 weeks ago		We are seeing site performance return to normal. We will continue to monitor everything as the day continues.
4 months, 2 weeks ago		We are working on performance issues related to the search system and will update shortly.
5 months, 2 weeks ago		It might be too soon to call it, but it appears everything is back up and processing again. We will keep an eye on it.
5 months, 2 weeks ago		Our hosting provider (Heroku) is having issues that are impacting our availability. We are working with them to mitigate and restore access as fast as possible. You can follow this status page for information or their status page (https://status.heroku.com/) More updates to come.
5 months, 2 weeks ago		We are experiencing an Up Stream issue and are actively investigating.
8 months ago		All services appear to be functioning like normal again. We will continue monitoring the situation.
8 months ago		It appears that an Amazon AWS global issue is impacting our connectivity to services that our software depends on, affecting performance. Our team is monitoring the situation closely.
8 months ago		We're seeing general performance issues now. The development team is investigating.
8 months ago		Our search database is showing errors and we are temporarily disabling it. The development team is investigating.
10 months, 2 weeks ago		Everything is back to normal now!
10 months, 2 weeks ago		We're seeing issues with Calendar entries loading and the development team is investigating. You should still be able to see a list of your appointments at /appointments.
11 months, 1 week ago		The issue has been resolved.
11 months, 1 week ago		We are currently experiencing issues and the development team is investigating.
1 year ago		We have isolated the issue and pushed a fix, and Ticket Charges are now working properly.
1 year ago		We've received reports of Ticket Charges not working for some users and we are currently investigating the cause.
1 year, 1 month ago		We’ve reverted an update to the site and standard functionality on the invoices page has returned.
1 year, 1 month ago		We are seeing a high error rate and are investigating, we'll provide updates here every few minutes.
1 year, 2 months ago		Maintenance has completed and the site is back online. Thanks!
1 year, 2 months ago		We are down for maintenance - we anticipate the downtime being less than 30 minutes. We will update every 15 minutes here.
1 year, 3 months ago		We have brought the search database back up and are continuing to watch for any long load times or search issues.
1 year, 3 months ago		We had to bring down our search data base temporarily after observing longer than normal load times. We will continue to update as we have more info.
1 year, 3 months ago		Our search database is going to remain disabled for a few more minutes but the site has recovered and is usable.
1 year, 3 months ago		Our new search database is not behaving well, we are temporarily disabling it. Engineers are working on it.
1 year, 3 months ago		We are going to mark the application server as up and continue adding notes to the background processor here on the status site.
1 year, 3 months ago		We have a mitigation idea we are testing out that might have background processes working shortly. S3 is still very broken.
1 year, 3 months ago		The search database is going to be outdated until the background processes start up again, except for Tickets - which should be updating in the search db properly.
1 year, 3 months ago		Although the application servers are working, due to an Amazon S3 outage our background processes are blocked. Our platform provider isn't allowing us to push updates while S3 is broken.
1 year, 3 months ago		We have returned to normal activity and will continue to monitor.
1 year, 3 months ago		We’re investigating issues with making invoices
1 year, 3 months ago		We have returned to normal activity.
1 year, 3 months ago		We are investigating a issue with print job services not sending. We will update as we have more information.
1 year, 4 months ago		Returned to normal activity. We will continue to monitor.
1 year, 4 months ago		Our new search database we installed a few days ago is not performing well, we are mitigating.
1 year, 4 months ago		Returned to normal activity
1 year, 4 months ago		We're seeing intermittent issues with pages timing out. The team is investigating.
1 year, 4 months ago		resolved
1 year, 4 months ago		We are aware of an SSL issue causing some PDFs to not load, we are working on it and it should be fixed in a few minutes.
1 year, 5 months ago		One of our 8 web servers was serving error pages for a couple minutes causing a high error rate, things seem to be back to normal
1 year, 6 months ago		Back to normal.
1 year, 6 months ago		We're seeing a delay in search indexing at the moment and the development team is investigating.
1 year, 8 months ago		One webserver in our fleet crashed causing a small percentage of total traffic to serve error pages for a few minutes while we found the issue and put up replacement webservers. All clear now.
1 year, 8 months ago		We're currently seeing an issue and the team is investigating.
1 year, 9 months ago		Engineers have deployed the fix and the service is stable, including Tickets and Invoices.
1 year, 9 months ago		Engineers are aware of the issue and are working on applying the fix now.
1 year, 9 months ago		Due to the HTTP 500 errors, a subset of users might experience issues creating Tickets and Invoices.
1 year, 9 months ago		We are currently experiencing HTTP 500's on our end due to some notifications issues.
2 years ago		All clear!
2 years ago		Part of our hosting infrastructure is experiencing a platform issue, we are working to replace that service with a backup now
2 years ago		We're currently experiencing intermittent slowness connecting with an external server.
2 years, 3 months ago		Note about what happened - we were seeing high error rates but mostly the app was working. The only symptom was high CPU load on all app servers, it seems to be a miss-behaving DoS prevention tool, restarting the app servers brought performance back to normal. We will be patching that tool today.
2 years, 3 months ago		All clear now!
2 years, 3 months ago		We're investigating issues with pages not loading.
2 years, 4 months ago		All clear!!
2 years, 4 months ago		We’re investigating issues with print jobs and emails.
2 years, 4 months ago		All clear
2 years, 4 months ago		Expect delays on print jobs and such for the next few minutes.
2 years, 4 months ago		Search performance returned to normal.
2 years, 4 months ago		We are re-enabling search to see how it performs, expect another update in 20 minutes
2 years, 4 months ago		We're investigating issues with our search provider
2 years, 6 months ago		Maintenance was a few minutes slower than anticipated but everything is back to normal
2 years, 6 months ago		We will be performing scheduled maintenance in a few minutes, we expect the app to be unavailable for about 15 minutes. We'll post here every 10 minutes if we have to go over that.
2 years, 7 months ago		A stuck database transaction at the same time we were trying to modify the database caused all access to be delayed longer than anticipated, causing a brief outage.
2 years, 7 months ago		A subset of users may have issues accessing the RepairShopr website. We'll be back up soon.
2 years, 7 months ago		We determined a stuck database transaction prevented some invoices from being updated. This has been resolved
2 years, 7 months ago		We're seeing issues and the team is investigating.
2 years, 8 months ago		We're back online and the development team is closely monitoring performance.
2 years, 8 months ago		We're seeing issues and the development team is investigating.
2 years, 9 months ago		We're seeing a return to normal performance across our upstream service providers and RepairShopr appears to be running as normal again as well. We'll continue to monitor for any issues. This affected many major sites across the internet and others are also seeing a return to expected performance (http://bit.ly/1Pgvfim).
2 years, 9 months ago		We're continuing to see the effects of the Amazon outage affecting many major sites across the web (http://bit.ly/1Pgvfim). We're receiving reports that services are coming back online, but page loads remain extremely slow on a number of requests.
2 years, 9 months ago		There are multiple reports across the internet of an Amazon S3 issue which we're seeing affect page load times in RepairShopr. We're monitoring the issue for resolution and will post updates here.
2 years, 10 months ago		System performance returned to normal.
2 years, 10 months ago		We had an issue with our search database, and it’s disabled for now while we troubleshoot
2 years, 10 months ago		We're seeing an issue with server response times and are looking into the issue.
3 years ago		We saw a spike in errors for 30 seconds or so while a process was restarting. System has returned to normal performance.
3 years ago		We're seeing an uptime issue and researching the cause. We expect to be back up shortly.
3 years ago		System performance has returned to normal.
3 years ago		We are seeing a higher than normal error rate due to server performance, we are investigating
3 years ago		We have implemented a few performance improvements that we think will drive to the root of the intermittent issues, we will continue to monitor.
3 years ago		We are seeing intermittent DB issues and should have it resolved shortly
3 years, 2 months ago		We were seeing errors coming from our connection to our database, we adjusted a setting and restarted a few services and the app seems to be running well again. We'll continue to investigate.
3 years, 2 months ago		We are currently down and investigating
3 years, 3 months ago		Maintenance is complete and everything checks out! Let us know if you see anything that seems out of the ordinary.
3 years, 3 months ago		One of the processes is going a little slower than it did in our tests, it's 9:34PM now and we anticipate we'll be online before 9:45PM.
3 years, 3 months ago		Looks like our time estimate was really close, a few more steps to run and we'll be back online. Hopefully just about 10 more minutes.
3 years, 3 months ago		We are performing a major upgrade to our software and will require some downtime. We anticipate this maintenance taking about 30 minutes. We will provide updates here if that changes.
3 years, 3 months ago		Page load times have returned to normal.
3 years, 3 months ago		We're seeing longer than normal response times from some pages and are currently investigating the issue.
3 years, 3 months ago		System performance has returned to normal.
3 years, 3 months ago		We're seeing higher than normal error rates in the app and are researching the issue.
3 years, 4 months ago		One of our network infrastructure providers experienced an issue. We will investigate with them on how to prevent this in the future.
3 years, 4 months ago		We are currently experiencing an issue. We are investigating and will keep you updated.
3 years, 4 months ago		We just had an issue with our rate limiting system, we are investigating - it seems to have been resolved already.
3 years, 4 months ago		The issue is resolved and we will continue to monitor
3 years, 4 months ago		Error levels have returned to normal. We are continuing to monitor.
3 years, 4 months ago		A service we rely on for caching is down. We will post an update soon.
3 years, 5 months ago		The issue has been resolved
3 years, 5 months ago		We're seeing a high rate of errors and investigating
3 years, 5 months ago		We resolved the Payment issue
3 years, 5 months ago		We know about an issue with Payments and a patch should be live in the next 10 minutes
3 years, 7 months ago		A request spike caused server instability. We rebooted the server immediately and resolved the issue. System performance has returned to normal.
3 years, 7 months ago		We are currently seeing abnormal errors and are working on it. We will keep you updated.
3 years, 7 months ago		We resolved the issue and are seeing error rates back to normal. The site is stable again.
3 years, 7 months ago		We identified the cause as a miss-behaving application server and are expecting it to be replaced in the next couple minutes. More updates to come.
3 years, 7 months ago		We are currently seeing abnormal errors and are working on it. We will keep you updated.
3 years, 7 months ago		We rolled back the change and will attempt it again after hours.
3 years, 7 months ago		We are having a problem with a new configuration and it's causing a high error rate, we are working on a fix now.
3 years, 7 months ago		Restarting of the app servers had made it stable again, we'll continue to monitor.
3 years, 7 months ago		We are seeing some of our app servers reporting a high level of errors, we are investigating.
3 years, 8 months ago		Error rates have returned to normal.
3 years, 8 months ago		We're seeing occasional 500 errors on some pages. We're looking into the issue and with post an update.
3 years, 8 months ago		PDF generation and printing were delayed for about 20 minutes. Everything should be caught up and running smoothly again.
3 years, 8 months ago		We're seeing an issue with printing and with some PDF generation that we're looking into and will provide an update shortly.
3 years, 8 months ago		This is a confirmed issue on Authorize.net's end. You can find updates here: https://twitter.com/AuthorizeNet.
3 years, 8 months ago		We're having reports of issues with Authorize.net and are looking into the issue.
3 years, 8 months ago		Error rates have returned to normal and the issue appears to be resolved
3 years, 8 months ago		We're seeing an elevated number of errors and are restarting a couple servers. You may see some failed page loads which should resolve if you refresh. We expect to have this resolved shortly.
3 years, 8 months ago		We had an 8 minute downtime a couple hours ago, and our host has not been specific yet about what happened. Here is a link to their status site, and we will update this in a few hours if there is more information. https://status.heroku.com/
3 years, 9 months ago		The payments issue should now be resolved.
3 years, 9 months ago		We're seeing an issue with payments processing. We should have a fix live in 10 minutes.
3 years, 9 months ago		Error rates have returned to normal and the issue appears to be resolved
3 years, 9 months ago		Our hosting platform is experiencing performance issues, we are looking into it.
3 years, 9 months ago		The elevated errors we were seeing previously have returned to normal
3 years, 9 months ago		We've seen some elevated error rates in the app and are looking into the issue
3 years, 9 months ago		Increasing the servers did clear out the backlog, we are seeing error rates back to normal. (near 0)
3 years, 9 months ago		We had a denial of service of sorts that clogged our load balancer a bit, and we are seeing some small percentage of traffic still getting timeout errors. We are increasing the number of application servers and think this will be cleared up shortly.
3 years, 10 months ago		The database upgrade is complete and we are back online.
3 years, 10 months ago		The process is going as planned, we'll provide an update in 15 minutes
3 years, 10 months ago		We are taking the site down for approximately 30 minutes for planned maintenance.
3 years, 10 months ago		After adding some new servers to the cluster and restarting the affected ones, we have seen the error rate go down and it appears all traffic is processing normally again. We will be keeping an eye on it, and following up with our platform provider.
3 years, 10 months ago		We are still investigating, it seems 95% of the traffic is being processed properly, although some portion of the platform is currently stuck rebooting. We will have another update from our platform provider shortly.
3 years, 10 months ago		We are seeing a high error rate and are investigating now, we'll update here every few minutes as we learn more.
3 years, 12 months ago		We are just starting this new status dashboard, we are looking in to a way to backfill historical outages - at this time, the data will be accurate going forward.

Status Legend

Scheduled maintenance
The service is currently down
The service is up
The service is experiencing intermittent problems
Delay in processing