Application Server

Our main application that you probably want to be up and running all the time

Time Status Message
1 week, 2 days ago Up All clear, PDFs are loading normally now.
1 week, 2 days ago Warning We're seeing issues with PDF generation and the team is investigating.
1 week, 4 days ago Up The upgrade is complete and everything is starting back up!
1 week, 4 days ago Maintenance We are down for scheduled maintenance. We are upgrading the primary DB server to a machine with 500GB of RAM. #bigserver #nomoreoutages
1 week, 5 days ago Up Performance has returned to normal. The development team will continue to monitor, but everything appears fine now.
1 week, 5 days ago Warning We're seeing issues adding ticket comments and other performance issues. The team is investigating.
3 weeks, 1 day ago Up System should be back to normal now.
3 weeks, 1 day ago Warning We're seeing intermittent issues with page loads and the development team is working on it.
3 weeks, 5 days ago Up We are observing normal traffic.
3 weeks, 6 days ago Warning We are starting to see page loads go back to normal, but are still closely watching traffic.
3 weeks, 6 days ago Maintenance The team is continuing to work on the load times.
3 weeks, 6 days ago Up Traffic is returning to normal and the team is carefully monitoring traffic on the site.
3 weeks, 6 days ago Down We experienced a brief outage caused by high latency. The team is bringing services back to normal.
3 weeks, 6 days ago Warning We are experiencing high latency on the app. The team is actively working on bringing performance back in line. Will update as we have more information.
1 month ago Up Latency has returned to normal. The devs are continuing to monitor traffic closely and further refine search performance.
1 month, 1 week ago Warning Ticket Comment and invoice line item searching are causing search performance degradation. We have temporarily disabled search for those and will bring them back online as soon as possible (you can still add line items to invoices normally).
1 month, 1 week ago Warning We are seeing some high latency causing slow performance on the site. Some search query performance is effected. The devs are working on getting this resolved as fast as possible.
1 month, 1 week ago Up Performance has returned to normal.
1 month, 1 week ago Warning We're seeing slow page loads and the development team is investigating.
1 month, 2 weeks ago Up It appears page load times are back to normal but we will have a team monitoring closely.
1 month, 2 weeks ago Down We had a production database issue that caused everything to timeout and show the Oops error page. Developers are working on it now and it should be running but more slowly while it recovers.
1 month, 2 weeks ago Warning We're seeing slow page loads and the development team is investigating.
1 month, 2 weeks ago Up We are not down but getting lots of reports that we are down. This is probably due to some broader 'internet weather' type issues and we'll keep an eye on it. Try other internet connections if you can, like tethering to a cell phone/etc
1 month, 3 weeks ago Up All clear, back to normal.
1 month, 3 weeks ago Down We're having issues but the development team is working on it.
1 month, 3 weeks ago Up Performance is much closer to normal now. The site should be totally usable now.
1 month, 3 weeks ago Warning We're seeing performance issues and the development team is investigating.
2 months, 2 weeks ago Information We are proceeding with replacing the search cluster and the underlying tech. Please report any search issues you find so we can make sure we are tracking any open issues thoroughly.
2 months, 2 weeks ago Up The search database is functioning in a limited state and we are planning to swap out the underlying infrastructure as soon as possible (probably tomorrow). This is urgent and we will do everything we can to do this with zero downtime.
2 months, 2 weeks ago Delayed We are seeing better stability currently. In case you missed our earlier message: "We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster.
2 months, 2 weeks ago Warning New Update for everyone. We are experiencing another search index issue with The Search All The Things top field. Search will work in a reduced state during this time. Work continues to mitigate this and we apologize for the inconvenience. More updates to come as we have more information.
2 months, 2 weeks ago Delayed We are now putting half our resources in setting up a whole new search database with a new host while half of the team tries to continue monitoring and working on the old search cluster and keep it running as best we can. We are attempting to just disable search for a few minutes while the server is overloaded and then quickly switch it back on when it stabilizes. We hope to have this stabilized later today - possibly could go another day if we do have to move hosts for the entire search cluster.
2 months, 2 weeks ago Warning We are experiencing another search index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com.
2 months, 2 weeks ago Warning We're seeing issues with the search again and the development team is working on it.
2 months, 2 weeks ago Up We have re-enabled the main search system so everything should be back to normal. We will continue to monitor. We can't rule out new issues coming up tomorrow during peak load times based on what we saw so we're cautiously calling this incident closed and we'll be working on more safeguards into tomorrow. Expect a full writeup of the incident in the coming days as we take time to post-mortem.
2 months, 2 weeks ago Warning Page performance is returning to normal. The Search remains working but with degraded performance. You can search for most things in their respective tabs until we get Search back up. We apologize and recognize the inconvenience this is causing.
2 months, 2 weeks ago Warning We are seeing some slower load times and are actively working on it. We will update as we have more information.
2 months, 2 weeks ago Warning We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue.
2 months, 2 weeks ago Maintenance We are deploying a fix for the Search All the Things issue and hope to have it live shortly. We are continuing to monitor the issue.
2 months, 2 weeks ago Warning We are experiencing a Search Index issue with The Search All The Things top field. The team is working to get this solved ASAP. Search will work in a reduced state during this time. If something is not returning a result please report this to help@repairshopr.com.
2 months, 2 weeks ago Up We are seeing site performance return to normal. We will continue to monitor everything as the day continues.
2 months, 2 weeks ago Warning We are working on performance issues related to the search system and will update shortly.
3 months, 2 weeks ago Up It might be too soon to call it, but it appears everything is back up and processing again. We will keep an eye on it.
3 months, 2 weeks ago Warning Our hosting provider (Heroku) is having issues that are impacting our availability. We are working with them to mitigate and restore access as fast as possible. You can follow this status page for information or their status page (https://status.heroku.com/) More updates to come.
3 months, 2 weeks ago Down We are experiencing an Up Stream issue and are actively investigating.
6 months ago Up All services appear to be functioning like normal again. We will continue monitoring the situation.
6 months ago Warning It appears that an Amazon AWS global issue is impacting our connectivity to services that our software depends on, affecting performance. Our team is monitoring the situation closely.
6 months ago Warning We're seeing general performance issues now. The development team is investigating.
6 months ago Warning Our search database is showing errors and we are temporarily disabling it. The development team is investigating.
8 months, 1 week ago Up Everything is back to normal now!
8 months, 1 week ago Warning We're seeing issues with Calendar entries loading and the development team is investigating. You should still be able to see a list of your appointments at /appointments.
9 months, 1 week ago Up The issue has been resolved.
9 months, 1 week ago Down We are currently experiencing issues and the development team is investigating.
10 months, 2 weeks ago Up We have isolated the issue and pushed a fix, and Ticket Charges are now working properly.
10 months, 2 weeks ago Warning We've received reports of Ticket Charges not working for some users and we are currently investigating the cause.
12 months ago Up We’ve reverted an update to the site and standard functionality on the invoices page has returned.
12 months ago Warning We are seeing a high error rate and are investigating, we'll provide updates here every few minutes.
1 year ago Up Maintenance has completed and the site is back online. Thanks!
1 year ago Maintenance We are down for maintenance - we anticipate the downtime being less than 30 minutes. We will update every 15 minutes here.
1 year, 1 month ago Up We have brought the search database back up and are continuing to watch for any long load times or search issues.
1 year, 1 month ago Warning We had to bring down our search data base temporarily after observing longer than normal load times. We will continue to update as we have more info.
1 year, 1 month ago Up Our search database is going to remain disabled for a few more minutes but the site has recovered and is usable.
1 year, 1 month ago Warning Our new search database is not behaving well, we are temporarily disabling it. Engineers are working on it.
1 year, 1 month ago Up We are going to mark the application server as up and continue adding notes to the background processor here on the status site.
1 year, 1 month ago Information We have a mitigation idea we are testing out that might have background processes working shortly. S3 is still very broken.
1 year, 1 month ago Maintenance The search database is going to be outdated until the background processes start up again, except for Tickets - which should be updating in the search db properly.
1 year, 1 month ago Warning Although the application servers are working, due to an Amazon S3 outage our background processes are blocked. Our platform provider isn't allowing us to push updates while S3 is broken.
1 year, 1 month ago Up We have returned to normal activity and will continue to monitor.
1 year, 1 month ago Maintenance We’re investigating issues with making invoices
1 year, 1 month ago Up We have returned to normal activity.
1 year, 1 month ago Warning We are investigating a issue with print job services not sending. We will update as we have more information.
1 year, 2 months ago Up Returned to normal activity. We will continue to monitor.
1 year, 2 months ago Warning Our new search database we installed a few days ago is not performing well, we are mitigating.
1 year, 2 months ago Up Returned to normal activity
1 year, 2 months ago Warning We're seeing intermittent issues with pages timing out. The team is investigating.
1 year, 2 months ago Up resolved
1 year, 2 months ago Delayed We are aware of an SSL issue causing some PDFs to not load, we are working on it and it should be fixed in a few minutes.
1 year, 3 months ago Warning One of our 8 web servers was serving error pages for a couple minutes causing a high error rate, things seem to be back to normal
1 year, 4 months ago Up Back to normal.
1 year, 4 months ago Delayed We're seeing a delay in search indexing at the moment and the development team is investigating.
1 year, 6 months ago Up One webserver in our fleet crashed causing a small percentage of total traffic to serve error pages for a few minutes while we found the issue and put up replacement webservers. All clear now.
1 year, 6 months ago Warning We're currently seeing an issue and the team is investigating.
1 year, 7 months ago Up Engineers have deployed the fix and the service is stable, including Tickets and Invoices.
1 year, 7 months ago Down Engineers are aware of the issue and are working on applying the fix now.
1 year, 7 months ago Down Due to the HTTP 500 errors, a subset of users might experience issues creating Tickets and Invoices.
1 year, 7 months ago Warning We are currently experiencing HTTP 500's on our end due to some notifications issues.
1 year, 10 months ago Up All clear!
1 year, 10 months ago Warning Part of our hosting infrastructure is experiencing a platform issue, we are working to replace that service with a backup now
1 year, 10 months ago Warning We're currently experiencing intermittent slowness connecting with an external server.
2 years, 1 month ago Information Note about what happened - we were seeing high error rates but mostly the app was working. The only symptom was high CPU load on all app servers, it seems to be a miss-behaving DoS prevention tool, restarting the app servers brought performance back to normal. We will be patching that tool today.
2 years, 1 month ago Up All clear now!
2 years, 1 month ago Warning We're investigating issues with pages not loading.
2 years, 1 month ago Up All clear!!
2 years, 1 month ago Warning We’re investigating issues with print jobs and emails.
2 years, 2 months ago Up All clear
2 years, 2 months ago Warning Expect delays on print jobs and such for the next few minutes.
2 years, 2 months ago Up Search performance returned to normal.
2 years, 2 months ago Warning We are re-enabling search to see how it performs, expect another update in 20 minutes
2 years, 2 months ago Warning We're investigating issues with our search provider
2 years, 4 months ago Up Maintenance was a few minutes slower than anticipated but everything is back to normal
2 years, 4 months ago Maintenance We will be performing scheduled maintenance in a few minutes, we expect the app to be unavailable for about 15 minutes. We'll post here every 10 minutes if we have to go over that.
2 years, 5 months ago Up A stuck database transaction at the same time we were trying to modify the database caused all access to be delayed longer than anticipated, causing a brief outage.
2 years, 5 months ago Warning A subset of users may have issues accessing the RepairShopr website. We'll be back up soon.
2 years, 5 months ago Up We determined a stuck database transaction prevented some invoices from being updated. This has been resolved
2 years, 5 months ago Warning We're seeing issues and the team is investigating.
2 years, 6 months ago Up We're back online and the development team is closely monitoring performance.
2 years, 6 months ago Down We're seeing issues and the development team is investigating.
2 years, 7 months ago Up We're seeing a return to normal performance across our upstream service providers and RepairShopr appears to be running as normal again as well. We'll continue to monitor for any issues. This affected many major sites across the internet and others are also seeing a return to expected performance (http://bit.ly/1Pgvfim).
2 years, 7 months ago Warning We're continuing to see the effects of the Amazon outage affecting many major sites across the web (http://bit.ly/1Pgvfim). We're receiving reports that services are coming back online, but page loads remain extremely slow on a number of requests.
2 years, 7 months ago Warning There are multiple reports across the internet of an Amazon S3 issue which we're seeing affect page load times in RepairShopr. We're monitoring the issue for resolution and will post updates here.
2 years, 8 months ago Up System performance returned to normal.
2 years, 8 months ago Warning We had an issue with our search database, and it’s disabled for now while we troubleshoot
2 years, 8 months ago Down We're seeing an issue with server response times and are looking into the issue.
2 years, 10 months ago Up We saw a spike in errors for 30 seconds or so while a process was restarting. System has returned to normal performance.
2 years, 10 months ago Down We're seeing an uptime issue and researching the cause. We expect to be back up shortly.
2 years, 10 months ago Up System performance has returned to normal.
2 years, 11 months ago Warning We are seeing a higher than normal error rate due to server performance, we are investigating
2 years, 11 months ago Up We have implemented a few performance improvements that we think will drive to the root of the intermittent issues, we will continue to monitor.
2 years, 11 months ago Warning We are seeing intermittent DB issues and should have it resolved shortly
3 years ago Up We were seeing errors coming from our connection to our database, we adjusted a setting and restarted a few services and the app seems to be running well again. We'll continue to investigate.
3 years ago Down We are currently down and investigating
3 years ago Up Maintenance is complete and everything checks out! Let us know if you see anything that seems out of the ordinary.
3 years ago Information One of the processes is going a little slower than it did in our tests, it's 9:34PM now and we anticipate we'll be online before 9:45PM.
3 years ago Information Looks like our time estimate was really close, a few more steps to run and we'll be back online. Hopefully just about 10 more minutes.
3 years ago Maintenance We are performing a major upgrade to our software and will require some downtime. We anticipate this maintenance taking about 30 minutes. We will provide updates here if that changes.
3 years, 1 month ago Up Page load times have returned to normal.
3 years, 1 month ago Warning We're seeing longer than normal response times from some pages and are currently investigating the issue.
3 years, 1 month ago Up System performance has returned to normal.
3 years, 1 month ago Warning We're seeing higher than normal error rates in the app and are researching the issue.
3 years, 2 months ago Up One of our network infrastructure providers experienced an issue. We will investigate with them on how to prevent this in the future.
3 years, 2 months ago Down We are currently experiencing an issue. We are investigating and will keep you updated.
3 years, 2 months ago Information We just had an issue with our rate limiting system, we are investigating - it seems to have been resolved already.
3 years, 2 months ago Up The issue is resolved and we will continue to monitor
3 years, 2 months ago Warning Error levels have returned to normal. We are continuing to monitor.
3 years, 2 months ago Down A service we rely on for caching is down. We will post an update soon.
3 years, 2 months ago Up The issue has been resolved
3 years, 2 months ago Down We're seeing a high rate of errors and investigating
3 years, 3 months ago Up We resolved the Payment issue
3 years, 3 months ago Warning We know about an issue with Payments and a patch should be live in the next 10 minutes
3 years, 4 months ago Up A request spike caused server instability. We rebooted the server immediately and resolved the issue. System performance has returned to normal.
3 years, 4 months ago Warning We are currently seeing abnormal errors and are working on it. We will keep you updated.
3 years, 5 months ago Up We resolved the issue and are seeing error rates back to normal. The site is stable again.
3 years, 5 months ago Warning We identified the cause as a miss-behaving application server and are expecting it to be replaced in the next couple minutes. More updates to come.
3 years, 5 months ago Down We are currently seeing abnormal errors and are working on it. We will keep you updated.
3 years, 5 months ago Up We rolled back the change and will attempt it again after hours.
3 years, 5 months ago Down We are having a problem with a new configuration and it's causing a high error rate, we are working on a fix now.
3 years, 5 months ago Up Restarting of the app servers had made it stable again, we'll continue to monitor.
3 years, 5 months ago Delayed We are seeing some of our app servers reporting a high level of errors, we are investigating.
3 years, 6 months ago Up Error rates have returned to normal.
3 years, 6 months ago Warning We're seeing occasional 500 errors on some pages. We're looking into the issue and with post an update.
3 years, 6 months ago Up PDF generation and printing were delayed for about 20 minutes. Everything should be caught up and running smoothly again.
3 years, 6 months ago Delayed We're seeing an issue with printing and with some PDF generation that we're looking into and will provide an update shortly.
3 years, 6 months ago Up This is a confirmed issue on Authorize.net's end. You can find updates here: https://twitter.com/AuthorizeNet.
3 years, 6 months ago Warning We're having reports of issues with Authorize.net and are looking into the issue.
3 years, 6 months ago Up Error rates have returned to normal and the issue appears to be resolved
3 years, 6 months ago Warning We're seeing an elevated number of errors and are restarting a couple servers. You may see some failed page loads which should resolve if you refresh. We expect to have this resolved shortly.
3 years, 6 months ago Warning We had an 8 minute downtime a couple hours ago, and our host has not been specific yet about what happened. Here is a link to their status site, and we will update this in a few hours if there is more information. https://status.heroku.com/
3 years, 6 months ago Up The payments issue should now be resolved.
3 years, 6 months ago Warning We're seeing an issue with payments processing. We should have a fix live in 10 minutes.
3 years, 7 months ago Up Error rates have returned to normal and the issue appears to be resolved
3 years, 7 months ago Warning Our hosting platform is experiencing performance issues, we are looking into it.
3 years, 7 months ago Up The elevated errors we were seeing previously have returned to normal
3 years, 7 months ago Warning We've seen some elevated error rates in the app and are looking into the issue
3 years, 7 months ago Up Increasing the servers did clear out the backlog, we are seeing error rates back to normal. (near 0)
3 years, 7 months ago Warning We had a denial of service of sorts that clogged our load balancer a bit, and we are seeing some small percentage of traffic still getting timeout errors. We are increasing the number of application servers and think this will be cleared up shortly.
3 years, 7 months ago Up The database upgrade is complete and we are back online.
3 years, 7 months ago Information The process is going as planned, we'll provide an update in 15 minutes
3 years, 7 months ago Maintenance We are taking the site down for approximately 30 minutes for planned maintenance.
3 years, 8 months ago Up After adding some new servers to the cluster and restarting the affected ones, we have seen the error rate go down and it appears all traffic is processing normally again. We will be keeping an eye on it, and following up with our platform provider.
3 years, 8 months ago Information We are still investigating, it seems 95% of the traffic is being processed properly, although some portion of the platform is currently stuck rebooting. We will have another update from our platform provider shortly.
3 years, 8 months ago Down We are seeing a high error rate and are investigating now, we'll update here every few minutes as we learn more.
3 years, 9 months ago Information We are just starting this new status dashboard, we are looking in to a way to backfill historical outages - at this time, the data will be accurate going forward.

Status Legend

  • Maintenance

    Scheduled maintenance

  • Down

    The service is currently down

  • Up

    The service is up

  • Warning

    The service is experiencing intermittent problems

  • Delayed

    Delay in processing