Downtime last night
Last night around 9PM PST, Some of the Lighthouse mongrels started acting funny and returning errors. By the time I had logged in to take a look, Ezra at Engineyard had already restarted the mongrels. However, the app didn’t come right back up properly. After going through some log files, I found the cause: a log file had grown to enormous size and filled the drive up. After clearing the log and restarting, everything came right back up. Total downtime was about 15 minutes.
I immediately filed a ticket to EY to get some extra logs from some new plugins that were installed, and to get a hard drive size increase on our slices. Sorry for the troubles lately. We’re working on some stuff behind the scenes to be able to keep better tabs on how the application is performing.
Update We actually had some more downtime yesterday (Feb 5th) relating to the same plugin (different problem though). The offending plugin has been updated.
Sorry, comments are closed for this article.
Active Reload is a small company founded by Rick Olson and Justin Palmer. Rick is a member of the Ruby on Rails core team, while Justin is a member of the Prototype Javascript Library core team. Together we build rocking software with a focus on awesomeness.
-
Beautifully simple issue tracking. With a beautiful Mac-like interface, email and subversion integration, Lighthouse is the perfect project management tool. We took the suck out of issue tracking.
-
Web-based Subversion Interface Manage Subversion permissions and browse through your changesets and files with ease. Warehouse takes the complicated and mundane tasks of Subversion and makes it easy.
-
We've changed the way you manage your blog. With a beautiful interface, multi-site support, and infinitely extendible through plugins, Mephisto is a fresh take on what a blogging application should be.
-
May29permalink
Read your tweets for messages prefixed with @foo and post to that user. http://pastie.caboo.se/65927
# Read tyler-durden's tweets for any messages starting with '@project-mayhem' # # $ ruby twitter_bot.rb tyler-durden project-mayhem my-password # # Stores the last modified and highest id in a yaml file in the pwd by the anme of 'twitter_bot.cache' # Should block against dupes and be easy against twitter's servers -
May17permalink
Displaying nice relative dates
<%= relative_time(Time.now) %> # today <%= relative_time(1.day.ago) %> # yesterday <%= relative_time(1.day.from_now) %> # tomorrow <%= relative_time_span([Time.now, 5.days.from_now]) %> # May 17th - 22nd -
May16permalink
Rick: no more lightbox? Justin: no, I banished it because it was useless the way it was being used. Rick: ha, it was cool man Justin: yeah, Zune music sharing is cool too. Rick: touche -
May10
Scrobbler: Last.fm for Ruby —Ruby Library for accessing the AudioScrobbler web service.
permalink
Copyright © 2007 Active Reload, LLC. | Powered by Mephisto
Hosting and Servers provided by our good friends over at Engine Yard
Discussion
Logs filling up the HDs – sounds familiar. Has happened to me several times at EY, including just the night before last. Seems like a weak point.
You solve this problem? What plugins your update if it not secret.
I can’t say which plugin right now. It’s been brought up with the author and fixed however. I’ll be able to say more later though.
Ben: sure, just make sure EY sets up chronolog for you. I just asked them to set it up for log/*.log. I have the bj and viking plugins creating their own logs now too.