Gamma Ray Bursts
11 June 2005
Once in a while the Digital Routes server displays odd spikes in activity. When this happens the load average soars beyond 180, then drops back to 0.2. This is the signature of something blowing up, hitting a resource limit, and being terminated by the OS. The problem was that I had no clue what was at fault. By the time I could get 'top' running the show was already over and everything was returning to normal.
It struck me that what I was experiencing was similar to the Gamma Ray Bursts[?] which astronomers were so perplexed about. About once a day astronomers will notice a gigantic explosion in the sky which emits an unbelievable amount of energy. By the time they swing their big telescopes over to the offending area, the event is gone. For forty years they couldn't even figure out if these events were originating within our solar system or from distant galaxies. Extremely frustrating.
So I did what the astronomers did[?]. I built a rapid autonomous system that starts a pre-scripted sequence of observations the moment it detects an event. In my case, the culprit turned out to be an astonishingly buggy Python script. With the benefit of hindsight I should have suspected it from the beginning; I've been patching and repatching that script for years, trying to convince myself that it was salvageable. Time for a rewrite.
Unrelated quote from a week ago: