I came home from a week long company hackathon to see my newrelic error reporting going crazy …

sphinx error driving new relic CRAZY

My free bugsnag account had already maxed out of the 2000 error allocation I had for the entire month (I’ll usually get 100 in a month … maybe).

I immediately knew from the exception in new relic that something was wrong with Sphinx, and I figured it was pretty bad because on each page load of news pages, I hit sphinx to give me a list of related news stories, which meant that the crawlers that hit my site every second of the day could be generating loads of exceptions.

However, when I went to pull up one of my news pages … it loaded just fine …

hmmm.

I hit the search page, and got a 500 error. Weird as that may be, I’d encountered that before. I immediately logged into my vps and ran

`rake ts:restart`

at the console and was ready to drop my mic and moonwalk back to my sofa to catch up on “The Americans”.

Then I looked at my new relic account and noticed that the same error was still coming in 10 minutes after it should have been fixed.

hmmmmmmmmmmmm.

I tried hitting the urls specified in the errors and got the 500 error. I realized that some urls were exhibiting the problem while others weren’t.

I’d never seen that before :\

My google-fu quickly turned up this non-upvoted gem that helped me fix the issue. Basically you just have to rotate your sphinx index because that error means parts of it have gone “stale”.

the command I used to accomplish this was

`/usr/bin/indexer –rotate xx_core –config /path/to/your/sphinx/production.sphinx.conf`

your indexer command might be located somewhere different though; to locate mine I just used the `whereis indexer` command

You’ll want to change xx_core to match the name of your index (which you can find by looking in your sphinx.conf file)

Hope this helps you out!