ResqueFailure server middleware

[fbjork]

Added an optional ResqueFailure server middleware that enqueues jobs in the failure queue that can be retried in the Resque dashboard.

[fbjork]

When I try to register the middleware in my Rails app like this:

Sidekiq::Processor.middleware.register do
  use Middleware::Server::ResqueFailure, Sidekiq::Client.redis
end

I get the following error:

/Users/fredrik/Projects/test_app/config/initializers/sidekiq.rb:3:in `<top (required)>': uninitialized constant Sidekiq::Processor (NameError)
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/engine.rb:556:in `block (2 levels) in <class:Engine>'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/engine.rb:555:in `each'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/engine.rb:555:in `block in <class:Engine>'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/initializable.rb:30:in `instance_exec'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/initializable.rb:30:in `run'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/initializable.rb:55:in `block in run_initializers'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/initializable.rb:54:in `each'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/initializable.rb:54:in `run_initializers'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/application.rb:96:in `initialize!'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/railtie/configurable.rb:30:in `method_missing'
    from /Users/fredrik/Projects/Banjo/banjo-api/config/environment.rb:5:in `<top (required)>'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/application.rb:83:in `require_environment!'
    from /Users/fredrik/.rvm/gems/ruby-1.9.3-p0/gems/railties-3.1.3/lib/rails/commands.rb:39:in `<top (required)>'
    from script/rails:11:in `require'
    from script/rails:11:in `<main>'

[mperham]

You can only register it when running in the server, not in the client but there's no clean to do this at the moment. You can just add this hack for now:

Sidekiq::Processor.middleware.register do
  use Middleware::Server::ResqueFailure, Sidekiq::Client.redis
end if defined?(Sidekiq::Processor)

[mperham]

And I don't want features to have Resque in their name. Call it FailureRetry or something like that but it's ok to make it compatible with Resque.

[ryanlecompte]

How are you initializing the middleware here? In a separate initializer? It looks like you're doing it in a place where the sidekiq gem hasn't been required yet by rails.

[mperham]

And on the server-side, you use Sidekiq::Manager.redis, not the Client.redis. Confusing, I know. Needs to be refactored.

[fbjork]

How about naming it FailureJobs instead?

[fbjork]

I updated my initializer to this:

Sidekiq::Client.redis = Sidekiq::RedisConnection.create(namespace: 'resque')

Sidekiq::Processor.middleware.register do
  use Sidekiq::Middleware::Server::FailureJobs, Sidekiq::Client.redis
end if defined?(Sidekiq::Processor)

When I queue up a job that fails I now get this error:

E, [2012-02-16T15:44:51.491070 #38700] ERROR -- : Sidekiq::Processor crashed!
NoMethodError: undefined method `del' for nil:NilClass
/Users/fredrik/.rvm/gems/ruby-1.9.3-p125/gems/activesupport-3.1.3/lib/active_support/whiny_nil.rb:48:in `method_missing'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/server/unique_jobs.rb:12:in `ensure in call'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/server/unique_jobs.rb:12:in `call'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/server/airbrake.rb:9:in `call'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/chain.rb:76:in `call'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/middleware/chain.rb:76:in `invoke'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/processor.rb:38:in `block in process'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/processor.rb:67:in `stats'
/Users/fredrik/Projects/Personal/sidekiq/lib/sidekiq/processor.rb:37:in `process'
/Users/fredrik/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
/Users/fredrik/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/Users/fredrik/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'

[mperham]

Use Sidekiq::Manager.redis, not Sidekiq::Client.redis, within the register block.

[fbjork]

Changing to Sidekiq::Manager.redis still gives me the same error, additionally the failed jobs don't get added to the failed queue anymore.

[jcoene]

Hi there, thanks for your work on this. I think I've tracked this down, though I'm not sure how @mperham wants to resolve it given that it involves how we use any non-default middleware.

Your initializer code is calling Sidekiq::Processor.middleware.register, which is setting up the default chain (since it doesn't exist). The problem is that Sidekiq::Manager.redis is not established, yet is being referenced when it adds the default UniqueJobs middleware. That class is getting nil as its redis instance variable and throwing the errors we see.

As a quick and dirty fix, I'm explicitly setting the redis value on both client and manager in my initializer, which has resolved the error for me. Here's my initializer:

Sidekiq::Client.redis = Sidekiq::RedisConnection.create(:namespace => "resque")
Sidekiq::Manager.redis = Sidekiq::RedisConnection.create(:namespace => "resque") if defined?(Sidekiq::Manager)

Sidekiq::Processor.middleware.register do
  use Sidekiq::Middleware::Server::FailureJobs, Sidekiq::Manager.redis
end if defined?(Sidekiq::Processor)

Any thoughts on what it would take to improve this behavior? At the moment it seems like using any non-default middleware would require this workaround.

[mperham]

If you can clean up this pull request so it applies cleanly, I'll merge it in.

[fbjork]

Sure thing, I'll make it work with the new middleware API shortly.