Simple, efficient background processing for Ruby Sidekiq simple, efficient message or background job processing for ruby

Restarting Sidekiq

What is the correct way to restart sidekiq. It seems to cache my workers' code when I start it, so every time I make a change to my workers I need to restart it. I'm doing this with Ctrl/C, but the process takes a long time to wind down and return me to the prompt.

Is there a way to force a restart with immediate effect?

I'm using the latest version with Sinatra running via POW.

Source: (StackOverflow)

"SSL_read: cert already in hash table" when sending mail asynchronously

I keep getting an OpenSSL::SSL::SSLError with a message of SSL_read: cert already in hash table when sending delayed e-mails out asynchronously with ActionMailer.

We use Sidekiq to send all of our e-mails out asynchronously. When posing this question as an issue in the Sidekiq GitHub repo, I was told that Sidekiq doesn't know anything about or manage the SSL connection.

Our app is hosted on Heroku, which is running OpenSSL 0.9.8k 25 Mar 2009.

We've seen this error several times in other jobs and have found that sometimes the jobs get processed, but sometimes they don't.

Is this an OpenSSL threading problem in which multiple Sidekiq threads are attempting to use the same SSL connection? Is a there a fix to this?

Here's the stacktrace we're getting:

    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `sysread_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `read_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:153:in `rbuf_fill`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:932:in `recv_response`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:903:in `block in data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:942:in `critical`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:896:in `data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `block in send_message`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:852:in `rcptto_list`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `send_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:113:in `block in deliver!`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:521:in `start`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:112:in `deliver!`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:2129:in `do_delivery`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `block in deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:456:in `block in deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `block in instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications/instrumenter.rb:20:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:454:in `deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/extensions/action_mailer.rb:20:in `perform`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:49:in `block (3 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm- `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm- `perform_action_with_newrelic_trace`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm- `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-failures-0.3.0/lib/sidekiq/failures/middleware.rb:10:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.7.0/lib/sidekiq-unique-jobs/middleware/server/unique_jobs.rb:15:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/active_record.rb:6:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/logging.rb:22:in `with_context`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:7:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:48:in `block (2 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:105:in `stats`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:47:in `block in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:86:in `do_defer`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:37:in `process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create

Source: (StackOverflow)

Are there console commands to look at whats in the queue and to clear the queue in Sidekiq?

I'm used to using delayed_jobs method of going into the console to see whats in the queue, and the ease of clearing the queue when needed. Are there similar commands in Sidekiq for this? Thanks!

Source: (StackOverflow)

Can multiple sidekiq instances process the same queue

I'm not familiar with the internals of Sidekiq and am wondering if it's okay to launch several Sidekiq instances with the same configuration (processing the same queues).

Is it possible that 2 or more Sidekiq instances will process the same message from a queue?

I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine

Source: (StackOverflow)

How to clear all the jobs from sidekiq?

I am using sidekiq for background tasks in Rails application. Now the numbers of jobs becomes more, so i want to clear all the jobs.I tried the following command in console

but it was giving following error.

NameError: uninitialized constant Sidekiq::Queue 

So kindly suggest me a way to clear all the jobs from sidekiq.

Source: (StackOverflow)

How can I password-protect my /sidekiq route (i.e. require authentication for the Sidekiq::Web tool)?

I am using sidekiq in my rails application. By Default, Sidekiq can be accessed by anybody by appending "/sidekiq" after the url. I want to password protect / authenticate only the sidekiq part. How can i do that?

Source: (StackOverflow)

How do I reset my sidekiq counters?

In my sidekiq dashboard, I see on the left a box with the counters

Processed 168
Failed 111
Busy 0
Scheduled 0
Retries 0
Enqueued 0

How do I reset them all to 0?

Source: (StackOverflow)

Gracefully shutting down sidekiq processes

Does anyone know how to find sidekiq's pidfile to gracefully shut it down? Running ps ax | grep sidekiq and then running sidekiqctl stop <pid from grep> consistently gives a no such pidfile error? Cntl-C and Cntl-D also seem to have no effect.

Closing the process window and reopening a new window doesn't kill the process as it appears to be running as a daemon.

The only consistent fix I've found is rebooting.

Source: (StackOverflow)

sidekiq to cancel list to scheduled jobs

I have several scheduled jobs running like this:

MyWorker.perform_at(3.hours.from_now, 'mike', 1)

I am wondering, if later, say an hour later, I feel like I want to cancel this job, how would I go about doing that?

Source: (StackOverflow)

Sidekiq not deallocating memory after workers have finished

I have about six Sidekiq worker which perform JSON crawling. Dependent on the endpoint's dataset size they finish between 1min and 4h. Especially, watching the long one, which takes 4h, I see a very slight increase of memory over time.

It's not a problem, until I want to schedule the same worker jobs again. The memory is not deallocated and stacks up, until I run into the Linux OOM Killer which gets rid of my Sidekiq process.

Memory leak? I watched the number of different objects in ObjectSpace:

ObjectSpace.each_object.inject( { |count, o| count[o.class] += 1 }

There is not really an increase there, the set of hashes, arrays, etc. stays the same, short increases are swept away by the Garbage Collector and gc.stat[:count] tells me, that the Garbage Collector is working, too.

Even after the worker finishes, e.g. I get the [Done] logged and no workers are busy any more, the memory is not deallocated. What are the reasons for that? Can I do something against this? Write a finalizer?

The only current solution: Restart the Sidekiq process.

I am on Ruby 2.0.0 and use Ruby MRI.

For the JSON parsing I use Yajl, thus a C binding. I need it because it seems the only fast JSON parser that properly implements streamed reading and writing.

Source: (StackOverflow)

Starting sidekiq on Heroku

I'm having a problem getting sidekiq up and running on my Heroku deployed Rails app. I have my app working fine in development (and on Heroku without sidekiq).

I created a Procfile with:

worker: bundle exec sidekiq

If I run heroku ps, the only process I see is web.1. Should I see one for sidekiq?

I do get an error Redis::CannotConnectError (Error connecting to Redis on localhost:6379) in my heroku logs.

UPDATE: Found I probably needed heroku addons:add redistogo. Still not working. I feel I'm missing some basic configuration.

Is there something I need to do to get Redis up and running for my Heroku app?

I've been using Redis/sidekiq for about a day... so this is new to me.



Source: (StackOverflow)

Sidekiq not processing queue

What possible reasons can Sidekiq prevent from processing jobs in the queue? The queue is full. The log file sidekiq.log indicates no activity at all. Thus the queue is full but the log is empty, and Sidekiq does not seem to process items. There seem to no worker processing jobs. Restarting Redis or flush it with FLUSHALL or FLUSHDB as no effect. Sidekiq has been started with

bundle exec sidekiq -L log/sidekiq.log

and produces the following log file:

2013-05-30..Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-30..Running in ruby 1.9.3p374 (2013-01-15 revision 38858) [i686-linux]
2013-05-30..See LICENSE and the LGPL-3.0 for licensing details.
2013-05-30..Starting processing, hit Ctrl-C to stop

How can you find out what went wrong? Are there any hidden log files?

Source: (StackOverflow)

Best way to monitor for completion of a Sidekiq job?

I'm using a Sidekiq worker to complete some requests to Facebook after a user signs-in for the first time. Typically the task takes around 20 seconds or so.

I'd like to load some information on to the page using an ajax request as soon as the sync is completed, but am unsure as to the best way to check for the job completion with Javascript.

One possibility would be to configure the Sidekiq worker to set a cookie after the rest of the jobs are done. Then I can use a setTimeout function to keep checking for the cookie before calling the load function. But I'm unsure whether this is the best way to be doing this. Could I be using Redis instead?

Source: (StackOverflow)

Why is "❨╯°□°❩╯︵┻━┻" with such an encoding used for a method name?

I came across following method in sidekiq gem. Its just invoked from test_sidekiq.rb.

def self.❨╯°□°❩╯︵┻━┻
  puts "Calm down, bro"

This is the only link I was able to find on SO.

  1. Google can't understand ❨╯°□°❩╯︵┻━┻. Why doesn't Ruby complain about this encoding?
  2. What is the purpose of this method (not much looking at its body)?
  3. Why did author @mike-perham use this name? Just for fun, or testing some boundaries?

Source: (StackOverflow)

Sidekiq jobs stuck in enqueue

Sidekiq has been working in development mode just perfectly. Now that I am trying to use it in production, all the jobs are just sitting in enqueue and aren't ever being run. Could anyone point me in the right direction as to how to solve this issue?

Source: (StackOverflow)