smallseo.info

sidekiq

Simple, efficient background processing for Ruby Sidekiq simple, efficient message or background job processing for ruby

How can I password-protect my /sidekiq route (i.e. require authentication for the Sidekiq::Web tool)?

I am using sidekiq in my rails application. By Default, Sidekiq can be accessed by anybody by appending "/sidekiq" after the url. I want to password protect / authenticate only the sidekiq part. How can i do that?


Source: (StackOverflow)

How do I reset my sidekiq counters?

In my sidekiq dashboard, I see on the left a box with the counters

Processed 168
Failed 111
Busy 0
Scheduled 0
Retries 0
Enqueued 0

How do I reset them all to 0?


Source: (StackOverflow)

Gracefully shutting down sidekiq processes

Does anyone know how to find sidekiq's pidfile to gracefully shut it down? Running ps ax | grep sidekiq and then running sidekiqctl stop <pid from grep> consistently gives a no such pidfile error? Cntl-C and Cntl-D also seem to have no effect.

Closing the process window and reopening a new window doesn't kill the process as it appears to be running as a daemon.

The only consistent fix I've found is rebooting.


Source: (StackOverflow)

sidekiq to cancel list to scheduled jobs

I have several scheduled jobs running like this:

MyWorker.perform_at(3.hours.from_now, 'mike', 1)

I am wondering, if later, say an hour later, I feel like I want to cancel this job, how would I go about doing that?


Source: (StackOverflow)

How to clear all the jobs from sidekiq?

I am using sidekiq for background tasks in Rails application. Now the numbers of jobs becomes more, so i want to clear all the jobs.I tried the following command in console

 Sidekiq::Queue.new.clear

but it was giving following error.

NameError: uninitialized constant Sidekiq::Queue 

So kindly suggest me a way to clear all the jobs from sidekiq.


Source: (StackOverflow)

Can multiple sidekiq instances process the same queue

I'm not familiar with the internals of Sidekiq and am wondering if it's okay to launch several Sidekiq instances with the same configuration (processing the same queues).

Is it possible that 2 or more Sidekiq instances will process the same message from a queue?

UPDATE:
I need to know if there is a possible conflict, when running Sidekiq on more than 1 machine


Source: (StackOverflow)

Are there console commands to look at whats in the queue and to clear the queue in Sidekiq?

I'm used to using delayed_jobs method of going into the console to see whats in the queue, and the ease of clearing the queue when needed. Are there similar commands in Sidekiq for this? Thanks!


Source: (StackOverflow)

"SSL_read: cert already in hash table" when sending mail asynchronously

I keep getting an OpenSSL::SSL::SSLError with a message of SSL_read: cert already in hash table when sending delayed e-mails out asynchronously with ActionMailer.

We use Sidekiq to send all of our e-mails out asynchronously. When posing this question as an issue in the Sidekiq GitHub repo, I was told that Sidekiq doesn't know anything about or manage the SSL connection.

Our app is hosted on Heroku, which is running OpenSSL 0.9.8k 25 Mar 2009.

We've seen this error several times in other jobs and have found that sometimes the jobs get processed, but sometimes they don't.

Is this an OpenSSL threading problem in which multiple Sidekiq threads are attempting to use the same SSL connection? Is a there a fix to this?

Here's the stacktrace we're getting:

    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `sysread_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/openssl/buffering.rb:175:in `read_nonblock`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:153:in `rbuf_fill`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:932:in `recv_response`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:903:in `block in data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:942:in `critical`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:896:in `data`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `block in send_message`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:852:in `rcptto_list`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:663:in `send_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:113:in `block in deliver!`
    [PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/smtp.rb:521:in `start`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:112:in `deliver!`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:2129:in `do_delivery`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `block in deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:456:in `block in deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `block in instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications/instrumenter.rb:20:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.3/lib/active_support/notifications.rb:159:in `instrument`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/actionmailer-4.0.3/lib/action_mailer/base.rb:454:in `deliver_mail`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `deliver`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/extensions/action_mailer.rb:20:in `perform`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:49:in `block (3 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:122:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/sidekiq.rb:30:in `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:339:in `perform_action_with_newrelic_trace`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.7.1.182/lib/new_relic/agent/instrumentation/sidekiq.rb:21:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-failures-0.3.0/lib/sidekiq/failures/middleware.rb:10:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.7.0/lib/sidekiq-unique-jobs/middleware/server/unique_jobs.rb:15:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/active_record.rb:6:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/logging.rb:22:in `with_context`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/server/logging.rb:7:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:124:in `block in invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `call`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/middleware/chain.rb:127:in `invoke`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:48:in `block (2 levels) in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:105:in `stats`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:47:in `block in process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:86:in `do_defer`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.17.2/lib/sidekiq/processor.rb:37:in `process`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize`
    [PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create

Source: (StackOverflow)

Restarting Sidekiq

What is the correct way to restart sidekiq. It seems to cache my workers' code when I start it, so every time I make a change to my workers I need to restart it. I'm doing this with Ctrl/C, but the process takes a long time to wind down and return me to the prompt.

Is there a way to force a restart with immediate effect?

I'm using the latest version with Sinatra running via POW.


Source: (StackOverflow)

Sidekiq worker is leaking memory


Using the sidekiq gem - I have sidekiq worker that runs a process (git-tf clone of big repository) using IO.popen and tracks the stdout to check the progress of the clone.

When I am running the worker, I see that sidekiq memory is getting larger over the time until I get kernel OOM and the process get killed. the subprocess (java process) is taking only 5% of the total memory.

How I can debug/check the memory leak I have in my code? and does the sidekiq memory is the total of my workers memory with the popen process?
And does anyone have any idea how to fix it?

EDIT
This is the code of my worker - https://gist.github.com/yosy/5227250

EDIT 2
I ran the code without sidekiq, and I have no memory leaks.. this is something strange with sidekiq and big repositories in tfs


Source: (StackOverflow)

Is my understanding of Unicorn, Sidekiq and DB Pool size correct?

I have Unicorn, Sidekiq and Postgres setup.

I am trying to understand the right configuration to set up so that I don't hit the maximum db connection limit. In Opsworks, the m1.small Postgres RDS instance can have a max of 121 connections.

I have a db pool size of 5.

Consider this. Sidekiq and Unicorn are its own process. So the db pool size for each process is 5. Correct me if my understanding here is wrong.

If I have 5 unicorn process', that means 5*5=25 database connections

Now this is the part where I am slightly confused, since Sidekiq is multithreaded. If Sidekiq has a concurrency of 5. and the db pool size is also set to 5. Does that mean 25 potential db connections at a given time too?

This means, for one instance, I could have 50 db connections?


Source: (StackOverflow)

Why does Sidekiq not close old connections?

My Rails 4.1 application uses Sidekiq to run measurements in Celluloid actors. It continously runs out of database connections. It appears that Sidekiq opens 2 connections per job, and old connections never get closed properly.

What the system does

  • Every 15 minutes, I start a MeasurementWorker. By calling MeasurementWorker.perform_async(measurement.id). It does this:
    class MeasurementWorker
      include Sidekiq::Worker
      sidekiq_options retry: false, backtrace: true

      def perform(measurement_id, force = false)
        ActiveRecord::Base.connection_pool.with_connection do
            Measurement.find(measurement_id).run
        end
      end
    end
  • Inside this measurement, when I call .run, it does this:
    # various checks if measurement can be run at all, using AR
    # ...
    begin
      ActiveRecord::Base.connection_pool.with_connection do
      # (I used to have a Timeout.timeout here, but removed it for the
      # sake of simplification)
        @connection = MeasurementConnection.new do |event_info|
          event = Event.new
          event.assign_attributes(event_info)
          event.save
        end
        while @connection.measuring?; end
      end # with_connection
    rescue Exception => e
      # an exception happened, e.g. something during the measurement itself
      # log error (left out here for brevity)
    else
      # all went fine, 
      # save this measurement via AR
    ensure
      # Close and terminate the actor to free up the websocket,
      # if it is still actively measuring something.
      if @connection
        if @connection.alive? and @connection.measuring?
          @connection.close
        end
        while @connection.alive?
          @connection.terminate
          sleep(0.01)
        end
      end
    end
  • The MeasurementConnection is a simple Celluloid actor. There is no AR-related code inside this actor.

Configuration

  • Unicorn: concurrency set to 3
  • Sidekiq 3.3.0: concurrency set to 50, and in the initializer:
    Sidekiq.configure_server do |config|
      if defined?(ActiveRecord::Base)
        config = Rails.application.config.database_configuration[Rails.env]
        config['pool']              = Sidekiq.options[:concurrency] + 2
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 5
        ActiveRecord::Base.establish_connection(config)
      end
    end
  • database.yml: pool set to 60, reaping frequency 5
  • PostgreSQL 9.3: maximum connections are 2000, no other modifications

The problem: too many connections open

When I check SELECT * FROM pg_stat_activity;, I see that there are some old connections open with no Sidekiq workers busy, and some new where they are:

16661 measurement 6354  16384 measurement unicorn worker[0] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50775 2015-02-20 12:52:48.572551+01   2015-02-20 13:05:05.773178+01 2015-02-20 13:05:05.773565+01 f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6406  16384 measurement unicorn worker[2] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50776 2015-02-20 12:53:59.636414+01   2015-02-20 13:04:53.930305+01 2015-02-20 13:04:53.931+01  f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6687  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50801 2015-02-20 13:00:05.14621+01    2015-02-20 13:04:49.558589+01 2015-02-20 13:04:49.558835+01 f idle  COMMIT
16661 measurement 7042  16384 measurement unicorn worker[1] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50997 2015-02-20 13:00:34.874675+01   2015-02-20 13:00:35.376593+01 2015-02-20 13:00:35.376979+01 f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6668  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50781 2015-02-20 13:00:04.883553+01   2015-02-20 13:04:19.108365+01 2015-02-20 13:04:19.108567+01 f idle  COMMIT
16661 measurement 6669  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50782 2015-02-20 13:00:04.908349+01   2015-02-20 13:03:57.683036+01 2015-02-20 13:03:57.683236+01 f idle  COMMIT
16661 measurement 6672  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50786 2015-02-20 13:00:04.962251+01   2015-02-20 13:04:32.395137+01 2015-02-20 13:04:32.395344+01 f idle  COMMIT
16661 measurement 6674  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50788 2015-02-20 13:00:04.98456+01    2015-02-20 13:04:32.396335+01 2015-02-20 13:04:32.39652+01  f idle  COMMIT
16661 measurement 6676  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50790 2015-02-20 13:00:05.006847+01   2015-02-20 13:04:19.059628+01 2015-02-20 13:04:19.059831+01 f idle  COMMIT
16661 measurement 6678  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50792 2015-02-20 13:00:05.029448+01   2015-02-20 13:04:23.730293+01 2015-02-20 13:04:23.730523+01 f idle  COMMIT
16661 measurement 6680  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50794 2015-02-20 13:00:05.051932+01   2015-02-20 13:04:49.557435+01 2015-02-20 13:04:49.557633+01 f idle  COMMIT
16661 measurement 6684  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50798 2015-02-20 13:00:05.124225+01   2015-02-20 13:03:51.693799+01 2015-02-20 13:03:51.694034+01 f idle  COMMIT
16661 measurement 6690  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50804 2015-02-20 13:00:05.168099+01   2015-02-20 13:04:54.849239+01 2015-02-20 13:04:54.849459+01 f idle  COMMIT
16661 measurement 6693  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50807 2015-02-20 13:00:05.189661+01   2015-02-20 13:04:18.688459+01 2015-02-20 13:04:18.688732+01 f idle  COMMIT
16661 measurement 6696  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50810 2015-02-20 13:00:05.210659+01   2015-02-20 13:03:57.68424+01  2015-02-20 13:03:57.684483+01 f idle  COMMIT
16661 measurement 6699  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50813 2015-02-20 13:00:05.231641+01   2015-02-20 13:04:04.962397+01 2015-02-20 13:04:04.96258+01  f idle  COMMIT
16661 measurement 6701  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50815 2015-02-20 13:00:05.252357+01   2015-02-20 13:04:41.685372+01 2015-02-20 13:04:41.685594+01 f idle  COMMIT
16661 measurement 6706  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50820 2015-02-20 13:00:05.273301+01   2015-02-20 13:04:23.733488+01 2015-02-20 13:04:23.733681+01 f idle  COMMIT
16661 measurement 7003  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50967 2015-02-20 13:00:09.004487+01   2015-02-20 13:02:02.036429+01 2015-02-20 13:02:02.036696+01 f idle  COMMIT
16661 measurement 7005  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50969 2015-02-20 13:00:11.118961+01   2015-02-20 13:02:48.341078+01 2015-02-20 13:02:48.341294+01 f idle  COMMIT
16661 measurement 7006  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50970 2015-02-20 13:00:12.245408+01   2015-02-20 13:03:04.300372+01 2015-02-20 13:03:04.300575+01 f idle  COMMIT
16661 measurement 7007  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50971 2015-02-20 13:00:12.648636+01   2015-02-20 13:03:01.855616+01 2015-02-20 13:03:01.85588+01  f idle  COMMIT
16661 measurement 7008  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50972 2015-02-20 13:00:12.956139+01   2015-02-20 13:03:13.840023+01 2015-02-20 13:03:13.840466+01 f idle  COMMIT
16661 measurement 7009  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50973 2015-02-20 13:00:13.02424+01    2015-02-20 13:02:50.115996+01 2015-02-20 13:02:50.116259+01 f idle  COMMIT
16661 measurement 7010  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50974 2015-02-20 13:00:13.0909+01   2015-02-20 13:03:09.968+01  2015-02-20 13:03:09.968284+01 f idle  COMMIT
16661 measurement 7014  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50976 2015-02-20 13:00:14.929822+01   2015-02-20 13:03:20.183195+01 2015-02-20 13:03:20.183467+01 f idle  COMMIT
16661 measurement 7020  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50980 2015-02-20 13:00:22.498892+01   2015-02-20 13:03:29.887257+01 2015-02-20 13:03:29.887599+01 f idle  COMMIT
16661 measurement 7021  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50981 2015-02-20 13:00:22.898087+01   2015-02-20 13:03:39.689939+01 2015-02-20 13:03:39.69798+01  f idle  COMMIT
16661 measurement 7022  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50982 2015-02-20 13:00:23.215846+01   2015-02-20 13:03:03.918339+01 2015-02-20 13:03:03.918613+01 f idle  COMMIT
16661 measurement 7023  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50983 2015-02-20 13:00:23.930861+01   2015-02-20 13:03:51.504525+01 2015-02-20 13:03:51.512786+01 f idle  COMMIT
16661 measurement 7025  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50985 2015-02-20 13:00:24.409999+01   2015-02-20 13:03:16.000375+01 2015-02-20 13:03:16.006178+01 f idle  COMMIT
16661 measurement 7027  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50986 2015-02-20 13:00:25.786321+01   2015-02-20 13:03:22.631091+01 2015-02-20 13:03:22.631353+01 f idle  COMMIT
16661 measurement 7045  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50998 2015-02-20 13:00:48.004036+01   2015-02-20 13:03:39.717124+01 2015-02-20 13:03:39.722956+01 f idle  COMMIT

In total, there are 34 connections, but I only ran 15 measurements.

In sidekiq.log, they all show as done:

2015-02-20T12:00:04.879Z 6235 TID-osgop8md0 MeasurementWorker JID-1cdcd44bf41fefe9ddca21ac INFO: start
2015-02-20T12:00:04.907Z 6235 TID-osgoox8hg MeasurementWorker JID-79f4d1ff6692248682ba93dd INFO: start
2015-02-20T12:00:04.939Z 6235 TID-osgoos38k MeasurementWorker JID-09f95fbccd2438d17916d425 INFO: start
2015-02-20T12:00:04.940Z 6235 TID-osgoorlmo MeasurementWorker JID-be1a57871f26146e9884107e INFO: start
2015-02-20T12:00:04.959Z 6235 TID-osgoow16k MeasurementWorker JID-e296efec897c23629b96e99f INFO: start
2015-02-20T12:00:04.968Z 6235 TID-osgoouytg MeasurementWorker JID-c6a57700872b7fe427e33664 INFO: start
2015-02-20T12:00:04.984Z 6235 TID-osgooz63k MeasurementWorker JID-f7448eaffe109882130497ca INFO: start
2015-02-20T12:00:04.998Z 6235 TID-osgoozvzs MeasurementWorker JID-c55c04f3424268fba50ec048 INFO: start
2015-02-20T12:00:05.014Z 6235 TID-osgooyr6c MeasurementWorker JID-01bd303e953fd2998fe3f8d1 INFO: start
2015-02-20T12:00:05.030Z 6235 TID-osgon0ums MeasurementWorker JID-6949c5c81b4c254046f0c585 INFO: start
2015-02-20T12:00:05.042Z 6235 TID-osgomw0g8 MeasurementWorker JID-cc03b717f81dd6fb0f58a946 INFO: start
2015-02-20T12:00:05.111Z 6235 TID-osgomlno8 MeasurementWorker JID-10eebcd76113f3565d8265ca INFO: start
2015-02-20T12:00:05.113Z 6235 TID-osgomjzzw MeasurementWorker JID-b0536d9a029faed0ba8eb5d3 INFO: start
2015-02-20T12:00:05.114Z 6235 TID-osgomhyms MeasurementWorker JID-1f6624314afd8e0ae611599f INFO: start
2015-02-20T12:00:05.115Z 6235 TID-osgolgzoc MeasurementWorker JID-24e87960d7e7fbd871037dd3 INFO: start
2015-02-20T12:02:02.333Z 6235 TID-osgomw0g8 MeasurementWorker JID-cc03b717f81dd6fb0f58a946 INFO: done: 117.291 sec
2015-02-20T12:02:48.983Z 6235 TID-osgoox8hg MeasurementWorker JID-79f4d1ff6692248682ba93dd INFO: done: 164.077 sec
2015-02-20T12:02:50.688Z 6235 TID-osgomlno8 MeasurementWorker JID-10eebcd76113f3565d8265ca INFO: done: 165.577 sec
2015-02-20T12:03:02.429Z 6235 TID-osgolgzoc MeasurementWorker JID-24e87960d7e7fbd871037dd3 INFO: done: 177.314 sec
2015-02-20T12:03:04.581Z 6235 TID-osgoorlmo MeasurementWorker JID-be1a57871f26146e9884107e INFO: done: 179.641 sec
2015-02-20T12:03:09.453Z 6235 TID-osgop8md0 MeasurementWorker JID-1cdcd44bf41fefe9ddca21ac INFO: done: 184.573 sec
2015-02-20T12:03:10.362Z 6235 TID-osgoouytg MeasurementWorker JID-c6a57700872b7fe427e33664 INFO: done: 185.394 sec
2015-02-20T12:03:14.232Z 6235 TID-osgomjzzw MeasurementWorker JID-b0536d9a029faed0ba8eb5d3 INFO: done: 189.118 sec
2015-02-20T12:03:16.347Z 6235 TID-osgoos38k MeasurementWorker JID-09f95fbccd2438d17916d425 INFO: done: 191.408 sec
2015-02-20T12:03:20.398Z 6235 TID-osgoow16k MeasurementWorker JID-e296efec897c23629b96e99f INFO: done: 195.439 sec
2015-02-20T12:03:22.947Z 6235 TID-osgomhyms MeasurementWorker JID-1f6624314afd8e0ae611599f INFO: done: 197.833 sec
2015-02-20T12:03:30.212Z 6235 TID-osgooz63k MeasurementWorker JID-f7448eaffe109882130497ca INFO: done: 205.228 sec
2015-02-20T12:03:39.931Z 6235 TID-osgooyr6c MeasurementWorker JID-01bd303e953fd2998fe3f8d1 INFO: done: 214.918 sec
2015-02-20T12:03:39.936Z 6235 TID-osgon0ums MeasurementWorker JID-6949c5c81b4c254046f0c585 INFO: done: 214.906 sec
2015-02-20T12:03:51.694Z 6235 TID-osgoozvzs MeasurementWorker JID-c55c04f3424268fba50ec048 INFO: done: 226.696 sec

So, for every measurement, Sidekiq seems to open 2 connections, but it never closes them. What should I do?


Source: (StackOverflow)

Run Sidekiq as daemon on Ubuntu

How can I run sidekiq as daemon on Ubuntu?

If I run bundle exec sidekiq -D I get invalid option: -D, is there any way to run it without some other controller, like god, upstart...?


Source: (StackOverflow)

How to start sidekiq in RubyMine?

i'm using RubyMine 6.3.

I tried to configure sidekiq in RubyMine but i can't start it.

How to configure and start sidekiq in RubyMine?


Source: (StackOverflow)

Configuring Unicorn & Sidekiq correctly on Heroku

I was getting ActiveRecord::StatementInvalid (PG::Error: SSL error: decryption failed or bad record mac errors so I followed this guide about deploying Unicorn to Heroku and it seems to have fixed it. However under caveats it shows how to configure Resque for such a setup - would I have to do something similar with Sidekiq?

Sample code from Heroku:

before_fork do |server, worker|

  ...

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|

  ...

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

This is what I currently have set up:

config/unicorn.rb

worker_processes 2
timeout 30
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
end  

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end
  defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end

config/initializers/sidekiq.rb

require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

Sidekiq.configure_server do |config|
  config.redis = { :size => 6 }
end

Procfile

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -c 4

Source: (StackOverflow)