smallseo.info

sidekiq

Simple, efficient background processing for Ruby Sidekiq simple, efficient message or background job processing for ruby

Run Sidekiq as daemon on Ubuntu

How can I run sidekiq as daemon on Ubuntu?

If I run bundle exec sidekiq -D I get invalid option: -D, is there any way to run it without some other controller, like god, upstart...?


Source: (StackOverflow)

Why does Sidekiq not close old connections?

My Rails 4.1 application uses Sidekiq to run measurements in Celluloid actors. It continously runs out of database connections. It appears that Sidekiq opens 2 connections per job, and old connections never get closed properly.

What the system does

  • Every 15 minutes, I start a MeasurementWorker. By calling MeasurementWorker.perform_async(measurement.id). It does this:
    class MeasurementWorker
      include Sidekiq::Worker
      sidekiq_options retry: false, backtrace: true

      def perform(measurement_id, force = false)
        ActiveRecord::Base.connection_pool.with_connection do
            Measurement.find(measurement_id).run
        end
      end
    end
  • Inside this measurement, when I call .run, it does this:
    # various checks if measurement can be run at all, using AR
    # ...
    begin
      ActiveRecord::Base.connection_pool.with_connection do
      # (I used to have a Timeout.timeout here, but removed it for the
      # sake of simplification)
        @connection = MeasurementConnection.new do |event_info|
          event = Event.new
          event.assign_attributes(event_info)
          event.save
        end
        while @connection.measuring?; end
      end # with_connection
    rescue Exception => e
      # an exception happened, e.g. something during the measurement itself
      # log error (left out here for brevity)
    else
      # all went fine, 
      # save this measurement via AR
    ensure
      # Close and terminate the actor to free up the websocket,
      # if it is still actively measuring something.
      if @connection
        if @connection.alive? and @connection.measuring?
          @connection.close
        end
        while @connection.alive?
          @connection.terminate
          sleep(0.01)
        end
      end
    end
  • The MeasurementConnection is a simple Celluloid actor. There is no AR-related code inside this actor.

Configuration

  • Unicorn: concurrency set to 3
  • Sidekiq 3.3.0: concurrency set to 50, and in the initializer:
    Sidekiq.configure_server do |config|
      if defined?(ActiveRecord::Base)
        config = Rails.application.config.database_configuration[Rails.env]
        config['pool']              = Sidekiq.options[:concurrency] + 2
        config['reaping_frequency'] = ENV['DB_REAP_FREQ'] || 5
        ActiveRecord::Base.establish_connection(config)
      end
    end
  • database.yml: pool set to 60, reaping frequency 5
  • PostgreSQL 9.3: maximum connections are 2000, no other modifications

The problem: too many connections open

When I check SELECT * FROM pg_stat_activity;, I see that there are some old connections open with no Sidekiq workers busy, and some new where they are:

16661 measurement 6354  16384 measurement unicorn worker[0] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50775 2015-02-20 12:52:48.572551+01   2015-02-20 13:05:05.773178+01 2015-02-20 13:05:05.773565+01 f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6406  16384 measurement unicorn worker[2] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50776 2015-02-20 12:53:59.636414+01   2015-02-20 13:04:53.930305+01 2015-02-20 13:04:53.931+01  f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6687  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50801 2015-02-20 13:00:05.14621+01    2015-02-20 13:04:49.558589+01 2015-02-20 13:04:49.558835+01 f idle  COMMIT
16661 measurement 7042  16384 measurement unicorn worker[1] -c /home/web...E production -D -l0.0.0.0:8080 127.0.0.1   50997 2015-02-20 13:00:34.874675+01   2015-02-20 13:00:35.376593+01 2015-02-20 13:00:35.376979+01 f idle  SELECT COUNT(*) FROM "measurements"
16661 measurement 6668  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50781 2015-02-20 13:00:04.883553+01   2015-02-20 13:04:19.108365+01 2015-02-20 13:04:19.108567+01 f idle  COMMIT
16661 measurement 6669  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50782 2015-02-20 13:00:04.908349+01   2015-02-20 13:03:57.683036+01 2015-02-20 13:03:57.683236+01 f idle  COMMIT
16661 measurement 6672  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50786 2015-02-20 13:00:04.962251+01   2015-02-20 13:04:32.395137+01 2015-02-20 13:04:32.395344+01 f idle  COMMIT
16661 measurement 6674  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50788 2015-02-20 13:00:04.98456+01    2015-02-20 13:04:32.396335+01 2015-02-20 13:04:32.39652+01  f idle  COMMIT
16661 measurement 6676  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50790 2015-02-20 13:00:05.006847+01   2015-02-20 13:04:19.059628+01 2015-02-20 13:04:19.059831+01 f idle  COMMIT
16661 measurement 6678  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50792 2015-02-20 13:00:05.029448+01   2015-02-20 13:04:23.730293+01 2015-02-20 13:04:23.730523+01 f idle  COMMIT
16661 measurement 6680  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50794 2015-02-20 13:00:05.051932+01   2015-02-20 13:04:49.557435+01 2015-02-20 13:04:49.557633+01 f idle  COMMIT
16661 measurement 6684  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50798 2015-02-20 13:00:05.124225+01   2015-02-20 13:03:51.693799+01 2015-02-20 13:03:51.694034+01 f idle  COMMIT
16661 measurement 6690  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50804 2015-02-20 13:00:05.168099+01   2015-02-20 13:04:54.849239+01 2015-02-20 13:04:54.849459+01 f idle  COMMIT
16661 measurement 6693  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50807 2015-02-20 13:00:05.189661+01   2015-02-20 13:04:18.688459+01 2015-02-20 13:04:18.688732+01 f idle  COMMIT
16661 measurement 6696  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50810 2015-02-20 13:00:05.210659+01   2015-02-20 13:03:57.68424+01  2015-02-20 13:03:57.684483+01 f idle  COMMIT
16661 measurement 6699  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50813 2015-02-20 13:00:05.231641+01   2015-02-20 13:04:04.962397+01 2015-02-20 13:04:04.96258+01  f idle  COMMIT
16661 measurement 6701  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50815 2015-02-20 13:00:05.252357+01   2015-02-20 13:04:41.685372+01 2015-02-20 13:04:41.685594+01 f idle  COMMIT
16661 measurement 6706  16384 measurement sidekiq 3.3.0 measurement [0 of 50 busy]  127.0.0.1   50820 2015-02-20 13:00:05.273301+01   2015-02-20 13:04:23.733488+01 2015-02-20 13:04:23.733681+01 f idle  COMMIT
16661 measurement 7003  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50967 2015-02-20 13:00:09.004487+01   2015-02-20 13:02:02.036429+01 2015-02-20 13:02:02.036696+01 f idle  COMMIT
16661 measurement 7005  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50969 2015-02-20 13:00:11.118961+01   2015-02-20 13:02:48.341078+01 2015-02-20 13:02:48.341294+01 f idle  COMMIT
16661 measurement 7006  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50970 2015-02-20 13:00:12.245408+01   2015-02-20 13:03:04.300372+01 2015-02-20 13:03:04.300575+01 f idle  COMMIT
16661 measurement 7007  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50971 2015-02-20 13:00:12.648636+01   2015-02-20 13:03:01.855616+01 2015-02-20 13:03:01.85588+01  f idle  COMMIT
16661 measurement 7008  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50972 2015-02-20 13:00:12.956139+01   2015-02-20 13:03:13.840023+01 2015-02-20 13:03:13.840466+01 f idle  COMMIT
16661 measurement 7009  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50973 2015-02-20 13:00:13.02424+01    2015-02-20 13:02:50.115996+01 2015-02-20 13:02:50.116259+01 f idle  COMMIT
16661 measurement 7010  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50974 2015-02-20 13:00:13.0909+01   2015-02-20 13:03:09.968+01  2015-02-20 13:03:09.968284+01 f idle  COMMIT
16661 measurement 7014  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50976 2015-02-20 13:00:14.929822+01   2015-02-20 13:03:20.183195+01 2015-02-20 13:03:20.183467+01 f idle  COMMIT
16661 measurement 7020  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50980 2015-02-20 13:00:22.498892+01   2015-02-20 13:03:29.887257+01 2015-02-20 13:03:29.887599+01 f idle  COMMIT
16661 measurement 7021  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50981 2015-02-20 13:00:22.898087+01   2015-02-20 13:03:39.689939+01 2015-02-20 13:03:39.69798+01  f idle  COMMIT
16661 measurement 7022  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50982 2015-02-20 13:00:23.215846+01   2015-02-20 13:03:03.918339+01 2015-02-20 13:03:03.918613+01 f idle  COMMIT
16661 measurement 7023  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50983 2015-02-20 13:00:23.930861+01   2015-02-20 13:03:51.504525+01 2015-02-20 13:03:51.512786+01 f idle  COMMIT
16661 measurement 7025  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50985 2015-02-20 13:00:24.409999+01   2015-02-20 13:03:16.000375+01 2015-02-20 13:03:16.006178+01 f idle  COMMIT
16661 measurement 7027  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50986 2015-02-20 13:00:25.786321+01   2015-02-20 13:03:22.631091+01 2015-02-20 13:03:22.631353+01 f idle  COMMIT
16661 measurement 7045  16384 measurement sidekiq 3.3.0 measurement [15 of 50 busy] 127.0.0.1   50998 2015-02-20 13:00:48.004036+01   2015-02-20 13:03:39.717124+01 2015-02-20 13:03:39.722956+01 f idle  COMMIT

In total, there are 34 connections, but I only ran 15 measurements.

In sidekiq.log, they all show as done:

2015-02-20T12:00:04.879Z 6235 TID-osgop8md0 MeasurementWorker JID-1cdcd44bf41fefe9ddca21ac INFO: start
2015-02-20T12:00:04.907Z 6235 TID-osgoox8hg MeasurementWorker JID-79f4d1ff6692248682ba93dd INFO: start
2015-02-20T12:00:04.939Z 6235 TID-osgoos38k MeasurementWorker JID-09f95fbccd2438d17916d425 INFO: start
2015-02-20T12:00:04.940Z 6235 TID-osgoorlmo MeasurementWorker JID-be1a57871f26146e9884107e INFO: start
2015-02-20T12:00:04.959Z 6235 TID-osgoow16k MeasurementWorker JID-e296efec897c23629b96e99f INFO: start
2015-02-20T12:00:04.968Z 6235 TID-osgoouytg MeasurementWorker JID-c6a57700872b7fe427e33664 INFO: start
2015-02-20T12:00:04.984Z 6235 TID-osgooz63k MeasurementWorker JID-f7448eaffe109882130497ca INFO: start
2015-02-20T12:00:04.998Z 6235 TID-osgoozvzs MeasurementWorker JID-c55c04f3424268fba50ec048 INFO: start
2015-02-20T12:00:05.014Z 6235 TID-osgooyr6c MeasurementWorker JID-01bd303e953fd2998fe3f8d1 INFO: start
2015-02-20T12:00:05.030Z 6235 TID-osgon0ums MeasurementWorker JID-6949c5c81b4c254046f0c585 INFO: start
2015-02-20T12:00:05.042Z 6235 TID-osgomw0g8 MeasurementWorker JID-cc03b717f81dd6fb0f58a946 INFO: start
2015-02-20T12:00:05.111Z 6235 TID-osgomlno8 MeasurementWorker JID-10eebcd76113f3565d8265ca INFO: start
2015-02-20T12:00:05.113Z 6235 TID-osgomjzzw MeasurementWorker JID-b0536d9a029faed0ba8eb5d3 INFO: start
2015-02-20T12:00:05.114Z 6235 TID-osgomhyms MeasurementWorker JID-1f6624314afd8e0ae611599f INFO: start
2015-02-20T12:00:05.115Z 6235 TID-osgolgzoc MeasurementWorker JID-24e87960d7e7fbd871037dd3 INFO: start
2015-02-20T12:02:02.333Z 6235 TID-osgomw0g8 MeasurementWorker JID-cc03b717f81dd6fb0f58a946 INFO: done: 117.291 sec
2015-02-20T12:02:48.983Z 6235 TID-osgoox8hg MeasurementWorker JID-79f4d1ff6692248682ba93dd INFO: done: 164.077 sec
2015-02-20T12:02:50.688Z 6235 TID-osgomlno8 MeasurementWorker JID-10eebcd76113f3565d8265ca INFO: done: 165.577 sec
2015-02-20T12:03:02.429Z 6235 TID-osgolgzoc MeasurementWorker JID-24e87960d7e7fbd871037dd3 INFO: done: 177.314 sec
2015-02-20T12:03:04.581Z 6235 TID-osgoorlmo MeasurementWorker JID-be1a57871f26146e9884107e INFO: done: 179.641 sec
2015-02-20T12:03:09.453Z 6235 TID-osgop8md0 MeasurementWorker JID-1cdcd44bf41fefe9ddca21ac INFO: done: 184.573 sec
2015-02-20T12:03:10.362Z 6235 TID-osgoouytg MeasurementWorker JID-c6a57700872b7fe427e33664 INFO: done: 185.394 sec
2015-02-20T12:03:14.232Z 6235 TID-osgomjzzw MeasurementWorker JID-b0536d9a029faed0ba8eb5d3 INFO: done: 189.118 sec
2015-02-20T12:03:16.347Z 6235 TID-osgoos38k MeasurementWorker JID-09f95fbccd2438d17916d425 INFO: done: 191.408 sec
2015-02-20T12:03:20.398Z 6235 TID-osgoow16k MeasurementWorker JID-e296efec897c23629b96e99f INFO: done: 195.439 sec
2015-02-20T12:03:22.947Z 6235 TID-osgomhyms MeasurementWorker JID-1f6624314afd8e0ae611599f INFO: done: 197.833 sec
2015-02-20T12:03:30.212Z 6235 TID-osgooz63k MeasurementWorker JID-f7448eaffe109882130497ca INFO: done: 205.228 sec
2015-02-20T12:03:39.931Z 6235 TID-osgooyr6c MeasurementWorker JID-01bd303e953fd2998fe3f8d1 INFO: done: 214.918 sec
2015-02-20T12:03:39.936Z 6235 TID-osgon0ums MeasurementWorker JID-6949c5c81b4c254046f0c585 INFO: done: 214.906 sec
2015-02-20T12:03:51.694Z 6235 TID-osgoozvzs MeasurementWorker JID-c55c04f3424268fba50ec048 INFO: done: 226.696 sec

So, for every measurement, Sidekiq seems to open 2 connections, but it never closes them. What should I do?


Source: (StackOverflow)

How to start sidekiq in RubyMine?

i'm using RubyMine 6.3.

I tried to configure sidekiq in RubyMine but i can't start it.

How to configure and start sidekiq in RubyMine?


Source: (StackOverflow)

Configuring Unicorn & Sidekiq correctly on Heroku

I was getting ActiveRecord::StatementInvalid (PG::Error: SSL error: decryption failed or bad record mac errors so I followed this guide about deploying Unicorn to Heroku and it seems to have fixed it. However under caveats it shows how to configure Resque for such a setup - would I have to do something similar with Sidekiq?

Sample code from Heroku:

before_fork do |server, worker|

  ...

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis.quit
    Rails.logger.info('Disconnected from Redis')
  end
end

after_fork do |server, worker|

  ...

  # If you are using Redis but not Resque, change this
  if defined?(Resque)
    Resque.redis = ENV['REDIS_URI']
    Rails.logger.info('Connected to Redis')
  end
end

This is what I currently have set up:

config/unicorn.rb

worker_processes 2
timeout 30
preload_app true

before_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn master intercepting TERM and sending myself QUIT instead'
    Process.kill 'QUIT', Process.pid
  end
  defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect!
end  

after_fork do |server, worker|
  Signal.trap 'TERM' do
    puts 'Unicorn worker intercepting TERM and doing nothing. Wait for master to send QUIT'
  end
  defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection
end

config/initializers/sidekiq.rb

require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

Sidekiq.configure_server do |config|
  config.redis = { :size => 6 }
end

Procfile

web: bundle exec unicorn -p $PORT -E $RACK_ENV -c ./config/unicorn.rb
worker: bundle exec sidekiq -e production -c 4

Source: (StackOverflow)

how to delete a job in sidekiq

I am using sidekiq in my rails app. Users of my app create reports that start a sidekiq job. However, sometimes users want to be able to cancel "processing" reports. Deleting the report is easy but I also need to be able to delete the sidekiq job as well.

So far I have been able to get a list of workers like so:

workers = Sidekiq::Workers.new

and each worker has args that include a report_id so I can identify which job belongs to which report. However, I'm not sure how to actually delete the job. It should be noted that I want to delete the job whether it is currently busy, or set in retry.


Source: (StackOverflow)

Sidekiq configuration for multiple environments

I have looked at multiple sources and tried various scenarios but couldn't resolve this hence the issue. Please point me in the right direction.

Like everybody I have 3 env (development, staging and production).

I have the following in my sidekiq.yml file

# Options here can still be overridden by cmd line args.
#   sidekiq -C config.yml  
---
:verbose: false
:namespace: xyz
:logfile: log/sidekiq.log
:concurrency:  25
:strict: false
:pidfile: tmp/pids/sidekiq.pid
:queues:
  - [stg_xyz_tests_queue, 10]
  - [stg_default_xyz_queue, 2]
  - [stg_xyz_default_queue, 3]
development:
  :verbose: true
  :concurrency:  15
  :queues:
    - [dev_xyz_queue, 10]
    - [dev_default_xyz_queue, 2]
    - [dev_xyz_default_queue, 3]
staging:
  :queues:
    - [stg_xyz_queue, 10]
    - [stg_default_xyz_queue, 2]
    - [stg_xyz_default_queue, 3]
production:
  :queues:
    - [prod_xyz_queue, 10]
    - [prod_default_xyz_queue, 2]
    - [prod_xyz_default_queue, 3]

With this I was hoping that when I start sidekiq with the command

RAILS_ENV=#{rails_env} bundle exec sidekiq -C config/sidekiq.yml

that it would pickup all the values from the configuration file and start sidekiq with the appropriate queues and log file at log/sidekiq.log but that doesn't work. Sidekiq starts but it only creates the stg_xyz_tests_queue, stg_default_xyz_queue and stg_xyz_default_queue no matter what environment we use.

The other approach I tried was using the following code in the config/environments/development.rb

  #configure Sidekiq for dev environment
  Sidekiq.configure_server do |config|
    config.options[:namespace] = "xyz"
    config.options[:concurrency] = 25
    config.options[:verbose] = true
    config.options[:strict] = false
    config.options[:logfile] = "log/sidekiq.log"
    config.options[:pidfile] = "tmp/pids/sidekiq.pid"

    queues = Array.new
    10.times do
      queues.push "dev_xyz_queue"
    end

    2.times do
      queues.push "dev_default_xyz_queue"
    end

    3.times do
      queues.push "dev_xyz_default_queue"
    end

    config.options[:queues] = queues
    puts "Sidekiq server config options for development => #{config.options.to_yaml}"
  end

With this the queues are created ok but the logfile is not created or written and I need to duplicate this code for all the 3 environments.

What is the best way to get sidekiq working seamlessly for my setup Thanks for your help in advance !!!


Source: (StackOverflow)

Sidekiq and rails 4 actionmailer never delivers emails

I've setup sidekiq in a rails 4 application for sending emails and processing background jobs. I have also devise which I am using devise_async and a typical contact form emailer. I am using gmail for sending emails.

If I remove sidekiq, it sends the emails normally through gmail (both devise and contact form) but when I am enabling it, it doesn't work (neither devise_async neither contact form). Sidekiq shows that the background jobs starts and finishes successfully (I also see them through the sinatra app that it's been processed) but the email never been delivered.

In development, the console shows that the emails sent (both with devise_async and from contact form, as they are processed with sidekiq successfully).

What I've already have try:

  • I've added the github branch for rails4 of sidekiq (I've also tried the master)
  • I've updated and check redis version to be greater than 2.4 (it's 2.6.14)
  • I am running sidekiq with bundle exec sidekiq

but nothing worked. I am using ruby 2.0.0-p247.

I have also setup image processing with sidekiq in this app and works perfectly both in development and in production.

I am not doing anything fancy but I am adding the code for completion:

My mailer code:

  def send_message
   @message = Message.new(message_params)
   if @message.save
     ContactMailer.delay.contact_form(@message.id)
   end
 end

My mailer:

  def contact_form(message_id)
    message = Message.where(id: message_id).first
    @email = message.email
    @message = message.text
    mail(to: "myemail@gmail.com", subject: "Message from contact form")
  end

and my configuration for production (which it works without sidekiq):

  config.action_mailer.delivery_method = :smtp
  ActionMailer::Base.smtp_settings = {
    :address              => "smtp.gmail.com",
    :port                 => 587,
    :domain               => "gmail.com",
    :authentication       => "plain",
    :enable_starttls_auto => true,
    :user_name            => "myusername",
    :password             => "mypassword"
  }
  config.action_mailer.default_url_options = { :host => "mydomain.com" }

And here a typical sidekiq queue:

    /home/john/.rvm/gems/ruby-2.0.0-p247@cluey/bundler/gems/sidekiq-4ed6eba8aff1/lib/sidekiq/rails.rb:14: warning: toplevel constant Queue referenced by Sidekiq::Client::Queue
    2013-07-19T09:47:56Z 20112 TID-1a5mkc INFO: Booting Sidekiq 2.5.2 with Redis at redis://localhost:6379/0
    2013-07-19T09:47:56Z 20112 TID-1a5mkc INFO: Running in ruby 2.0.0p247 (2013-06-27 revision 41674) [i686-linux]
    2013-07-19T09:47:56Z 20112 TID-1a5mkc INFO: See LICENSE and the LGPL-3.0 for licensing details.
    2013-07-19T09:47:56Z 20112 TID-1a5mkc INFO: Starting processing, hit Ctrl-C to stop
    2013-07-19T09:48:11Z 20112 TID-1u2z02 Devise::Async::Backend::Sidekiq JID-4a7e66a14deab112191e4b49 INFO: start
    2013-07-19T09:48:12Z 20112 TID-1u2z02 Devise::Async::Backend::Sidekiq JID-4a7e66a14deab112191e4b49 INFO: done: 1.214 sec
    2013-07-19T09:48:50Z 20112 TID-1u2z02 Devise::Async::Backend::Sidekiq JID-32191822b789f5b6896a5353 INFO: start
2013-07-19T09:48:51Z 20112 TID-1u2z02 Devise::Async::Backend::Sidekiq JID-32191822b789f5b6896a5353 INFO: done: 0.143 sec
2013-07-19T09:49:29Z 20112 TID-1u2z02 Sidekiq::Extensions::DelayedMailer JID-c1911e0c4b72295dc067d57f INFO: start
2013-07-19T09:49:29Z 20112 TID-1u2z02 Sidekiq::Extensions::DelayedMailer JID-c1911e0c4b72295dc067d57f INFO: done: 0.152 sec

I think it has to do with actionmailer and sidekiq but I don't know how to debug, everything seems to work but the emails never delivered.


Source: (StackOverflow)

Sidekiq worker not getting triggered

I am using Sidekiq for my background jobs:

I have a worker app/workers/data_import_worker.rb

class DataImportWorker
 include Sidekiq::Worker
 sidekiq_options retry: false

  def perform(job_id,file_name)
    begin
    #Some logic in it .....
  end
 end

Called from a file lib/parse_excel.rb

  def parse_raw_data
      #job_id and #filename are defined bfr
      DataImportWorker.perform_async(job_id,filename)   
  end

As soon as i trigger it from my action the worker is not getting called.. Redis is running on localhost:6379

Any idea why this must be happening. The Environment is Linux.


Source: (StackOverflow)

Background mailing through sidekiq and devise-async, queue fills up but mails aren't processed

I'm currently building a rails platform and I've used devise for authentication and now want to move the default devise emails into a background process using sidekiq. I'm using devise-async for this and have done the following:

Added the devise_async.rb file:

#config/initializers/devise_async.rb
Devise::Async.backend = :sidekiq

Added the async command to the devise model:

#user.rb
devise :database_authenticatable, :async #etc.

The versions of the gems are the following:

Devise 2.1.2
Devise-async 0.4.0
Sidekiq 2.5.3

The issue that I'm having is that the emails are passed in the sidekiq queue but the workers never execute sending the emails. I've also looked at devise async not working with sidekiq and he seemed to have the same problem. But I don't think I have an issue with the hostname command.

Any thoughts on the issue?


Source: (StackOverflow)

Clear sidekiq queue

I've this worker that runs for ever.

class Worker
  include Sidekiq::Worker
  sidekiq_options queue: "infinity", retry: true

  def perform(params)
    # ...
    self.class.perform_in(30.seconds, params)
  end
end

The problem is that I load workers on start up, like this. config/initializers/load_workers.rb

Rails.application.config.after_initialize do  
  if ENV["SIDEKIQ"] == "1"
    Worker.perform_async({})
  end
end

Using this to start sidekiq SIDEKIQ=1 sidekiq --verbose --environment production -C config/sidekiq.yml.

This means that old workers as to stop, both those currently running but also the ones being rescheduled.

I tried running this on start up (just before loading new works), but that didn't work.

q = []
q += Sidekiq::RetrySet.new.select { |job| job.klass.match(/Worker/) }
q += Sidekiq::Queue.new("infinity").select { |job| job.klass.match(/Worker/) }
q += Sidekiq::ScheduledSet.new.select { |job| job.klass.match(/Worker/) }
q.each(&:delete)

After 5-ish deploys there are bunch of duplicate workers in the queue scheduled for later. So, is there a way to clear everyting in one queue and prevent already running jobs from rescheduling?

I'm using sidekiq 3.0.


Source: (StackOverflow)

Stack level too deep when using carrierwave versions

I'm trying to use a sidekiq worker, which more or less saves a image file to database (using carrierwave). There are few files to save, which are a keyframes extracted from a video file. That's what that worker is about.

My image uploader has a few versions defined and looks as follows:

class KeyframeUploader < CarrierWave::Uploader::Base

  # ...

  # Keyframe thumbnail sizes
  version :small do
    process resize_to_fill: [180, 180], if: :square?
    process resize_to_fill: [320, 180], if: :not_square?
  end

  version :medium do
    process resize_to_fill: [460, 460], if: :square?
    process resize_to_fill: [640, 460], if: :not_square?
  end

  version :large do
    process resize_to_fill: [720, 720], if: :square?
    process resize_to_fill: [1280, 720], if: :not_square?
  end


  private

    # Checks if image is a square
    def square? file
      img = Magick::Image.read(file.path)
      img[0].columns == img[0].rows
    end

    # Oposite to #square?
    def not_square? file
      !square? file
    end

end

The thing is, when i'm trying to run my Sidekiq Worker, it throws Celluloid::FiberStackError: stack level too deep and the only way to fix that is to remove my version definitions. It works only if there are not any version assigned to the uploader.

I have tried moving a save process to another worker or using Carrierwave::Backgrounder but i'm allways getting the same result.

Have you any idea what can i do about it?


Edit: My stracktrace is:

SystemStackError: stack level too deep from /usr/local/rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/irb/workspace.rb:86


Source: (StackOverflow)

Sidekiq jobs stuck in enqueue

Sidekiq has been working in development mode just perfectly. Now that I am trying to use it in production, all the jobs are just sitting in enqueue and aren't ever being run. Could anyone point me in the right direction as to how to solve this issue?


Source: (StackOverflow)

Why is "❨╯°□°❩╯︵┻━┻" with such an encoding used for a method name?

I came across following method in sidekiq gem. Its just invoked from test_sidekiq.rb.

def self.❨╯°□°❩╯︵┻━┻
  puts "Calm down, bro"
end

This is the only link I was able to find on SO.

  1. Google can't understand ❨╯°□°❩╯︵┻━┻. Why doesn't Ruby complain about this encoding?
  2. What is the purpose of this method (not much looking at its body)?
  3. Why did author @mike-perham use this name? Just for fun, or testing some boundaries?

Source: (StackOverflow)

Best way to monitor for completion of a Sidekiq job?

I'm using a Sidekiq worker to complete some requests to Facebook after a user signs-in for the first time. Typically the task takes around 20 seconds or so.

I'd like to load some information on to the page using an ajax request as soon as the sync is completed, but am unsure as to the best way to check for the job completion with Javascript.

One possibility would be to configure the Sidekiq worker to set a cookie after the rest of the jobs are done. Then I can use a setTimeout function to keep checking for the cookie before calling the load function. But I'm unsure whether this is the best way to be doing this. Could I be using Redis instead?


Source: (StackOverflow)

Sidekiq not processing queue

What possible reasons can Sidekiq prevent from processing jobs in the queue? The queue is full. The log file sidekiq.log indicates no activity at all. Thus the queue is full but the log is empty, and Sidekiq does not seem to process items. There seem to no worker processing jobs. Restarting Redis or flush it with FLUSHALL or FLUSHDB as no effect. Sidekiq has been started with

bundle exec sidekiq -L log/sidekiq.log

and produces the following log file:

2013-05-30..Booting Sidekiq 2.12.0 using redis://localhost:6379/0 with options {}
2013-05-30..Running in ruby 1.9.3p374 (2013-01-15 revision 38858) [i686-linux]
2013-05-30..See LICENSE and the LGPL-3.0 for licensing details.
2013-05-30..Starting processing, hit Ctrl-C to stop

How can you find out what went wrong? Are there any hidden log files?


Source: (StackOverflow)