smallseo.info

resque

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Introducing Resque · GitHub

Have Rails 2.3.x ignore the i18n gem

I have a Rails 2.3.5 project that uses the localization features of Rails. I also happen to have Rails 3 beta installed (which depends on the i18n gem). Rails 2.3.5 will happily handle localization on it's own (without i18n installed), however if the i18n gem is available, it makes use of it.

Recently I upgraded my gems and now have version 0.3.7 and 0.4.0 of i18n installed. Rails, of course, wants to load and use the latest version which is causing errors in my project. I tried setting the gem version to 0.3.7 which gets around the errors in the web app. However, we're using resque and resque_mailer to delay the sending of messages. When the worker picks up the mailer job from the queue, it ignores my config.gem requirement in environment.rb and uses version 0.4.0 anyway.

Ideally, I'd like to tell Rails to just not use the i18n gem at all. How do I do that?

Update: As of beta 4, Rails 3 now requires i18n version 0.4.1. I don't see how more people aren't running into this problem as it would seem now if you have both Rails 2 and Rails 3 installed, you're going to run into this.


Source: (StackOverflow)

How to get resque-web to work on Heroku?

On my dev machine, I am able to type resque-web in a console and it launches a new tab on my browser which shows the Resque interface.

On Heroku, Cedar stack, how can I do the same thing? i.e. I would like to see Resque's interface for my Heroku app.

EDIT

in config/initializers/resque.rb

require 'resque'
require 'resque/server'

uri = URI.parse(APP_CONFIG['redis_to_go_url'])
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

# Load all jobs at /app/jobs
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }

in routes.rb

mount Resque::Server.new, :at => '/resque'

Everything works. I am now able to see the Resque web interface. However, I would like to protect this from public view. Possibly with a username and password. How can this be done?


Source: (StackOverflow)

How to stop God from leaving stale Resque worker processes?

I'm trying to understand how to monitor the resque worker for travis-ci with god in such a way that stopping the resque watch via god won't leave a stale worker process.

In the following I'm talking about the worker process, not forked job child processes (i.e. the queue is empty all the time).

When I manually start the resque worker like this:

$ QUEUE=builds rake resque:work

I'll get a single process:

$ ps x | grep resque
 7041 s001  S+     0:05.04 resque-1.13.0: Waiting for builds

And this process will go away as soon as I stop the worker task.

But when I start the same thing with god (exact configuration is here, basically the same thing as the resque/god example) like this ...

$ RAILS_ENV=development god -c config/resque.god -D
I [2011-03-27 22:49:15]  INFO: Loading config/resque.god
I [2011-03-27 22:49:15]  INFO: Syslog enabled.
I [2011-03-27 22:49:15]  INFO: Using pid file directory: /Volumes/Users/sven/.god/pids
I [2011-03-27 22:49:15]  INFO: Started on drbunix:///tmp/god.17165.sock
I [2011-03-27 22:49:15]  INFO: resque-0 move 'unmonitored' to 'init'
I [2011-03-27 22:49:15]  INFO: resque-0 moved 'unmonitored' to 'init'
I [2011-03-27 22:49:15]  INFO: resque-0 [trigger] process is not running (ProcessRunning)
I [2011-03-27 22:49:15]  INFO: resque-0 move 'init' to 'start'
I [2011-03-27 22:49:15]  INFO: resque-0 start: cd /Volumes/Users/sven/Development/projects/travis && rake resque:work
I [2011-03-27 22:49:15]  INFO: resque-0 moved 'init' to 'start'
I [2011-03-27 22:49:15]  INFO: resque-0 [trigger] process is running (ProcessRunning)
I [2011-03-27 22:49:15]  INFO: resque-0 move 'start' to 'up'
I [2011-03-27 22:49:15]  INFO: resque-0 moved 'start' to 'up'
I [2011-03-27 22:49:15]  INFO: resque-0 [ok] memory within bounds [784kb] (MemoryUsage)
I [2011-03-27 22:49:15]  INFO: resque-0 [ok] process is running (ProcessRunning)
I [2011-03-27 22:49:45]  INFO: resque-0 [ok] memory within bounds [784kb, 784kb] (MemoryUsage)
I [2011-03-27 22:49:45]  INFO: resque-0 [ok] process is running (ProcessRunning)

Then I'll get an extra process:

$ ps x | grep resque
 7187   ??  Ss     0:00.02 sh -c cd /Volumes/Users/sven/Development/projects/travis && rake resque:work
 7188   ??  S      0:05.11 resque-1.13.0: Waiting for builds
 7183 s001  S+     0:01.18 /Volumes/Users/sven/.rvm/rubies/ruby-1.8.7-p302/bin/ruby /Volumes/Users/sven/.rvm/gems/ruby-1.8.7-p302/bin/god -c config/resque.god -D

God only seems to log the pid of the first one:

$ cat ~/.god/pids/resque-0.pid
7187

When I then stop the resque watch via god:

$ god stop resque
Sending 'stop' command

The following watches were affected:
  resque-0

God gives this log output:

I [2011-03-27 22:51:22]  INFO: resque-0 stop: default lambda killer
I [2011-03-27 22:51:22]  INFO: resque-0 sent SIGTERM
I [2011-03-27 22:51:23]  INFO: resque-0 process stopped
I [2011-03-27 22:51:23]  INFO: resque-0 move 'up' to 'unmonitored'
I [2011-03-27 22:51:23]  INFO: resque-0 moved 'up' to 'unmonitored'

But it does not actually terminate both of the processes, leaving the actual worker process alive:

$ ps x | grep resque
 6864   ??  S      0:05.15 resque-1.13.0: Waiting for builds
 6858 s001  S+     0:01.36 /Volumes/Users/sven/.rvm/rubies/ruby-1.8.7-p302/bin/ruby /Volumes/Users/sven/.rvm/gems/ruby-1.8.7-p302/bin/god -c config/resque.god -D

Source: (StackOverflow)

Sending Devise emails through Resque

I'm trying to send Devise emails through Resque.

Regular emails are getting sent through Resque just fine. And Devise emails are sent just fine, but not Devise emails through Resque. I get "Could not find a valid mapping" which implies that my helper overrides aren't getting picked up.

I'm following this http://shaker.4-dogs.biz/2011/08/06/using-resque-to-send-mail-for-devise/

The odd thing is that to debug it I'm using a local copy of Devise and adding breakpoints in 'initialize_from_record' in Devise, which gets hit when I just use Devise alone. But when I send the Devise emails through resque the breakpoints don't get hit:

class ResqueMailer < Devise::Mailer
  include Resque::Mailer
end

config.mailer = "ResqueMailer"

And resque instead shows a packaged gem path and not my local source such as:

/Users/mm/.rvm/gems/ruby-1.9.2-p290@evergreen/gems/devise-1.4.9/lib/devise/mailers/helpers.rb:20:in `devise_mail'

Any idea why it's not using my local gem source AND/OR how to get Resque to send my Devise emails?


Source: (StackOverflow)

Best Ruby on Rails Architecture for Image Heavy App

I'm building an application that allows for large amounts of photo uploads at once, and wanted to know what the best setup would be to tackle this.

This is what I am using so far:

  • Jquery File Upload: allows users to drag and drop images
  • CarrierWave: Processes images and resizes them with ImageMagick
  • Amazon S3: CarrierWave uploads images to Amazon S3 through Fog
  • Heroku: for hosting

I'd like to allow users to be able to drag and drop a large amount of images onto a page, and then navigate to other pages while the upload is going on in the background. I'd also like pictures to appear as they finish uploading. I don't want this process to lock up the Heroku dynos, so I probably need to move the work to a background job but I'm not sure what to use for my situation.

What's the best setup for this type of app? What background worker gem should I use? Is Cloudinary a good idea?


Source: (StackOverflow)

Resque or Gearman - choosing the right tool for background jobs [closed]

We are developing a web application wherein with about 50% of write requests, we end up pushing data to multiple data stores and inserting and updating significant number of records in those data stores. To improve the response time, we want to process such requests asynchronously in the background.

Our web application is being written in Ruby on Rails.

Two solutions that I'm inclined towards are Resque and Gearman.

Resque: More info here: http://github.com/blog/542-introducing-resque Resque seems very well suited for Ruby, and it's specifically meant for background job processing. "Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work."

Gearman: It's not specifically meant only for background jobs for asynchronous processing, but that's something it can definitely do. Apparently more robust, or so it seems. Another advantage of Gearman is that while your client code might be in Ruby, the worker code could be in, let's say, PHP. Though right now we are completely Ruby on Rails app, who knows if in future we might want to use PHP or something else depending on the job at hand.

What would you recommend? Do you have experience with either of the two? What real life production challenges should I keep in mind while choosing between the two? And am I even comparing apple to apple here?


Source: (StackOverflow)

Rails: Exception in after_create stopping save

Simple question. I have a ActiveRecord model that I want to perform post processing on AFTER the record is saved. So in the model I have a queue_for_processing method that sticks a job onto my Resque queue. To make this execute after my record is successfully persisted I have written the following in my model:

after_create :queue_for_processing

Pretty simple. I had thought that everything was working as expected EXCEPT that last night my redis server went down and things went awry. My expectations were that the record would still be saved and I could process the job later manually. But the queue_for_processing method is throwing an exception (expected behavior) and stopping the record from saving.

Am I misunderstanding how after_create works? Or is my understanding correct and something funky happening?

Thanks.


Source: (StackOverflow)

Postgres error on Heroku with Resque

I don't know enough Postgres to understand the message.

PG::Error: SSL error: decryption failed or bad record mac : SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = '"users"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum

And here is the callstack got from the resque backend

/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `async_exec'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `exec_no_cache'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:663:in `block in exec_query'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:662:in `exec_query'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1264:in `column_definitions'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:858:in `columns'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/schema_cache.rb:12:in `block in initialize'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `yield'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `default'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `columns'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:237:in `columns_hash'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/delegation.rb:7:in `columns_hash'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:330:in `find_one'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:311:in `find_with_ids'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:107:in `find'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/querying.rb:5:in `find'
/app/app/workers/compute_worker.rb:5:in `perform'

Explanation: My worker is only doing some writing operations into the databases. Nothing complicated. When I execute the worker's task for the first time, it works, and the second time it fails with the above message. If I restart the worker, it will work only the first time. I already run

heroku pg:restart
heroku db:restart
heroku db:migrate

... without success.

On my local computer(using mysql) it works perfectly. Any idea ?


Source: (StackOverflow)

What is the proper way to setup and use php-resque?

I am trying to use php-resque to queue and execute ffmpeg conversions on my server. I understand broadly how it should work, but I am having some trouble with the details and can not find any tutorials. Specifically, I don't understand where I should place my job classes, and how to give the classes to my workers and start my workers. The read me only says "Getting your application underway also includes telling the worker your job classes, by means of either an autoloader or including them."

Hopefully someone can outline the overall structure of using php-resque.


Source: (StackOverflow)

deploying redis to heroku unable to connect

ive been trying to get resque to work with heroku. i can successfully get it to work in development mode, however when i try pushing to heroku i get

Errno::ECONNREFUSED (Connection refused - Unable to connect to Redis on 127.0.0.1:6379):

i then read and followed http://blog.redistogo.com/2010/07/26/resque-with-redis-to-go/

i put the configurations listed in the site but got the following error

SocketError (getaddrinfo: nodename nor servname provided, or not known):

i put in my initializers/resque.rb

Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }

ENV["redis://redistogo:11111111111111111@lab.redistogo.com:9254/"] ||= "redis://heroku_username:heroku_password@host:9254/"
uri = URI.parse(ENV["redis://redistogo:1111111111111111@lab.redistogo.com:9254/"])
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

however it throws the error mentioned above. on my dev mode now, i get the error as well.

i tried using my heroku username (im using the add on from heroku), putting my password to heroku, and changing the port to 9254. however i keep getting the socket error now. what am i doing wrong?

help would be much appreciated. thank you

UPDATE.

@kevin

i tried

uri = URI.parse(ENV["my_url_string"] || "redis://localhost:9254/" )
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

in a initializer/redis.rb as well but i get the following error

Errno::ECONNREFUSED (Connection refused - Unable to connect to Redis on 127.0.0.1:6379):

are the numbers in the error, ie 127.0.0.1:6379 significant? ive checked my redis gui app and also from heroku config that my port number is 9254

REDISTOGO_URL       => redis://redistogo:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@lab.redistogo.com:9254/

did you have any other configuration settings? thanks for helping!

FINAL UPDATE.

i fixed it. i can't believe it! my complete solution is

uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.redis = REDIS

verbatim. it works without explicitly setting the url because i guess heroku tries to set it up for me already


Source: (StackOverflow)

How to check Resque worker status to determine whether it's dead or stale

The default resque web interface says that I have 5 of 7 workers working. I don't understand how this could be happening.

I'm on heroku, so when my dyno restarts, it should spin down existing dynos and workers, then spin up new ones. So, I'm assuming some of these workers are stale, but resque thinks there are so many more workers working than there should be... (there should only be 1)

How can I check whether any of these are stale or dead? I expect to see only 1 worker working.

Eventually, I expect I'll do whatever this SO post says: How do I clear stuck/stale Resque workers?, but first I'd like to know how to determine whether a worker should be removed... I don't want to blindly unregister workers...

Apologies if this is an obvious question. I'm new to resque.

Thanks!


Source: (StackOverflow)

Redis with Resque and Rails: ERR command not allowed when used memory > 'maxmemory'

When using redis, it gives me the error:

ERR command not allowed when used memory > 'maxmemory'

The info command reveals:

redis 127.0.0.1:6379> info
redis_version:2.4.10
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:1881
uptime_in_seconds:116
uptime_in_days:0
lru_clock:1222663
used_cpu_sys:0.04
used_cpu_user:0.04
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:1
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:930912
used_memory_human:909.09K
used_memory_rss:1269760
used_memory_peak:931408
used_memory_peak_human:909.58K
mem_fragmentation_ratio:1.36
mem_allocator:libc
loading:0
aof_enabled:0
changes_since_last_save:4
bgsave_in_progress:0
last_save_time:1333432389
bgrewriteaof_in_progress:0
total_connections_received:1
total_commands_processed:2
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master

Is the used_memory high? I'm a complete redis noob. If so, how does this problem occur and how should I proceed from here? This same error is all occurring in production (Heroku), so any help is greatly appreciated. Thank you.


Source: (StackOverflow)

Rails Resque workers fail with PGError: server closed the connection unexpectedly

I have site running rails application and resque workers running in production mode, on Ubuntu 9.10, Rails 2.3.4, ruby-ee 2010.01, PostgreSQL 8.4.2

Workers constantly raised errors: PGError: server closed the connection unexpectedly.

My best guess is that master resque process establishes connection to db (e.g. authlogic does that when use User.acts_as_authentic), while loading rails app classes, and that connection becomes corrupted in fork()ed process (on exit?), so next forked children get kind of broken global ActiveRecord::Base.connection

I could reproduce very similar behaviour with this sample code imitating fork/processing in resque worker. (AFAIK, users of libpq recommended to recreate connections in forked process anyway, otherwise it's not safe )

But, the odd thing is that when I use pgbouncer or pgpool-II instead of direct pgsql connection, such errors do not appear.

So, the question is where and how should I dig to find out why it is broken for plain connection and is working with connection pools? Or reasonable workaround?


Source: (StackOverflow)

Gems/Services for autoscaling Heroku's dynos and workers

I want to know if there are any good solutions for autoscaling dynos AND workers on Heroku in a production environment (probably a different solution for each of those, as they are pretty unrelated). What are you/companies using, regarding this?

I found lots of options, but none of them seem really mature for a production environment. There is Heroscale, which seem to introduce some latency as it does not run locally, and I also heard of some downtime. There are modifications of delayed_jobs, which have not been updated for a long time, and there are some issues with current bundlers. There is also some alternatives related to reque, which seem not to handle very well some HTTP exceptions, which results in app crashing, and others which seem to need an always-running worker to schedule other workers, and may also suffer from some HTTP exceptions problems.

Well. In the end. What is being used, nowadays, for autoscaling Heroku's dynos and workers on a production environment with Rails3?

Thanks in advance.


Source: (StackOverflow)

How to bridge the testing using Resque with Rspec examples?

I've a confusion while implementing Resque in parallel with Rspec examples. The following is a class with expensive method .generate(self) class SomeClass ... ChangeGenerator.generate(self) ... end

After implementing resque, the above class changed to the following and added a ChangeRecorderJob class.

class SomeClass
  ...
  Resque.enqueue(ChangeRecorderJob, self.id)
  ...
end

class ChangeRecorderJob
  @queue = :change_recorder_job

  def self.perform(noti_id)
    notification = Notification.find(noti_id)    
    ChangeGenerator.generate(notification)
  end
end

It works perfectly. But I have 2 concerns.

Before, my example spec used to test the whole stack of .generate(self) method. But now since I pushed that into Resque job, how can I bridge my examples to make that same test green without isolating? Or do I have to isolate the test??

And lastly, if I have 10 jobs to enque, do I have to create 10 separate job classes with self.perform method?


Source: (StackOverflow)