smallseo.info

resque

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. Introducing Resque · GitHub

Resque, Devise and admin authentication

Using Resque and Devise, i have roles for User, like:

User.first.role #=> admin
User.last.role #=> regular

I want to setup an authentication for Resque. So, inside config/routes.rb i have:

namespace :admin do
  mount Resque::Server.new, :at => "/resque", :as => :resque
end

And, of course it's accessible for all logged in users.

Is there any way to use a role from User.role? It should be accessible only by users with 'admin' role.

Thanks a lot.


Source: (StackOverflow)

Best Ruby on Rails Architecture for Image Heavy App

I'm building an application that allows for large amounts of photo uploads at once, and wanted to know what the best setup would be to tackle this.

This is what I am using so far:

  • Jquery File Upload: allows users to drag and drop images
  • CarrierWave: Processes images and resizes them with ImageMagick
  • Amazon S3: CarrierWave uploads images to Amazon S3 through Fog
  • Heroku: for hosting

I'd like to allow users to be able to drag and drop a large amount of images onto a page, and then navigate to other pages while the upload is going on in the background. I'd also like pictures to appear as they finish uploading. I don't want this process to lock up the Heroku dynos, so I probably need to move the work to a background job but I'm not sure what to use for my situation.

What's the best setup for this type of app? What background worker gem should I use? Is Cloudinary a good idea?


Source: (StackOverflow)

Sending Devise emails through Resque

I'm trying to send Devise emails through Resque.

Regular emails are getting sent through Resque just fine. And Devise emails are sent just fine, but not Devise emails through Resque. I get "Could not find a valid mapping" which implies that my helper overrides aren't getting picked up.

I'm following this http://shaker.4-dogs.biz/2011/08/06/using-resque-to-send-mail-for-devise/

The odd thing is that to debug it I'm using a local copy of Devise and adding breakpoints in 'initialize_from_record' in Devise, which gets hit when I just use Devise alone. But when I send the Devise emails through resque the breakpoints don't get hit:

class ResqueMailer < Devise::Mailer
  include Resque::Mailer
end

config.mailer = "ResqueMailer"

And resque instead shows a packaged gem path and not my local source such as:

/Users/mm/.rvm/gems/ruby-1.9.2-p290@evergreen/gems/devise-1.4.9/lib/devise/mailers/helpers.rb:20:in `devise_mail'

Any idea why it's not using my local gem source AND/OR how to get Resque to send my Devise emails?


Source: (StackOverflow)

Inspect and retry resque jobs via redis-cli

I am unable to run the resque-web on my server due to some issues I still have to work on but I still have to check and retry failed jobs in my resque queues.

Has anyone any experience on how to peek the failed jobs queue to see what the error was and then how to retry it using the redis-cli command line?

thanks,


Source: (StackOverflow)

Resque vs Sidekiq? [closed]

I am currently using Resque for my background process but recently I heard a lot of huff-buff about sidekiq. Could anybody compare/differentiate?

In particular I would like to know is there a way to monitor programmatically whether a job is completed in sidekiq


Source: (StackOverflow)

How can I delete specific jobs from Resque queue without clearing the whole queue?

I'm using Resque workers to process job in a queue, I have a large number of jobs > 1M in a queue and have some of the jobs that I need to remove ( added by error). Crating the queue with the jobs was not an easy tasks, so clearing the queue using resque-web and adding the correct jobs again is not an option for me.

Appreciate any advice. Thanks!


Source: (StackOverflow)

Postgres error on Heroku with Resque

I don't know enough Postgres to understand the message.

PG::Error: SSL error: decryption failed or bad record mac : SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull FROM pg_attribute a LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum WHERE a.attrelid = '"users"'::regclass AND a.attnum > 0 AND NOT a.attisdropped ORDER BY a.attnum

And here is the callstack got from the resque backend

/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `async_exec'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1139:in `exec_no_cache'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:663:in `block in exec_query'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:280:in `block in log'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:275:in `log'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:662:in `exec_query'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:1264:in `column_definitions'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/postgresql_adapter.rb:858:in `columns'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/connection_adapters/schema_cache.rb:12:in `block in initialize'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `yield'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `default'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:228:in `columns'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/model_schema.rb:237:in `columns_hash'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/delegation.rb:7:in `columns_hash'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:330:in `find_one'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:311:in `find_with_ids'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/relation/finder_methods.rb:107:in `find'
/app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.2/lib/active_record/querying.rb:5:in `find'
/app/app/workers/compute_worker.rb:5:in `perform'

Explanation: My worker is only doing some writing operations into the databases. Nothing complicated. When I execute the worker's task for the first time, it works, and the second time it fails with the above message. If I restart the worker, it will work only the first time. I already run

heroku pg:restart
heroku db:restart
heroku db:migrate

... without success.

On my local computer(using mysql) it works perfectly. Any idea ?


Source: (StackOverflow)

What is the proper way to setup and use php-resque?

I am trying to use php-resque to queue and execute ffmpeg conversions on my server. I understand broadly how it should work, but I am having some trouble with the details and can not find any tutorials. Specifically, I don't understand where I should place my job classes, and how to give the classes to my workers and start my workers. The read me only says "Getting your application underway also includes telling the worker your job classes, by means of either an autoloader or including them."

Hopefully someone can outline the overall structure of using php-resque.


Source: (StackOverflow)

deploying redis to heroku unable to connect

ive been trying to get resque to work with heroku. i can successfully get it to work in development mode, however when i try pushing to heroku i get

Errno::ECONNREFUSED (Connection refused - Unable to connect to Redis on 127.0.0.1:6379):

i then read and followed http://blog.redistogo.com/2010/07/26/resque-with-redis-to-go/

i put the configurations listed in the site but got the following error

SocketError (getaddrinfo: nodename nor servname provided, or not known):

i put in my initializers/resque.rb

Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }

ENV["redis://redistogo:11111111111111111@lab.redistogo.com:9254/"] ||= "redis://heroku_username:heroku_password@host:9254/"
uri = URI.parse(ENV["redis://redistogo:1111111111111111@lab.redistogo.com:9254/"])
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

however it throws the error mentioned above. on my dev mode now, i get the error as well.

i tried using my heroku username (im using the add on from heroku), putting my password to heroku, and changing the port to 9254. however i keep getting the socket error now. what am i doing wrong?

help would be much appreciated. thank you

UPDATE.

@kevin

i tried

uri = URI.parse(ENV["my_url_string"] || "redis://localhost:9254/" )
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

in a initializer/redis.rb as well but i get the following error

Errno::ECONNREFUSED (Connection refused - Unable to connect to Redis on 127.0.0.1:6379):

are the numbers in the error, ie 127.0.0.1:6379 significant? ive checked my redis gui app and also from heroku config that my port number is 9254

REDISTOGO_URL       => redis://redistogo:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@lab.redistogo.com:9254/

did you have any other configuration settings? thanks for helping!

FINAL UPDATE.

i fixed it. i can't believe it! my complete solution is

uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.redis = REDIS

verbatim. it works without explicitly setting the url because i guess heroku tries to set it up for me already


Source: (StackOverflow)

How to check Resque worker status to determine whether it's dead or stale

The default resque web interface says that I have 5 of 7 workers working. I don't understand how this could be happening.

I'm on heroku, so when my dyno restarts, it should spin down existing dynos and workers, then spin up new ones. So, I'm assuming some of these workers are stale, but resque thinks there are so many more workers working than there should be... (there should only be 1)

How can I check whether any of these are stale or dead? I expect to see only 1 worker working.

Eventually, I expect I'll do whatever this SO post says: How do I clear stuck/stale Resque workers?, but first I'd like to know how to determine whether a worker should be removed... I don't want to blindly unregister workers...

Apologies if this is an obvious question. I'm new to resque.

Thanks!


Source: (StackOverflow)

Redis with Resque and Rails: ERR command not allowed when used memory > 'maxmemory'

When using redis, it gives me the error:

ERR command not allowed when used memory > 'maxmemory'

The info command reveals:

redis 127.0.0.1:6379> info
redis_version:2.4.10
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:1881
uptime_in_seconds:116
uptime_in_days:0
lru_clock:1222663
used_cpu_sys:0.04
used_cpu_user:0.04
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:1
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:930912
used_memory_human:909.09K
used_memory_rss:1269760
used_memory_peak:931408
used_memory_peak_human:909.58K
mem_fragmentation_ratio:1.36
mem_allocator:libc
loading:0
aof_enabled:0
changes_since_last_save:4
bgsave_in_progress:0
last_save_time:1333432389
bgrewriteaof_in_progress:0
total_connections_received:1
total_commands_processed:2
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master

Is the used_memory high? I'm a complete redis noob. If so, how does this problem occur and how should I proceed from here? This same error is all occurring in production (Heroku), so any help is greatly appreciated. Thank you.


Source: (StackOverflow)

Rails Resque workers fail with PGError: server closed the connection unexpectedly

I have site running rails application and resque workers running in production mode, on Ubuntu 9.10, Rails 2.3.4, ruby-ee 2010.01, PostgreSQL 8.4.2

Workers constantly raised errors: PGError: server closed the connection unexpectedly.

My best guess is that master resque process establishes connection to db (e.g. authlogic does that when use User.acts_as_authentic), while loading rails app classes, and that connection becomes corrupted in fork()ed process (on exit?), so next forked children get kind of broken global ActiveRecord::Base.connection

I could reproduce very similar behaviour with this sample code imitating fork/processing in resque worker. (AFAIK, users of libpq recommended to recreate connections in forked process anyway, otherwise it's not safe )

But, the odd thing is that when I use pgbouncer or pgpool-II instead of direct pgsql connection, such errors do not appear.

So, the question is where and how should I dig to find out why it is broken for plain connection and is working with connection pools? Or reasonable workaround?


Source: (StackOverflow)

Gems/Services for autoscaling Heroku's dynos and workers

I want to know if there are any good solutions for autoscaling dynos AND workers on Heroku in a production environment (probably a different solution for each of those, as they are pretty unrelated). What are you/companies using, regarding this?

I found lots of options, but none of them seem really mature for a production environment. There is Heroscale, which seem to introduce some latency as it does not run locally, and I also heard of some downtime. There are modifications of delayed_jobs, which have not been updated for a long time, and there are some issues with current bundlers. There is also some alternatives related to reque, which seem not to handle very well some HTTP exceptions, which results in app crashing, and others which seem to need an always-running worker to schedule other workers, and may also suffer from some HTTP exceptions problems.

Well. In the end. What is being used, nowadays, for autoscaling Heroku's dynos and workers on a production environment with Rails3?

Thanks in advance.


Source: (StackOverflow)

How to bridge the testing using Resque with Rspec examples?

I've a confusion while implementing Resque in parallel with Rspec examples. The following is a class with expensive method .generate(self) class SomeClass ... ChangeGenerator.generate(self) ... end

After implementing resque, the above class changed to the following and added a ChangeRecorderJob class.

class SomeClass
  ...
  Resque.enqueue(ChangeRecorderJob, self.id)
  ...
end

class ChangeRecorderJob
  @queue = :change_recorder_job

  def self.perform(noti_id)
    notification = Notification.find(noti_id)    
    ChangeGenerator.generate(notification)
  end
end

It works perfectly. But I have 2 concerns.

Before, my example spec used to test the whole stack of .generate(self) method. But now since I pushed that into Resque job, how can I bridge my examples to make that same test green without isolating? Or do I have to isolate the test??

And lastly, if I have 10 jobs to enque, do I have to create 10 separate job classes with self.perform method?


Source: (StackOverflow)

How to deploy resque workers in production?

The GitHub guys recently released their background processing app which uses Redis: http://github.com/defunkt/resque http://github.com/blog/542-introducing-resque

I have it working locally, but I'm struggling to get it working in production. Has anyone got a:

  1. Capistrano recipe to deploy workers (control number of workers, restarting them, etc)
  2. Deployed workers to separate machine(s) from where the main app is running, what settings were needed here?
  3. gotten redis to survive a reboot on the server (I tried putting it in cron but no luck)
  4. how did you work resque-web (their excellent monitoring app) into your deploy?

Thanks!

P.S. I posted an issue on Github about this but no response yet. Hoping some SO gurus can help on this one as I'm not very experienced in deployments. Thank you!


Source: (StackOverflow)