08 Jun '13, 10pm

Active Record batch processing in parallel processes

Active Record batch processing in parallel processes Published almost 2 years ago Active Record provides find_each for batch processing of large number of records. However, when you are dealing with REALLY larger number of records ( I’m talking millions here ), find_each can become quite slow. One obvious solution is to use something like Resque : User .find_each {|user| Resque .enqueue(MyJob , user) } But this solution can feel a little heavy in certain cases. Enter forking! if GC .respond_to?(:copy_on_write_friendly= ) GC .copy_on_write_friendly = true end jobs_per_process = 100 process_count = 10 User .find_in_batches(:batch_size => jobs_per_process * process_count) do |group| batches = group.in_groups(process_count) batches.each do |batch| Process .fork do ActiveRecord ::Base .establish_connection # Do the actual work batch.each {|user| .. } end end Process .waitall en...

Full article: http://m.onkey.org/active-record-batch-processing-in-para...

Tweets

.@SingaporePoly launches own record label, run by its staff and students

.@SingaporePoly launches own record label, run ...

by Eddino Abdul Hadi, Music Correspondent Wong Ay Shyuan dreams of making it as a Mando-pop violinist-singer in the region...

New Irish Record for Tori Pena in the pole vault, next stop @ETCDublin2013

New Irish Record for Tori Pena in the pole vaul...

athleticsireland.ie 08 Jun '13, 8pm

Hours after hearing news of her selection for the European Cup in Dublin in two weeks time, Finn Valley athlete Tori Pena,...

@checker Have a look at the official Rails guid...

guides.rubyonrails.org 15 Jun '13, 8am

But this approach becomes increasingly impractical as the table size increases, since User.all.each instructs Active Recor...

Check out the on-demand webinar on #element14 a...

element14.com 07 Jun '13, 4pm

Join Newark element14, NXP Semiconductor and Spansion for an exclusive webinar featuring element14’s LPC4357-EVB developme...