Nick Sieger: Category ruby

Tweaking IRB

Nick Sieger — Sun, 23 Apr 2006 04:02:00 +0000

IRB (Interactive Ruby Shell) is one of those tools that a hacker learning Ruby hopefully discovers right away. It’s an extremely useful way to learn the language, verify hunches, test assumptions, and get immediate feedback. IRB promotes learning by doing, which is the best way of making something stick in your head. (You can even try an online version of irb without even installing Ruby!)

The first order of business when using IRB is to setup your preferences. If you haven’t done so already, create the file ~/.irbrc (%USERPROFILE%.irbrc on windows native ruby). .irbrc is just a regular ruby script where you can run arbitrary ruby code at the start of your IRB session. Add the following to .irbrc:

require 'irb/completion'
ARGV.concat [ "--readline", "--prompt-mode", "simple" ]

This sets up usage of readline in your session and turns on TAB completion, making IRB feel as comfortable as regular old bash. Now you can type Kernel::<TAB> and get a list of available methods! Good.

Next, the thing that you find yourself doing after using IRB for a while is cutting and pasting code from your console buffer over to your text editor. Don’t have Ruby’s reflection rules down yet? Not sure whether to use instance_eval or module_eval when working on that metaprogramming hack? Working inside Rails’ script/console and searching for the right ActiveRecord finder options? No matter how good your terminal program, you probably have to use the mouse to select text out of it to copy to your text editor, and hackers hate having to switch from the keyboard to the mouse when in the flow of programming.

So here’s a technique that will append commands entered in your IRB session to a file in your home directory (idea from ruby-talk:58931). Put the following in your .irbrc:

module Readline
  module History
    LOG = "#{ENV['HOME']}/.irb-history"

    def self.write_log(line)
      File.open(LOG, 'ab') {|f| f << "#{line}
"}
    end

    def self.start_session_log
      write_log("
# session start: #{Time.now}

")
      at_exit { write_log("
# session stop: #{Time.now}
") }
    end
  end

  alias :old_readline :readline
  def readline(*args)
    ln = old_readline(*args)
    begin
      History.write_log(ln)
    rescue
    end
    ln
  end
end

Readline::History.start_session_log

Now every line typed into IRB will immediately be saved into ~/.irb-history. Exercise left to the reader to bind a custom keystroke and macro to yank the last line out of that file and automatically paste into your text editor.

Long-time bash users know that the shell maintains a history of commands across sessions so that you can access commands you typed yesterday. Wouldn’t it be nice to do this in IRB as well? Wish granted:

require 'irb/ext/save-history'
IRB.conf[:SAVE_HISTORY] = 100
IRB.conf[:HISTORY_FILE] = "#{ENV['HOME']}/.irb-save-history"

Happy ruby hacking! If you find any more handy IRB tips leave them at rubygarden, and let me know about them.

Footnote: I realize there is duplication and non-DRY happening here with two copies of your IRB history, but I came across these techniques at two different times, and the functions they serve seem different enough to potentially use them both. If you don’t like that, choose whichever is more appropriate for your needs.

(Hope this post serves your needs Dan.)

Virtual Ruby?

Nick Sieger — Tue, 02 May 2006 18:56:00 +0000

I had a chance to dabble around with Python a tiny bit today, building the XSV XML schema validator. Got it working fairly easily on my MacBook Pro after a minor detour. Later, I needed to build another copy on a shared linux dev box where I didn’t have root access. Now how does this work in Python? Fortunately, the answer was close at hand: a virtual python instance. Nice! You can create a replica of the main python install, and add your personal site-packages there. The only thing that gets copied is the python executable plus a bunch of symlinks to the main libraries and packages. After that point all you need to do is put your virtual python executable ahead of the main executable on your PATH.

If I were to install local packages/gems with Ruby, I’d probably either build my own Ruby or maintain a local packages directory and constantly customize RUBYLIB. Seems like a suboptimal solution. Has anyone attempted a virtual-ruby.rb that does the same as virtual-python.py?

Hackers who are fathers and _why

Nick Sieger — Fri, 05 May 2006 14:24:00 +0000

Spotted on ruby-talk today was this anecdote on a Ruby hacker/father and son, and I just had to share here.

I trotted my 11 year old son over to the Poignant Guide and he laughed like crazy at the Cartoon Foxes and did a little programming, but soon got bored. Wait a minute Alex, let’s try one more thing, Dwe.. Array. Wow, how do I make my R kill big time (He knows all about cheats). I showed him the character’s values and he promptly put in a hugh number for the Rabbit’s strength. He was delighted to see the larger and larger negative life values appear whenever he attacked. Dad, is this how all my Gameboy programs work? Yes, son. Hey, dad I want to do 6dof animation too, can you teach me? Yes son, here’s a few books for you – Computer Graphics, Foley et al and Physics for Game Developers, Bourg. Not sure he’ll read them right away, but the whole episode saved me another $49.95 Game Cube cartridge.

Rails is simpler than Office

Nick Sieger — Fri, 26 May 2006 03:32:00 +0000

Before my big blog drought at the beginning of the year, I had an entry queued up talking about some success I’d experienced with Rails. A lot in the Rails world has progressed since then, but I still think the story is worth documenting. Also, the code to generate a PDF of mailing labels may be useful to somebody out there.

I’ve had some Rails success lately building a home-use mailing list manager/rolodex application. There are plenty of ways that such a list could be maintained without resorting to a full web application framework such as Rails, but what the heck! The mailing list started life as an MS Access database; after my work computer was re-imaged I no longer had “access to Access” so it had a temporary layover in an Excel spreadsheet. Within the past couple of months I had moved it to a MySQL database as a way to nurture my fledgling Rails efforts.

Ok, so nothing real special so far, except that in order to print mailing labels (one of the primary reasons for keeping such a list) I’d have to export the names to a .csv file and do a mail merge with Word. Until the most recent mailing.

On a Saturday night I had the brainstorm to use Austin Ziegler’s PDF::Writer library to create a printable PDF directly from the Rails app, thus skipping the need to go through the mail merge rigamarole. Only a couple of hours of effort later, I had my mother-in-law’s Christmas mailing list printed out! Anyone who has ever done a mail merge with Word knows that clicking a single link to create the printable versions of the mailing labels is a huge improvement in usability. And finally, no MS bits were harmed in the production of this mailing!

My starting point in building the code to generate PDFs was this page in the Rails wiki. I decided to use the method that describes installing an “rpdf” template handler. Nowadays, you may as well use Josh Charles’ Rails PDF plugin, but for posterity I’ve packaged up my effort as a simple plugin as well (install into an existing Rails application with ./script/plugin install http://svn.caldersphere.net/svn/main/plugins/pdfrender).

With the plugin in place, all that’s necessary is a controller method to set up the data for the view, and the view code itself. The controller is as straightforward as you’d expect:

class AddressController < ApplicationController
  # ...

  def pdf
    @addresses = Address.find(:all, :order => 'last_name, first_name')
    render :layout => false
  end
end

The view code is a little more hairy but with a little thought the dimensioning and layout code could easily be DRY’d out.

FONT = "Times-Roman"
FONT_SIZE = 12

COLS = 3
LABELS_PER_PAGE = 30
LABELS_PER_COL = 10

# margins: .5in top & bottom, 0.19 in left and right
# table column widths: 2.63in | 0.13in | 2.63in | 0.13in | 2.63in
# table rows: 1in height

MARG_X = pdf.in2pts 0.19
MARG_Y = pdf.in2pts 0.5

CELL_Y = pdf.in2pts 1
CELL_X = pdf.in2pts 2.63

COL_PAD_X = pdf.in2pts 0.19

COL1_X = MARG_X 
COL2_X = COL1_X + CELL_X + COL_PAD_X
COL3_X = COL2_X + CELL_X + COL_PAD_X

CELL_PAD_X = pdf.in2pts 0.13
CELL_PAD_Y = pdf.in2pts 0.25
CELL_LINE_Y = FONT_SIZE + 2

def cell_x(col)
  [COL1_X, COL2_X, COL3_X][col] + CELL_PAD_X
end

def cell_y(row, line)
  MARG_Y + ((LABELS_PER_COL - row) * CELL_Y) - CELL_PAD_Y - (line * CELL_LINE_Y)
end

def add_label(row, col, addr, pdf)
  if addr
    pdf.add_text_wrap(cell_x(col), cell_y(row, 0), CELL_X, addr.name, FONT_SIZE)
    pdf.add_text_wrap(cell_x(col), cell_y(row, 1), CELL_X, addr.address, FONT_SIZE)
    pdf.add_text_wrap(cell_x(col), cell_y(row, 2), CELL_X, "#{addr.city}, #{addr.state} #{addr.zip}", FONT_SIZE)
  end
end

pdf.select_font(FONT)

pages = @addresses.length / LABELS_PER_PAGE
pages += 1 if (@addresses.length % LABELS_PER_PAGE) > 0

0.upto(pages - 1) do |page|
  start = page * LABELS_PER_PAGE
  address_page = @addresses[start..start+LABELS_PER_PAGE]

  0.upto(LABELS_PER_COL - 1) do |row|
    add_label(row, 0, address_page[row*COLS], pdf)
    add_label(row, 1, address_page[row*COLS+1], pdf)
    add_label(row, 2, address_page[row*COLS+2], pdf)
  end

  pdf.new_page unless page + 1 == pages
end

And that’s it! Avery labels in Rails!

.irbrc on Windows

Nick Sieger — Tue, 30 May 2006 16:57:00 +0000

Having trouble getting IRB to use a .irbrc file on Windows? The following seems to work:

Create the .irbrc file in your %USERPROFILE% directory.
Create an environment variable in the System Properties->Advanced Tab->Environment Variables area called HOME and set it to %USERPROFILE%.

An alternate approach is to create an environment variable called IRBRC and set it to the full path of the .irbrc file.

Perhaps IRB should be updated to look in %USERPROFILE% on Windows?

Plug: for handy .irbrc contents refer to this previous post.

QOTD

Nick Sieger — Tue, 06 Jun 2006 15:03:00 +0000

Spotted the following thread on ruby-talk:

Pat Maddox: I hate when languages put a condom on my code.
Gennady Bystritsky: What does it make your code, then? ;-)
Mat Schaffer: Pregnant!

RailsConf begins

Nick Sieger — Fri, 23 Jun 2006 14:10:00 +0000

I got lucky with a last minute ticket and found a way to get to the First International Rails Conference in Chicago.

That’s me, hope to see you this weekend!

RailsConf: Dave Thomas keynote

Nick Sieger — Fri, 23 Jun 2006 15:12:00 +0000

Dave Thomas, Professional Cassandra

Intro

Dave starts by dissecting a google search and pointing out how the sponsored links sidebar contains a wealth of paid ads for various topics:

Hosting
Consulting
Commercial IDEs and tools
Alternatives to RoR!

Moving on to rubyforge, Dave examines some download stats, and points out that there have been 536835 downloads of Rails through gem install rails

Dave examines some Google trends mapping of ruby on rails vs the following, showing RoR catching up on some and overtaking others:

websphere
jboss
tapestry
spring framework
zend

3 Problems for Rails

Dave proceeds to give his own, tailored for Rails, version of David Hilbert’s famous “23 unsolved problems”…except due to time constraints, can only discuss three.

PDI

An acronym, PDI, has emerged on the Rails core team as a way to encourage people to contribute. Please Do Investigate

The challenge is to get new features to be proved out in the community before taking it into the core, the Rails core team is only 12 and has limited bandwidth, and that is the idea behind PDI

The three problem areas are:

Data integration
CRUD
Deployment

Data integrations

Dave mentions that he and David “disagree mildly” on the point of using constraints in the database.

automatic validation based on schema
work with foreign keys - make it easy to define in a migration
add a “belongs_to” relationship if a foreign key constraint is detected
primary key support, e.g., non-integer keys, particularly in migrations, and composite primary keys

The argument here is that real-world applications, and legacy databases today require these features, and Rails needs to be able to have answers to those questions if it wants to be viable in that area.

Dave is asking for distributed transactions now. Should we be pushing Rails into such an enterprisey direction? If you really need XA transactions, shouldn’t you be banished to a java world?

Non-database models would be a fabulous addition to rails. ActiveRecord::Base currently assumes that it’s talking to a sinle relational datastore. If we could integrate multiple disparate data soures, it would make integrations, REST data sources, and mashups much easier. Microsoft is doing this with LINQ today, and if Rails wants to keep up, it will need this.

Real-world CRUD

Dave is arguing that we should beef up scaffolding and make it useful. Scaffolding was the star of the initial 10 minute Rails video that started everything, but yet it’s the ugliest web 1.0 interface ever! No AJAX, no automatic, DRY, in-browser validation. But should we, can we really genericize these interfaces? Maybe a scaffolding widget/component system? It can and probably will happen, but at the momet I’m inclined to think that this will complicate things. Do we want Rails view to turn into Wicket widgets or ASP.NET components? It’s a completely different way of writing web programs. We’ll have to see how it plays out.

Deployment

Dave argues for improved deployment. Yes we have Capistrano, and it’s arguably the best web deployment mechanism in existence, but the web of knowledge required to deploy a Rails application is too large, varied and unwieldy.

Capistrano is a push model, ideal for small shops where the developers manage and maintain everything, but it’s not the real world. Larger shops have separate development and admin departments. They have stricter requirements for where files go, pre and post-deployment hooks, roles, passwords, security, etc. Capistrano currently falls down a bit in this environment.

Dave discusses cap --deploy-on: a hypothetical extension to Capistrano that would allow staged deployment to multiple servers, and the possibility to introduce workflow between developers and administrators. What if ISPs had a standardized infrastructure that would allow hobbyists to issue a cap --deploy-on and instantly have their apps running up on TextDrive, DreamHost, etc. To go further, what if we could deploy from rubygems? gem deploy <name> --on cap://my.isp.com. Typo installation would literally be a single command.

Closing

All developers need to be happy, so let’s make their lives easier!

RailsConf: Mike Clark -- Introduction to Capistrano

Nick Sieger — Fri, 23 Jun 2006 16:45:00 +0000

Introduction to Capistrano

Mike Clark has been Java-free for 15 months and 16 days. He’s here to talk about deployment with Capistrano.

First off, props to Jamis! There are several hundred people in the room and it’s a tribute to him.

Capistrano is about making it easy to deploy web applications. “Cap”, the short name has quickly become jargon in the Rails community. “Stop wasting time, just cap it!”

Capistrano is built to scale. From a single machine all the way up to clusters, it is intended to make application deployment as simple as the push of a button. That’s priceless for reducing friction as your project nears its first production live date, as well as subsequent maintenance releases.

Configuring Capistrano - create a recipe

Set the application name

set :application, "depot"

Set the repository

set :repository, "http://svn.yourhost.com/#{application}/trunk"

Roles

role :app, "app01.example.com", "app02.example.com"
role :web, "web01.example.com", "web02.example.com"
role :db, "db.example.com", :primary => true

Deployment Root

set :deploy_to, "/Library/Rails/#{application}"

Setup

Run cap setup once to setup the deployment directory structure on all the roles you’ve configured:

depot
  `- releases
  `- shared
       `- log
       `- system

First time deploy

Run cap update_code symlink. This checks out the code, adds a “current” symlink to the code on the remote machine that points to the timestamped releases directory where the code was checked out. You can then manually start your web server(s) if you wish.

New Release

Run cap deploy. The newly committed code to your repository gets pulled, a new release created, and it also restarts fcgi processes.

Rollback release

Run cap rollback. The “current” symlink gets updated to the previous release, and the fcgi processes get restarted.

Scheduled downtime

Run cap disable_web. A maintenance screen is put up during the maintenance period. This is great when you’re in firefighting mode and you don’t want to think, just get the page up there. To enable again, use cap enable_web.

Customization

Here are examples of several useful, real-world tasks. The fourth task shows aggregation of tasks together.

desc "locate ruby"
task :which_ruby, :roles => [:app] do
  puts "You're running:"
  run "which ruby"
end

task :current_revision do
  run "echo Current rev is #{revision}"
  run "echo Current rev is at #{releast_path}"
end

task :uptime, :roles => [:app, :web, :db] do
  run "uptime"
end

task :status do 
  which_ruby
  current_revision
  uptime
end

Channels and streams

Mike showed a log file tailing task here. I failed to capture the full task code, so you’re on your own here.

Hooks

task :before_deploy, :roles => [:app] do
  cleanup
end

task :after_update_code, :roles => [:app] do
  production_config = "path/to/database.yml"
  run "cp #{production_config} #{current_path}/config/database.yml"
end

Multiple configs

if where == "production"
  set :user, "ops_gal"
else
  set :user, "dev_guy"
end

Task libraries

You can creat a file full of extra, reusable tasks and simply require them in. The files can be published and installed as gems or placed anywhere on your load path.

Capistrano isn’t just for Rails

It’s great at shuttling files around and executing remote commands on the server. Mike describes some automation James Duncan Davidson did to replicate new servers into a cluster with the push of a button.

Assumptions

Capistrano, out of the box comes with some assumptions:

Remote servers talk POSIX
Same deploy structure and password on each machine
Web application uses FastCGI with Apache or Lighty

It looks like you can bend some of these assumptions if you are willing to write your own custom deployment tasks.

Take Home

Capistrano provides easy, consistent, worry-free deployment of new releases. The friction to deploying your app just got simpler, so you can go enjoy your life!

Notes and Questions

Can you use SSH public keys to authenticate? Yes, this is the most typical scenario.
You can use sudo to execute remote commands – this is built-in.
Can you/would you want to check out subsets of files on each server, rather than pulling all the code everywhere? So far most of our stuff has been on the same server, so this hasn’t been an issue. It seems like you would need to override deployment tasks for some roles.
Sometimes I want to deploy a lot, but what if the app is big, lots of media files? It can take a longer time to pull all the code. One solution I’ve seen is to check out once and rsync everywhere else. I haven’t had to do that myself.
Who can I talk to for promoting my own extensions into Capistrano. You probably don’t. Instead, publish your extensions, blog them, and see if it takes.
deploy_with_migrations – use with caution as it’s not as easy to rollback, and not all migrations can be rolled back cleanly.

RailsConf: Stefan Kaes - Rails Performance

Nick Sieger — Fri, 23 Jun 2006 20:40:00 +0000

Stefan starts by citing a factor of 4-5 improvement in performance in Rails over the last year.

Performance, broken down

Latency – how fast
Throughput – how many
Utilization – how idle is the cpu
Cost efficiency – performance per unit cost

For completeness calculate the min, max, mean and standard deviation of these metrics and use the deviation as your guide for how reliable the data is.

Tools

Log files (level > Logger::DEBUG)
Rails Analyzer Tools (Eric Hodel)
Benchmarker (script/benchmarker)
DB vendor tools
Apache bench (ab or ab2)
httperf
railsbench (Stefan Kaes)

Railsbench

Railsbench measuress raw performance of rails request processing. It’s configured using config/benchmarks.yml and config/benchmarks.rb. These files let you control which requests get benchmarked, whether to create a new session when benchmarking them, etc.

Profiling Tools

Ruby Profiler
Zen Profiler
rubyprof
Rails profiler script
Ruby Performance Validator

At this point Stefan gave an overview of RPV, which appears to be a nifty tool that lets you get typical hotspot tree views of where time is spent in code. It currently only runs on Windows.

Session containers

In memory – if you server crashes…oops. Also doesn’t scale.
File system – easy to set up, scales with NFS, but slower than…
ActiveRecordStore – easy to set up since it comes with Rails, but much slower than…
SQLSessionStore – which uses the same table structure as ActiveRecordStore, but was written by Stefan to overcome performance issues with ActiveRecordStore. Setup is more involved.
memcached – slightly faster than SQLSessionStore, scales best, but setup is also more involved.
DrbStore – distributed ruby store

Caching

Full pages – fastest, complete pages are served on the filesystem. Web server bypasses appserver for rendering. If you have private pages, you can’t use it.
Actions – pages are cached after an action is rendered. The user ID can be used as part of the storage key.
Fragments – fragments can be cached in memory, on the file system, in a DrbStore, or in memcached. Memcached scales the best but doesn’t support expiring fragments by regular expression.

ActionController

Stefan recommends avoiding components, and replacing them with helpers or partials. He has not found a use for them.

ActionView

Don’t create unnecessary instance variables in the controller; creating them in the view with instance_variable_set and accessing with instance_variable_get is slow.

Helpers

pluralize – don’t use the inflector if you don’t need to, it’s expensive.
linkto and urlfor are among the slowest helpers, since they need to use routes. Instead, if you have page with lots of links, you might consider hard-coding the links. This reduces the amount of GC by up to 50% and the GC time down by a few percentage points (11.3% to 8.7% of total processing time).

ActiveRecord

use the :include option to prefetch associations, it avoids extra onesy-twosy SQL statements.
use piggy-backing plugin for has_one or belongs_to relationships – allows you to retrieve extra attributes from additional tables in the same fetch query.
Field values are retrieved from the DB mostly as strings, so type conversion happens on each access, which can be slow.

Language-level and miscellaneous issues

Method calls are the slowest – don’t needlessly create method abstractions
Short-circuit intermediate results to improve performance
Cache results in instance variables or class variables
Don’t call ObjectSpace.each_object on each request

Ruby Memory Management

designed for batch scripts, not long-running servers.
no generational garbage collection.
this is suboptimal for Rails because ASTs are stored on the heap (biggest portion of non-garbage for Rails apps), and get processed/traversed more often than they need to be
Railsbench includes a patch to allow one to recompile Ruby and tweak the garbage collector.

Rails Template Optimizer

Stefan has started a project to “compile” templates.
The idea is to cache results of some ERb scriptlets and essentially “compile” or replace the template with one that has more expressions expanded or inlined.
Code forthcoming; I assume you can stay tuned to Rails Express for news.

Questions

There was a question on JRuby – Stefan replied that it would certainly solve GC issues, but he doesn’t know if it’s in a state to be able to benchmark Rails requests.
What are your recommendations for a web server. I don’t have any.
Is horizontal or vertical scaling better? I don’t know, I’ve been focused on making single requests go fast, so I don’t have enough experience.