Nick Sieger: Tag railsconf

Gig: Speaking at RailsConf Europe 2007

Nick Sieger — Fri, 14 Sep 2007 05:33:00 +0000

Speaking of keeping busy, I’ll be speaking alongside my colleague Craig McClanahan at RailsConf Europe in Berlin next week.

Sun is a Diamond Sponsor at RailsConf again, just like in Portland last May. Part of that sponsorship money pays for a brief keynote spot (filled by Craig) as well as a session or two. So no, I didn’t get my spot through an accepted proposal submission, but that doesn’t mean that the session is going to be a big marketing shill.

No, actually Craig and I are part of a small group at Sun that’s embracing Rails in a big way, and we’re going to be launching a site built mostly on Rails later this fall. We’re taking what we think are some novel approaches to building a Rails-based application and we thought we’d share some of those thoughts with you rather than drone on for the session about how great Sun is and what snazzy tools we make. (Although expect to see a subtle plug or two for Sun hardware and tools. Call it product placement rather than overt selling.)

I titled the session “Rails Hydra” because the central idea of the structure of our application is not one Rails app, but many. The UI and views don’t even talk to a database; instead they make use of ActiveResource and RESTful web services, talking to the models living in other Rails applications in the backend. One key point is we’re deploying .war files to JRuby running on Glassfish, thus avoiding headaches of morbidly multiplying Mongrel math. We’ll elaborate on this arrangement and talk about some of the other tools and tricks we’re using.

Also, Charlie, Tom and Ola will be there, so we’ll certainly have a JRuby summit at some point. Stop by and say hello!

JRuby Q & A at RailsConf

Nick Sieger — Sun, 01 Jun 2008 22:07:33 +0000

These are the notes I took during our “panelish” Q & A session on Sunday afternoon at RailsConf.

Exceptions (behavior between ruby and java)
- Embedding: JSR-223 preferred, BSF fallback
Java integration, mocking, proxying, extension
Multiple VM support
- multiple JRuby instances can be run in the same VM
- JavaSand (google it) -- Ola’s port of _why’s (freaky freaky) sandbox
Startup time performance gain recently
- -Xbootclasspath: VM does not verify classes in the “boot” classpath
- -Xverify:none is not recommended because of runtime-generated code
- Nailgun to keep VM running and “send commands” to it
Compiling Ruby code to JVM bytecode
- Keep # of class files low
- Walk AST and dump out high-level, abstracted operations (local variable access, dynamic invocation) and low-level bytecode is built for each of those
- 4096 -- maximum # of methods to be compiled
- Compiled methods are shared between runtimes
- Future:
- raise/eliminate overhead of compiled methods
- share AST, reduce memory load
Community tasks/actions
- Projects needing help (e.g., ActiveHibernate)
- Adoption, acceptance, blogging
- Documentation (http://wiki.jruby.org/ and others), screencasts
- Who’s using it?
Sun’s commitment
- Ruby vs. Groovy vs. Python vs. Scala vs. other langs
- Rails vs. Grails vs. Lift vs. JSF vs. Struts 2 vs. ???
- all of the above
- improve the JVM’s support for all dynamic languages
- JRuby is a full open source project, not owned or controlled by Sun, with history that extends years before main devs hired by Sun
Ruby programming language, JRuby is an implementation
- community evangelism of Ruby the language vs. implementations
- despite MVM, FFI and emerging areas that are not standard yet
How do you pitch JRuby/Rails in a legacy environment?
- Consider demonstrating running application without discussing technology specifics
- IBM and JDK 1.5 issues have been reported
- Free support as long as these issues are new and help improve JRuby and the community!
Testing
- JRuby enables more agile testing
- Without production risks
- Ceremony vs. Essence discussion
Terracotta/DSO
- Jonas Boner blogged about it over a year ago
Windows/Mongrel/ImageMagick issues
- ImageVoodoo is an imaging library using Java2D, comes with Java
- MiniMagick also reportedly works
Rails benchmarks
- Micro-benchmarks are problematic and usually don’t lead to measurable application speedups
- Community needs a real-world, full application
- Antonio Cangiano working on new application-level benchmarks?
- “Richards” -- Smalltalk-originated application benchmark
JSR-292 (“invokedynamic”)
- Actually extension of “invokeinterface” bytecode
- Call site structure/method handles
- Expose the dynamic nature of the JVM (get the Java-specific stuff out of the way)
- Language-specific calling semantics, method invocation
- “Punching a hole” through the JVM
- Comments on JSR-292 Early Draft Review currently ongoing

RailsConf: Stefan Kaes - Rails Performance

Nick Sieger — Fri, 23 Jun 2006 20:40:00 +0000

Stefan starts by citing a factor of 4-5 improvement in performance in Rails over the last year.

Performance, broken down

Latency -- how fast
Throughput -- how many
Utilization -- how idle is the cpu
Cost efficiency -- performance per unit cost

For completeness calculate the min, max, mean and standard deviation of these metrics and use the deviation as your guide for how reliable the data is.

Tools

Log files (level > Logger::DEBUG)
Rails Analyzer Tools (Eric Hodel)
Benchmarker (script/benchmarker)
DB vendor tools
Apache bench (ab or ab2)
httperf
railsbench (Stefan Kaes)

Railsbench

Railsbench measuress raw performance of rails request processing. It’s configured using config/benchmarks.yml and config/benchmarks.rb. These files let you control which requests get benchmarked, whether to create a new session when benchmarking them, etc.

Profiling Tools

Ruby Profiler
Zen Profiler
rubyprof
Rails profiler script
Ruby Performance Validator

At this point Stefan gave an overview of RPV, which appears to be a nifty tool that lets you get typical hotspot tree views of where time is spent in code. It currently only runs on Windows.

Session containers

In memory -- if you server crashes...oops. Also doesn’t scale.
File system -- easy to set up, scales with NFS, but slower than...
ActiveRecordStore -- easy to set up since it comes with Rails, but much slower than...
SQLSessionStore -- which uses the same table structure as ActiveRecordStore, but was written by Stefan to overcome performance issues with ActiveRecordStore. Setup is more involved.
memcached -- slightly faster than SQLSessionStore, scales best, but setup is also more involved.
DrbStore -- distributed ruby store

Caching

Full pages -- fastest, complete pages are served on the filesystem. Web server bypasses appserver for rendering. If you have private pages, you can’t use it.
Actions -- pages are cached after an action is rendered. The user ID can be used as part of the storage key.
Fragments -- fragments can be cached in memory, on the file system, in a DrbStore, or in memcached. Memcached scales the best but doesn’t support expiring fragments by regular expression.

ActionController

Stefan recommends avoiding components, and replacing them with helpers or partials. He has not found a use for them.

ActionView

Don’t create unnecessary instance variables in the controller; creating them in the view with instance_variable_set and accessing with instance_variable_get is slow.

Helpers

pluralize -- don’t use the inflector if you don’t need to, it’s expensive.
linkto and urlfor are among the slowest helpers, since they need to use routes. Instead, if you have page with lots of links, you might consider hard-coding the links. This reduces the amount of GC by up to 50% and the GC time down by a few percentage points (11.3% to 8.7% of total processing time).

ActiveRecord

use the :include option to prefetch associations, it avoids extra onesy-twosy SQL statements.
use piggy-backing plugin for has_one or belongs_to relationships -- allows you to retrieve extra attributes from additional tables in the same fetch query.
Field values are retrieved from the DB mostly as strings, so type conversion happens on each access, which can be slow.

Language-level and miscellaneous issues

Method calls are the slowest -- don’t needlessly create method abstractions
Short-circuit intermediate results to improve performance
Cache results in instance variables or class variables
Don’t call ObjectSpace.each_object on each request

Ruby Memory Management

designed for batch scripts, not long-running servers.
no generational garbage collection.
this is suboptimal for Rails because ASTs are stored on the heap (biggest portion of non-garbage for Rails apps), and get processed/traversed more often than they need to be
Railsbench includes a patch to allow one to recompile Ruby and tweak the garbage collector.

Rails Template Optimizer

Stefan has started a project to “compile” templates.
The idea is to cache results of some ERb scriptlets and essentially “compile” or replace the template with one that has more expressions expanded or inlined.
Code forthcoming; I assume you can stay tuned to Rails Express for news.

Questions

There was a question on JRuby -- Stefan replied that it would certainly solve GC issues, but he doesn’t know if it’s in a state to be able to benchmark Rails requests.
What are your recommendations for a web server. I don’t have any.
Is horizontal or vertical scaling better? I don’t know, I’ve been focused on making single requests go fast, so I don’t have enough experience.

Spring and Summer Gigs

Nick Sieger — Fri, 18 Apr 2008 17:25:16 +0000

Just a quick note on some upcoming gigs. I’ll be speaking next month at both JavaOne and RailsConf, giving an update on the JRuby deployment situation, which by the way is rapidly improving! Despite my proposal for a hands-on tutorial on JRuby deployment not getting accepted for RailsConf, I’ve been asked to fill one of the Sun-sponsored talk slots again. So, much like RailsConf EU last fall, if you come listen I promise you will not get a heavy sales pitch from me! Do stop by, I may even have a couple goodies in store for you.

I also feel privileged to be speaking at RubyFringe this summer, where I haven’t yet determined exactly what I’ll be talking about (although it will probably have some connection to JRuby).

Hope to see you at one of these upcoming events!

Update: We just re-jiggered the schedule of 3 JRuby-related talks at JavaOne, including mine, so that they flow better from beginning to end of the week. My talk is now on Thursday morning instead of Wednesday morning. In addition, Charlie and Tom’s is on Tuesday, and Ola’s is on Wednesday. If you’re attending JavaOne and are interested in JRuby-related talks, please make sure you log into Schedule Builder and re-check your schedule.

RailsConf 2007 Opening Keynote: David Heinemeier Hansson

Nick Sieger — Fri, 18 May 2007 17:10:39 +0000

Rails 2.0

Where we’ve been

David is surprised and proud of the community that we already have, and wants us to be comfortable with where we are, and not always looking toward the future. We have:

Million gem downloads
Hundreds of plugins
10k users on the rubyonrails-talk mailing list
Ruby job descriptions (asking for 3 years RoR experience, longer than David)
Books, books, books (and not just English books, but non-English titles as well), surpassing VBA, Perl, and Python in book sales
IDEs from NetBeans, Borland, Aptana, etc.

Rails 2.0 is not going to be the “Unicorn”. It’s not going to be a total rewrite, it actually has a release schedule, it will not break backwards-compatibility. Instead, it will build upon what we already have, and continue the philosophy of building on what is useful and needed. In fact, 95% of what’s in 2.0 already works today, in the edge. Example, a simple controller that handles three formats of input/output, with a person resource for accessing the data from a remote server.

class PeopleController < ApplicationController
  ...
  def create
    @person = Person.create(...)
    respond_to do |format|
      format.html { redirect_to person_url(@person) }
      format.xml  { render :status => :created, :location => person_url(@person), ... }
      format.js {
        render :update do |js|
          ...
        end
      }
    end
  end
end

class Person < ActiveResource::Base
  self.site = "http://example.com/"
end

David then goes into a live demo of the new scaffold resource, which by appearance is identical to the old scaffolding, except it comes pre-baked with a REST-ful XML interface. He then adds support for a text format with a couple of lines of code, jumps into IRB, defines an active resource, and proceeds to change the data remotely.

If you want to add search to your controller, you can do it in a DRY way, and all the format/view work you’ve done will benefit:

class PeopleController < ApplicationController
  def index
    if params[:name]
      @people = Person.find :all, :conditions => ["name like ?", "#{params[:name]}%"]
    else
      @people = Person.find :all
    end
    ...
  end
end

David points out that 37signals, Shopify, Fluxiom, et. al. are real sites, with non-trivial domains that are still well executed in Rails, so it’s not just about simple scaffolding demos.

In Rails 2.0, ActiveResource will be bundled with Rails, and ActionWebService will not.

Friends of Rails

AJAX!
REST!
Atom? -- Atom should be more native to Rails
Openid? -- Openid is not necessarily something that needs to be used by all, but still a strong ally.

9 other things I like about Rails 2

Breakpoints are back -- no longer depends on Binding.of_caller; instead Rails depends and builds upon ruby-debug by Kent Sibilev.
HTTP Performance -- streamlining .js and .css, even though it feels better to break up Javascript and CSS into many little pieces, and gzip them

<%= javascript_include_tag :all, :cache => true %>
<%= stylesheet_link_tag :all, :cache => true %>

We can also fake out the browser and configure multiple asset hosts (4) you can maximize browser connections

config.action_controller.asset_host = 'assets%d.highrisehq.com'

Query cache

ActiveRecord::Base.cache do
  # actions here are cached
end

Rendering and MIME types -- bake the MIME convention into the template, and separate from the rendering mechanism people/index.html.erb people/index.xml.builder people/index.rss.erb people/index.atom.builder
config/initializers replacing config/environment. Initializers are .rb files in the config/initializers directory of your app that are automatically loaded during initialization time.
Sexy migrations

create_table :people do |t|
  t.integer :account_id
  t.string :first_name, :last_name, :null => false
  t.text :description
  t.timestamps
end

HTTP authentication (authenticate_or_request_with_http_basic, authenticate_with_http_basic)
The MIT assumption -- the licensing question -- make it easier to understand
Spring cleaning -- getting rid of the cruft -- stay tuned!

RailsConf Europe: Hydra

Nick Sieger — Sat, 06 Oct 2007 12:04:00 +0000

On September 19, Craig and I presented our talk at RailsConf, which appears to have been well received. It went off mostly without a hitch, if it wasn’t for a couple of hiccups in the demos. I apparently didn’t practice them enough, because a couple of critical steps were either missed or I did them out of order and confused myself. But that’s ok, because I’m releasing the demo steps, source and slides here so you can try them out for yourself.

So, download the zip and follow along. The contents look like this:

1-active-resource-basics.txt
2-make-resourceful.txt
3-atom-roller.txt
4-service-chatter.txt
RailsConfEurope-Hydra.pdf
demo/
demo-baked/

The demo steps are in the text files; I’d recommend going in the order specified. It turns out the third isn’t really a demo but more of a code review, because it also requires you to have Roller set up and I’m not going into those details for now. If you want to just run the demos, skip to demo-baked, the finished product.

For those of you who didn’t see the talk, here’s the basic message.

Look at your basic MVC Rails app.

Why not consider spliting it into two? ActiveResource allows you to access a RESTful resource in your Rails application like it was just another model.

This might seem like overkill for a simple application, but what if you had an e-commerce application domain like this?

Splitting up your code into separate Rails applications encourages encapsulation, reduces potential coupling, and gives you more flexible deployment options. Basing interactions upon REST and HTTP means that you can more easily mash up data or create caching strategies, given proper usage of ETags/Last-Modified and/or cache-control headers. The great thing is that existing HTTP reverse proxies can be used without having to mix the caching code in with your application code.

In the application we’re building, we have a number components that are not Rails-based. To expose them to our environment, we’ve taken the strategy of exposing a simple REST web service for the component, and then it can be consumed by the other applications using ActiveResource. The REST web service can either be implemented in the component’s native language/technology, or in some cases we’ve written a wrapper service in Rails since Rails makes it so easy to build REST interfaces. In that case, Rails is pure integration -- RESTful glue.

The idiom that makes this all possible is the uniform interface. In HTTP, this means addressability (each resource gets a unique URI) coupled with the HTTP method verbs HEAD, GET, POST, PUT, and DELETE. Inside your Rails applications, it’s the ActiveRecord interface. If you keep your controllers skinny, you can boil the interface down to the following set of methods (in this case, for the prototypical blog post model):

Post.new/Post.create
Post.find
@post.save
@post.update_attributes
@post.errors
@post.destroy

And in fact, this is precisely what ActiveResource provides, and it’s enabled by duck-typing. Walking through the demos illustrates this pretty well, as you’ll basically swap ActiveRecord for ActiveResource with no noticeable difference, all the way down to validation errors in the scaffolded forms (which I was unable to demo in the talk due to the hiccups).

We’ve found that when you’re making RESTful web services, the controllers largely become boilerplate because of the uniform interface. make_resourceful has been a boon in that regard, as the demos also show. There are several plugins that help you DRY up your controllers (other approaches include resources_controller), so you have some choices there.

We mentioned some drawbacks in our experience with ActiveResource, which have largely been addressed for the upcoming Rails 2.0 release.

Finally, we noted that deployment could be a pain with so many Rails applications to keep running. To that end, we are leveraging JRuby and Glassfish to make this a non-issue, as we simply WAR up our Rails applications with warbler and let Glassfish take care of the rest. Performance is still an open question, but we plan to roll up our sleeves and make sure this combination really hums.

Enjoy the demos! Feel free to drop me an email if you have any questions or troubles with them.

RailsConf 2007: Chris Wanstrath: Kickin' Ass with Cache-fu

Nick Sieger — Sat, 19 May 2007 20:34:00 +0000

Chris is here to talk about games, since he used to work for Gamespot. He coded PHP, which is like training wheels without the bike. He had to sit in a glass cube and help keep the site running during E3 last year. There were 100 gajillion teenage boys during their lunch break hitting refresh, and it all blew up. Couldn’t even gzip the responses, because the servers heated up to much. They served 50M pages in a day, without downtime. They did it with Memcache.

Memcache is a distributed hash -- multiple daemons running on different servers. Developed by Livejournal for their infrastructure, you just put up the servers, and they just work.

Should you use Memcache? No. YAGNI, UYRDNI (unless you really do need it).

Rails and Memcache

Fragments, Actions, Sessions, Objects, cache it all. You can use:

memcache-client (by Robot-coop guys/Eric Hodel). Marshal.unload is 40 times faster than Object.new/loading from the database.
CachedModel -- integration with ActiveRecord
Fragment Cache Store
Memcache session store

...or...

`cache_fu`

Or, acts_as_cached. It knows about all the aforementioned objects, with a single YAML config file (config/memcached.yml). Word to the wise: don’t use names in your server config file. Use IPs, avoid BIND and connections to the servers with every connection. Don’t let DNS outages bring down your servers.

get_cache
expire_cache

This is all you need -- if you’re using set_cache, you probably don’t understand how the plugin works. Expire cache on the “after save” hook, which allows you to cache ID misses as well.

class Presentation < ActiveRecord::Base
  acts_as_cached
  after_save :expire_cache
end

Example: only cache published items

class Presentation < ActiveRecord::Base
  acts_as_cached :conditions => 'published = 1'
end

Cached-scoped-finders (if somebody thinks of a good name, let Chris know). The idea is to move custom finder logic to a method on your model, and then wrap a cache-scoping thingy around it. cache_fu ties this up nicely by giving you a cached method on AR::Base.

class Topic < ActiveRecord::Base
  def self.weekly_popular
    Topic.find :all, ...
  end
end

Topic.cached(:weekly_popular)

Adding date to cache key with alias_method_chain:

def self.cache_key_with_date(id)
  ...
end

class << self
  alias_method_chain :cache_key, :date
end

Cached loads by ID: Topic.find(1, 2, 3) moves to Topic.get_cache(1, 2, 3), which can parallelize calls to memcached and bring them back as they’re ready.

user_ids = @topic.posts.map(&:user_id).uniq
@users = User.get_cache(user_ids)

You can also cache associations, so that you’re navigating associations via Memcache.

Cache overrides

class ApplicationController < ActionController::Base
  before_filter :set_cache_override
  def set_cache_override
    ActsAsCached.skip_cache_gets = !!params[:skip_cache]
  end
end

reset_cache: Slow, uncached operations can sometimes queue up and wedge a site. Instead, issue cache resets on completion of a request, rather than expiring beforehand. That way, requests that continue to pile up will still use the cached copy until the rebuild is complete.

class Presentation < ActiveRecord::Base
  after_save :reset_cache
end

Versioning: a way to expire cache on new code releases

class Presentation < ActiveRecord::Base
  acts_as_cached :version => 1
end

Deployment: Chris recommends using Monit to ensure your Memcache servers are up.

libketama: consistent hashing that gives you the ability to redeploy Memcache servers without invalidating all the keys.

Q: Page caching? A: Nginx with native Memcache page caching, but outside of Rails domains.

Lots of other questions, but dude, Chris talks too fast!

RailsConf 2007: Bradley Taylor: Virtual Clusters

Nick Sieger — Sat, 19 May 2007 19:30:06 +0000

How does Rails figure into virtualization? Bradley will cover this topic with examples and case studies. Along the way, hardware items may be mentioned, but are not critical. Really, it’s about the design of the clusters, not the bits of plumbing you use to connect them up.

Virtualization is partitioning of physical servers that allow you to run multiple servers on it. Xen, Virtuozzo, VMWare, Solaris containers, KVM, etc. Bradley uses Xen. The virtual servers share the same processor (hopefully multi-core), memory, storage, network cards (but with indepenent IP addresses), etc., but run independently of each other. VPS, slice, container, accelerator, VM, it’s all the same. Memory, storage, and CPU can be guaranteed with the virtualization layer.

Why would you do this? Consolidate servers for less hardware and cost; Isolate applications -- bad apps don’t drag the server down, contain intrusions, use different software stacks; Replicate -- easily create new servers and deploy in a standardized and automated way; Utilize -- take advantage of all CPU, memory, storage, resources; Allocate resources, give a server exactly what it requires, grow/shrink up and down, and balance them. Bradley says, “Once you go to virtualization you won’t want to go back. Do the simplest thing that could possibly work.”

Virtual clusters, then, are a bunch of servers cooperating toward a common goal -- if you have many versions or copies of one thing. More than one customer, more than one version of software, etc.

For Rails, this means a lot of things: you can have many development environments and stages, take advantage of memory isolation, protect against PHP/Java, and make multiple-server scaling accessible.

Examples

Two servers for production and staging
Three for web/db/staging
Mixed languages -- instead of 1x1GB server use 3x300MB servers
High availability applications with fewer servers
Multiple applications -- one server per application
Standardized roles/appliances -- mail, ftp, dns, web, db

EastMedia

They can incubate customers in separate images
Dev/staging/production servers
Shared SVN/trac
2 physical servers => 8 virtual servers

Boom Design

Again, multiple stages
Customer staging, with lower uptime requirements
Low-traffic apps on a single server, but everything else gets its own dedicated server
2GB memory spread across 9 virtual servers

RailsConf 2007: Saturday Morning Keynotes

Nick Sieger — Sat, 19 May 2007 17:22:42 +0000

Cyndi Mitchell -- ThoughtWorks Studios

Enterprise (the “e” word)

Before IT got involved, “enterprise” was a bold new venture. Toyota manufacturing, Skype disruption of telephony.

Enterprise in terms of IT has come to mean bloatware, incompetence, corruption, waste of time, no value.

So this is the battle: The enterprise (to boldly go where no man has gone before) we need to reclaim vs. the bloatware/competence/corruption/fear-based selling etc.

RubyWorks -- package stack with haproxy, mongrel, monit through an RPM repository

For JRuby support, call Ola.

Tim Bray -- Web Guy from Sun Microsystems

Change the world that are better than just using a cool web framework: http://pragmaticstudio.com/donate/

Sun loves Ruby. Ruby and Rails, that is. The impact of the Ruby language is going to be at least as big as Rails is for web development.

Sun provided servers for Ruby 2.0 development, and can provide servers for your potentially cool, worthy, open source project, just drop Tim an email.

A few more obligatory plugs for NetBeans and Sun sponsoring the conference. “Pre-alpha,” he says. Hmm, I wonder what Tor would say about that!

JRuby: when would you use JRuby vs. Ruby? If you have no pain, keep using C Ruby. But if you have management concerns, deployment concerns, etc. then by all means do try it!

Obligatory handshake/sandal connection with ThoughtWorks and Cyndi -- running Mingle (and cruisecontrol.rb) with JRuby.

Sun: “Hi, the answer is Java, what was the question?” So why would Sun want to support Ruby? Well, you guys are programmers. Programmers who deliver quality software fast. And those programmers need computers, and OSes, and web servers, and support and services, etc. Plug, plug, plug.

How do you make money on free products? Sun has open-sourcing Java, Solaris, even Sparc. Joyent is open-sourcing their stuff. Where does the money come from? 1. Adoption 2. Deployment 3. Monetization at the point of value

What if we win? Are our problems over? No, we’ll have to deal with Java. And .NET. And PHP. From the audience: And COBOL. The Network Is The Computer. The Network Is Heterogeneous. Deal with it. So how do we interoperate?

Just Run Java (and JRuby, of course!, and JavaScript, and PHP, etc.)
Use Atom/REST. Everything should have a publish button. Don’t use WS-DeathStar or WCF or WSIT.

Developer issues: Scaling, Static vs. Dynamic, Maintainability, Concurrency, Tooling, Integration, Time to Market. Which two of these matter the most?

Tim’s final assertion: Maintainability and Time to Market, and that’s why we’re all at RailsConf.

RailsConf 2007: Evan Weaver: Going Off Grid

Nick Sieger — Fri, 18 May 2007 19:33:31 +0000

Evan is talking about leaving Rails as a full-stack framework and remixing bits and pieces for integration projects. He’s doing it in the context of a case study on Bio: a project at the University of Delaware working with DNA data in large SQL databases. Evan states that all of bioinformatics is an integration problem. (Me: That’s probably true of any research project where data is coming from multiple, varied sources. So where does Rails fit in this?)

So how do you cope with this? Use the Rails console as an admin interface, mapping AR onto the legacy schema.

Shadow (gem install shadow) is a REST-ful record server -- a small Mongrel handler that allows you to manipulate the database remotely. It uses dynamic ActiveRecord classes that are created and trashed for each request.

Parallelization -- uses the Sun 1 grid engine that distributes shell scripts across 128 nodes. Used for job and backend processing.

bioruby/bioperl/biopython -- bioinformatics libraries in other languages -- bioruby is not complete, but we still want to use Ruby, so he looked at ways of integrating Ruby with other languages. No RubyInline for Perl or Python, no up-to-date direct/C bindings. He ended up building a socket-level interface into python.

Admin tools to consider -- streamlined, active_scaffold, autoadmin, Django (manage.py inspectdb; manage.py syncdb; manage.py runserver). (Wow, come to RailsConf, get a Django demo. Unexpected surprise!)

Extending Rails -- has_many_polymorphs for easy creation directed graphs

Frustrating AR tidbits: has_many_through has a huge case statement, with sql strings everywhere, and tightly intertwined classes. Ugh.

Scaling big webapps: AR/SQL is not the way. Instead, go to a hyper-denormalized model, where the DB is just a big hash. This leads to things like berkeleydb, memcached, madeleine, etc. and MySQL just becomes a persistence store for memcache. One key is moving joins at write-time, so that reads don’t need to re-join associations. You’re essentially duplicating/caching the data out to each association, but this makes sharding/splitting of data easier. Example: Flickr user photos vs. photos placed in a group.

Evan doesn’t believe that SQL is a viable data store for webapps -- I think he means large-scale webapps. Not everyone who’s trying to build a web application will run into these kinds of issues, so your mileage may vary. Still, it’s refreshing to see more people rebel against the incumbent 30-year gorilla of SQL.