Posted by Nick Sieger
Sun, 19 Oct 2008 18:30:52 GMT
Project Kenai
Project Kenai has been open for business for over a month, and I’m just writing about it now?
Blogging is hard. Let’s go shopping, m’kay?
http://www.flickr.com/photos/zen/1229934/ on flickr
There’s no excuse for not writing until now about the JRuby on Rails project that’s kept me busy at work since I joined Sun almost 18 months ago. (Ok, there are a few lame ones. Twitter. Conference travel. Presidential political news distractions. Also, Tim’s write-up filled in my side of the story immediately after the launch.)
So, better late than never. Actually, by looking back on the first month of operation I think that I can give you a better idea of where we’re going, supported by what we’ve done (as well as what we haven’t done), rather than what I might have said we’re going to do.
Word of Mouth
One thing we haven’t done is put the heavy Sun marketing blitzkrieg operation to work on our behalf. Word has made its way around the blogs, and even onto a few tech news sites, but we’re still in a growth phase for the project, and we’d rather earn respect quietly through a site that people find useful instead of shouting the word from the mountain tops.
We have established that we intend to embrace change, having deployed three additional releases since launch. During that time, we’ve added a new feature, fixed bugs and UI inconsistencies, and worked on performance and infrastructure issues.
We’ve also seen community participation grow. We’ve had over 2100 people join, 79 projects have been created, and we’re starting to see real activity in those projects. These are modest but respectable numbers.
JRuby on Rails
As you’ve heard, kenai.com runs on JRuby and Glassfish and uses bits of software I’ve worked on like activerecord-jdbc, Warbler and JRuby-Rack. Having worked on JRuby itself and Rails support for JRuby for a couple years now, this is a personal validation of all that work. One of the things I’ve relished the most working on Project Kenai for the past year is to be able to build infrastructure software for and based upon real-world use. The JRuby story continues to get stronger every day, with things like thread-safety in the upcoming Rails 2.2 release adding fuel to the fire.
(Not Yet) More Than Just a Forge
One of the things that might have caught your eye when you came to the site is the slogan More Than Just a Forge. That’s silly, you might say to yourself. What do they have here that I can’t get at another project hosting site? And if you said that, I’d heartily agree with you – the marketers are just getting a little antsy.
However, that doesn’t mean we don’t have plans. For now, we’re taking the Gmail Launch strategy, starting with a simple, solid foundation, and gradually inviting more and more to participate, stabilizing and growing the platform, and soliciting feedback from our user base. We do already have a healthy amount of requests on our UserVoice page, and while we hope to make good on a number of those, we also plan to do some new things that aren’t being done elsewhere. Stay tuned, and I hope to be able to reveal some more in the coming months as we start to roll out the implementation of those plans.
In the meantime, if you’d like an invite to create a project, drop me an email. If you have comments or requests, you can share them with me privately via email, on the site in the forums, or on our UserVoice page.
Tags jruby, kenai, rails | no comments
Posted by Nick Sieger
Thu, 10 Jul 2008 05:10:27 GMT
The other day several people chimed in wondering how I set up this blog with JRuby and Glassfish. One of the reasons I didn’t include the details in the post is that it’s not really much different than any JRuby/Glassfish/Warbler deployment, but in case you don’t know what that looks like, here are the basics.
Preconditions (Java)
I’m running on a Joyent Accelerator, which runs OpenSolaris, which has JDK 6 installed by default. If you’re running on some flavor of Linux, hopefully there’s a package available for you to install, otherwise you may have to download a self-extracting binary.
This step is actually straightforward; not at all as problematic as you might expect of a piece of Java technology! In the parent directory where you want Glassfish to be installed (substituting the name of the Glassfish jar you downloaded as appropriate):
java -Xmx256m -jar glassfish-installer-v2ur2-b04-sunos_x86.jar
cd glassfish
chmod -R +x lib/ant/bin
./lib/ant/bin/ant -f setup.xml
Start Glassfish.
./bin/asadmin start-domain
You may want to add GLASSFISH/bin to your path so that you can run the Glassfish asadmin command from anywhere.
On Solaris, SMF is the subsystem that is used to ensure services are started at boot time (among other things). Glassfish works nicely with SMF. On other systems, there may be /etc/rc.d init scripts out there, or you can roll your own (asadmin start-domain and asadmin stop-domain).
Install JRuby
Download JRuby and unpack it somewhere. I recommend adding JRUBY_HOME/bin to the end of your path, so it doesn’t clash with Matz-Ruby.
Install Warbler and activerecord-jdbcmysql-adapter
In addition to Warbler, I’m using the activerecord-jdbcmysql-adapter to connect to the blog’s database. Both can be installed with Rubygems:
jruby -S gem install warbler activerecord-jdbcmysql-adapter
With Rails 2 and up, the application’s config/database.yml file should be updated for adapter: jdbcmysql:
<% jdbc = defined?(JRUBY_VERSION) ? 'jdbc' : '' %>
development:
adapter: <%= jdbc %>mysql
encoding: utf8
database: testapp_development
username: root
password:
socket: /tmp/mysql.sock
# same for test/production...
Otherwise, you need to jump through some extra environment.rb configuration hoops.
Configure Warbler
Warbler needs to be told about any gems that your application uses. To generate a Warbler configuration file:
jruby -S warble config
The file is generated at config/warble.rb. In it, modify the following sections:
config.gems = ["activerecord-jdbcmysql-adapter"]
...
config.webxml.jruby.min.runtimes = 2
config.webxml.jruby.max.runtimes = 4
Build and deploy the .war
jruby -S warble
asadmin deploy --contextroot / blog.war
(--contextroot / makes the application rooted at / in the server, rather than at /blog which would be the default.)
At this point, the blog application is up and running on port 8080. I had previously been running the blog with an Apache/.htaccess-based setup reverse-proxying to mongrel, so all I had to do was change the port. I haven’t touched it since.
But is this right for you?
Chances are, this setup is overkill for a simple blog. If you’re going to try it, I’d recommend at minimum running on a VPS with at least 1G of memory. But once you get the core pieces in place, updating and re-deploying the application is really just as simple as the last two commands. It’s mundane and boring in its simplicity. But boring is good when you don’t want to worry about having to keep Mongrel running, or max out the memory in your server and make it unstable.
Tags jruby, rack, rails, warbler | 5 comments
Posted by Nick Sieger
Wed, 04 Jun 2008 21:57:00 GMT
The long-delayed and much-awaited 0.8.1 release is here. It fixes
quite a few reported bugs (but not quite all). See the full changelog
below for details.
Please help me make more frequent releases of ar-jdbc by submitting patches against the trunk of
activerecord-jdbc. Test cases appreciated also.
File bugs in JRuby’s JIRA. Use the “ActiveRecord-JDBC” component when filing them.
You can check out the source here:
svn co http://jruby-extras.rubyforge.org/svn/trunk/activerecord-jdbc
git clone git://github.com/nicksieger/activerecord-jdbc-adapter
Recently I started keeping a mirror of activerecord-jdbc-adapter on Github. Feel free to watch or fork and send me patches via git format-patch as well.
One of the cool new things in this release is a JDBC version of sqlite3 using the Zentus Sqlite JDBC driver. The crazy thing is how the driver is created. The original sqlite3 codebase is cross-compiled to MIPS and the resulting output coverted to Java bytecode using NestedVM! The adapter is still in its early stages, but basic stuff seems to be working. Try it out using jruby -S gem install activerecord-jdbcsqlite3-adapter.
Let me know how the release works for you!
Changes in 0.8.1
- Now sporting a JDBC sqlite3 adapter! Thanks Joseph Athman.
- Added support for InterSystems Cache database (Ryan Bell)
- Fix for JRUBY-2256
- JRUBY-1638, JRUBY-2404, JRUBY-2463: schema.table handling and Oracle
NUMBER fixes (Darcy Schultz & Jesse Hu)
- Add structure dump and other DDL-ish for DB2 (courtesy abedra and
stuarthalloway)
- Fix missing quotetablename function under Rails 1.2.6 and earlier
- Small tweaks to jdbc.rake to select proper config
- JRUBY-2011: Fix MSSQL string un-quoting issue (Silvio Fonseca)
- JRUBY-1977, 17427: Fix information_schema select issue with MSSQL (Matt Burke)
- 20479: Improve gettablename for MSSQL (Aslak Hellesøy)
- 20243: numerics improvements for MSSQL (Aslak Hellesøy)
- 20172: don’t quote table names for MSSQL (Thor Marius Henrichsen)
- 19729: check for primary key existence in postgres during insert
(Martin Luder)
- JRUBY-2297, 18846: retrying failing SQL statements is harmful when
not autocommitting (Craig McMillan)
- 10021: very preliminary sybase support. (Mark Atkinson) Not usable
until collision w/ sqlserver driver is resolved.
- JRUBY-2312, JRUBY-2319, JRUBY-2322: Oracle timestamping issues
(Jesse Hu & Michael König)
- JRUBY-2422: Fix MySQL referential integrity and rollback issues
- JRUBY-2382: mysql string quoting fails with ArrayIndexOutofBoundsException
Tags jruby, rails | no comments
Posted by Nick Sieger
Thu, 08 May 2008 17:31:00 GMT
Continuing in the spirit of Conference-Driven Development, I’m happy to announce the first public release of JRuby-Rack! You can use it to run Rails, Merb, or any Rack-compatible application inside a Java application server.
Also released today is Warbler 0.9.9, which has been updated to bundle JRuby-Rack.
In addition to providing as seamless a connection as possible between the servlet environment and Rack, JRuby-Rack (along with Warbler) is also bridging the gap between Ruby and Java web development. Some of the things it does are:
- Makes the Java servlet context and servlet request available to Ruby through special variables in the Rack environment
- Servlet request attributes from Java are passed through and available in the Rack environment. Request attributes can override Rack variables such as
PATH_INFO, QUERY_STRING etc.
- Configures Rails deployment options such as page caching directories and session handling automatically and optimally for the servlet environment.
I’ve also included the beginnings of some extensions that should help integrate Rails with existing Java web frameworks, servlets, JSPs, and other code. For example, you can invoke a Rails request from within a JSP with a tag:
<jruby-rack:rails path="/projects/activity" params="layout=none"/>
You can set servlet and session attributes and forward to other servlets and JSPs from your Rails controllers:
class DemoController < ApplicationController
def index
servlet_request["hello"] = "world!"
session["rails"] = "Visible to java!"
forward_to "/attributes.jsp"
end
end
and read them from within the servlet or JSP:
<dl>
<dt><tt>servlet_request["hello"] | request.getAttribute("hello")</tt></dt>
<dd><%= request.getAttribute("hello") %></dd>
<dt><tt>session["rails"] | session.getAttribute("rails")</tt></dt>
<dd><%= session.getAttribute("rails") %></dd>
</dl>
This is just the beginning of this kind of integration, and I’m interested where people take it. I think this provides a nifty way to start integrating Rails bits into existing applications or reuse existing Java web application code.
I’ve tagged the release with an 0.9 version number. I believe the bits are ready for serious use, but could use some help pounding out a few more bugs before calling it 1.0. So jruby -S gem install warbler today, try it out, and bring plenty of feedback to the JRuby user list!
Tags jruby, merb, rack, rails | 17 comments
Posted by Nick Sieger
Thu, 17 Jan 2008 23:48:00 GMT
Next up in our performance series: Builder::XChar. (Another fine Sam Ruby production!) While this piece of code in the Builder library strikes me as perfectly fine, it also tends to slow down quite a bit with larger documents or chunks of text.
Our path to the bottleneck is as follows: ActiveRecord::Base#to_xml => Builder::XMLMarkup#text! => String#to_xs => Fixnum#xchr. Consider:
require 'rubygems'
gem 'activesupport'
require 'active_support'
require 'benchmark'
module Benchmark
class << self
def report(&block)
n = 10
times = (1..10).map do
bm = measure(&block)
puts bm
bm
end
sum = times.inject(0) {|s,t| s + t.real}
mean = sum / n
sumsq = times.inject(0) {|s,t| s + t.real * t.real}
sd = Math.sqrt((sumsq - (sum * sum / n)) / (n - 1))
puts("Mean: %0.6f SDev: %0.6f" % [mean, sd])
end
end
end
page = File.open("page.xml") {|f| f.read }
Benchmark.report do
20.times { page.to_xs }
end
On Ruby and JRuby, this produces:
$ ruby to_xs.rb
21.430000 0.400000 21.830000 ( 22.022769)
21.530000 0.360000 21.890000 ( 22.005737)
21.540000 0.370000 21.910000 ( 22.065165)
21.530000 0.370000 21.900000 ( 22.028591)
21.500000 0.350000 21.850000 ( 21.990395)
21.550000 0.370000 21.920000 ( 22.033164)
21.520000 0.360000 21.880000 ( 21.984129)
21.550000 0.370000 21.920000 ( 22.116802)
21.550000 0.370000 21.920000 ( 22.051421)
21.520000 0.380000 21.900000 ( 22.084736)
Mean: 22.038291 SDev: 0.041985
$ jruby -J-server to_xs.rb
79.112000 0.000000 79.112000 ( 79.112000)
81.480000 0.000000 81.480000 ( 81.481000)
84.745000 0.000000 84.745000 ( 84.745000)
84.384000 0.000000 84.384000 ( 84.384000)
121.933000 0.000000 121.933000 (121.933000)
85.533000 0.000000 85.533000 ( 85.532000)
82.762000 0.000000 82.762000 ( 82.763000)
82.090000 0.000000 82.090000 ( 82.090000)
81.298000 0.000000 81.298000 ( 81.299000)
80.774000 0.000000 80.774000 ( 80.773000)
Mean: 86.411200 SDev: 12.635700
(Hmm, I must have accidentally swapped in some large program in the middle of that JRuby run. The perils of benchmarking on a desktop machine. I don’t claim that the numbers are scientific, just illustrative!)
Fortunately, the fix again is very simple, and has previously been acknowledged. The latest (unreleased?) Hpricot has a new native extension, fast_xs, which is an almost drop-in replacement for the pure-ruby String#to_xs. (Almost, because it creates the method String#fast_xs instead of String#to_xs. ActiveSupport 2.0.2 and later take care of aliasing it for you). Unbeknownst to me, I ported fast_xs recently as part of upgrading JRuby extensions that have Java code in them. And so it happens to come in handy at this time. The patch for that is here.
I have the latest Hpricot gems on my server, so you can install it yourself (for either Ruby or JRuby):
gem install hpricot --source http://caldersphere.net
or
jruby -S gem install hpricot --source http://caldersphere.net
With that installed, the script now produces these results:
$ ruby to_xs.rb
0.460000 0.080000 0.540000 ( 0.537793)
0.420000 0.070000 0.490000 ( 0.501965)
0.430000 0.070000 0.500000 ( 0.501359)
0.400000 0.070000 0.470000 ( 0.484495)
0.400000 0.070000 0.470000 ( 0.479995)
0.400000 0.070000 0.470000 ( 0.469118)
0.390000 0.070000 0.460000 ( 0.468864)
0.390000 0.070000 0.460000 ( 0.465009)
0.390000 0.060000 0.450000 ( 0.452902)
0.390000 0.070000 0.460000 ( 0.466881)
Mean: 0.482838 SDev: 0.024926
$ jruby -J-server to_xs.rb
0.882000 0.000000 0.882000 ( 0.883000)
0.832000 0.000000 0.832000 ( 0.832000)
0.851000 0.000000 0.851000 ( 0.850000)
0.837000 0.000000 0.837000 ( 0.837000)
0.846000 0.000000 0.846000 ( 0.846000)
0.843000 0.000000 0.843000 ( 0.843000)
0.835000 0.000000 0.835000 ( 0.835000)
0.825000 0.000000 0.825000 ( 0.826000)
0.830000 0.000000 0.830000 ( 0.830000)
0.834000 0.000000 0.834000 ( 0.833000)
Mean: 0.841500 SDev: 0.016379
Tags jruby, performance, rails, ruby | 3 comments
Posted by Nick Sieger
Tue, 06 Nov 2007 15:00:00 GMT
Just out is ActiveRecord-JDBC 0.6, the post-RubyConf release.
The sparkly new feature is Rails 2.0 support. In the soon-to-be-released Rails 2.0 (edge), Rails will automatically look for and load an adapter gem based on the name of the adapter you specify in database.yml. Example:
development:
adapter: funkdb
...
With this database configuration, Rails will attempt to load the activerecord-funkdb-adapter gem, require the active_record/connection_adapters/funkdb_adapter library, and call the method ActiveRecord::Base.funkdb_connection in order to obtain a connection to the database. (This is the mechanism used to off-load non-core adapters out of the Rails codebase.)
We can leverage this convention to make it easier than ever to get started using JRuby with your Rails application. So, the first thing new in the 0.6 release is the name. You now install activerecord-jdbc-adapter:
jruby -S gem install activerecord-jdbc-adapter
But wait, there’s more! We also have adapters for four open-source databases, including MySQL, PostgreSQL, and two embedded Java databases, Derby and HSQLDB. And, for your convenience, we’ve bundled the JDBC drivers in dependent gems, so you don’t have to go hunting them down if you don’t have them handy.
Check this out. Get a fresh copy of JRuby 1.0.2, unpack it, and add the bin directory to your path. Install the adapter:
$ jruby -S gem install activerecord-jdbcderby-adapter --include-dependencies
Successfully installed activerecord-jdbcderby-adapter-0.6
Successfully installed activerecord-jdbc-adapter-0.6
Successfully installed jdbc-derby-10.2.2.0
In your Rails application, freeze to edge Rails (soon to be Rails 2.0).
rake rails:freeze:edge
Re-run the Rails command, regenerating configuration files.
jruby ./vendor/rails/railties/bin/rails .
Currently, Rails 2.0 uses openssl for the HMAC digest used in the new cookie session store, so we have to install the jruby-openssl gem:
jruby -S gem install jruby-openssl
Now, update your config/database.yml as follows:
development:
adapter: jdbcderby
database: db/development
Re-run your migrations, and you should now see a Derby database footprint in the db/development directory.
$ ls -l db/development
total 24
-rw-r--r-- 1 nicksieg nicksieg 38 Nov 6 08:24 db.lck
-rw-r--r-- 1 nicksieg nicksieg 4 Nov 6 08:24 dbex.lck
drwxr-xr-x 5 nicksieg nicksieg 170 Nov 6 08:24 log/
drwxr-xr-x 65 nicksieg nicksieg 2210 Nov 6 08:24 seg0/
-rw-r--r-- 1 nicksieg nicksieg 882 Nov 6 08:24 service.properties
drwxr-xr-x 2 nicksieg nicksieg 68 Nov 6 08:24 tmp/
That’s it! To re-emphasize, to make your application run under JRuby, no longer will you need to a) find and download appropriate JDBC drivers, b) wonder where they should be placed so that JRuby will find them, or c) make custom changes to config/environment.rb. All that’s taken care of you if you use one of the following adapters:
activerecord-jdbcmysql-adapter (MySQL)activerecord-jdbcpostgresql-adapter (PostgreSQL)activerecord-jdbcderby-adapter (Derby)activerecord-jdbchsqldb-adapter (HSQLDB)
If you need to connect to a different database, you’ll still need to place your database’s JDBC driver jar file in the appropriate place and use the straight activerecord-jdbc-adapter. Also note that in this case, and for Rails 1.2.x in general, you’ll still need to add that pesky require statement to config/environment.rb.
As always, there are bug fixes too (though we haven’t been tracking exactly which ones are fixed). We’re starting to file ActiveRecord-JDBC bugs in the JRuby JIRA now, and will be putting in future AR-JDBC versions to target soon too. So, please file new bugs in JIRA (and select component “ActiveRecord-JDBC”) rather than in the antiquated Rubyforge tracker.
Tags activerecord, jruby, rails | 9 comments
Posted by Nick Sieger
Thu, 25 Oct 2007 03:36:00 GMT
People have been asking for a while how fast JRuby runs Rails. (Of course, “fast” has always been a relative term.) We haven’t been quick to answer the question, because frankly we didn’t know. We hadn’t been building real Rails applications on JRuby ourselves yet, and there was no definitive word from the crowd either.
Recently, several guys from ThoughtWorks have been working on a Rails petstore application and benchmark to get to the heart of the matter. Discussion has been heated on the JRuby mailing list, but results have not been conclusive yet.
In the project I’m working on, we’ve committed to using and deploying on JRuby. Eventually we were going to reach the point where we’d need to find out how well our application runs. So today I began running a simple single request benchmark on a relatively busy page. The numbers turned out to be rather surprising:
(The raw data is available here.)
Now, MRI (C Ruby) will always run about the same speed no matter how many runs you give it, but it’s well known that the JVM needs time to warm up. And indeed it does; after 250 iterations, Mongrel running on JRuby finally surpasses MRI. The JRuby/Goldspike/Glassfish combo comes close as well.
Some details about the setup:
- I ran the tests on my MacBook Pro Core 2 Duo 2.4 GHz. I didn’t disable one of the cores for the tests, which means that JRuby has an advantage over MRI because it can use both (native threads at work). However, the test script ran the requests serially, which means that the advantage was minimal.
- The application is indeed of the “hydra” variety; the setup is nearly identical to the second diagram on that page. So a single request is passing through not one, but two Rails applications in addition to touching the database. It rendered an HTML ERb view with data from an ActiveResource-accessed RESTful service. The applications are based on Rails 1.2.3.
- MRI version is using Ruby 1.8.6 and Mongrel 1.0.1.
- JRuby Mongrel is also version 1.0.1 (details on installing it here)
- JRuby on Glassfish used Glassfish 2 and Goldspike 1.4, deployed in war files via Warbler.
- The two JRuby setups used JDK 1.5 and were tweaked to disable ObjectSpace and use the “server” VM (-server argument to the JVM).
The main point I wish to make with these numbers is that JRuby performance is there today, and still has room to grow. There’s no longer any doubt in my mind. Yes, this is a simplistic application benchmark run on a developer’s machine, but it’s a real application. The test may not be exacting in precision, but I see enough in the numbers to believe that this will be replicable to production environments. The plot thickens!
Tags jruby, rails, ruby | 1 comment
Posted by Nick Sieger
Fri, 14 Sep 2007 05:33:00 GMT
Speaking of keeping busy, I’ll be speaking alongside my colleague Craig McClanahan at RailsConf Europe in Berlin next week.
Sun is a Diamond Sponsor at RailsConf again, just like in Portland last May. Part of that sponsorship money pays for a brief keynote spot (filled by Craig) as well as a session or two. So no, I didn’t get my spot through an accepted proposal submission, but that doesn’t mean that the session is going to be a big marketing shill.
No, actually Craig and I are part of a small group at Sun that’s embracing Rails in a big way, and we’re going to be launching a site built mostly on Rails later this fall. We’re taking what we think are some novel approaches to building a Rails-based application and we thought we’d share some of those thoughts with you rather than drone on for the session about how great Sun is and what snazzy tools we make. (Although expect to see a subtle plug or two for Sun hardware and tools. Call it product placement rather than overt selling.)
I titled the session “Rails Hydra” because the central idea of the structure of our application is not one Rails app, but many. The UI and views don’t even talk to a database; instead they make use of ActiveResource and RESTful web services, talking to the models living in other Rails applications in the backend. One key point is we’re deploying .war files to JRuby running on Glassfish, thus avoiding headaches of morbidly multiplying Mongrel math. We’ll elaborate on this arrangement and talk about some of the other tools and tricks we’re using.
Also, Charlie, Tom and Ola will be there, so we’ll certainly have a JRuby summit at some point. Stop by and say hello!
Posted in java | Tags gig, rails, railsconf | 1 comment
Posted by Nick Sieger
Tue, 04 Sep 2007 02:48:40 GMT
This week I was working on integrating the latest JRuby 1.0.1 and Goldspike 1.3 releases into our environment, when my frustration hit a fever pitch.
See, I had always thought that the .war packaging side of Goldspike was a little clunky and
un-ruby-like, but I didn’t see a clear path to fixing it. I had heard little complaints about it here
and there: the little configuration DSL didn’t give you enough control or wasn’t documented well
enough; the fact that it downloads libraries from the internet during assembly (convenient, but not
safe or reproducible for production deployments).
Also, in my own opinion it took the wrong approach to packaging Rails in a .war file. It puts the
Rails application directory structure into the root of the .war file where any web server or Java
application server might mistakenly serve up your code as static content. The Java .war file spec has
this special directory called WEB-INF expressly for the purpose of hiding that stuff away, so why
not use it?
And then, suddenly Goldspike was packaging up my entire Rails application directory, .svn directories
and everything. So I set out to fix this once and for all.
And so I present Warbler. A little bird who chirpily steps up to the task of assembling your
Rails application into a Java Web Archive (.war). Here, get it:
gem install warbler
And then, in the top directory of your Rails application,
warble
Those two steps are all it takes to make a .war file, including your application and recent versions
of JRuby and Goldspike, that’s deployable to your favorite Java application server.
There are a number of points about Warbler worth mentioning.
Does one thing, well
Warbler only packages, and doesn’t care about anything else, like how to dispatch servlet requests to
Rails. This will allow for more runtime servlet binding mechanisms to take advantage of Warbler in
the future.
Fast and lightweight
50% less code than the Goldspike packaging plugin, yet does the job quickly and efficiently.
Sane defaults
Warbler only packages code that you need to run the application, omitting database migrations and
tests. If your application is self-sufficient (no external dependencies), then the out-of-the-box
configuration will probably work for you. Public HTML/images/javascript/stylesheets go in the root of
the webapp, where Java webservers expect them to be.
Documented, flexible configuration
Need to customize your configuration? Run warble config and edit config/warble.rb. All the
options are there, commented and documented.
Need to change out the bundled JRuby/Goldspike versions? warble pluginize makes a copy of Warbler
in the vendor/plugins area of your application, allowing you to change the .jar files in the
vendor/plugins/warbler-0.9/lib directory. Warbler then makes his nest in your project’s list of
rake tasks (as rake -T | grep war shows)
rake war # Create trunk.war
rake war:app # Copy all application files into the .war
rake war:clean # Clean up the .war file and the staging area
rake war:gems # Unpack all gems into WEB-INF/gems
rake war:jar # Run the jar command to create the .war
rake war:java_libs # Copy all java libraries into the .war
rake war:public # Copy all public HTML files to the root of the .war
rake war:webxml # Generate a web.xml file for the webapp
Warbler even omits himself in the .war file produced when running in plugin mode, since you won’t
need him at runtime. It’s the little details that matter.
Give him a try and let me know if it makes your life deploying Rails applications to JRuby on Java appservers easier!
Tags jruby, rails, warbler | 13 comments
Posted by Nick Sieger
Wed, 23 May 2007 05:51:36 GMT
I was fortunate to be in town right after RailsConf and attended the inaugural geekSessions event on Rails scalibility. The event went off without a hitch: it was well attended, City Club is a classy place, and there was decent food and an open bar. I don’t know the SF geek/startup scene, but pretty much all of the few guys I know were there along with a ton of other folks. My only complaint would have been to let it run at least 30 minutes longer. Socializing was good too, but it seemed like the conversation was just getting started.
Here are some notes for you in my typical rapid-fire style – hope they’re useful to you.
Ian McFarland
Case study: divine caroline
Servers:
- Load balancer
- Apache + mongrel
- MySQL
- SOLR
Ruby is slow. Rails is slow. Unoptimized app was slow – 7 pages/sec with ab. So how can Rails possibly be? 150 pv/s with a simple text render. This formed a sort of upper-bound, that ruled out fragment/action/partial caching, etc. This brought the throughput to 3500 pv/s. Except for page caching limitations:
- Cache coherency
- Writes are more expensive
- Page caching is not applicable to as many pages as you think
But measure first. Pivotal built a drop-in page caching extension to deal with cache coherency issues (soon to be at http://rubyforge.org/projects/pivotalrb)
Jason Hoffman
Jason somehow has the distinction of the first four commits in the Rails repository. Joyent/TextDrive/Strongspace.
If your application is successful, you’re going to have a lot of machines. What happens when you have 1000s of machines, 100s of TB, 4 locations, etc. Is this really a Rails issue? In a typical Joyent setup, Rails is only one of 26+ processes on the server stack. So scaling it really doesn’t mean much more than scaling any application. Object creation in Ruby is fast, sockets and threads are slow. So forget sockets and threads.
Instead, use DNS, load balancers, evented mongrels, JRuby/Java, DBMSes (not just RDBMS; LDAP, filesystem, etc.), Rails process doing Rails only, static assets going through a static server, federate and separate as much as you can.
Jeremy LaTrasse
Jeremy’s job is about safety nets; about knowing the underlying infrastructure. Is the hardware/OS/stack important? Can you build safety nets around those so that you can spare cycles when you need to intrude into the system to troubleshoot?
Twitter is in a unique position with the volume of traffic to be able to find some pretty tough bugs, like the recent backtrace issue.
Bryan Cantrill
Measure first! Like Ian said. Is software information? Or a machine? It’s both. Nothing else in human existence can claim this. 3 weeks after Bryan joined Sun, he was working with Jeff (ZFS architect) debugging an issue when Jeff retorted, “Does it bother you that none of this exists? It’s just a representation of some plastic and metal morass in a backroom” (slightly paraphrased).
We’ve been living with bifurcated code – “if DEBUG; print something” ad nauseum. But this has a cost. So dev code deviates from production code. But we can’t get the data we want, where it matters, in production. Bryan goes on to describe the aforementioned backtrace issue and how it saved Twitter 33% CPU. So don’t pre-optimize, but you’ve got to be prepared to go get the data. In production.
Q & A
What’s the best way to move from one database to two databases (MySQL), when you scale past the volume of reads that overwhelms one?
Jason doesn’t like the replication approach, it’s not fault tolerant. Reference to Dr Nic’s magic multi-connections gem. Reference to acts_as_readonly. Don’t rely on things that are out of your control, start reading/writing to multiple locations, at the application level. Jeremy: So do you want to be in the business of writing SQL or C extensions to Rails? What about MySQL proxy? Seems ok, but I might not trust it in production. MyTop/InnoTop will tell you about your query volume.
Virtualization: 4 virtual servers w/ web servers on top of a single physical server? Why?
Jason: Free BSD 4.9 on early pentium was the perfect balance of utilization. 18 CPUs by 64G RAM with virtual servers gets us back to that level of utilization. Bryan: Not all virtualization solutions are equivalent! (Solaris containers/zones plug.)
RDBMSes are not good for web applications? Why? Can you give some examples?
Jason: It depends on when you want to join. When people are clicking, or pre-assembled. Look at your application and put the data together before people request it. Why does YouTube need an RDBMS? It serves a file that people can comment on.
Mention of Dabble DB, ZFS, Jabber, Atom, Atom over Jabber, etc. as ways of innovative ways of storing objects, data, etc. GData/GCal most certainly does not store its Atom files in an RDBMS.
Sell Rails apps and have the customer deploy it? What options are available?
Ian: JRuby on Rails with a .war file is an interesting approach. What operational issues/ways to help with scaling remote deployments? Jeremy: Log files are the first line of defense. Jason: Corporate IT are comfortable with Java.
The pessimist in me says that my servers are going to fall over after 5 users. How can I be prepared/not be optimistic about a traffic spike?
Ian: Load test the crap out of the app. Find out the horizontal scaling point. Use solutions like S3 for images. Make sure you can scale by throwing hardware at it. Eventually single points of failure will overcome you (such as a single database), but you can wait until you get to that point before doing something about it.
Jason: You can benchmark your processes, and get an idea of what they can do. Most people that want to do something will be look at your stuff, and maybe signup. So front-load and optimize your signup process, possibly by taking it out of Rails.
Jeremy: Conversations with Zed, DHH, etc. have pointed out that sometimes “Rails isn’t good at that, take it out of Rails.” Same thing for the database. Split those things out into a different application.
Bryan: Do your dry land work, know your toolchain, so that when the moment comes, you can dive in and find the problem.
We have a migration that takes a week to run because of text processing. GC was running after every 10th DB statement. Used Rails bench GC patch to overcome the issue with the migration. Any issue running these?
Jason: We run those GC modifications and a few more in production, and they’re fine.
Most comversations revolve around items like database is slow, or Ruby is slow. How can we use DTrace to streamline the process?
Jeremy: We spent 20 minutes over lunch (plus some preparation) to find a Memcache issue. It’s worth it to spend a little time to learn the tool.
Bryan: “Awk is God’s gift to all of us.” When DTrace was being reviewed inside of Sun, folks commented “This reminds us of awk.” “Thanks!”
Jason: We’re putting a tracing plugin in Rails as a remote process to collect data from a running app. Apple has shown a commitment to get this in Leopard. Textual and graphical output are possible. I believe in DTrace a lot, and the tooling and documentation will go beyond its current state of an experts tool.
Lastly, what one closing thing would you like to say about Rails scalability?
Ian: Measure.
Jason: Don’t use relational databases.
Jeremy: I thought it was a Joyent sales pitch.
Bryan: Use DTrace (with Joyent accelerators of course).
Tags rails, ruby | 2 comments | no trackbacks
