Nick Sieger: Tag rubyconf2006

RubyConf: Graphics with Ruby

Nick Sieger — Fri, 20 Oct 2006 20:18:00 +0000

Geoff Grosenbach is waxing on pagefuls of numbers condensed into a small, tidy graph that increases the amount of information you can communicate on a page.

The cartoon fox makes his second appearance of the day in one of Geoff’s sparklines

It is in our hands, the hands of the programmer, to show designers what the range of visual representation capabilities are.

Libraries

Scruffy -- SVG graphs, no dependencies (i.e., RMagick)
GNUPlot -- the old standby
MRPlot -- scientific plots
PNG -- line and font drawing in pure ruby from Seattle.rb
Gruff -- depends on RMagick
Sparklines -- depends on RMagick
[Ploticus and RRD were mentioned during the talk as well]

Applications

Automatically generating image mastheads with a font mask, a gradient and a cloud image showing through the mask
UrbanDrinks.com -- plotting bars on a timeline showing happy hours in Portland
“Scene graph” -- rendering multiple layers of images/icons on the filesystem into a composite graphics “scene”
BillMonk.com -- rendered image with multiple components that allows you to circumvent cross-site or crippled javascript issues
In a Rails controller:
- Generate and cache (with caches_page) an image with text
- Register a new mime type and use responds_to {|type| type.jpg { ... } }
Requisite reference to Edward Tufte

Techniques

Comparisons -- show two competing trendlines on a graph
Multivariate analysis -- stockhive.com stock chart rendering
Content is king -- be judicious

RubyConf: Matz Roundtable

Nick Sieger — Sat, 21 Oct 2006 02:59:38 +0000

An annual tradition at RubyConf is the “Roundtable”, where any member of the audience can come up and ask Matz any question. Transcript follows with partial paraphrasing.

Q. Will Ruby ever get the features of evil built in? Will we get become? No.

Q. If Symbol is a “frozen” String, why do we need Symbol? I don’t have a complete answer to that question -- following tradition.

Q. What’s the most unusual architecture Ruby has ever run on? Ruby...I can’t think of one...JRuby? Some guys ran compiled Ruby on the NECS supercomputer. [From the audience] Symbian.

Q. As Ruby gains acceptance, it becomes more resistance to change. How do you keep agile while gaining momentum? We have forked off 1.8, so if you want stable, use Ruby 1.8 forever.

Q. The array patch -- is there any chance of a backport for those of us who want 1.8 forever? Could be, but the current patch has a bug in 1.9.

Q. Why isn’t YARV the only VM for 1.9? [Comment about register-based VM] I’m not sure whether stack-based or register-based VMs are better...[THAT GUY!]...[Given the hook]

Q. It seems like momentum is based on strong metaprogramming facilities. With 1.9 there are more user-friendly features. Will code eventually be data, or are we just patching on techniques on a tool that is being used more heavily? An extreme way to do metaprogramming is to use Lisp, but for practicality we have to start somewhere. I’m not sure that s-expressions are the way to go. We’re not going to have macros. It will remain similar to what we have in 1.9. The ability to serialize code would be really powerful -- will that happen? Disclosing internal state is tricky because it changes. It could be a platform-specific feature.

Q. Recently for 1.9, there was a patch for Array and String to be in the object header. Was this profiled? It seems like it would be an incredible performance problem. It was done for slowness of malloc and gc. In some cases it could run slower, but it benchmarked at about 5% faster. It should be tested thoroughly.

Q. Ruby 2.0 will not have green threads or continuations, correct? Are there specific reasons for that decision? It is difficult to implement green threads in YARV.

Q. People want to see development move quickly, but may not be able to contribute code. Can you suggest ways for those programmers to contribute? Submit patches to the list. What is the obstacle to joining us? [Question restated] Submit testing, submit documentation. Contribute to RubyCentral? (looks at Chad Fowler)

Q. The CSV library is useful, but slow. Can we see FasterCSV in stdlib? It’s ok to replace the csv library with FasterCSV, as long as the compatibility issues are resolved.

Q. I want to thank you because I love writing in Ruby every day. I tried to write Python first. Recently an unrelated upgrade broke my Python apps. Will we be able to manage multiple versions of Ruby? You’ll have to have two versions, 1.8, and future ones.

Q. What would you say is the most important feature of your personality that has given you success? Endurance? Can you elaborate? It’s easy to design a language, many of them disappear in a year or two. I’ve been working on Ruby for 13 years.

Q. I wanted to see in IRB how the class was implemented at runtime. (He means an uneval feature) [Audience member (Eric Hodel?) mentioned ParseTree]

Q. If there wasn’t Ruby, what would you be programming in? Some language of my own not named Ruby.

Q. Do you still look at RCRchive? Yes. I’ll talk about it in the keynote.

Q. I would like a feature abstraction that would allow Ruby to fork itself rather than actually forking. Can you see this added to core? We have to define the behavior of that feature. We have to have a good name for the method. Then there will be no problem to add it to core.

Q. Why do you need to bind methods to an object of the same original module? If you bind a method taken from String class to Array class, it will crash, and we need to prevent it.

Q. What do you think of Ruby.NET? I’m pretty open to new implementations. Should there be a standard (language specs) that other projects should follow? Charles started work for a written 1.8 spec, and I’ll help him with that if I have time.

Rubyconf Wrap-Up

Nick Sieger — Tue, 24 Oct 2006 22:51:55 +0000

Whew! Back home from my first RubyConf, it’s taken me a couple days to collect some parting thoughts. As you might have noticed, I was pretty busy last weekend.

First of all, what an awesome and welcoming community. It’s going to sound cliché, but there are so many intelligent and motivated people walking around that you can’t help but be inspired to roll up your sleeves and get your hands dirty.

There were definitely some high points for me. The beauty and power of the language, even after using it for almost two years, still amazes me. Pretty much every piece of code I saw, whether in a presentation or looking over someone’s shoulder, had a clear purpose and communicated its intent better than any general-purpose machine language I have seen. The simplicity of Evan’s new Ruby-in-ruby VM, the syntax integration tricks of John’s RubyCLR project, the forthcoming RubyOSA APIs, and Geoffrey’s graphics programs, are all great testaments to Ruby’s power.

There was an implementer’s summit on Friday night, which I attended (see also coverage here and here). There are now at least 8 active implementations of Ruby (Ruby, Yarv, JRuby, Cardinal, Rubinius, MetaRuby, Ruby.NET, IronRuby), and two interop bridges (RubyCLR and RubyCocoa)! The biggest news was that there are plans to revive the Ruby testing project (formerly the Rubicon) and share as many tests as possible among the implementations.

RejectConf was a huge success, due largely to the indefatigable Adam Keys. Kevin Tew has a decent wrap-up of the talks that occurred. Charlie’s demo of NetBeans in-place refactoring feature drew a couple oohs and ahs and even one f-bomb. Heckle, in time, should be an awesome tool as well. Big thanks to zenspider for coordinating it. It’s destined to become an annual tradition. Perhaps the organizers of future RubyConfs could account for it in the budget?

On a lighter note, there were quite a few humorous moments that kept popping up. A summary may read like a list of inside jokes, so here’s some context. THAT GUY is a reference to a disclaimer in Zed’s talk about the know-it-all guy who always pipes up during your talk with skepticism. THAT GUY kept getting called out during the rest of the conference. Ani, the developer evangelist from Microsoft was pretty thick-skinned. She was heckled constantly about MS, Vista, and everything else, and still kept a smile on her face. And of course you already watched Adam’s one-act play, right?

My note-taking streak wasn’t quite perfect; I didn’t take notes Kevin’s mkmf talk nor Rich’s talk about indi, and I slept in and missed Justin’s Streamlined talk. Also, the beer was flowing for RejectConf, and despite the quality summer of code talks, I was spent. Fortunately, you can fill in the blanks by following along with Curt Hibbs and the rest of the blogosphere. Thanks for tuning in, and I hope you got something worthwhile here. See you next year!

RubyConf: Rinda in the Real World

Nick Sieger — Sat, 21 Oct 2006 19:01:35 +0000

Speaker: Glenn Vanderburg

Rinda is a distributed coordination system, by Masatoshi SEKI, based on work by David Gelernter (called Linda). It’s similar to JavaSpaces, but more in the Ruby spirit. It uses DRb for communication (also by Seki-san).

There are several existing tutorials on Rinda, but none with a broad, real-world applicability. As Glenn started to go deeper with Rinda, flaws began to be exposed.

Rinda Basics

The TupleSpace is in the conceptual middle, as a bag of tuples “arrays”. Participants can write tuples into the space, or take them out (usually according to a set of conditions).
Tuples are usually requests to do some work by some unknown requester, or responses to that work.
A RingServer is a broadcast/multicast lookup service for finding Rinda tuplespaces.
Templates are used to find tuples to take. Template#match requires that the tuples have the same length, and that all the non-nil elements are triple-equal (===).
In practice arrays are always used in the #take and #write methods, and they’re implicitly converted to the appropriate object.

Protocol design

When you’re deciding what to store/read in the tuples, you’re essentially designing a communication protocol. So you need to take the extra precautions required as you would when designing any API or protocol, including documentation, evolution of changes, process workflow, etc.

Parts of tuples: command/request, identifiers, and associated parameters/data. Usually templates will match on the command, sometimes on the ID, but never on the data.
Strings work well in tuples, because you can use Regexps to select them.
Numbers work well, because you can match them with Ranges
Symbols don’t work so well (at least until Symbol < String)
Communication patterns -- synchronous communication is not a good fit for a Rinda architecture (e.g., a request for status)

Deployment

DRb does not marshal code/behavior (unlike Java and RMI). This is a limitation that forces you to consider how to use custom objects in the tuples, because the code must be shared (and thus jointly upgraded) across all participants.

Multiple processes are probably more reliable for the various components of the architecture (RingServer, TupleSpace, clients) rather than using green threads, although this is just a hypothesis.

The TupleSpace is the single point of failure -- if the process with it crashes, when it starts back up, it’s likely to have a new DRb URL, so it’s helpful to have a proxy around the tuple space in clients that can rediscover it if it crashes.

Be wary of multiple ring servers on the same subnet! Behavior may be unexpected. You may notbe using the tuple space that you wanted, and when the other ring server goes away, so does your tuple space.

DRb does not have any security built into it, such as unforgeable object IDs, encrypted transport, authentication, etc. This can be a problem in some situations.

There is no persistence by default, so consider adding some for crash resistance, or deal with occasional loss of tuples.

There are no transactions, so the requester will never know if it still processing or was lost.

As Ruby matures and gains exposure, some libraries that have been good enough for a while may need reconsideration. As a case in point, Rinda and DRb haven’t been updated since February of 2004.

Q & A

Q. Are write/take atomic? Yes, if you timeout, the tuple will still be there. Two workers cannot get the same tuple.

Q. Where would you use this instead of traditional message queues (reinvent the wheel)? There’s more flexibility in this architecture. (Justin Gehtland) In this particular case, they have a JMS backend, but the Ruby code could not assume connectivity to that backend.

Q. Is there a common correlator design pattern? Tie response types to request types to limit the pool of potential matches. In the situation where there is a single requester, it’s simple to just generate sequential numbers. In a P2P situation, you may need to use a hostname or PID to help distinguish. [Sounds like a GUID/UUID system to me.]

Q. Have you used expiry dates to grab abandoned tuples? No, sounds like an interesting possibility.

RubyConf: Your Ruby in My CLR

Nick Sieger — Mon, 23 Oct 2006 14:16:00 +0000

John Lam wanted to build a photo-flash-card application using Avalon and Indigo and Flickr, but also using Ruby as the implementation language. So along the way he decided to build an interop layer (a bridge) between Ruby and the CLR to do it.

Now that John has joined Microsoft, his new mission (bigger picture) is to further dynamic language implementations on the CLR.

Bridging type systems

Dynamic methods in the CLR allow you to do better than simply invoking the reflection API.

Ruby          |  C               |  CLR
============================================
shadow class  |  dynamic method  |  instance

Polymorphic inline caching -- caching method dispatches on different call sites based on the assumption that types don’t change that often
Generate shadow classes and method stubs using const_missing and method_missing
Overload resolution happens in the method shims (a one time cost) to choose, e.g., which constructor to use for System::Collections::ArrayList.new
Integration is done to make the CLR feel more Rubyish

Implementation

This changes identity (proxied object):
```
ArrayList.new.as(IEnumerable)
```
This is less Rubyish, but identity is preserved:
```
IEnumerable.get_enumerator(ArrayList.new)
```

There are trade-offs and warts to a bridge approach to Ruby integration on top of a platform such as the CLR: there is a need to inject artificial type information occasionally to be able to construct CLR objects (e.g., arrays -- Array.of(Int32).new(3)). Generics are evil! Simple stuff doesn’t seem so bad: List.of(Int32).new, but there’s more pain to be had (see John for details). John also built a RubyInline-like implementation for the CLR languages too, to allow for getting things done (even if it’s dirty). Finally, method overloading is a problem, especially when there is no equivalent Ruby type -- this gave way to instance_shim which is a sort of aliasing method that mixes in type metadata for use by the interop layer.

On the other hand, there are many places where Ruby (even in bridged mode) can make the experience of developing on the CLR better. Implementing CLR interfaces is a feature that allows Ruby objects to cross to the CLR side, (e.g., adding IEnumerable to Ruby Array). Performance across the CLR boundary (marshalling data) is ~100 times slower than C#, but still fast (3 million calls/second). Huge benefits are gained from using DSLs in Ruby to help with the implementation of the interop layer. Also, RubyCLR allows mixing in methods into CLR types, so we can re-skin APIs that feel clunky in Ruby. This is really leveraging the power of Ruby in the best possible way.

My take is that it looks like the RubyCLR project will probably not be seeing much further development, unless John finds a willing maintainer -- but this is speculation, I haven’t confirmed with John. Yet, the problem of impedance matching between type systems is a recurring theme in the dynamic language arena, and so John’s work is valuable in helping us to understand this issue.

More Info

RubyConf 2006 Begins

Nick Sieger — Fri, 20 Oct 2006 19:04:00 +0000

The RubyConf room is filling up this morning. I’ll be doing my best to live-blog the conference here so stay tuned!

RubyConf: History of Ruby

Nick Sieger — Fri, 20 Oct 2006 19:06:00 +0000

Takahashi-san is here to present on the history of Ruby, an apparently thankless task, because none of the other original Rubyists are historians. Takahashi is the co-author of two Japanese books on Ruby, Enjoy Ruby and Ruby Recipe Book. He also has the “Takahashi method” of presentation named after him. His talk presented an informative timeline of Ruby, the details of which were a bit tricky to capture. If I transcribed anything erroneously, please let me know.

Pre-history age

Born 24th of February 1993. Without code!
Matz and Keiju-san proposed the name first.
Thus one of the philosophies of Ruby came to be -- that the name of things matters. Matz: “I guess Ruby is cool”. Keiju: “I also like coral”. Matz: “oops”.

Ancient age

Ruby is in public -- release 21 December 1995 -- ruby-0.95.
ruby-list ML was launched. First mail: ruby-0.95 test failed. Subsequently 3 versions of Ruby were released in two days.
No CVS repository at the time. Anonymous CVS was to come in 1999.
25 December 1996 -- Ruby 1.0 released.
1 July 1997: Matz announces that Netlab hired him to be a full-time Ruby developer.
22 Septempber 1997: an article was published on Ruby -- the first article on the web about Ruby.
15 May 1998: RAA launched, maintained manually by Matz.
7 December 1998: Ruby home page was in English, but very simple.

Middle

Ruby is spreading in Japan during this time. The community is growing around Japanese programming language designers and programmers who do not understand English. Finally they have a tool that they can embrace and establish their own opinions and choices.
27 October 1999: Matz and Keiju’s book is published, the first Ruby book
More Ruby books would follow in 2001-2002 (~20 books -- a bubble). But the bubble popped in 2003.
4 November 1999: Ruby workshop
There were some Perl and Ruby/Perl conferences during this time also.
26 May 2001: YARPC -- Yet Another Ruby and Perl Conference
9 August 2003: Lightweight language (LL) -- lightweight language workshop (LL Saturday) in 2003. PHP, Perl, Ruby and Python were present. LL Weekend, LL Day and Night, and LL Ring would follow in 2004-2006.
LL Ring: 300 attendees talking about LLs in a real boxing ring.

Modern

Ruby spreads outside of Japan
16 Feb 2002 -- ruby-talk ML surpasses ruby-list ML.
ruby-talk was started in December of 1998, but the first posts are almost all Japanese authors writing in English.
SunWorld in Februrary 1999 has an article entitled “New choices for scripting” including Ruby.
February 2000: IBM Developerworks article on the “latest open source gem from Japan”.
InformIT article by Matz also in 2000.
15 December 2001: Programming Ruby by the Pragprogs (1st edition of the Pickaxe).
RubyConf.new(2001)
Ruby Kaigi -- first Japanese Ruby conference didn’t happen until 2006, it turns out only because of a dinner of Japanese rubyists at RubyConf 2005 decided that it would be fun.

Contemporary

Rails age -- the killer application for Ruby
We all know what happened, so we’ll skip this part.

RubyConf: Sydney and Rubinius

Nick Sieger — Fri, 20 Oct 2006 19:07:00 +0000

Update: Evan has posted code and has a page set up for the project.

Evan Phoenix (nee Webb), of Seattle.rb, is presenting on Sydney and Rubinius, an experiment in improving the ruby interpreter. Sydney has died, and Rubinius has risen from its ashes, appropriately.

Why

Why would you write a new Ruby interpreter? It’s fun, it’s a good challenge.
What’s wrong with the existing interpreter -- are you hating on Matz? Of course not.

Today’s Ruby interpreter is like a big dump truck -- sometimes a little slow, but it works for us. YARV is like the red, shiny fire truck. Both big and complex. Rubinius, by comparison, is like a dune buggy. Fast, light, but you’re going to get sand in your eyes if you drive it a lot.

The project, admittedly, is naive.

Simple architecture and implementation.
As little background magic as possible
No opaque C backend
Leverage axiom of simple == powerful
Less magic means more introspection
- More control for the developer
- Richer introspection: Backtrace, MethodTable objects

What was Sydney?

Giant patch to 1.8.2 that included reentrancy and thread-safety
Turned out to be a major PITA
CRuby uses a large number of C globals, references to which had to be tracked and fixed

Transition to Rubinius

Ruby borrowed a lot from Smalltalk, so why not try an implementation based on the same concepts?
Prototype A ported the blue-book implementation to Ruby
It worked and validated the basic concept and approach
Prototype B took ideas from A but implemented a bytecode interpreter and compiler. Used RubyInline to access raw memory operations.
At this time the goal emerged to have a translator which could take a prototype and bootstrap itself into C code.
Prototype S was a manual translation of Prototype B into C code to make the implementation quicker.
Prototype W was created to translate parts of Prototype B so that there is a maintainable core in Ruby code itself.

Questions

Q. Since you were starting over, could you use a platform-independent library to ease the process, such as APR? Yes -- currently using String and PointerArray from glib.

Q. How is performance? Too early to tell -- I hope to know by the end of the conference. Prototype S became runnable and usable on the plane here.

Q. Can you clarify the goal? To create a Ruby interpreter in Ruby that can translate itself out into a C interpreter.

Q. Have you figured out how to link in external libraries in a platform independent way? No. My hope is that the decision will be made to write a common framework for translating to system calls, e.g., SWT.

Q. Have you looked at PyPy? (similar project for Python) Yes, and it’s f-in complicated. It worries me actually.

Q. Could you have it generate backend code in another language/platform (Java bytecode, CLR)? Yes, I certainly hope so, otherwise I’m wasting my time.

Q. How will you add native thread support in a cross-platform way? I hope I won’t have to, by leveraging external tools.

Q. If you’re building a Ruby-to-C translator, why write a Ruby interpreter at all? If I didn’t, what would I translate? You still need some core engine to translate. Would it be a subset of Ruby? Yes.

Q. Looks very similar to Squeak, have you looked at Squeak code and talked to Squeak people? Looked at code a lot, I’ve really stolen all of their ideas. I haven’t talked to the folks yet because I’m afraid they might laugh at me.

Resulting Works

SydneyParser: Used parser from Sydney and stole ParseTree’s algorithm for generating a sexp that represents the Ruby code.
SegfaultProtection: detects a segfault in an extension, saves the Ruby interpreter, and raises a memory fault exception instead.

The Nitty Gritty (Red Pill)

All components separated by APIs for swappability
Garbage collector: baker two-space copy collector, and a train GC
Bytecode interpreter: small set of instructions driven by tests and need, so there are no extraneous operations
Compiler: written completely in Ruby, using ParseTree and SexpProcessor. Intended to compile itself to be used as a base compiler for Prototype S.

Future

Other backends -- Java, Smalltalk

RubyConf: Zed Shaw: Fuzzing

Nick Sieger — Sat, 21 Oct 2006 02:51:38 +0000

What is Fuzzing?

Throw random stuff at an app until it breaks, with maliciously crafted inputs. It’s the randomness, stupid.
Creating a model for an attack
Sampling, survival analysis and mean-time-between-failures (MTBF)

Limitations

Shallow -- errors come right away, but then few and far between
Not smart
Not always needed

Disadvantages

Destructive -- don’t do it against your production system!
Potentially expensive

Advantages

Fun making developers go insane
Finds bugs even in closed source software
Easy to do when you have the right tools
Excellent for regression/load/DDOS/pen testing
Large existing base of tools (links from Zed’s site)

Demo/Usage

Designed to be a simple data container for all the HTTP objects, so you can easily store and replay requests
Randomness engine (RC4 cipher) generates random bytes, numbers, chars, base64, etc.
Data collection (ten runs of ten samples, spits out .csv files)
Session management (dump cookie management)
Rails security test (see also my post on this subject)
Chunked encoding test
Mongrel test suite -- test GET vs. PUT to see if there is any difference in performance between the two methods

Other Ideas

Random ruby scripts from a grammar
Automatic random AR fixtures
Thrash functions for unit tests
Random thrashing of other protocols
Release RFuzz’s HTTP client separately as an alternative to net/http
Hpricot and RWB inclusion
Utu -- HCI research to see if it’s possible to measure how programmers interact

RubyConf: John Long: Radiant CMS

Nick Sieger — Sat, 21 Oct 2006 02:54:00 +0000

What is Radiant?

No-fluff, lightweight CMS for small teams
Simplicity over features
A little more than a blogging engine
Made for designers and programmers (techies)
Tag-based template language
Total control over output
Plugin extension mechanism under development
Content sites, not portal software

Getting Started

Install: gem install radiant
Generate a new application: mkdir demo && cd demo && radiant .
Configure database: cp config/database.sqlite.yml config/database.yml

Installation types

Instance mode vs. application mode (whether or not you have Rails source present). Instance mode also makes it possible to clone and share customize Radiant applications.
Base application includes an admin interface

Pages, Snippets and Layouts

Hierarchical page management
Create page in several states (draft, reviewed, published) in Textile or Markdown with slug, breadcrumb, and layout
Snippets are small chunks of content that can be shared between pages (to DRY up your content)
Layouts that can be broken down into components of the layout (sidebar, etc.)

Text Filters

Textile (RedCloth), Markdown (BlueCloth), SmartyPants
Vanilla HTML (no filtering)

Custom filters possible. [This is my own example below, not John’s.]

    class MyFilter < TextFilter::Base
  register 'profanity filter'
  def filter(txt)
    txt.gsub(/(damn|ass|shit)/i, '####')
  end
end

Radiant is powering the new ruby-lang.org site. Overall, a slick, well-thought out, polished, extendable CMS done very much in the philosophy of Ruby and Rails. Check it out!