Nick Sieger: Visualization of Ruby's Grammar

Comment on Visualization of Ruby's Grammar by Anonymous Coward

2007-01-29T23:06:00+00:00

Yes, what Brendan said. The biggest difference between these graphs is that some of them, like ANSI C, have straight lines of nodes, and some, like JavaScript, have loopy-looking “chains” involving about twice as many nodes. However, this is totally an artifact of the particular grammar used by the artist, and it doesn’t have anything to do with the languages C and JavaScript themselves. (That is, you could easily rewrite the C grammar to produce loopy chains, or the JavaScript grammar to produce straight lines.)

The culprits are JavaScript’s foo, fooNoIn, fooTail, and fooNoTail productions, where C just has foo. Rewriting the grammar would get rid of those productions, but it would add complications to the yacc file that don’t show up in the diagram.

In short, the diagrams may look pretty to some people (not me), but they don’t say anything about the underlying language standards, and they don’t even give much useful information about the artist’s particular grammar files.

Comment on Visualization of Ruby's Grammar by Brendan Eich

2006-12-22T22:42:58+00:00

occam: long paths simply mean precedence hierarchy, not “red tape” that obstructs programmers. Expressions using the minimal parenthesization work as in C (and Java, and …).

Another wrinkle that does not make red tape for users: the “noIn” parameterization of productions used by ECMA-262, to forbid “in” expressions in the head of “for” loops. This forks the expression grammar, and uglies up the graph considerably, but again it is no hardship for users.

I’ve noted in keynotes about JavaScript this year that I was under orders from Netscape management to “make it look like Java” in 1995. If I had my druthers, it would have looked more like Self, Logo, Smalltalk, or even HyperTalk. In that elseworld, the graph for JS’s grammer would be simpler, for sure.

Many languages claim C as an ancestor; lots of hackers know (most of) C’s syntax (and gcc nags us to overparenthesize when we mix bitwise and equality ops, since the precedence is misordered in that part of the grammar).

JS is one such C-derived language. Nothing much to do about it at this point. It has its pluses and minuses.

/be

Comment on Visualization of Ruby's Grammar by Joe

2006-11-28T20:18:10+00:00

And Tcl?

Comment on Visualization of Ruby's Grammar by occam

2006-11-26T07:52:03+00:00

With the Ruby graph, it seems to me that the congested primary means that there’s less context to remember about using various items in the language. So, more of the language is a “first class citizen”. In other words, you can do almost anything from anywhere in Ruby which is probably why it seems so elegant and powerful (and concise).

Javascript seems very ordered graphically, but also very obstructionist since there are some very long paths to get to certain items. Lots of order, but lots of red tape too.

Java seems like somewhere in the middle.

That’s my first intuition from these graphs.

Comment on Visualization of Ruby's Grammar by wert

2006-11-26T00:24:17+00:00

“But over time we seem to have become more accepting of this degree of complexity,”

No it is just that knowledgeable people are not heard anymore in all the noise that the fan-boys make. Take for example ruby it is a total piece of shit but only because there is a lot of noise about it do people think it is good.

Comment on Visualization of Ruby's Grammar by eric

2006-11-24T16:03:08+00:00

NICK, could you also do ada 2005, D, and possibly a C++ to compare with your c??:) thanks man, hope to see D and Ada 2005 especially:D!!

Comment on Visualization of Ruby's Grammar by Jules

2006-11-24T15:20:31+00:00

jer wrote: The simple fact that they have quotes for string literals or whatnot is just sugar and not strictly required.

Doh, but it’s in the language so it counts as extra syntax. You cannot implement string literals from inside the Io language (not 100% sure though).

Comment on Visualization of Ruby's Grammar by Harald Korneliussen

2006-11-24T07:04:54+00:00

I wonder what the Ada 2005 syntax tree would look like… or one of the newer Fortrans, or C++. Interesting that Ruby’s grammar isn’t LALR(1)/LL(1). That confirmed my suspicion that modern languages aren’t always. I think the first language I came across that made me wonder “Won’t this require a rather more complex parser?” was SML. But for all I know I was wrong there.

Comment on Visualization of Ruby's Grammar by jer

2006-11-24T00:32:55+00:00

Jules wrote: Io has more syntax than that (literals for example), and so has Lisp.

Literals in Io are messages just like any other, they’re not syntactically special. The simple fact that they have quotes for string literals or whatnot is just sugar and not strictly required.

Comment on Visualization of Ruby's Grammar by tdphuc@centrum.cz

2006-10-31T18:38:11+00:00

download

Comment on Visualization of Ruby's Grammar by Robert Feldt

2006-10-30T09:45:19+00:00

Very nice visualisations!

One thing that should be noted about Ruby’s grammar is that the lexer is (mildly) context-dependent. So the full complexity of the grammar(s) is not captured in these diagrams. Also this is why the Ruby grammar is not LALR(1)/LL(1) or some other of the simple grammar classes.

Comment on Visualization of Ruby's Grammar by rue

2006-10-30T04:47:41+00:00

Ed, take a peek at http://xruby.com/default.aspx. He supposedly has a full Ruby grammar written for ANTLR.

Comment on Visualization of Ruby's Grammar by Nick

2006-10-29T22:48:45+00:00

Rather than spend more time generating however many formats, I’ve uploaded the .dot files, so you can use GraphViz to generate whatever format you’d like.

ruby.dot c.dot java.dot ecmascript.dot python.dot

As Bil said, the originals are big enough to read the text, but possibly not hi-res enough for printing, if you want to do such a thing. Generating a pdf or svg from the dot file should serve you well in that respect.

Comment on Visualization of Ruby's Grammar by Bil Kleb

2006-10-29T21:00:29+00:00

JD and Robin: follow the image links to Flickr, and you’ll find hi resolution originals. –Bil

Comment on Visualization of Ruby's Grammar by Ed Borasky

2006-10-29T17:39:28+00:00

That is interesting … it seems like the existence of an IDE for Antlr grammars would be a good reason to start using Antlr as the tool set of choice for Ruby implementations … anyone have an contrasting opinion?

You might want to post this to the ruby-core and YARV mailing lists.

Comment on Visualization of Ruby's Grammar by JD

2006-10-29T14:19:34+00:00

Oh yeah and chiming in with robin… making these already made graphs available in some format like SVG, PDF …etc – where zooming in to get a clear read on the textis possible – would be a great addition to already great work…

– JD

Comment on Visualization of Ruby's Grammar by JD

2006-10-29T14:16:37+00:00

Interesting stuff. Being a comp.lang hobbyist, I’m curious at seeing the EBNF, C++, C#, and myriad of other language grammars that exist graphed out this way. (Yes I mean EBNF as language grammar itself)… Also I agree with the previous poster who indicated that “just because it’s complex doesn’t mean it’s inherently bad” (paraphrasing)

later

Comment on Visualization of Ruby's Grammar by Robinganemccalla@gmail.com

2006-10-29T13:07:10+00:00

Is there any way I can get a higher resolution version of the image? It looks nice but I can’t actually see the text.

Comment on Visualization of Ruby's Grammar by Michael Shigorin

2006-10-29T12:21:55+00:00

Thanks, rather interesting!

Comment on Visualization of Ruby's Grammar by AC

2006-10-29T09:08:27+00:00

Really FOOLISH. Know and think about “pseudo-simplicity”

Comment on Visualization of Ruby's Grammar by Jules

2006-10-28T20:42:55+00:00

Io has more syntax than that (literals for example), and so has Lisp.

Forth’s syntax is the simplest: words. Literals are handled from inside the language.

Comment on Visualization of Ruby's Grammar by quag

2006-10-28T19:45:40+00:00

This is a little bit of a joke. I think the equivalent graph for Io is http://www.quag.geek.nz/io/message.png . I imagine that lisp, scheme and a few other languages would have similar graphs.

Comment on Visualization of Ruby's Grammar by Bruce Perens

2006-10-28T19:10:04+00:00

The centrality of the primary node in the parser indicate that it is complex and multivalent. I think the main implication of that is that the lexical analyzer does a lot of work on that node that does not appear in the parser.

Language gurus used to prefer languages that parsed simply - for example ones that were pure LR(1). It surely makes it easier to explain how the language is parsed. Ruby’s 8000-line parser would offend language purists, if there were any left. But over time we seem to have become more accepting of this degree of complexity, and nobody seems to care greatly about explaining how Ruby is parsed as long as it works.

Then again, Ruby doesn’t seem to have confusion-producing elements like C’s declarations. Certainly the parser doesn’t get in the way of the programmer.

Bruce

Comment on Visualization of Ruby's Grammar by phil

2006-10-28T17:56:23+00:00

“Bison-to-ANTLR converter” - had no idea that such a beast existed. Seems that there was a project to convert Ruby’s yacc parser to ANTLR, does this mean that it’s trivial to convert the Ruby parser over to ANTLR?

The ultimate goal of many of these projects is to be able to parse Ruby in Ruby - really looking forward to that.

Comment on Visualization of Ruby's Grammar by Giles Morant

2006-10-28T10:53:00+00:00

The graphs for C and Python are very different to the ECMAscript, Ruby and Java.

To be honest though, I’m not sure what I’m looking at.

Is the ideal to be lots of many-linked items? Fewer items? Long lines of items?

Visualization of Ruby's Grammar

2006-10-27T16:48:00+00:00

As part of the momentum surrounding the Ruby implementer’s summit, I have decided to take on a pet project to understand Ruby’s grammar better, with the goal of contributing to an implementation-independent specification of the grammar. Matz mentioned during his keynote how parse.y was one of the uglier parts of Ruby, but just how ugly?

Well, judge for yourself. Below is a grammar dependency graph generated using ANTLRWorks and GraphViz. The steps I took are as follows. I took parse.y, stripped all C definitions, code and actions from it to give a bare YACC definition. Next, I did the equivalent of gsub(/[kt]([A-Z]+)/, '1') (since ANTLR’s convention is to have lexer tokens named starting with a capital letter). I then used the Bison-to-ANTLR converter to generate an ANTLR 2.x grammar, which I hand-modified to produce a v3 grammar. Opening the resulting grammar in ANTLRWorks allows you to generate a DOT file from which GraphViz can then generate a jpeg image. I’ve also included visualizations of the Java 1.5 and Javascript (ECMAScript) grammars for comparison.

I haven’t even begun to absorb all the meanings from this picture, but one stark difference between Ruby and the other two is the node in the middle of the picture with a high concentration of outgoing edges. That node is called primary in the grammar definition, and it is probably one of the reasons that Ruby syntax is so flexible and forgiving. A primary node’s direct children apparently represent a large portion of the syntax, and explain why in Ruby a single statement can either be a literal, a method invocation (or series of them), a standalone expression (such as a < b), all the way up to larger syntactic groupings such as if ... else ... end and begin ... rescue ... end, among many others.