Predictably Meta

Algorithms That Govern Our Lives Turntable.fm defies this trend,...

Tue, 05 Jul 2011 22:47:26 -0400

Algorithms That Govern Our Lives

Turntable.fm defies this trend, and that’s why it’s so good. A minor victory against our algorithmic overlords.

Deal with it.

Sat, 23 Apr 2011 01:25:05 -0400

Deal with it.

npm install jison

Mon, 14 Jun 2010 22:02:32 -0400

Tinkering with new projects should be easy. You should be able to click a link or invoke a command and BAM, there goes your Sunday afternoon. I’m not sure how many people build parsers on the weekend, but hey, Jison is easy to install now anyway.

If you’re using Node.js you should already have the excellent npm installed. If not, that’s easy:

curl -L http://github.com/isaacs/npm/tarball/master | tar xz --strip 1
cd npm
sudo make

Now install jison:

sudo npm install jison

Crazy easy. Now go read the docs for challenges.

Lambda Calculus Evaluator

Sun, 25 Apr 2010 17:52:57 -0400

After writing Jison, I began more experimentation with other facets of programming language design, namely interpreters. I was first exposed to their workings two years ago in a programming languages course at my university, where we wrote an interpreter for a minimal functional language using SML. As an experiment, I used the same technique to evaluate lambda calculus expressions.

Well, that was fun and all, but since I’m doing most of my language experimentation in JavaScript these days, I decided to port the lambda calculus evaluator to JavaScript (and put up a web interface.) It uses call-by-value evaluation semantics, as did the SML original. I used Jison for parsing, naturally.

Sure, it’s Turing Complete, but I doubt you would want to (or could) code your next blogging engine using it.

Anyway, have fun!

Replication Speed: CouchDB vs. MongoDB

Tue, 20 Apr 2010 16:03:39 -0400

I’m posting this because I hadn’t seen this comparison done before. Therefore, it might be useful to someone else. Maybe.

This was a rough test to see how long a slave would take to catch up with its master, which was sustaining about 500 document insertions per second with about 5 million documents already stored. The general verdict was that MongoDB replication is much faster, at least for this particular data model (i.e. logging data.) For CouchDB, inserting millions of small documents is probably not the most efficient way to do things.

I set up Ubuntu 9.10 VMs with 2 GB of RAM each for the master and two slaves. The first slave was set to replicate with the master from the beginning, to see if it ever lags (it was negligible for both DBMSs.) The other slave was used to see how long replication took to catch up.

CouchDB (v0.11)

For CouchDB, after 5,015,630 documents had been inserted over the course of 2 hours and 48 minutes, I started continuous replication on the third node and waited for it to catch up.

At 9,266,860 documents total, the third node had finally caught up with the master, after another 2 hours and 47 minutes, almost the same amount of time it had been offline.

For shorter intervals, the rate of replication was better. For example, it took 16 minutes for an empty database to catch up to a master that had been running for 30 minutes sustaining 500/document insertions per second.

MongoDB (v1.4)

MongoDB took considerably less time, completing the sync of 5,015,639 documents in under 7 minutes. Roughly half of the time was spent creating the new databases on disk and filling them with zeros, and the other half applying document inserts.

If shutdown improperly the mongod node needs to be repaired (with mongod --repair), which took around 11 minutes for 5 million records, ~4 GB of data. There really isn’t an improper way to shutdown CouchDB, so you can boot and start replicating immediately.

Conclusion

We’ll probably use MongoDB for dumping log data. It’s simple enough and the data is not exactly mission critical, so MongoDB’s shortcomings (durability, master-master limitations) aren’t enough of a deterrent.

I really like CouchDB’s replication system. It reminds me of git and allows for the same fully distributed model, but our cluster is not large or diverse enough to really take advantage of those features. I would have picked CouchDB if not for the dramatic difference in replication speed. But hey, no one DBMS is a hammer for all nails.

Extra Notes on Size

Documents were all clones of this JSON:

{"play_time":"19:02:20","file_name":"8821_20100224-1100.xml","playlist":"none","event":"playlist","player_id":9861,"play_date":"2010-03-02"}

CouchDB

Master: 5,015,630 docs, 5.0 GB
Slave: 5,015,630 docs, 9.3 GB

CouchDB after sync completed (compacted)

Master: 9,266,860 docs, 7.4 GB (3.6 GB)
Slave: 9,266,860 docs, 18.6 GB (3.6 GB)
Slave2: 9,266,860 docs, 13.9 GB (3.6 GB)

MongoDB was the same size on all nodes, or just under 4 GB each after 5 million+ documents (as you may know, MongoDB adds new database chunks up to 2 GB as needed.)

Slides and code from "Build Your Own Programming Language with JavaScript"

Tue, 20 Apr 2010 10:09:10 -0400

I gave a Track B presentation this year at JSConf titled “Build Your Own Programming Language with JavaScript”. The conference was incredible, but I’ll post about that later, perhaps. I’ve pushed the code and slides from the talk to github for people to play with.

A Common, Universal Scripting Language

Tue, 30 Mar 2010 13:37:47 -0400

Where is it? Hint: JavaScript. There’s an interpreter on every consumer platform and it’s already the most widely used scripting language in the world^{[citation needed]}. The core language is also very simple, and beginner friendly by design. Heck, every programmer already knows about half of the language through dealing with JSON. Throw in functions and you can get by.

It may not be the ideal scripting language, but it is the closest we have to a universal one.

Localized JSON Lint

Sat, 13 Mar 2010 18:10:33 -0500

I stumbled upon jsonlint.com recently, a service that sends your JSON data to a server for validation and returns the results. I decided to create a pure JavaScript version, which makes it is easy to host your self. It also comes as a CommonJS module in case you want to include it in some other tool.

CoffeeScript compiler now uses Jison

Wed, 24 Feb 2010 20:48:45 -0500

The new CoffeeScript self compiler uses Jison for generating the parser. The whole build script runs on Node.

Jeremy, the creator of CoffeeScript, spurred me to make some quick performance improvements (and bug fixes) to Jison so that the CoffeeScript grammar could build in a timely fashion. And while there are still many improvements to be made, handling such a large and complex grammar is quite a feat!

Jeremy also reported that parsing times are quick, though that’s more of a testament to v8, I’m sure.

Jison, now with more Bison flavor.

Fri, 29 Jan 2010 15:16:50 -0500

I pushed a new version of Jison (0.1.5) the other day that adds some interesting features, namely the ability to use Bison-like grammar definitions and Flex-like lexer definitions.

Bison flavored grammars

The following is a valid Bison grammar:

%token NUMBER

%%

E
    : E '+' T
    | T
    ;

T
    : NUMBER
    ;

Now it is also a valid Jison grammar. You can generate a parser using it as you would with the JSON format:

narwhal bin/jison examples/calculator.jison

This will generate just the parser, so you won’t be able to parse anything with it just yet (we’ll need a lexer too, which I detail later.) And as you can see, the file format ends with a .jison extension to distinguish it, as it is not fully compatible with Bison.

Jison will ignore %token declarations, as seen in the above grammar. It only recognizes %left, %right, %nonassoc, %start, and %prec declarations, which carry the same semantics as with Bison.

Semantic Actions

Semantic actions should look familiar:

/* description: Parses end executes mathematical expressions. */

%left '+' '-'
%left '*' '/'
%left '^'
%left UMINUS

%%

S
    : e EOF
        {print($1); return $1;}
    ;

e
    : e '+' e
        {$$ = $1+$3;}
    | e '-' e
        {$$ = $1-$3;}
    | e '*' e
        {$$ = $1*$3;}
    | e '/' e
        {$$ = $1/$3;}
    | e '^' e
        {$$ = Math.pow($1, $3);}
    | '-' e
        {$$ = -$2;} %prec UMINUS
    | '(' e ')'
        {$$ = $2;}
    | NUMBER
        {$$ = Number(yytext);}
    | E
        {$$ = Math.E;}
    | PI
        {$$ = Math.PI;}
    ;

This is valid for both Bison and Jison. The difference is that Jison does not attempt to match braces inside of actions like Bison does, so you’ll have to wrap your action in a different set of braces if they contain a closing brace, e.g.:

nonterminal
    : TOKEN
        {{$$ = '}';}}

In this case we wrapped double braces around the action so it knows the single brace is not the end of the action.

The nicest thing about this format over JSON is that porting Bison grammars is much simpler, though you’ll still have to remove all of the C code and duplicate the logic in a separate JavaScript module. Jison also doesn’t handle semantic value data types (not really necessary with a dynamically typed language like JavaScript,) mid-rule actions and error recovery, yet, among other things. Feel free to post an issue requesting your favorite missing feature.

A bit of an aside, but I found the process of bootstrapping this really interesting. You end up with a grammar that can parse itself.

Named Semantic Values

I decided to add some sugar for accessing semantic values within actions by name instead of position. Normally, you’d have to use the position of the corresponding nonterminal in the production, prefixed by a dollar sign $, e.g.:

 exp:    ...
         | '(' exp ')'
             { $$ = $2; }

Now, you can also access the value by using the name of the nonterminal instead of its position, e.g.:

 exp:    ...
         | '(' exp ')'
             { $$ = $exp; }

If the rule is ambiguous (the nonterminal appears more than once,) append a number to the end of the nonterminal name to disambiguate the desired value:

 exp:    ...
         | exp '+' exp
             { $$ = $exp1 + $exp2; }

Association by name leads to a looser coupling (and is easier to grok.)

Defining Lexers

Note that the new grammar format does not allow embedded lexer definitions. This is because there’s a new, separate format for defining lexers, which also serves to keep things more modular.

The format has a more restricted form of regular expression syntax, but no less expressive. The main difference is that strings that should match exactly must be wrapped in quotes. Here’s an example of the format:

%%
\s+                   {/* skip whitespace */}
[0-9]+("."[0-9]+)?\b  {return 'NUMBER';}
"*"                   {return '*';}
"/"                   {return '/';}
"-"                   {return '-';}
"+"                   {return '+';}
"^"                   {return '^';}
"("                   {return '(';}
")"                   {return ')';}
"PI"                  {return 'PI';}
"E"                   {return 'E';}
<<EOF>>               {return 'EOF';}

You can also use macros:

D                     [0-9]

%%
\s+                   {/* skip whitespace */}
{D}+("."{D}+)?\b      {return 'NUMBER';}
"*"                   {return '*';}
"/"                   {return '/';}
"-"                   {return '-';}
"+"                   {return '+';}
"^"                   {return '^';}
"("                   {return '(';}
")"                   {return ')';}
"PI"                  {return 'PI';}
"E"                   {return 'E';}
<<EOF>>               {return 'EOF';}

The format ultimately compiles to JavaScript regular expressions, so use the appropriate escape sequences and such. The file extension for this format is .jisonlex.

JSON-to-Jison

The update includes a utility for converting grammars from the old JSON format to the new Jison format. Just run your JSON file through json2jison to perform the conversion, e.g.:

narwhal bin/json2jison examples/basic.json

This will create a file called basic.jison in the current directory, which should be ready to run through Jison as-is.

If the JSON grammar has an embedded lex specification, json2jison will create a .jisonlex file also, though it will likely need correction (the conversion is lossy, to say the least.) E.g, running:

narwhal bin/json2jison examples/calculator.json

will create calculator.jison and calculator.jisonlex. Compare the generated calculator.jisonlex to the correct one found in examples/calculator.jisonlex to see the difference. Once the syntax of the lex specification is corrected, you can generate a complete parser like so:

narwhal bin/jison calculator.jison calculator.jisonlex

passing both the grammar and lexer specs.

Node Compatible

I should note that Jison is now compatible with node.js when included using require from a module. The examples from the README should work on any CommonJS system that implements the Module specification.

Tim Caswell used Jison as part of his CoffeeScript compiler that runs on Node.

In the future

There’s tons more to implement from Bison, and the grammar should also become more compatible over time, in terms of syntax and features that are supported by Jison.

I also have many topics to cover on using Jison, which I hope to write about more.

Until then, stay parsey, my friends.

Orderly.js Web Version

Thu, 31 Dec 2009 13:23:59 -0500

I’ve commited a minified, dependency-free version* of orderly.js for use on the web. It introduces an orderly object, with parse and compile methods.

Usage is straight forward:

<script src="orderly.js"></script>
<script>
    var orderlySource = "array {};";

    var jsonSchemaSource = orderly.compile(orderlySource);

    var jsonSchemaObject = orderly.parse(orderlySource);
</script>

Edit: View the online demo here.

*Excpet for a JSON stringifier, to account for browsers that don’t include native JSON yet.

Jison: An API for creating parsers in JavaScript

Mon, 28 Dec 2009 14:39:37 -0500

Bottom-up parsing

When you hear of someone writing a parser, it is usually of the top-down, recursive descent variety. Visualizing the source as an abstract syntax tree (AST), this means the parser tries to match the root, and by doing so, may recursively try to match branches and stems of the AST as it works its way down.

Bottom-up parsing is different:

Bottom-up parsing (also known as shift-reduce parsing) is a strategy for analyzing unknown data relationships that attempts to identify the most fundamental units first, and then to infer higher-order structures from them. It attempts to build trees upward toward the start symbol. It occurs in the analysis of both natural languages and computer languages. - Wikipedia

Because most bottom-up parsers require a parse table (a complex matrix of instructions for the parser PDA) they are usually not handwritten. Instead, a parser generator such as Yacc, Bison, ocamlyacc, etc are used to generate the parser in the target language. For Bison and Yacc, the language is C, for ocamlyacc, the language is, well, ocaml.

Jison

Jison generates bottom-up parsers in JavaScript. Its API is similar to Bison’s, hence the name. It supports many of Bison’s major features, plus some of its own. If you are new to parser generators such as Bison, and Context-free Grammars in general, a good introduction is found in the Bison manual. If you already know Bison, Jison should be easy to pickup.

A brief warning before proceeding: the API is ridiculously unstable right now. The goal is to mirror Bison where it makes sense, but we’re not even there yet. Also, optimization has not been a main focus as of yet.

Briefly, Jison takes a JSON encoded grammar specification and outputs a JavaScript file capable of parsing the language described by that grammar specification. You can then use the generated script to parse inputs and accept, reject, or perform actions based on the input.

Installation

Prerequisite: To run Jison from the command line, you’ll need to have Narwhal installed and available from your PATH.

Clone the github repository:

git clone git://github.com/zaach/jison.git

Using Jison from the command line

Now you’re ready to generate some parsers:

cd jison
narwhal bin/jison examples/calculator.json

This will generate a calculator.js file in your current working directory. This file can be used to parse an input file, like so:

echo "2^32 / 1024" > testcalc
narwhal calculator.js testcalc

This will print out 4194304.

Using Jison from a CommonJS module

You can generate parsers programatically from JavaScript as well. Assuming Jison is in your commonjs environment’s load path:

// mygenerator.js
var Parser = require("jison").Parser;

var grammar = {
    "lex": {
        "rules": [
           ["\\s+", "/* skip whitespace */"],
           ["[a-f0-9]+", "return 'HEX';"]
        ]
    },

    "bnf": {
        "hex_strings" :[ "hex_strings HEX",
                         "HEX" ]
    }
};

var parser = new Parser(grammar);

// generate source, ready to be written to disk
var parserSource = parser.generate();

// you can also use the parser directly from memory

// returns true
parser.parse("adfe34bc e82a");

// throws lexical error
parser.parse("adfe34bc zxg");

Using the generated parser

So, you have generated your parser through the command line or JavaScript API and have saved it to disk. Now it can be put to use.

As demonstrated before, the parser can be used from the command line:

narwhal calculator.js testcalc

Though, more ideally, the parser will be a dependency of another module. You can require it from another module like so:

// mymodule.js
var parser = require("./calculator").parser;

function exec (input) {
    return parser.parse(input);
}

var twenty = exec("4 * 5");

Or, more succinctly:

// mymodule.js
function exec (input) {
    return require("./calculator").parse(input);
}

var twenty = exec("4 * 5");

Specifying a language

The process of parsing a language involves two phases: lexical analysis (tokenizing) and parsing, which the Lex/Yacc and Flex/Bison combinations are famous for. Both lexing and parsing information can be included in the Jison grammar specification for a language.

For example, here is calculator.json:

{
    "comment": "Parses end executes mathematical expressions.",

    "lex": {
        "rules": [
           ["\\s+",                    "/* skip whitespace */"],
           ["[0-9]+(?:\\.[0-9]+)?\\b", "return 'NUMBER';"],
           ["\\*",                     "return '*';"],
           ["\\/",                     "return '/';"],
           ["-",                       "return '-';"],
           ["\\+",                     "return '+';"],
           ["\\^",                     "return '^';"],
           ["\\(",                     "return '(';"],
           ["\\)",                     "return ')';"],
           ["PI\\b",                   "return 'PI';"],
           ["E\\b",                    "return 'E';"],
           ["$",                       "return 'EOF';"]
        ]
    },

    "operators": [
        ["left", "+", "-"],
        ["left", "*", "/"],
        ["left", "^"],
        ["left", "UMINUS"]
    ],

    "bnf": {
        "S" :[[ "e EOF",   "print($1); return $1;"  ]],

        "e" :[[ "e + e",   "$$ = $1+$3;" ],
              [ "e - e",   "$$ = $1-$3;" ],
              [ "e * e",   "$$ = $1*$3;" ],
              [ "e / e",   "$$ = $1/$3;" ],
              [ "e ^ e",   "$$ = Math.pow($1, $3);" ],
              [ "- e",     "$$ = -$2;", {"prec": "UMINUS"} ],
              [ "( e )",   "$$ = $2;" ],
              [ "NUMBER",  "$$ = Number(yytext);" ],
              [ "E",       "$$ = Math.E;" ],
              [ "PI",      "$$ = Math.PI;" ]]
    }
}

This is all that is needed for Jison to generate a parser capable of recognizing the language and executing the semantic actions.

More examples can be found in the examples/ directory.

JSON isn’t so bad, but a more author friendly format is in the works. It will be similar to Bison’s .y files.

Sharing scope

In Bison, code is expected to be lexically defined within the scope of the semantic actions. E.g., chunks of code may be included in the generated parser source, which are available from semantic actions.

Jison is more modular. Instead of pulling code into the generated module, the generated module is expected to be required and used by other modules. This means that if you want to expose functionality to the semantic actions, you can’t rely on it being available through lexical scoping. Instead, the parser has a yy property which is exposed to actions as the yy free variable. Any functionality attached to this property is available in both lexical and semantic actions through the yy free variable.

An example from orderly.js:

var parser = require("./orderly/parse").parser;

// set parser's shared scope
parser.yy = require("./orderly/scope");

// returns the JSON object
var parse = exports.parse = function (input) {
    return parser.parse(input);
};
...

The scope module contains logic for building data structures, which is used within the semantic actions.

Lexical Analysis

Jison includes a rather rudimentary lexer, though any module that supports the basic lexer API could be used in its place. Jison’s lexer uses the lex key of the JSON grammar spec, where the rules for matching a token are defined along with the action to execute on a match. Usually, the action will return the token which is used by the Jison parser. A custom lexer could be used instead with it’s own methods of tokenizing.

TODO: More on this.

Parsing algorithms

Like Bison, Jison can recognize languages described by LALR(1) grammars, though it also has modes for LR(0), SLR(1), and LR(1). It also has a special mode for generating LL(1) parse tables (requested by my professor,) and could be extended to generate a recursive descent parser for LL(k) languages in the future. But, for now, Jison is geared toward bottom-up parsing.

*LR(1) mode is currently not practical for use with anything other than toy grammars, but that is entirely a consequence of the algorithm used, and may change in the future.

Real world example

I wrote a parser for Orderly using Jison. The benefits I found were:

If modeled after the normative language grammar, it is guaranteed to recognize the correct language.
Adding new syntax is straight forward.
It was much faster to develop than if I were to attempt implementing a (top-down) parser from scratch. But for others not used to grammar specifications, this might not be the case.
Improvements to Jison mean you only have to regenerate your parsing module to see the benefits.

Pie-in-the-sky ideas for the future

Practical, minimal LR(1) parsing
GLR parsing, which can recognize ambiguous grammars and natural languages

Orderly.js: Orderly to JSONSchema Compiler

Sun, 27 Dec 2009 16:19:06 -0500

Orderly is a textual format for describing JSON. Orderly can be compiled into JSONSchema. It is designed to be easy to read and write.

I decided to give Jison a real-world spin by creating an Orderly to JSONSchema compiler for JavaScript. After about three days, interspersed with lots of eating and family vists, I have a working implementation, which I call Orderly.js.

Usage

var orderly = require("orderly");

var orderlySource = "array {};";

var jsonSchemaSource = orderly.compile(orderlySource);

var jsonSchemaObject = orderly.parse(orderlySource);

print(JSON.stringify(jsonSchemaObject));

An API for building parsers

I must say, Jison made this really simple. I don’t consider myself a speedy coder in the slightest, probably the opposite, but I was able to crank this out in a (holiday) weekend. Jison takes care of the hard parts of parsing, all you need is a working grammar* and the appropriate hooks. I’ll be writing more on Jison shortly.

*This can be non-trivial, though if the language you are parsing already specifies a grammar, as was the case with Orderly, you’re set!

Stay parsey, my friends.

Wed, 23 Dec 2009 10:44:13 -0500

I hinted on twitter that I was implementing some kind of compiler in JavaScript. It was a project I began to help study for a compilers course I was taking, which also gave me an opportunity to earn extra credit. Though, really, I just thought it was a friggen cool thing to implement. I ended up focusing on the compiler front-end to create a parser generator, which I call Jison.

Anyway, the semester ended, I received the extra credit, and pushed the source to github. I’m still working on the initial documentation to include in a more formal announcement this week.

The demo page that was built to showcase some of the features to my professor is also available. It turned out to be a very useful tool for this particular niche in its own right. The page can help you visually debug CFGs, so it is helpful regardless of the parsing tool the grammar is intended for, or would be helpful for simply learning about CFGs and parsing algorithms.

I’ll post a follow up with documentation and more examples soon. For now, check out the JSON parser.

Stay parsey, my friends.

Random Notes on Computability

Mon, 23 Nov 2009 21:13:36 -0500

I felt like venting a bit. Let’s take a brief stroll through computability land.

The difference is computability.

Computability is important to a programmer. It’s fundamental to the act, even. Let’s just say it’s basically everything. Why waste cycles trying to compute the uncomputable! Or better yet, why introduce bugs trying to compute a problem using an insufficient computation model!

Computation models

You already know about these, even if you don’t realize it. There are three we encounter frequently:

DFAs
PDAs
Turing Machines

Deterministic Finite Automata

DFAs can be specified using regular expressions. Yes, a regular expression is just a convenient DSL for specifying a state machine. The regex engine in your favorite programming language adds on more power than a vanilla regular expression, but there are still some fundamental limitations to consider.

For instance, whenever you have to count, dynamically, you’ll need something more powerful. You’ll need some memory to go along with that state.

Say, you want to match all strings with the same number of leading 0s as trailing 1s. Informally specified as:

0ⁿ1ⁿ

Remember, n is dynamic, it depends on the string. Now, think of a simple regular expression to match that. No, DFAs are not powerful enough to recognize this pattern.

It also can’t recognize HTML or XML, for this same reason. You can get away with a limited subset of HTML or XML, but not the full language.

For that, you need at least a PDA.

Push-Down Automata

A PDA is like a DFA, but it can track where it’s been using a stack. The stack, along with the string input, is used to determine the next transition. A stack machine!

PDAs can be specified using a Context-free Grammer.

Okay, maybe you don’t encounter these frequently. That’s because a more powerful model is usually available.

Turing Machines

If you’re using a programming language, one that is Turing complete, well, you’re building Turing machines. A Turing Machine is an algorithm. A program is just a big (or little) algorithm. This is the most sophisticated level of computation you’ll encounter.

Turing machines can be specified using, well, Turing-complete programming languages.

Compilers

The stages of a compiler require progressively higher levels of computation:

Lexer: DFA
Parser: PDA
Type Checker, Code Gen: Turing Machine

Of course, you’d only actually need a TM to implement all stages, but as good software engineering principles hold, less is more.

I’ll work on writing more coherent blog posts in the future. Not saying that’s my new years resolution though.

Momentum and The Web

Thu, 19 Nov 2009 21:58:00 -0500

Recent events have my mind spinning (pun semi intended,) so excuse the perspective-rather-than-directive here.

Ahem.

There’s nothing stopping the web now. Not to say that there was doubt before, but man, there’s nothing stopping it now. This is the fastest I’ve ever seen it move (granted, my observing eye awoke to a stagnant age where the blue E ruled all.) The momentum of the Web has become the momentum of the Software industry. It’s really exciting from where I’m sitting.

RIAs are a nice gimmick. Flash was always annoying. Silverlight almost elicits rage (the tech is nice, but, urghh, why???) Now the days of every proprietary platform having it’s own dedicated app store seem numbered. The web already has the largest catalog of applications of any platform today, and it’s free and open for anyone to participate in.

Now to organize it.

An App Store for the Open Web ?

Search engines are great for finding things you already know you want, but there are tons of useful web applications out there that you’ll never see because you never knew they were exactly what you needed. And it’s a shame, because both parties are missing out: the creators and the consumers. The need for a better catalog is obvious.

Another interesting sub plot is the proposed Death of the URL. This would indeed lessen the freedom of the web, and its appeal to many. So, an app store for the open web that abstracts away the URL might not be so great…

But, well, URLs are still vastly useful on the modern web, right? People exchange them all the time over Twitter, Digg, Reddit, ect. There’s already a tremendous amount of data about web apps based on the links that are traded. Let’s build an app store with that in mind.

The way things are going, you are either with the web, or you are being deprecated by it. Choose wisely!

Number.prototype

Fri, 13 Nov 2009 00:59:21 -0500

Monkey-patching the Number global in JavaScript seems innocuous, and usually, that would be correct. I mean, who actually uses methods on Number instances, right? Your only worry is that someone else had the same idea in a another library, with different semantics. Other than that, you’re fine. Anyway, this is my favorite Number trick:

Number.prototype.times = function(funct){
  if(typeof funct === 'function') {
    for(var i = 0;i < Math.floor(this);i++) {
      funct(i);
    }
  }
  return this;
}

(20).times(function(){
    print("You're a jerk.");
});

Reads better than a for loop.

Bonus. Repeat a string 20 times:

new Array(21).join("I know.");

Flip Text Jetpack

Wed, 11 Nov 2009 20:22:05 -0500

I was surprised when I didn’t find a Jetpack that already did this. Who doesn’t love the flipped-text Unicode trick? ˙op ǝɹns I Now you can flip text at will.

Another Way to Run Narwhal

Wed, 07 Oct 2009 14:49:10 -0400

What’s Narwhal? Ahem:

Narwhal is a cross-platform, multi-interpreter, general purpose JavaScript platform. It aims to provide a solid foundation for building JavaScript applications, primarily outside the web browser. Narwhal includes a package manager, module system, and standard library for multiple JavaScript interpreters. - narwhaljs.org

It basically enables you to do things in JavaScript you might do in other scripting languages, like Python or Ruby. It’s one implementation of the CommonJS specification. If you’re in to JavaScript, you should get acquainted with it.

The default engine uses Rhino, but there’s also a XULRunner engine, allowing you to use Narwhal from within Firefox extensions and other XUL applications. Really exciting stuff!

I wanted a quick way test Narwhal/XPCOM modules, so I made an engine using xpcshell to do just that. It shares most of its libs with the xulrunner engine.

To set it up:

git clone git://github.com/tlrobinson/narwhal.git narwhal
cd narwhal/engines
git clone git://github.com/Gozala/narwhal-xulrunner.git xulrunner
git clone git://github.com/zaach/narwhal-xpcshell.git xpcshell
cd xpcshell
bin/install
cd ../..
NARWHAL_PLATFORM=xpcshell bin/narwhal
js>

It should work on Linux and OS X. Windows support is certainly possible. I’ll have a hack at it one day.

You can learn more about what can be done with Narwhal on their site.

Cross-site HTTP Request for jsonip

Fri, 03 Jul 2009 17:17:22 -0400

I’ve added cross-site HTTP request support for jsonip, a micro-service I introduced previously that retrieves a client’s IP address. Now, with HTTP access controls and a browser that supports them (such as Firefox 3.5,) any site can use a regular XmlHTTPRequest to utilize the service instead of JSONP. This might not be terribly useful today, as most browsers don’t support HTTP access controls, but the arrival of Firefox 3.5 beacons the future of possibilities.

Here’s a new cross-site HTTP request usage example:

<script>
  var req = new XMLHttpRequest();  
  req.open('GET', 'http://jsonip.appspot.com', true);  
  req.onreadystatechange = function (e) {  
    if (req.readyState === 4) {  
      if(req.status === 200) { 
        var ip = JSON.parse(req.responseText); 
        alert(ip.address);
      } else {  
        alert("Error loading page\n");  
      }
    }  
  };  
  req.send(null); 
</script>

You’ll notice it looks just like a regular XmlHTTPRequest, except the domain does not have to be your own. My service at jsonip.appspot.com is using special headers that allow this to work instead of greeting you with the usual permission errors.

The source code of the python script I use for the service is below. You’ll notice where the special headers are added:

import os
import cgi

form = cgi.FieldStorage()
callback = form.getvalue('callback','')

address = cgi.escape(os.environ["REMOTE_ADDR"])

json = '{"ip": "'+address+'", "address":"'+address+'"}'

#Allow cross domain XHR
print 'Access-Control-Allow-Origin: *'
print 'Access-Control-Allow-Methods: GET'

if callback != '':
  print 'Content-Type: application/javascript'
  result = callback+'('+json+');'
else:
  print 'Content-Type: application/json'
  result = json

print ''
print result

The magic happens here, with these printed headers:

print 'Access-Control-Allow-Origin: *'
print 'Access-Control-Allow-Methods: GET'

The first header means any domain can use this service cross-domain. Alternatively, you could have a comma seperated list of domains instead of a * to allow requests only from those domains. I want my service open to all sites, so I use a *.

The second header is self-explanatory (as if the first wasn’t:) there’s no need to support any other HTTP methods for this simple service, so I limit them to GET.

There are more access control headers available, so do make sure to read the article before trying these out. Some configurations could be dangerous, depending on your service.

Other jsonip Changes

JSON output uses double quotes for property names, as per the JSON specification
Added address property to use instead of the old ip property to retrieve the IP address. I’ve left the ip property for backwards compatibility.

The old, JSONP method of use still works as expected:

<script>
// callback function
function getip(jsonip){
  alert(jsonip.address) // alerts the ip address
}
</script>
<script type="text/javascript" src="http://jsonip.appspot.com/?callback=getip"> </script>