Jeremey's Weblog

1 Million Processes in 98 Characters

For the #twitcode contest (see: Twitter contest: what can you code in 130 characters?)

This Erlang code (98 characters) spawns N processes in a ring and sends a message through the entire ring. Try N=1000000 (1 million processes!!). In 135 characters, I could pass M messages through the ring, but there was no way to get that under 130. So this passes 1 message.

a(N)->X=b(self(),N),X!m.
b(P,0)->P;b(P,N)->b(spawn(fun()->l(P)end),N-1).
l(N)->receive T->N!T end.

Complete with module definition and export:

-module(r).
-export([a/1]).
a(N)->X=b(self(),N),X!m.
b(P,0)->P;b(P,N)->b(spawn(fun()->l(P)end),N-1).
l(N)->receive T->N!T end.

To run:

$ erl +P 2000000
Erlang R13B (erts-5.7.1) [source] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.7.1  (abort with ^G)
1> c(r).
{ok,r}
2> P=spawn(fun()->r:a(1000000) end).
<0.39.0>
3>

It should run in under a minute (w/ 1 million processes) on most machines, maybe noticeably less. Obviously 1000 processes or so is instantaneous.

Technorati Tags: programming, twitter, twitcode, erlang

Yelp Crosses The Line

Yelp sent me an MGD 64 ad in email. I'm offended. Seriously. How can a site dedicated to reviews (and thus, by inference, to "quality") send me an email with an ad for such a wretched beer, sold by a mega-corp, the enemy of micro-brews, quality, and ethics?! I yelp no more. They crossed the line. Yelp, you're dead to me.

Uploaded with plasq's Skitch!

Technorati Tags:
yelp, beer, ad, bad

Erlang FTW!

If you write code and you haven't learned Erlang, you should do so, right now. It will stretch your thinking, for sure, but it may do much more than that.

Erlang represents a fundamental shift in the way we think about solving problems. The fact that it is a functional language is part of the necessary reality that makes Erlang successful, but it is not the primary reason. What makes Erlang very special is its concurrency. Erlang is a language for "concurrent" programming and a virtual machine (like Java) for running compiled Erlang code.

The Java language is almost always considered in light of its runtime environment, the JVM. Erlang even more so, it would not be what it is without the runtime that implements it. So when I say Erlang, I mean the whole package, not just the language.

Erlang is oriented around concurrency on a massive scale. The fundamental unit of thinking in Erlang code is the process. One Erlang interpreter/virtual machine can handle millions of concurrent processes. The processes interact by passing immutable messages (everything is immutable BTW, Erlang uses single-assignment variables).

Guess what? The Internet IS concurrency. Practically any piece of software connected to a network these days must handle concurrency. Most languages do a pathetic job of this (or no job at all). They don't give us tools to build concurrent systems with any kind of ease. Most languages make it quite difficult, in fact.

Erlang makes it so easy and so natural that you begin to think about solving problems differently. The ability to run millions of concurrent processes can radically change your approach to a problem. What are social networking and large user-driven sites always running into? Scalability. How do you scale with all these users?

There are two problems here, the first being how do you scale to handle the load, and the second being what do you have to sacrifice in terms of code elegance, simplicity, etc. to get there? Will the complexity of your system grow with the scale? More little optimizations that disguise the real logic of the application?

What if you could run a unique process for every single user on your system? How would that change your approach?

That is what Erlang makes possible. Instead of a pool of workers crunching away in serial, individualized processes working in parallel, on a massive scale.

Plus, it is built to cluster, and all of its message-passing works identically across a cluster as it does locally. So if one machine can handle a million or millions of processes, how many processes could you handle on a large cluster?

One of the exercises in the Erlang book is to write a process ring benchmark. The idea is that you create a ring of N processes and then pass M messages through the ring and see how long it takes. On my Mac Pro (couple years old, quad-core 2.4GHz, 6G RAM), I ran the benchmark with 1000 messages and 1 million processes (!!). That means Erlang passed 1 billion messages. The benchmark ran in 8 minutes, which boils down to 2 million messages every second, or 2 messages every 1 microsecond. That's amazing. And powerful.

Another interesting tidbit: the statistics report "run" time and user time. User time was half of the run time. Why? Single-assignment + message passing = no locking, no contention = SMP-friendly. Erlang is built to run on multi-core computers. So it spent twice as much time "on processor" as it took in "real life" because it was using the multiple CPUs/cores offered by the Mac Pro.

Here's the code:

-module(ring).
-compile(export_all).

run(M,N) ->
   Pid = start(N,M,self()),
   statistics(runtime),
   statistics(wall_clock),
   send_messages(Pid, M),
   receive
      finished ->
         void
   end,
   {_, Time1} = statistics(runtime),
   {_, Time2} = statistics(wall_clock),
   io:format("~p ~p ~n", [Time1, Time2]).

start(N,M,RunPid) ->
   spawn(fun() -> start_master(N,M,RunPid) end).

send_messages(Pid, M) ->
   for(1,M,fun() -> Pid ! {init, message} end).

for(M,M,F) ->
   F();
for(N,M,F) ->
   F(),
   for(N+1,M,F).

start_master(N,M,RunPid) ->
   LastPid = start_ring(self(), N),
   master_loop(LastPid, RunPid, M).

start_ring(Pid, 1) ->
   spawn(fun() -> loop(Pid) end);

start_ring(Pid, N) ->
   NextPid = spawn(fun() -> loop(Pid) end),
   start_ring(NextPid, N-1).

loop(NextPid) ->
   receive
      stop ->
         NextPid ! stop;
      Any ->
         NextPid ! Any,
         loop(NextPid)
   end.

master_loop(Pid, RunPid, 0) ->
   RunPid ! finished,
   io:format("Finished~n"),
   receive
   after 100 ->
      Pid ! stop
   end;

master_loop(NextPid, RunPid, M) ->
   receive
      stop ->
         NextPid ! stop,
         io:format("Master stopping ~n");
      {init, Msg} ->
         NextPid ! Msg,
         master_loop(NextPid, RunPid, M);
      _Any ->
         master_loop(NextPid, RunPid, M-1)
   end.

Why Twitter?

A friend recently asked me why I like Twitter. I find this a difficult question to answer, much like explaining Twitter is difficult. You can explain what is is in a couple sentences, but to explain why you would want to use it is much more difficult. That's not because there aren't good reasons and everyone on Twitter is just being silly... quite the contrary, I find it difficult because there are lots of reasons, but none of them are easy to quantify.

It's a little bit like "why do you like to hang out with your friends?" or even "why are your friends your friends?" Quantifying an answer may not be so easy, it's complex and involves more than simple logical thought processes.

Now, some people (a lot of people now, it seems) are on Twitter because it's "the next big thing" and they're in PR or media or "social media" and Twitter is just another channel to deliver a message or content. That's not me. I'm not there to promote anything, and I largely ignore those who are.

I started on Twitter in part because there was a growing population of Houston folks, across a bunch of my interests (start-ups, software, photography, music, etc.), getting on Twitter. The majority of the people I follow are local, and Twitter has been a catalyst for networking with good people on both business and personal levels, and continuing to interact with them.

I've tried to describe Twitter as "social networking at the right speed", meaning it's "faster" than Facebook, MySpace, or email, but "slower" than IM. You don't expect to have a conversation like you would on IM, but you do expect to have conversations. They're just slower and more fault tolerant.

Part of what makes Twitter work is that it's perfectly flat. You don't join the right groups, friend the right bands, etc., you're just you. Everyone is just themselves (except for those in the PR camp, but we're ignoring them). People don't get silo-ed into buckets, you just get the people, with all their interests, all their comments on the world. The same people will be talking about photography one minute, programming the next minute, music after that, etc. You get a picture of a whole person, and discover that people are pretty interesting.

And on the other end of the spectrum, Twitter is global, so I have occasional interactions with people from all over the world, again in this flat space that's just a bunch of people. The "just a bunch of people" flatness and connectedness seems to have fostered a culture of respect and openness, vs. the somewhat flame-war-ish culture that has developed (to a certain extent) around blogs.

Twitter is direct. I can work out lunch arrangements or a meeting about a business opportunity very quickly, with multiple people, from anywhere, as long as I have at least SMS access. This gets at the heart of what Twitter is... Twitter is another level of indirection (for you C programmers), or another layer (for you networking people), for connecting people, and that, combined with its simplicity and ease of operation, makes it very powerful. It's a big lever and a light touch is all you need.

Twitter is casual. Most of the conversations I have on Twitter are the kinds of conversations I might have with friends. They aren't stuck in some "virtual" reality, they extend into day-to-day life. Most people on Twitter are real and they share mundane things right next to big things, and that's part of what makes it refreshing. Humans "do life" in little moments, in ordinary activities. Real relationships come from sharing those things, and the flatness, immediacy, and simplicity of Twitter allow that kind of sharing to actually work on the Internet.

Finally, Twitter is a powerful tool for "grass-roots" information. All of that connectedness and genuine interaction leads to a lot of real-time information sharing, everything from restaurant criticism to traffic reports to emergencies, with real results. Real people help each other, inform each other, etc. This is what the media have picked up on and part of why they now can't speak a sentence without saying something about Twitter.

To really "get" Twitter, you need to really use it and follow people and interact and grow your "network" enough to where these things start to happen. So there is some intentional effort involved. Maybe I hit it at the right time, but it has really been worth whatever effort I've put in. My life is significantly enriched because of the people I know on Twitter.

So that's some of what I love about Twitter. Maybe it answers the question. You can follow me on Twitter if you like. I follow real people back most of the time.

Also check out @BrentO's Twitter FAQ.

Technorati Tags:
love, twitter

Twitter, Then and Now

Then: "Hey, everyone... cool to hang out!"

Now: "Will someone tell Ashton Kutcher and Oprah to get off my damn lawn?!"

Watching Twitter "evolve" is sorta like watching the Internet "evolve" in time-lapse.

Sigh. :)

Technorati Tags:
twitter, oprah, devolve

Get Lisp? Not So Much

The Lisp function has called itself again, it seems. Or maybe it calls itself often, and I have just failed to notice it until now (see http://xkcd.com/297/). In any case, Lisp is the programming language that programmers are taught to venerate. Ogle its perfection, its code and data dual nature, its simplicity, its elegance, and so on.

I wholeheartedly support bashing new concepts into programmer brains using Lisp. However, I fail to see its appeal for writing real software. Part of developing software systems involves programmers typing in code, reading that code, and re-reading that code 6 months later. Other programmers are going to come along and read that code, too. If you've ever looked at the source code for a large Lisp system, you may notice that its syntax does not lend itself to being understood. It is difficult to read, and even more difficult to interpret, because there is SO MUCH LIST PROCESSING GOING ON (Lisp = List Processing).

Lisp is difficult, in part because it represents new concepts to most programmer brains, but also in part because it's just tedious. Very nice that you can represent every data structure as a list or list of lists, but that doesn't mean you should. Just because it is possible to treat all things as exactly the same and have no keywords doesn't mean it makes for better software. I think it just makes for easier compilers. Lists just aren't a natural way to represent data to humans.

All that list processing produces code that is syntax-heavy (parentheses and list processing) in a language that essentially has no innate syntax. It's difficult to read, just like almost every Perl script is difficult to read. There's a lot of syntax happening. Why? Because it makes the compiler's job really easy. Ummm, why do I care if the compiler has an easy time of things? The compiler should work hard to make my life easy, and to make it possible for me to write better, more maintainable, more legible, simpler code. In Lisp, the programmer is responsible for constructing the entire AST himself.

Programming doesn't have to be difficult to be powerful. I reject that notion as the supposition of lesser programmers who've bought into a culture that worships Lisp and its kin. Learning Lisp may grow your brain (a good thing), but deploying real software in it seems like less of a good idea. To think that we've failed to come up with any more good programming language ideas in 40 years is ridiculous.

The Lisp culture reminds me of the "if you only knew what we knew" arguments that get trotted out by "national security" people every so often.

Tell you what... let's have a contest. You (hypothetical Lisp worshiper) implement a Twitter clone in Lisp, and I will implement it in a language of my choice (NOT Lisp, probably Erlang). You must provide an HTTP-based API in addition to a pleasing web UI. Both must support username/password authentication. You must be able to demonstrate support for 500,000 concurrent users with sustained message rates over 1000 messages/sec. You must store all messages. 50% of the users will have 50 followers. 35% will have 200 followers. 10% will have 1000 followers. 5% will have 10,000 followers.

When both systems are complete, we will note how many hours (roughly) it took to build each system, and how many lines of code (this is not always a good metric, but we won't abuse its relevance), and what deployment requirements there are to reach the scalability goals.

Then we will take a good programmer who has not programmed extensively in either language, and ask him to understand each set of source code and ask him to make some modifications. We'll subjectively analyze how things go in each system.

I think we will find that the non-Lisp system is less code, more intelligible, easier to extend, faster, and easier to deploy.

Lisp as a religion needs to die. Lisp as a set of interesting ideas worth learning should stay.

Technorati Tags:
programming, software, lisp

Programmers and Philosophies

Dear programmers of the world: take your pet "philosophies" and shove them where the sun don't shine. You're building an app that you want to SELL and your "philosophy" prevents you from implementing a major feature that many users want? You can rot.

You want a philosophy? How about this: build great apps that people actually want to us and that solve real problems for people.

For instance, Twitter clients. I love Twitter. I hate Twitter client developers (except TweetDeck). Why? Because they don't bloody listen to users. TweetDeck has how many users? Despite the mega-suck AIR UI? Why? Because it solves a major problem (grouping Twitter users), and solves it well. Is it perfect? No. Is is 100,000 times better than every other Twitter client because it makes a decent attempt? Yes.

Would I buy TweetDeck? I might. Would I buy a fantastic Cocoa app that had a similar implementation of groups along with a nice UI, low memory footprint, etc.? Absolutely. I would pay $40 for a high-quality Cocoa Twitter client. Is there one? Nope. Not even one.

Tweetie looks fantastic, but it is DOA without groups. I love everything else about it, and it pains me to shut it off and start TweetDeck, but that's what I gotta do, because a beautiful interface to an un-grouped Twitter feed is like the most perfect baseball bat ever made when I need to chop down a tree. Sorry, it doesn't work. It's dead.

If you want to implement what people need and then make it perfect by beating Twitter into implementing groups and transition from your local implementation to an API-based implementation, fine. Just do it. What a waste of an opportunity.

Technorati Tags:
twitter, tweetie, programming, philosophy

Software Development Meme (8 Months Late)

@BrentO posted this over 8 months ago, and I just now saw it, while googling myself in the hopes of finding one of my old resumes laying around on the Internet!

Anyway, here goes... this is a very deep topic for me, so bear with me. :)

How old were you when you first started programming?

I think my first programming experience was in grade school, on TRS-80 machines ("trash-80s"), in BASIC. This was a mischievous child's dream, because although most people could bang out the programs the teacher wrote on the board, almost none of them knew what to do when their screen filled with the same lines of text over and over, ad infinitum. DoS (Denial of Service) at its best! (10 PRINT "YOUR MOMMA WEARS COMBAT BOOTS". 20 GOTO 10.) I vaguely recall building some incredibly simple game at the time, but it's been so long I may be making it up.

How did you get started in programming?

Not long after starting high school, our family got our first personal computer, a Leading Edge 8088 machine. It was awesome with its 640K of RAM. All we'll ever need, right Bill? :) Not long after, I produced a DOS manual for my parents, explaining the intricacies of things like "dir". Probably a year later (sophomore year?) I tried to teach myself Pascal. I had started working at a bicycle shop, so I had a little money, and I bought the Borland Pascal compiler which must have come with some introductory book, or something.

In any case, Pascal was a disaster. It was a very frustrating experience, mostly because the language felt so artificially limited. I did manage to program a little space game on our 4-color CGA display that would go into hyperspace and shoot laser beams, but I got tired of Pascal and let the whole thing sit dormant.

But, in the beginning of my senior year of high school, my mom began taking an intensive career-retraining course at DePaul University. It was a 9-ish month program designed to launch your career in IT software development (COBOL, JCL, etc.). I didn't pay much attention until she began taking a C programming class, got stuck, and asked me for help.

Let me tell you, to a born hacker not yet aware of his calling, who had only worked in BASIC and Pascal, programming in C was like injecting some kind of super-drug. THIS was freedom, this was power, this was programming. Pointers were what I'd been looking for. Raw, type-unsafe power, baby.

What was your first language?

Although I dabbled in BASIC and Pascal, I consider C my first language. It's the one I really learned, the one that clicked, the one that I didn't need to read books about or study. I just got it.

What was the first real program you wrote?

I would probably say the Oberon compiler I wrote in college was my first real program (I was taking Compiler Design). It was limited, and only worked on a subset of the language, but it was real. And it really compiled code. The assignment was to target a virtual machine platform with your basic operations, but my friend and I decided to target the real SPARC processor instead. This actually turned out to be fortuitous because the SPARC architecture included overlapping register windows, so I was able to implement all of the required functionality (we had to be able to compile a certain set of algorithms) without implementing a stack.

Very few things make you feel more like master of the universe than writing your own compiler and having it actually generate machine code for a real machine.

What languages have you used since you started programming?

Yikes. This is a long list, but here goes, in rough order of appearance:

• C
• C++
• ML
• SQL
• Visual Basic
• COBOL
• shell (bourne and c)
• lex and yacc
• Tcl
• HTML
• Lisp
• Python
• Java
• Javascript
• Perl
• PHP
• CSS
• Haskell
• Ruby
• Objective-C
• Smalltalk
• Erlang

What was your first professional programming gig?

In college I began working for some friends who formed a consulting company. They were working on a couple contracts with a pharmaceutical company and a bank. This was where I had the unfortunate experience of being subjected to Visual Basic. There was also some C work, but all of it was tedium. I quit not too long after joining, ostensibly to start my own company with another friend, doing more interesting stuff.

The first gig I enjoyed, and stayed at, was with a tiny startup in Silicon Valley. My professor at DePaul recommended me to a friend of his, we had one phone interview, and I left Chicago bound for Palo Alto, 2 days after my last class finished, sight unseen, etc. We were building tools to put scientific journals and magazines like Science on the web (yes, the web, in 1994, pre-Netscape).

If you knew then what you know now, would you have started programming?

Certainly. It's in my blood. I would start with different languages if I had it to do over again (well, I would still do C first, but Smalltalk would be soon after).

If there is one thing you learned along the way that you would tell new developers, what would it be?

One? Are you kidding? :) Here's a few things:

• When the C++ and Java programmers tell you how the world works, don't believe them. Their world is too small.
• Learn Smalltalk, Erlang, and Python.
• Study EJB 3.0+ and recent JBoss. They represent an enormous amount of mature thought on deploying real applications. There is much there to learn, whether used in that environment or not.
• Do something meaningful.
• You won't really be any good for ten years, at least.
• If you don't get it, find something you do get, and do that instead.

What's the most fun you've ever had ... programming?

Usually, whatever I'm doing right now, for any value of now. I've been very fortunate to work on interesting projects that continue to stretch me.

Though, if I had to pick one, handing a live phone call from VoIP to GSM and back with no break in audio was pretty sweet.

Who are you calling out?

I dunno, I'll figure that out later. :)

A Month...

Blog? What blog? Oh, this one. Right. I've been using Twitter so much that I forgot to ball up some thoughts into something cohesive and stick them here! Or, for vi fans:

vi life
:g/blog/s//twitter/g
ZZ

So what happened in the last month? PHOTOCAMP!! Work! JEEeeeeee! Twitter!

PhotoCamp will be its own post. So will some JEE stuff. I pretty much adore Twitter. Did I mention Twitter?

Hello, 2009

I love the holidays, but I'm always a little relieved when they're over, too. 2009 should be a very interesting year, I'm looking forward to it!

To welcome 2009, here's a shot of Gabriel at the park on New Year's Day:

Definitely click through and then click the "Better Large" link. It is much better large.

50mm f/1.4

I totally love this... I shot this image at ISO 100 at 1/60th of a second! In not all that much natural light, indoors! f/1.4 is a whole new world of lots of light and shallow depth of field (I have an f/2.8 lens, but it has to be 6 ft from the subject, and I have the f/1.8 50mm, but its focusing is too unreliable for regular use).

Fun! Looking forward to sticking this thing on a 5D Mark II sometime soon!

Technorati Tags:
photography, lens, 50mm

The Christmas Picture!

I forgot to post it! Here comes the cuteness...

Technorati Tags:
photography, christmas, kids

Christmas Cards!

Our Christmas cards from Apple arrived, very cool! (designed in iPhoto using a template... I didn't color correct or anything on these pics of the cards, apologies)

Guadalajara Tacos (and Bakery)

Read my review of Guadalajara on Yelp. The short summary: it's excellent. A couple pictures to accompany the review:

The mess, post-tacos:

Guadalajara:

Technorati Tags:
food, houston, tacos

Saturn V Is REALLY BIG

We went to NASA (Johnson Space Center) recently (first time for me and the kids), and the last stop is Rocket Park where they take you into the building the Saturn V rocket is laying down in. Cool thing is it's not a mockup, it was the rocket that would have been Apollo 18 had that mission happened.

Anyway... IT'S FREAKIN HUGE!!! It's taller than the Statue of Liberty!! UN-believable. They shot that thing into space?!

Very much worth the trip to see it. Here are some pics which will utterly fail to really convey the size of this thing...

This is a shot of the first two stages (there's another almost half of the rocket to the left, not in the shot):

Stage 1 and the main engines:

Just one of the main engines:

Technorati Tags:
photography, saturn v, rocket

Hario Hand Coffee Grinder

UPDATED: more pics, see below

Max (owner of Catalina) got some Hario hand grinders in and I bought one on the hopes that it would replace my Kitchen Aid ProLine for French press and Chemex brewing.

The Hario has ceramic conical burrs, a "drive shaft", a simple adjustment mechanism, and a crank... along with a glass bowl to catch the grounds and a plastic bowl for beans. One burr (large, outer) is fixed to the plastic bowl and the other (small, inner) is fixed to the shaft. The adjustment "dial" is just a nut on the shaft, really, which "screws" the shaft up and down, thereby moving the small burr closer or further away from the large burr.

I have mixed opinions on this thing, but I'll give you the punch line first: it's roughly as good as my Kitchen Aid for French press or drip grinds, and much better for espresso grinds. I'm not sure it competes with my MACAP espresso grinder, but it is impressive for fine grinds.

My mixed opinions come from the design of the shaft and the way the small burr moves. The shaft is only fixed to the "body" of the grinder in one place: the top. This means that the small burr (fixed to the end of the shaft) is free to float around away from a perfect center line. At the very fine end of the grind spectrum (espresso), this is almost non-existent because the burrs are so close together. The small burr just doesn't have any room to move laterally.

However, for coarse grinds, the burr has plenty of room to move, and it does. This means an inconsistent grind. You get some (not too many) very fine particles along with some big chunks. This is too bad, because the burrs are clearly very good, and a better shaft/body design would turn this from pretty good to fantastic (DIY project?).

However, the results are no worse (or not significantly) than my Kitchen Aid, which cost $200. The Hario costs a quarter of that.

Here is a coarse grind w/ the Hario:

And a slightly finer grind from the Kitchen Aid, for comparison:

And compared to the Kitchen Aid in size:

Bottom line, I like the Hario, but I think it could be significantly improved. It's competitive with my Kitchen Aid costing four times as much, and significantly better for finer grinds. I probably give a slight edge to the Kitchen Aid for French press grinds, but a definite edge to the Hario for espresso grinds.

And there's the "hand" part... expect to get a workout grinding enough coffee for 2 large cups of French press. :)

UPDATED WITH MORE PICS:

Handle and adjustment assembly:

Grind adjuster:

Outer burr (bottom view):

Inner burr:

Inner burr and shaft:

Bottom view up through shaft "housing":

Handle and adjuster disassembled:

Technorati Tags:
coffee, espresso, grinder, hario

Best. Turkey. Ever.

This year's Thanksgiving turkey was the best ever, and also the easiest, so I thought I'd share. It's basically Alton Brown's recipe with some minor variation in brine ingredients. First, go get yourself a remote thermometer... I think ours was 18 bucks at Target (it's a meat thermometer with a cord and a remote control).

The key here is the brine... you're going to get far more flavor into the bird by soaking it in brine for 16 hours than you ever will by doing all kinds of hijinks in the oven. So, the brine recipe:

• 1 gallon vegetable stock, garlic-free if possible (I used Imagine organic veg stock)
• 1 cup kosher salt
• 1.5 cups brown sugar (I used light)
• large handful black/mixed peppercorns
• small handful smoked peppercorns
• small handful whole cloves
• fresh sage (leaves from several stalks)
• fresh thyme (a small bunch)
• fresh rosemary (a small bunch)
• 1+ gallon ice water (I emphasized ice)

AB also recommends allspice berries, Whole Foods didn't have them.

Heat the stock on the stove until it's hot (no need to boil it), add the salt and sugar, get it dissolved. Turn off the heat and add everything else except the ice water. Cool to room temperature.

Get your fresh or thoroughly thawed turkey out of the fridge and remove giblets, neck, and tail (don't cut anything, just remove whatever "extras" are in your bird). Stash that stuff in the fridge. Regarding thawing, if you're going to thaw in a fridge, allow 5 days.

Put the brine solution into a 10 gallon drink cooler (the orange work coolers you see around, they're tall and sorta skinny). Put a little of the ice water in. Put the turkey in, breast side down. Cover with ice water so the turkey is submerged, but you don't need more than that. I stirred the brine around at this point. Stash outside (or somewhere cool/cold) for 16 hours.

After 16 hours, preheat your oven to 500 degrees. Remove the turkey from the brine, drain. Truss the turkey well (this is really important for even cooking, actually). If you don't know how, watch AB do it.

Lightly coat in a high-temperature oil like safflower or canola (I used safflower); this will help it brown nicely. Grind salt and pepper onto the skin (to flavor the skin itself, which will get nice and crispy and yummy). Put the turkey on a rack in a roasting pan (or I suppose w/o a rack is fine).

Here comes the one "trick"... we're going to use the thermometer so we can take the turkey out at precisely the right doneness. Any other way, it's all guesswork. Read your thermometer instructions, some of them can't be in the oven at 500 degrees (like mine). So I did the 500 degree section and 30 minutes of 350 degrees w/o the thermometer and then stuck it in at that point. YMMV. Whenever you do insert the probe from the thermometer, insert it into the breast from the top, maybe 2/3 of the way toward the rear, so that the end of the probe is in the thickest part of the meat and not touching bone.

So, either with thermometer now, or with the plan to add it later, stick the turkey in your 500 degree oven for 20-30 minutes. There was so much sizzle I only let it go 20. After the stint at 500, turn the oven down to 350. If your thermometer isn't in yet, give it 30 minutes and then insert the thermometer. AB recommends setting the target temp on the remote to 151. I did 155, and out of the oven my turkey cruised to 170, so 151 is probably perfect (the final target being 165).

When the alarm goes off (make sure you turn it on!), remove the turkey, cover loosely with foil. DO NOT REMOVE THE THERMOMETER YET. Let it rest a good long time before carving (30 minutes minimum, more is fine). My turkey left the oven at 3pm and we ate at 5:30. Remove the thermometer when you go to carve.

A carving tip that really worked for me: cut the whole breast off in one chunk, and then cut slices across it (i.e. slice perpendicular to the length). That way you cut across the grain and the slices are much less tough. You can even cut them pretty thick if you like.

With this recipe, and really controlling them temperature, everything was fantastic. The white meat was succulent, the dark meat was done, the skin was crispy and delicious.

So:

• Brine
  
• Truss
  
• Control/monitor temperature
  
• Rest it
  

Technorati Tags:
recipe, turkey, thanksgiving

Five Characteristics of Good Coffee

Jim Hughes "tagged" me in this post to blog about 5 characteristics of good coffee. I (like he) generally don't do these blog tag game things, but this one I can write about. But I may not pass the tag along.

I think I will focus on the process of making good coffee, instead of the characteristics of the finished product, as I think that will be more useful, but also the final product is so complex (hundreds of unique flavor compounds) that it would be difficult to describe. So, what follows are five characteristics of the process of making great coffee.

1 - Beans. Great coffee starts with great beans. There are many variables that factor into great beans, but I will focus on two: source and roast. A roaster must decide how to choose the beans to roast, and how to obtain them. Roasters that build direct relationships with farms that are aiming to produce a high quality product (generally called "specialty coffee") tend to also be the ones with the most care and knowledge of coffee and the coffee industry, and thus they tend to be the best roasters. If the roaster doesn't know why he is carrying/roasting a specific bean, move along. Second, the roast itself... this is critical, as coffee is useless without being roasted. The roast is an alchemic transformation of a green bean into a little package of potential flavor (think potential energy). The flavors in coffee come from the oils in the beans, developed in the roast. If you see those oils all over the surface of your beans, well they're not in the beans anymore, are they?

How do you pick a good roaster? Find one that is passionate about coffee and its potential flavors and complexity, that sources his beans directly from farms or at least goes and experiences these farms and builds relationships with the farmers, that knows why he has chosen a particular bean and how he roasted it, and that won't sell you a particular bean in more than one roast (good roasters generally feel that there is a "right roast" for each bean).

2 - Freshness. Coffee should be consumed within two weeks of the day it is roasted, in most cases. So buy from good local roasters, buy a little (e.g. 1 pound), and buy it often. Even more important - it should be ground immediately before use. Grinding should be the last step prior to combining the ground coffee with water. Coffee contains many volatile compounds which distribute flavor into brewing coffee, and one of the more important is CO2 (carbon dioxide). Roasted coffee loses much of its CO2 within the first two weeks post-roast, and once it is ground, it loses the vast majority of whatever CO2 it had within 60 seconds. So prepare your brewing system, prepare your water, then grind and brew.

3 - Grind. Coffee (the drink) is not made from whole beans, it's made with ground coffee. Coffee beans must be ground to the appropriate fineness for the type of brewing you are doing, and this will require some experimentation (generally, coarse for French press, finer for drip). Also, as mentioned above, you really want to grind as the last step before brewing. To grind with any consistency, you really need a burr grinder. Blade grinders just pulverize the same coffee particles over and over, so you get a very uneven grind.

Why does this matter? Because brewing coffee is really about extracting the flavors and oils from the ground beans. If you have an uneven grind, you will have "dust" bits along with some larger chunks... the larger chunks won't get completely extracted, and the dust bits will get massively overextracted. Overextraction means bitterness. This is why a lot of coffee is bitter (among many factors).

4 - Quantity and Time. You need to use the right amount of coffee for your brewing method, and unless you've spent some time working on your coffee, you're probably not using enough. Again, the problem is overextraction... if you stick a teaspoon of ground coffee in a 12-cup French press, you will overextract that coffee and end up with bitter (and probably "weak") coffee. Real coffee geeks speak in grams, but if you develop a fair approximation by volume, that's fine. Start somewhere in the vicinity of two tablespoons of grounds per 6-8oz of water, and go from there. Experiment. Your grind will be a factor here too, because a finer grind will extract more quickly.

Brewing time (in methods where you control this, e.g. French press) is also a major factor. This will require some experimentation, but generally 3-4 minutes is the right ballpark, if the other variables are in line.

5 - Water and Temperature. Use good water. Filtered water, spring water, whatever... just use good water. Then, use that water at the right temperature (in brewing methods where you control this). Boiling is too hot! You want to be around 200 degrees. Find a reliable, consistent way to control your water temperature.

Finally, I leave you with a characteristic that is NOT a characteristic of good coffee: bitter. Good coffee is many things, but bitter should not be a prominent one. Bitterness generally results from poor extraction (too hot, too long, too little coffee, poor grind).

Make some good coffee! FWIW, a French press and a Solis Maestro Plus grinder will get you on your way. Spend your money on the grinder, buy a cheap French press. Expensive drip brewers won't make good coffee unless every variable is right. Control the variables yourself.

Technorati Tags:
coffee, tagged

Alan Wayne Blankenship: 1970-2008

How do I start? My friend is gone. We weren't so close over the last couple years, but there were years where our families did a lot of life together. We got married, we had babies, we spent time. I developed a deep love for Alan and his family that has never abated.

I was shocked to hear he'd fallen from a malfunctioning ladder and landed on concrete last Thursday, and of course was very concerned. But on Saturday another friend posted on Facebook that he was at the hospital saying goodbye, and that they were going to pull life support, and then it really hit me hard. It hadn't been clear how grave the situation was, and suddenly we had to go.

So we went to the hospital and were graciously welcomed by the family, and I got to say goodbye to my friend. Seeing him made it much more real and personal, but I'm so glad we went. He died two minutes after they removed the respirator.

Alan was a good man. He was someone you could depend on, someone who did things right. It turns out the previous owners of our house hired Alan to do their roof, not too long before we bought it. When Hurricane Rita was threatening, I called Alan for an estimate of what kind of wind speed the roof could handle, and we stayed put based on that. Rita didn't hit us, but Ike did, and not one shingle left our roof.

I probably laughed more with and around Alan in the last 10 years than anyone else. He had a knack for laughing at himself and situations that was disarming and genuine without being (too) silly.

Alan was a thinker and a wrestler and a struggler. He wrestled with big ideas, from politics to theology to faith, and we spent many an hour discussing and debating over the years. I am confident that his struggle is over and he has found his peace.

He loved his family and his kids, and it is for them that I am deeply grieved. I feel an overwhelming deep knot in my gut, an intense heaviness. I have imagined the same situation in our family, my kids losing one of us, and I can't begin to even think about it.

Alan, we will see you again soon, my friend. But not yet.

Americano

Just gotta say... I love a good americano. I basically don't brew drip coffee anymore, if I want a cup of black coffee (which I do, frequently), I make an americano. A proper americano is made w/ hot water in your cup and then you pull your shot onto the water. That's it. It is NOT a shot pulled for a really long time; that will massively overextract the coffee. Just a normal shot as if for a latte, pulled into a cup of hot water (not too much water).

P.S. you do want to pull the shot after you put the water in the cup... that way you get all that delicious crema on top. YUM.

Technorati Tags:
coffee, espresso, americano