Something anyone trying out NodeJS should know up front

TL;DR – scroll down till you see the heading you’re looking for.

I really love JavaScript as a language, and there were lots of things about NodeJS that appealed to me. So when I had the opportunity to use NodeJS on a significant project about a year ago I went for it. The app pulled data from several web services, mashed them together, and posted results and commands to other services: There was a lot of async. NodeJS should be really good at this.

After two months of serious coding I found my code was littered all over with error checking code — not error handling code as in, “I can compensate for this error,” but simple error checking code: This callback gives me an `err` as first parameter. Hmm, I guess I better not ignore it. If it’s not empty I better that’s set to something, and pass it to my own callback. So I end up with this All. Over. The. Place:

function getCoolResult(x, y, callback) {
   lookupSomeXData(x, function(err, xData) {
      if(err) {
      combineYWithXData(y, xData, function(err, coolResult) {
        if(err) {
        callback(null, coolResult);

To be clear, my beef with NodeJS was that, in order to do it right, I needed those error-checking warts (two in the example above). I found myself thinking, “This is like writing C in the 90′s,” where one is required to check return codes after every function call.

I don’t like it. I want my core business code to be really focused on business concerns, not infrastructure concerns. I want to only worry about errors that I can actually do something intelligent and contextual with.

I ran this project for two months, then decided to rewrite all the server side code in C#. I’m a consultant/contractor and the rewrite was on my own dime, so I didn’t make the decision lightly. I’m really glad I did for this project, because:

  1. Deleting all the error checking code felt great. Like StrongBad-cleaning-his-inbox great. Go ahead, watch that little video and imagine yourself cleansing your code of all these little error-checking warts.
  2. I dropped several flakey and dubious third-party NPM packages and used built-in .NET libraries or more mature Nuget packages instead, resulting in less lines of code and a more stable app.
  3. There were big chunks of code I wrote myself for node that I replaced with a third-party Nuget package. At the time I could find no NPM counter-part for some of these things.

The C# code shrunk drastically, was cleaner and more stable. Probably slower executing, but not enough to be an issue for this project.

TL;DR starts here:

So I think NodeJS is pretty cool, and it may be the right tool for certain kinds of jobs, and if you’re interested in exploring it, you should! But there’s something important that Christopher Langton distilled down for me today (in comments) that I wish I’d known a year ago:

You ultimately will need to conform to the handling of known errors or pass them on, as this has become the only implementation that all modules apply.

My translation: every callback will have a little error checking fragment up front. Yeah, it’s kinda annoying, but just do it and move on. You can live with that.

(Promises can help, but they’re not ubiquitous in NodeJS.)

Posted in JavaScript, Programming | Leave a comment

Fueling Creative Effort

This is something I wrote around 2009. It’s a realization that I’ve kept reflecting on from time to time these past few years, and has influenced the way I encourage my kids in some positive activity. It’s a bit rambling – I’ve bolded the important parts.

Timmy had been working on building comic books for a while, but then lost interest. He still liked to tell stories all the time. He’s invented a super hero named Fireman (with some certain super powers that you might predict). He developed quite a number of episodes, but they’re all in his mind, they change a lot, and from my recollection the stories seem to change each time. Allison and I encourage him to write these stories out. I really felt this would help him develop as a story creator/teller. Understandably our encouragement didn’t turn into immediate action in this area. :)

Yesterday he went to the library and picked up a Hulk comic book. This morning I saw by his bookmark that he was almost through. Later I found him on the computer; he was using Notepad to type out a Fireman script. I excitedly read his story, then showed him how to use Word to add a little text style and correct his spelling mistakes. I’m really excited that he took the initiative to do this. I think writing out stories in Word will help with him develop his story creating/telling, help his spelling, get him a jump-start exploring the features of Word, and many other things, I’m sure.

What prompted me to write about this is what stimulated him to take this next step: He experienced something cool that someone else had created, and was moved by it to create his own thing along those same lines.

Now that I think about it, he had the same response over the Christmas break to board games. We played a few games, and before I even knew what he was doing he had created his own board game. It wasn’t very balanced — if you ran out of money you went to jail, and the only way to buy yourself out of jail was for other players to volunteer to give you money. But he’s made refinements since then that I haven’t had the opportunity yet to check out.

I have noticed the same response in myself. I loved board games, and made up a lot of them. One actually turned out pretty well, although it had a bit of a dark twist. The object of the game was to kill all the other players. On the back of a Chutes and Ladders board I drew a clue-like set of rooms with a square grid between rooms to travel. Each room had some game cards in it. The cards were either weapons, defense against weapons, or ammunition for the weapons that needed it. Each weapon had a range, so if you got within 3 squares of another player, and you had a gun and a bullet, you can shoot them. But if they had a bullet-proof vest then they’d remain in the game. Another weapon was the rope. If someone attacked you with the rope you’d be okay if you had the rope-defense card: the neck brace. Kinda silly. I remember playing it with my mom and dad; Mom was concerned, but played anyway. My Dad was less concerned, and I think had to admit that it was actually kinda fun. Since the game was transformed from Chutes and Ladders, I also decorated the game box. I crossed out Chutes and Ladders and wrote Murder for the title. The box originally had a bunch of kids running up the ladder of a slide, sliding down, then running back around to the ladder. I drew in weapons that the kids were carrying, so it became a bunch of kids chasing each other with knives, guns, and ropes.

Although the game definitely had a dark twist to it, I remember having a very clear conscience about the whole thing.

A little later I had the same type of response to video games. After playing a good video game, a fire was kindled in me to create a video game myself. Some of these fires would die out very quickly, stifled by the actual difficulty of creating these things. But a handful of them were seen through to creations of noteworthy significance for someone of my age at the time. These things would be my book, Future Experience, in fifth grade; the Murder board game; and Cirkup’s Quest, a platform video game written in BASICA.

I wandered around a bit with this, so I’ll note the main “breadcrumb” that I wanted to record here: the creative effort of others can inspire creativity in us and or kids.

Posted in Programming for Kids | Leave a comment

Inventing Worlds

I know there are some things about Software Development I really like, while there are other parts that I really don’t like. I’ve often wondered, mostly as fleeting thoughts lost in the bustle of a work day, what causes the difference. Why do some days just fly by, and I can code for a solid work day, and get home and all I want to do is keep coding? Why do other days drag on and on, and I can’t wait to be done?

I was glancing through Chapter One of this book (can read online for free) and this paragraph crystalized some thoughts around this. I really like inventing worlds.

“Every other science deals with the universe as it is. The physicist’s job, for example, is to understand how the world works, not to invent a world in which physical laws would be simpler or more pleasant to follow. Computer scientists, on the other hand, [do].”

That’s why starting new software projects is so fun for me – when you lay the groundwork, you are inventing a world. You’re composing the natural laws for a new universe. The abstractions you create in code become its atoms and molecules.

Not all universes are created equal. If you don’t do a good job creating your universe, it will become overly complex and difficult to operate in. If you do a good job, you can create something like The Architect of The Matrix describes:

“The first matrix I designed was quite naturally perfect, it was a work of art, flawless, sublime.”

It’s the world creation part I really enjoy. It’s where you can “listen to the code tell you what it wants to be.” Generally a new software project has a lot of that up front, then over time it transitions to a more “construction” phase. The opportunity to create new abstractions (new types of atoms and new natural laws) never completely goes away, but it does diminish. Projects where the opportunity for that level of creativity is lowest are the ones that are the least fun for me to work on.

Posted in General PC Usage, Programming | Leave a comment

Securely accessing RavenDB / RavenHQ over HTTP

I want to use a RavenDB database hosted in the cloud by RavenHQ from a NodeJS app. I didn’t find a JavaScript client or NPM package that looked very mature, so I looked into straight HTTP API access. This took me a little time to figure out – I wasn’t able to find a step-by-step guide, so here’s one:

1. Get an API key from the RavenHQ website.

I guess they’re GUIDs. They’re straightforward to set up and configure with the access you’d like. Let’s pretend mine is 19ecad1c-d90e-4a98-a2e7-e79f40881cef.


2. Hit the OAuth server to exchange your API key for an access token.

Make an HTTP GET request to the following URL. Include the two headers shown.

  • Api-Key: 19ecad1c-d90e-4a98-a2e7-e79f40881cef
  • grant_type: client_credentials

You should get back a JSON document that looks something like this:

   Body: "{\"UserId\":\"19ecad1c-d90e-4a98-a2e7-e79f40881cef\",\"AuthorizedDatabases\":[{\"Admin\":false,\"TenantId\":\"your-database\",\"ReadOnly\":false}],\"Issued\":63504123456789.6}", 
   Signature: "ABigLongNastyBase64Thing=" 


3. Access your RavenHQ server with the appropriate Authorization Header.

Now you should be able to use the HTTP API as documented, including the entire access token from the previous step in an Authorization header, like this:

  • Authorization: Bearer {“Body”:”{\”UserId\”“, Signature: “ABigLongNastyBase64Thing=“}

To be clear, after the word “Bearer” and a space goes the entire JSON document you got back from the OAuth server, encoded as a string.

When you issue this request you should get an HTTP 200 response with JSON content matching what’s in your database.


4. Expiration

When the token expires your request will get a 401 Unauthorized response. Here are some of the interesting response headers that you’ll also get:

  • OAuth-Source:
  • WWW-Authenticate: Bearer realm=”Raven”, error=”invalid_token“,error_description=”The access token is expired

With this information you can detect that your token has expired and get a new one per Step 1 above.

Posted in JavaScript, JSON, RavenDB, REST | 3 Comments

Theodore Roosevelt – amazing story

My family heard a local historian, Steve Stark, talk about Theodore Roosevelt this past Sunday. Mr. Stark seemed to really like the guy, and from listening I found myself liking him as well. Steve briefly mentioned how TR had marched three hoodlums to Dickinson all by himself, and during the long sleepless journey also managed to read the book Anna Karenina. I was interested to look into it further and found this summary of the journey — really interesting:

I like this sentence that TR wrote; found it something to think about. It’s hard to relate, now-a-days, to conditions like these:

[The three criminals] belonged to a class that always holds sway during the raw youth of a frontier community, and the putting down of which is the first step toward decent government.

I wonder: people who would have belonged to “this class” in a frontier community – how do they get on in today’s community, in our community, which has “put down” this class and formed “decent government?”

  • Are they the criminals that our police pretty well protect us from today?
  • Does the influence of this culture redirect them toward a more beneficial course of life?
  • Do they find more sophisticated ways to carry out their basic thievery (white-collar crimes, etc)?

I suppose all three, depending on the combination of luck and ability dealt each individual.

Back to TR, between this story and the one where he boxed down a guy with two six-shooters pointed at him, apparently he really was quite a guy. As busy as he was, even as President he kept regular appointments to play with his children, even if it meant ending meetings with congressmen. That’s a good admonition and reminder for me.

Posted in History | Leave a comment

Nomic Candyland

King Kandy - candy-land PhotoRecently I heard of an interesting game called Nomic. You don’t need any equipment — just some creative and indulgent friends. Today I got together with my creative brother-in-law and my creative oldest son to play Nomic Candyland.

You start with the regulation Candyland rules, but add that at the beginning of every turn the player may propose a new rule to the game. If the rule passes by majority vote it takes effect at the beginning of the next turn.

We ended up with a pretty fun game:

  1. Every player gets 10 hitpoints; if you lose all your hitpoints you’re out of the game.
  2. Players would hold a hand of three cards and would select cards to play from the hand.
  3. All the special candy cards (lollipop, ice cream cone, etc.)  are weapons that can hit players up to three tiles away. As the game proceeded some weapons acquired special abilities, for example, eating the gingerbread man to gain hitpoints; the candy cane being a machine gun with higher range; the peanut being a hand grenade with a range of four inches in any direction on the game board. Landing on another player dealt one hitpoint of damage.
  4. If you reach the castle at the end of the map, you don’t win; you “level up” and return to the start of the map. Eventually it was added that leveling up allowed you to move faster and deal more damage.
  5. imageAt the end of the Candyland board game is “Gloppy.” In order to pass him you have to defeat him in battle. The rules of battle assigned relative value to different colored cards. “Uh-oh, Gloppy drew a green card, that’s not looking good for me.”
  6. Additional special passageways were invented.
  7. A robber was added that wandered the board according to certain rules, hitting players and stealing their cards.

We had fun. The final turn went something like, “I’ll play a red card; since I’m at level two I’ll make it a double-red. That jumps me over the robber so he can’t hurt me, and I land on the red player, damaging them down from three hitpoints to two. Finally I’ll shooting the red player with my candy cane, finishing him off to win the game.”

Posted in Uncategorized | Leave a comment

I am in the woods and I have gone crazy.

After reading books from The David Birkenhead series I was looking for some similar sci fi and decided on some Heinlein books. The only one on the shelve at the library was Variable Star.

The main character is very smart. He’s talking to one of the most powerful businessmen in the solar system — or rather, he’s being spoken at by this person. As the protagonist half-listens, his mind wanders about how he and this formidable leader are in such different stations in life, and look at things from such different perspectives, that if he speaks his mind to this man, he won’t understand, he’ll just think it’s crazy.

It reminds the main character of a story (from some googling this story may be true). A man named Russell wrote a song called I’m Lost in the Woods. Since the melody sounded African he wanted a Zulu background chorus for it. But when he talked to translators about how to translate the title of his song, they all told him it wasn’t possible to translate: Zulu don’t get lost in the woods, and so they don’t have that concept. There was no common ground for Russell to relate to the Zulu what his song was about. He had to settle for a translation of, “I am in the woods and I have gone crazy.”

First, that’s awesome for the Zulu. There’s a macho-coolness in being a people for whom getting lost in the woods is such a non-possibility that they don’t even have a way to say it.

Second, sometimes communication can be that way between different cultures, or people with vastly different backgrounds and/or perspectives. You can try to communicate something important, but all the other person can ascertain is that you must be crazy, because what you’re saying doesn’t make any sense. From his perspective, at least.

Last, I suppose if you’re on the receiving end, and are tempted to label the speaker as crazy, it might be better to be patient to listen to more, to be creative with your listening, and to delay negative judgments (like, “that guy’s nuts”) for as long as you can.

Posted in Softer Skills | 1 Comment

Lessons From Some Indie Sci-Fi

A technical blog I follow recently posted a side-note about some indie sci-fi books he was enjoying. I started reading the David Birkenhead series and I found it so exciting that I finished the series as far is available so far. There were a few little sections in Midshipman that I wanted to reflect on further.

Privilege and Responsibility

In the protagonist’s early years he comes under the tutelage of a great instructor, and he reflects on one of the early lessons he learned:

What my tutor wanted me to learn from the experience was that the higher a person climbs in life and the more power they accrue, the greater their obligations towards others become and the less personal freedom of action they enjoy. Life cannot in fact be lived ay other way without become a mockery of itself.

If the individuals involved have any honor, that is. Or empathy.

This realization by the hero is poignant because he is a freed slave in the process of making more and more of himself. As he does so, he finds himself obligated by a sense of duty and honor to take on more and more difficult tasks. He sense of commitment to the well-being of others less fortunate also drives him to a more difficult service than his former slavery. Furthermore, he sees both true and false dignitaries all around him, and he becomes aware of the great burden that the true dignitaries around him carry. Their sense of honor and their empathy for those less fortunate make their positions a real labor and heavy sense of responsibility, while false dignitaries only enjoy the privileges of their positions. On can see how this latter life makes a mockery of dignity and reveals a cold-hearted and false character.


Later in the book the protagonist and this same instructor are discussing fairness. In his adolescence the hero gives the too-often-used whine, “It’s not fair!” The instructor suggests others who have life even worse than the protagonist, then adds this little discourse:

“And then we might consider the destitute, the ignorant, the endless legions of the crippled…” He shook his head and sighed [I especially enjoy the genuine sympathy the greats in this book frequently show]. “Life’s never fair, David, not can it be made fair. No matter how hard people try. If I could share only one lesson with you, this would be the one. Every single person there’s every been was a universe unto themselves, and their path through life a unique, individual voyage that can’t be meaningfully compared with that of another. Nothing’s ever the same twice and therefore no two lives, or two people, are comparable.” He sighed. “So fairness is an illusion, David, and yearning for it is a waste of precious time.”

Being a father of several children, the “It’s not fair!” complaint comes around. I suppose that if what their really saying is, “That’s not just!” then I ought to intervene and fix the circumstances. But if things are just, but not fair, then the primary thing to focus on is helping my son or daughter to grow up a bit more.

Of course, adults also can waste their time contemplating how life isn’t fair. As adults, we still need to mature — grow up — from time to time.

Honor and Recognition

I’m just realizing that all three marks I made in my copy of Midshipman are discourses of the protagonist’s instructor. Here’s the bit this time:

“One of the traits I’m expected to encourage in my charges is a true sense of nobility. An understanding that being a nobleman is more about obligations than privileges, and that true honor is the greatest burden that [a man] can ever bear.” He shook his head, then turned to face me. “The greater one’s abilities and power, the greater their obligations. Some people never come to understand that, David. They become monsters like the Emperor… devoting their entire lives to seeking the very highest titles and decorations. To them [recognition or a crown] is more precious than gold. They’ll do anything – even commit terrible crimes – to obtain what they seek, all the while never having a clue about the substance that necessarily lies behind the symbol. Others, however, live quietly and try to do what’s right even when it’s not easy. They don’t seek medals or crowns or glory – instead, all they’re interested in is satisfying the demands of their own personal sense of honor. Yet, though this sort of man doesn’t chase after honors, somehow the honors tend to find him.”

I find these lessons, and the protagonist’s discovery of them and experience with them, to be a rich way to internalize these myself.

Parent’s Note

I find the David Birkenhead books exciting and thought provoking to read. I find the protagonist to be a truly an inspiring hero. I find myself wanting to share these stories together with my kids in their pre-teen or teen years, but there is some content that gives me pause. I’ll share these with you:

  • Throughout the books there is war going on, and some of the effects of the violence are mentioned, including the blood and dismembered body parts. I felt the book treated these subjects with an appropriately broad brush: mentioning them, but not dwelling on them.
  • In Midshipman the protagonist draws a suggestive picture in order to distract an adversary; a couple details are given about the picture that would probably fall between a PG-13 and R rating.
  • In Lieutenant the protagonist sees evidence left on some clothing that one character is raping another. In my opinion the book treats this serious and sad issue situation with due gravity. I mention it because parents will generally want to control when and to what extent their children are exposed to this sad and sick reality of life.
  • In Captain the protagonist struggles with the fact that the spouse he was uniquely intended for is not in his life, particularly in the area of the natural urges of a young man. He sees a doctor on the matter, who advises him to stay very busy and take long, cold showers. It’s not a major theme of the book; the whole things is described and dealt with in just a few pages.
  • Commodore and Admiral haven’t been released as of the time of this writing.
Posted in Leadership, Softer Skills | Tagged | 3 Comments

Book: The Goal

I’ve never before had a business book that kept me turning pages like The Goal: A Process of Ongoing Improvement. Someone on one of the blogs I read suggested this so I requested it from the local library. When the 384-page paperback showed up I groaned – I do not usually get through nonfiction of this size. This is a business novel, which teaches its business principles through a story, so that helped a lot. I enjoyed both the story and was thrilled with the subject matter so much it really kept me moving till I got to the end.

I recognized many of the principles it teaches being related to Lean manufacturing, but didn’t realize right away that this book was written a long time before Lean hit mainstream. For some reason that gives it more credibility in my mind.

The book is one man’s journey to debunk common manufacturing theory errors. He has the help of a “guru” type character. Of course every book where the protagonist has a guru, it’s important that the guru be scarce. You’ll find this frequently in fantasy books – the main character has a guide, but that guide suddenly disappears for long periods of time, leaving the protagonist to struggle and often experience failure. Consider Gandalf in any story he’s in, from The Hobbit through The Return of the King. That guy is disappearing (or dying, or having his staff broken) at the most inconvenient times! Leaving poor little hobbits and dwarves to fend for themselves, struggling (and learning and growing) in a cruel world.

Getting back to the book, here are some basic principles I found fun to ponder.

The success of a manufacturing plant can be improved by working on three values. In order of importance they are:

  1. Increasing throughput to sales.
  2. Decreasing inventory
  3. Decreasing operating expense.

To continually over time make changes in a manufacturing plant that move these numbers in these directions is to become more successful. In the book, the real value is in increasing throughput. The book purposely deemphasizes operating expenses.

  • Throughput is limited by the bottlenecks of the production system. The bottlenecks must be identified.
  • Trying to optimize anything besides the bottlenecks is a waste of time: it will not improve throughput.
  • Keep the bottleneck resources running! An hour lost for a bottleneck resource means an hour lost all the way up and down the system.

The book illustrates why throughput is limited by the bottlenecks by including a section about a boy scout troop hiking single-file for six miles.

    • The rate at which the last person of the troop travels the trail is throughput.
    • The amount of space between the first and last scouts is inventory.
    • The amount of effort each boy uses is operating expense.

One of the boys is a “bottleneck” because he’s a bit out of shape. When the bottleneck is in the middle of the single-file line, bad things happen. The faster folks up front get way ahead, representing high inventory. The folks behind him can’t go any faster than the bottleneck. Throughput of the entire system is constrained to the bottleneck, even when the other resources have excess capacity. Even the scouts up front, way ahead, aren’t helping matters because the product isn’t done until the last boy arrives at the camp. All they are doing is increasing inventory (waste) by always increasing the space between the first and last people.

The main character puts the bottleneck up front, and demands that everyone stay in line. Inventory at this point goes way down (good!) but throughput is still unchanged – they’re still moving along at the bottleneck’s pace. The boys complain about the slow-poke up front.

Then a very interesting thing happens. The protagonist gets everyone thinking about constructive ways to make the bottleneck faster. Before long they remove his backpack from him and share it among everyone who wants to go faster (they have excess capacity). Now the whole line speeds up:

    • Throughput it up! Everyone is moving faster.
    • Inventory is down! Everyone is still together.
    • Operating expense is down! The brisk yet steady pace is efficient for everyone.

The leadership team at the manufacturing plant applies these, and other principles, to their plant to great success. Here’s how they end up summarizing what they’ve done:

  1. Identify the system’s constraints (bottlenecks).
  2. Decide how to exploit the system’s constraints (focus everything on maximizing utilization on the constraint: minimize downtime, do a QA check on parts before they go through the bottleneck, etc.).
  3. Subordinate everything else to the above decision (don’t try to maximize non-bottleneck resources; make them work in a way that supports the bottleneck. The idea of a “cadence” in a production line is in line with this).
  4. Elevate the system’s constraints (try to improve the raw capacity of the bottlenecks, e.g., purchase more equipment or subcontract it out).
  5. Warning: if in this process a constraint is broken, go back to step 1, but do not allow inertia to cause a system’s constraint. (I actually don’t know what this means.)

The book concludes with a generalization of management. The hypothesis is that good management is properly figuring out (1) what to change, (2) what to change it to, and (3) how to change it. I think those might be clarifying steps to follow when one gets bogged down in a management situation.

Posted in Business, Lean | Leave a comment

Posting an AJAX JSON request with jQuery

After trying to Google this for the fifth time, and writing it from scratch for the seventh time, I’m sticking this snippet here. In AJAX apps, it’s always made sense to me to do JSON both ways, even though the jQuery default is form-data up and whatever down. Once ASP.NET MVC3 made JSON model binding turned on by default, something like the below is a natural next step.

(function ($) {
    $.postJSON = function (url, obj, callback) {
            url: url,
            type: 'POST',
            dataType: 'json',
            data: JSON.stringify(obj),
            contentType: 'application/json; charset=utf-8',
            success: callback
} (jQuery));

Note: JSON.stringify is built in to modern browsers now, but for older browser just find json2.js (commonly available) and include it on your page.

Posted in CodeMinder, JavaScript, JSON, Programming, REST | 1 Comment