John’s General Bobbins

 

Here

Tags

Subscribe



February 18, 2010

Noodling with Node.js

Been having a little play around with a new(ish) thing - Node.js. Only at a simple noddy level admittedly, but it's already one of the more interesting technologies I've come across recently. 

Essentially it's an extremely scalable event-driven web server. OK, it does a few other things, but the focus is definitely on low level HTTP and TCP.

It's JavaScript based, and built on top of Google's V8 open source JavaScript engine.

Standing on the shoulders

Built by a small Danish team, V8 is a really impressive JavaScript compiler. It's cross platform, very fast, and is the VM used inside the Chrome browser. The lead engineer is Lars Bak - the technical lead on Sun's HotSpot VM, and before that, Smalltalk. So it has some pedigree.

Then there is CommonJS. This is a JavaScript library spec, concentrating on server side development. Previously server side JS has been a rather fragmented affair, with multiple implementations. CommonJS attempts to define some standards in this area. Node.js had adopted the modules and unit testing from the CommonJs project.

Node.js builds on top of these, wrapping it all with an event driven model with non-blocking I/O and handy System/HTTP modules.

What it solves

An event based server solves certain specific problems that can occur when scaling up long running I/O or requests in more traditional architectures. Simon Wilison explains it best, naturally using hamsters, bunnies, and a hyperactive squid.

In a standard web server, like JEE, I/O and calls to other services will tend to be done synchronously within a thread. Scaling is achieved by creating a new thread per connection. If you've potentially long running processes in there, then you'll block that thread, using up increasing server resources or slowing down requests. Some of this can be mitigated by using MQ to add asynchronous handling for certain things, or Java NIO for async I/O, but it's interesting to see Node.js has this approach ingrained across the board.

Threaded models drop in performance as concurrency increases. The non-blocking event model stays relatively steady. Note it's not quite this simple - you're backend still has to scale. But it does remove at least one bottleneck.

The event driven model in Node.js uses the callback pattern to say "do this bit of possibly long running stuff, then call this other method when you're done". It can then carry on with other processing or other requests immediately.

Examples

Installing Node.js means a build and compile currently, but it's a very simple deal. Though note at the time of writing it needs a Mac or Linux (or any Posix type thing) - Windows isn't supported yet. See the Node.js website for more. Running is simply a case of typing "Node <your JS file>".

Non-blocking calls

The main thing Node.js add is callbacks across the board. All I/O become asynchronous. This means calls change slightly:


The second example happens to be an anonymous function, but could be any function defined elsewhere. The key is when you call any long running process, you also set what to call when it finishes. It the meantime you move on so no time is wasted.

This will be a really familiar idiom to anyone who's done much Ajax - it's the same idea, do something async & pass in a function to call when you're done. JavaScript just happens to be really well suited to this pattern with it's first order functions and closures.

Basic web server - Hello World!

Node.js has built in libraries for handling low level HTTP, TCP and DNS along with general I/O operations. Creating a basic web server is simple:

A Twitter client

You can also create an HTTP client relatively easy. Here's one to query Twitter (a fairly canonical example, adapted and annotated)

Practical use

It doesn't look like you'd want to write an full application using Node.js - it's far too low level. And it's not exactly mature yet. But some possible uses spring to mind:

  • Put it on a subdomain to have it handle Comet style long running/push requests
  • Similarly, use it to handle uploads that could go on a long time
  • As it contains an HTTP server & client, it could be used as a quick proxy or for routing requests
  • The HTTP client is easy to use and negligible CPU/memory when not active, could be left running to poll websites for availability or running tests
  • It's got a nice TCP library - makes talking across sockets a breeze. We used to have a WebSphere server talk to a backend machine over sockets - this would have been ideal for receiving and allocating those requests.
  • Combine with CouchDB simply for the bizarre situation of having all your tiers running JavaScript.

Seems to be something to keep an eye on...

Loading mentions Retweet

Comments [0]

January 5, 2010

Cardiff Montage

So I wanted to have a play around with iMovie and still photos - syncing to music, pans, transition timings and so on. 

Thought I'd stick the results up here too in case out of interest for anyone who knows Cardiff, or far away folks who wonder where I currently live (Cardiff Bay). 

Photos

The photos are mainly my own, plus a selection of the best Creative Commons shots I could find on Flickr. Credits to:

Music

The music is by an excellent local artist called Martyn Joseph, appropriately enough called Cardiff Bay. MP3s are available on his site (including some free ones!). This track is a live version, with Suzanne Vega's band backing.

Loading mentions Retweet
Filed under  //   photos   videos  

Comments [0]

January 5, 2010

Why we work

A post in which I waffle randomly and tangentially about working.  

Adieu

So, after 10 years of solid working I decided to take a short break at the end of last year. 

Being in the (admittedly very lucky) financial position of not having to dive into somewhere else immediately, I went and made the most of it. Cue visiting friends unseen in ages, a bit of travel, putting people up to stay, working through those long neglected todo lists. If you're ever in the situation where you get that chance, would certainly recommend it.

But as much as all that I wanted to take a step back and think what I like to do next work wise - and hence what I really wanted from work.

Ponos

Ponos was the Ancient Greek word for work. Alternative translations: Pain. Labour. Toil...

Ancient Greeks.  (photo creative commons from Flickr)

Now today of course it's slightly different for a lot of people. Software development's extremely lucky in that it's one of a few professions that can genuinely be fun, interesting and challenging, beyond the material reward. In fact many carry on and work for free on open source or private projects just for the enjoyment of it. I'm not sure how many accountants do that... 

Of course, in corporate environments this fun aspect can get hidden under the red tape on occasions. But generally it's something you can, and should, still get passionate about. Overall it's a damn good career to be in really.

It's all about the money, honey... 

This probably seems quite obvious - the main reason to work is to pay that mortgage, no? And the minor stuff, like food etc. Unless you've got enough investments to live off, win the lottery or marry a millionaire, you'll need to work for this reason.

But then I quit a highly paid job. This may have been slightly bonkers, given the economic climate. I was certainly valued, no doubt there - even offered an increase to stay. But the roles, priorities and people of the company had changed over the course of the last year. Hey, it turns out it's not all about the money.

So what is it you really look for in a job, once you're over a certain salary?

Futurama had it's career chips to remove all that hassle...

The essentials

There's a few things you always need. Or want at least. I guess the work equivalent of the bottom rows of Maslow's triangle.

  • Money - Yes, we all need it. Enough to live comfortable, ideally. 
  • Location - To commute or not to commute is the main question. Some people like that time on a train. Or prefer to live close by. Or even relocating. Personally speaking, Cardiff is a lovely place to live though. Always realised living here limits choices slightly more than London (of which have had several offers from companies in the past), but the quality of life is more important for now.
  • People - Generally speaking, you come to realise most people are nice. Certainly in our industry. A great team can make up for an awful lot, and help mask any other deficiencies. The last place we were all lucky - an awesome group who'd been together for a long time. Having others around is one of the reasons I decided freelancing wasn't really for me.

Passion

Beyond those basics, you need something that motivates you when working. There's a lot of nonsense management exercises that happen in large companies, but one of the few interesting ones was around this area. Different things, unsurprisingly, motivate different people. Some of the most common ones were (vaguely remembering here...):

  • Work itself - a pride in doing a job well, craftsmanship
  • Respect of peers - recognition, the status that you know your stuff
  • Creativity - problem solving and imagination
  • Challenge - are you being stretched
  • Growth - carrying on learning, having a career path
  • Stability - safety, insurance, even the daily routine
  • Personal life - does it give you a chance to have a full life outside work
  • Power - influence you have within (and even outside) the company
  • Salary - not just in monetary terms, but associated status for some 

Where your focus is changes as you go through life though. So if you've just had a child, you're far more likely to be after job security and money for a bit. If you're younger and fancy free you my be more about the life balance and future prospects.

Management roles (especially senior) when we did these sessions tended towards power and salary often, while developers often like intellectual challenge. Interestingly this mismatch being a reason why bad managers sometimes can't motivate teams - as they're projecting what would motivate themselves onto others. 

Work needs to give you a chance to use whatever motivates you in life generally.

Turns out personally I'm motivated mainly by creativity & problem solving, with a side order of respect for being good at it. And enough personal life not to be a total workaholic.

Work on stuff that matters

At a conference last year, Tim O'Reilly was urging developers to work on "stuff that matters" (one of his main themes in 2009 generally). Now of course that definition will vary from person to person, but it struck a chord in amongst all the tech-oriented sessions.

There's a lot of extremely smart people in this industry. But it's not always put to the best use for the wider good.

I've got a funny background. By some chance almost my entire family are in the "caring professions" or public sector. Nurses, doctors, social workers, teachers. There's even a nun in there. Some branches have the Catholic notion of doing "good" work. Barring a couple of legal types, I was one of the only people to go into the corporate world. And a financial company to boot... Software development is a curious thing though - which sector you work in doesn't actually make such a difference. The challenges are always roughly similar.

Still, having taken the corporate shilling, I preferred to work if at all possible on something that had some kind of vaguely worthwhile element. Most companies have varying levels of Corporate Social Responsibility that aim for this. Especially if there's a side effect on the bottom line... So after a while I ended up with one of my hats being in charge of accessibility of the web applications for disabled people (and usability more generally). Still working in the area I was interested in, but with some feeling that it was doing some good.

There's no need to always work for a charity, or something equally altruistic. Anything that helps the wider social community in some small way, even in large companies. 

Anyway

I think this has been a rambling wander on a theme. Most blog posts with be a little more focussed (maybe...). Heck, it'll probably be some meme about favourite films or some such.

But it's worth having a short think now and again about why you're working and what you're after. You know, before you retire.

Loading mentions Retweet
Filed under  //   project 52   work  

Comments [0]

January 3, 2010

Project 52

So, as one of my NY resolutions I though it'd be interesting to commit, in the loosest possible sense, to a year of this latest midi-blog. Over at http://project52.info/ several people have had the same idea. 

I'll try and post some kind of content here once a week. On average. Ish. Quite possibly may not post when on hols in Barbados/New York/Skegness. First one coming up shortly...

Hat tip to that there Ann (http://www.pixeldiva.co.uk/ & @pixeldiva) for pointing out the Project 52 site.

Loading mentions Retweet
Filed under  //   a bit meta   project 52  

Comments [0]

November 3, 2009

Blimey. I've got a blog again.

Well, I say blog. Somewhere between twitter and a full blog. A midi-blog, if you like.

Here be dragons?

What wonders will be found here? Well, the intention originally was largely work related things. That's general design, architecture, usability and accessibility (not just of the web, in the wider world too), software development and methodologies. But then that's how I started using twitter - sensibly, to keep in touch with people from conferences. After a while it turned a bit more random. So realistically a mix of odds and ends found on the internets with some commentary, some self created stuff, and occasional personal bits & bobs too. 

This isn't exactly the first incarnation. Some of you may remember me from such hits as Geocities '98, and a handwritten blog, before the term was coined, in the early 2000s. Over the past couple of years my online self has been restricted to the usual social networks - over there in links on the right. But now and again longer form writing is needed, so blogs it is again.
 
Credentials

In case you're a random visitor, best explain what position these posts are coming from. I've been a lead developer (and, oh, that title covers so many things...) for large financial types for 10 years. So that's a hella lot of experience with multi million pound enterprise cough developments. This has surprisingly involved some very forward thinking projects, with agile development, cutting edge web work and a very good name in accessibility.

Note that last point is largely thanks to ex colleague Mike Davies - who was one of the rare bloggers from corporate accessibility land. One of the many goaders into me restarting blogging. There's always been a lack of bloggers, speakers etc from this area of development for some reason - internal teams in big FTSE places, despite a lot on interesting work going on in that area. Perhaps a subject for a future post...

Loading mentions Retweet
Filed under  //   a bit meta   about  

Comments [0]