An alternative file server for Go

28 Dec 2015 16:19 GMT

Go's net/http package contains a simple FileServer that lets you map a request path to a directory and serves files under that path like a normal webserver. It's a handy way to serve static files from inside an otherwise dynamic web application.

Go's built-in FileServer is fine if you're just using it for development, but it has two shortcomings if you want to use it in production. First, when it can't find a requested file it responds with a very basic plain text 404 page. Second, when the user requests a path to a directory which does not contain an index page, it shows the directory listing, with no easy way to turn that off.

The FileHandler in the handlers package is an alternative file server that addresses both of these problems. FileHandler lets you specify your own handler for serving 404 pages, and it responds to any request for a directory that does not contain an index page with that handler, rather than show a directory listing. FileHandler is just a wrapper around Go's ServeFile function, so it still has all the goodness of Go's built-in file handling.

The handlers package also contains a simple NotFoundHandler which you can use with FileHandler. But you don't have to. FileHandler can use any type that satisfies the Handler interface to serve custom 404 pages when no matching file is found.

Decimals

16 Nov 2015 22:51 GMT

Decimals is a small library of functions for rounding and formatting base ten numbers in Go. Most of the software I write involves presenting numerical data, and Go's standard library lacks convenient methods for displaying numbers in a way humans find easy to read.

Decimals uses simple and consistent rules for rounding and formatting numbers across all its functions. There are other formatting libraries that offer a wider range of output formats, but decimals aims to make it as easy as possible to deal flexibly and extensibly with the most common case.

See the read me for a full write-up or jump straight into the GoDoc.

How to learn Go

11 Nov 2015 21:33 GMT

During the last two months I have been learning to program with Go. Go is a general purpose programming language which has features that make it well-suited for fast concurrent network applications. It is a very enjoyable language to use, in that it is terse, clear, well thought-through, and gets rid of a lot of the drudgery of programming, so that you can focus on building things rather than managing code.

This post is not a tutorial on programming with Go, but a recipe for how to learn the language. It consists of a short primer on the language and a guided reading list, which will take you from knowing nothing about Go to building complete applications.

It is aimed at people who already know at least one other programming language. If you want to learn Go as your first programming language then you should read An Introduction to Programming in Go, and then perhaps come back here for more later.

What sort of language is Go?

Go is a statically typed language, which means you have to declare what type of variable you are using, and convert between types when necessary. This is much less of a pain in practice than in theory, as Go has a number of conveniences for type handling, like a short assignment operator that infers the type of a new variable from what you assign to it. Static typing also has some benefits: it helps expose flawed reasoning earlier rather than later.

Go has pointers. Or at least, it has its own version of pointers. Pointers feel daunting at first if you are used to languages that don't use them, but they quickly become second nature. Pointers make explicit when you are passing by reference and by value, and let you decide which is most appropriate for your own functions and data structures case by case. In Go, a pointer is a special data type that allows for referencing behaviour but does not permit bare-metal (and potentially dangerous) memory-level hackery such as pointer arithmetic.

Go does not have objects, constructors or traditional object-oriented inheritance. Instead it has user-defined types, which can have methods attached to them, and interfaces, which define a set of methods that a type must have in order to implement them.

Go makes it easy to do concurrent programming safely. Go has a lightweight thread called a goroutine. Any function can be run within its own goroutine simply by calling it with the “go” keyword. Goroutines can communicate with one another through channels, so different threads can easily synchronise their tasks.

Go has playgrounds, which are online notebooks of Go code that you can run in the browser and share. Playgrounds are useful for drafting and testing functions and types in isolation. Go is compiled and can't be run interactively like Python or R, so playgrounds give you a similar sort of freedom to play and experiment with code.

Go is prescriptive. Its creators have strong opinions about how the language should be used, and encourage you not only to learn its idioms, but to understand the reasons for their language design choices. The source code to the standard library can be read online and the authors encourage you to explore it in order to learn how to use the language well.

How you react to this aspect of Go's culture will depend on your personality. I love it. I don't want to waste time fretting over stylistic formatting decisions or how best to organise code files. Go gives clear and unambiguous answers to the questions of how you should use the language, so you can concentrate on solving the programming problems as effectively as possible.

How to get going

Start with the Tour of Go. This is a short interactive guide to the most essential parts of the language that demonstrates its distinctive features. Because the code examples are shown as Go playgrounds it's easy to discover additional things about the language by testing your intuitions and modifying the example code as you go. Don't expect to learn the language from the tour though, it's just laying the foundations.

Next, install Go.

Then read How to Write Go Code. This takes you through setting up a Go workspace and the process of writing, building and running Go programs. From this point on you will probably want to start trying things out and it's good idea to get everything set up and ready.

Then, read Effective Go. This is the full introduction to the language. If you only want to read one thing to learn Go, it's this. It explains not just how the language works but why it works the way it does. It manages to fit an awful lot into a relatively short text.

How to keep going

The official Go Blog contains a back catalogue of articles exploring particular aspects of the language in depth. The article on slices is essential reading for anyone learning Go because slices are one of the most fundamental data structures in the language. The follow-up article on strings is also very helpful, as it shows you how to manipulate Unicode strings using the rune datatype. The article on error handling is another good one for beginners to read because Go does not use the standard try and catch approach.

The documentation for all of the packages in the standard library is available at GoDoc. It's well written and often tells you everything you need to start using a package effectively. You can also use the GoDoc website to read the documentation for any Go package hosted on GitHub. Just add the package path (e.g. github.com/mattn/go-sqlite3) to the GoDoc domain.

How to go further

Unit testing is built in to the standard library and is done with the testing package. The documentation will probably provide enough information if you're already familiar with unit tests, while Go(lang) Unit Testing for Absolute Beginners at JonathanMH's blog is a nice introductory tutorial on unit tests in Go which assumes no prior knowledge.

Database access in Go is done with the database/sql package, which provides a common interface to database operations across different databases. The Go database/sql tutorial is a step by step guide to the interface and how to use it with specific databases.

Go is so well suited to server side software there is an official tutorial on Writing Web Applications. Go has a flexible and powerful template system built into the standard library. The chapter on Templates in the GitBook Build web applications with Golang is a good introduction.

Final thoughts

Go is a great general purpose programming language, but that doesn't mean it's the best language for every purpose. Python and R are better for data analysis, and Go can't replace JavaScript for doing data visualisation in a browser. But what it does well, it does very well indeed. Although I have been using it for just a couple of months, I think it may be the most promising of the new generation of languages for building applications on the web.

Population Builder

17 Aug 2015 11:23 GMT

My latest experiment in data-driven web development is Population Builder, a web-app that lets you build a population estimate for a set of small areas in Great Britain.

Zoom into the map anywhere in Great Britain and the app will overlay boundaries of small areas. These areas are lower layer super output areas (LSOAs) in England and Wales, and data zones in Scotland. These are the smallest areas for which annual population estimates are published. (Currently the most recent estimates at this level are for mid-2013, but data for mid-2014 will be out in the autumn.)

A screenshot from the application showing a map of London.

Clicking or tapping these areas will select them, adding their estimated population to the total shown in the counter in the top right corner. Clicking or tapping “Get data” under the population estimate for the selected areas will take you to a results page, which shows a visualisation of the age-distribution of the population and provides a link to download the data.

I developed Population Builder in my spare time to contribute to a project taking place in Parliament to explore using the web to make data more directly available to the public. In my work I am often asked to produce ad-hoc population estimates for arbitrary small areas, and while the data is readily available, the process is time consuming. Population Builder tries to make that as easy as possible.

The software is still at a relatively early stage: it aims to perform its core task well, but I can imagine other features that would make it even more useful — linkable selections would be very nice, for example, but present some interesting challenges in this case.

Nevertheless, I felt it was sufficiently developed to release into the wild for feedback and suggestions. Consider this an open beta. Please let me know what you think.

How experienced is the new House of Commons?

25 May 2015 15:45 GMT

How much experience of Parliamentary service does the newly elected House of Commons have? The answer is that it depends on how you measure it. The following chart shows the mean and the median number of years served in the Commons by MPs elected at each general election since 1983.

Chart showing the mean and median years service by MPs elected at each general election

The mean is what people typically think of when they talk about averages — it's the total number of years that MPs in each group have worked in the Commons divided by the number of MPs in the group; while the median is the value which divides the group in half — in each group of MPs, half have fewer years of Parliamentary service than the median and half have more.

As the chart shows, the typical number of years service among MPs elected at each general election differs depending on whether you measure it using the mean or the median. What's more, the trends differ between the two measures.

Using the mean, the MPs elected in 1997 were the least experienced of any Parliament since 1983. But using the median, the group of MPs elected in 2001 were the least experienced.

So which is the more appropriate measure? Arguably, you need to consider both measures, because the difference between the two reflects the shape of the distribution of experience among the MPs in each group.

The following histograms show the distribution of Parliamentary experience among each cohort of MPs elected at general elections from 1983 to 2015. In each histogram the bars show the percentage of MPs whose total length of service in the House of Commons falls between the numbers shown on the X axis at the bottom of the chart.

Chart showing the distribution of Parliamentary experience among MPs elected at each general election

So around 29% of MPs elected at the 2015 General Election have fewer than four years experience in the Commons — these are the MPs elected for the first time at the 2015 General Election plus some of those elected at by-elections during the last Parliament.

The difference between the mean and the median in a group is determined by the overall shape of the distribution. In more normal distributions — the more convex and bell-shaped groups — the mean and the median are closer together; while in more skewed distributions — the more concave and L-shaped groups — the mean and the median are further apart.

Generally speaking, the mean is the better measure for normally distributed data, while the median is a better way to summarise data that is skewed. Here the skewness of the distribution varies among groups of MPs, so if you had to choose just one measure of Parliamentary experience, the median is the more robust statistic.

The distribution of experience among MPs elected in 2010 and in 2015 is similar, but not identical. Both distributions are skewed. But both the mean and the median years of service are almost exactly the same in both groups. And although it is possible to identify the group of MPs newly elected in 2010 as a more experienced group in 2015 (among those with four to eight years experience) a similar sized group of newly elected MPs also joined the House in 2015, while at the same time the proportion of MPs with more than 12 years service fell.

So while many ministers now have several years experience in government, the new House of Commons is on the whole no more experienced than the last. But the precise mix of experience among elected MPs is particular to each Parliament.

This article was originally published on Second Reading, the House of Commons Library blog.