11 Nov 2015 21:33 GMT
During the last two months I have been learning to program with Go. Go is a general purpose programming language which has features that make it well-suited for fast concurrent network applications. It is a very enjoyable language to use, in that it is terse, clear, well thought-through, and gets rid of a lot of the drudgery of programming, so that you can focus on building things rather than managing code.
This post is not a tutorial on programming with Go, but a recipe for how to learn the language. It consists of a short primer on the language and a guided reading list, which will take you from knowing nothing about Go to building complete applications.
It is aimed at people who already know at least one other programming language. If you want to learn Go as your first programming language then you should read An Introduction to Programming in Go, and then perhaps come back here for more later.
What sort of language is Go?
Go is a statically typed language, which means you have to declare what type of variable you are using, and convert between types when necessary. This is much less of a pain in practice than in theory, as Go has a number of conveniences for type handling, like a short assignment operator that infers the type of a new variable from what you assign to it. Static typing also has some benefits: it helps expose flawed reasoning earlier rather than later.
Go has pointers. Or at least, it has its own version of pointers. Pointers feel daunting at first if you are used to languages that don't use them, but they quickly become second nature. Pointers make explicit when you are passing by reference and by value, and let you decide which is most appropriate for your own functions and data structures case by case. In Go, a pointer is a special data type that allows for referencing behaviour but does not permit bare-metal (and potentially dangerous) memory-level hackery such as pointer arithmetic.
Go does not have objects, constructors or traditional object-oriented inheritance. Instead it has user-defined types, which can have methods attached to them, and interfaces, which define a set of methods that a type must have in order to implement them.
Go makes it easy to do concurrent programming safely. Go has a lightweight thread called a goroutine. Any function can be run within its own goroutine simply by calling it with the “go” keyword. Goroutines can communicate with one another through channels, so different threads can easily synchronise their tasks.
Go has playgrounds, which are online notebooks of Go code that you can run in the browser and share. Playgrounds are useful for drafting and testing functions and types in isolation. Go is compiled and can't be run interactively like Python or R, so playgrounds give you a similar sort of freedom to play and experiment with code.
Go is prescriptive. Its creators have strong opinions about how the language should be used, and encourage you not only to learn its idioms, but to understand the reasons for their language design choices. The source code to the standard library can be read online and the authors encourage you to explore it in order to learn how to use the language well.
How you react to this aspect of Go's culture will depend on your personality. I love it. I don't want to waste time fretting over stylistic formatting decisions or how best to organise code files. Go gives clear and unambiguous answers to the questions of how you should use the language, so you can concentrate on solving the programming problems as effectively as possible.
How to get going
Start with the Tour of Go. This is a short interactive guide to the most essential parts of the language that demonstrates its distinctive features. Because the code examples are shown as Go playgrounds it's easy to discover additional things about the language by testing your intuitions and modifying the example code as you go. Don't expect to learn the language from the tour though, it's just laying the foundations.
Next, install Go.
Then read How to Write Go Code. This takes you through setting up a Go workspace and the process of writing, building and running Go programs. From this point on you will probably want to start trying things out and it's good idea to get everything set up and ready.
Then, read Effective Go. This is the full introduction to the language. If you only want to read one thing to learn Go, it's this. It explains not just how the language works but why it works the way it does. It manages to fit an awful lot into a relatively short text.
How to keep going
The official Go Blog contains a back catalogue of articles exploring particular aspects of the language in depth. The article on slices is essential reading for anyone learning Go because slices are one of the most fundamental data structures in the language. The follow-up article on strings is also very helpful, as it shows you how to manipulate Unicode strings using the rune datatype. The article on error handling is another good one for beginners to read because Go does not use the standard try and catch approach.
The documentation for all of the packages in the standard library is available at GoDoc. It's well written and often tells you everything you need to start using a package effectively. You can also use the GoDoc website to read the documentation for any Go package hosted on GitHub. Just add the package path (e.g. github.com/mattn/go-sqlite3) to the GoDoc domain.
How to go further
Unit testing is built in to the standard library and is done with the testing package. The documentation will probably provide enough information if you're already familiar with unit tests, while Go(lang) Unit Testing for Absolute Beginners at JonathanMH's blog is a nice introductory tutorial on unit tests in Go which assumes no prior knowledge.
Database access in Go is done with the database/sql package, which provides a common interface to database operations across different databases. The Go database/sql tutorial is a step by step guide to the interface and how to use it with specific databases.
Go is so well suited to server side software there is an official tutorial on Writing Web Applications. Go has a flexible and powerful template system built into the standard library. The chapter on Templates in the GitBook Build web applications with Golang is a good introduction.
17 Aug 2015 11:23 GMT
My latest experiment in data-driven web development is Population Builder, a web-app that lets you build a population estimate for a set of small areas in Great Britain.
Zoom into the map anywhere in Great Britain and the app will overlay boundaries of small areas. These areas are lower layer super output areas (LSOAs) in England and Wales, and data zones in Scotland. These are the smallest areas for which annual population estimates are published. (Currently the most recent estimates at this level are for mid-2013, but data for mid-2014 will be out in the autumn.)
Clicking or tapping these areas will select them, adding their estimated population to the total shown in the counter in the top right corner. Clicking or tapping “Get data” under the population estimate for the selected areas will take you to a results page, which shows a visualisation of the age-distribution of the population and provides a link to download the data.
I developed Population Builder in my spare time to contribute to a project taking place in Parliament to explore using the web to make data more directly available to the public. In my work I am often asked to produce ad-hoc population estimates for arbitrary small areas, and while the data is readily available, the process is time consuming. Population Builder tries to make that as easy as possible.
The software is still at a relatively early stage: it aims to perform its core task well, but I can imagine other features that would make it even more useful — linkable selections would be very nice, for example, but present some interesting challenges in this case.
Nevertheless, I felt it was sufficiently developed to release into the wild for feedback and suggestions. Consider this an open beta. Please let me know what you think.
25 May 2015 15:45 GMT
How much experience of Parliamentary service does the newly elected House of Commons have? The answer is that it depends on how you measure it. The following chart shows the mean and the median number of years served in the Commons by MPs elected at each general election since 1983.
The mean is what people typically think of when they talk about averages — it's the total number of years that MPs in each group have worked in the Commons divided by the number of MPs in the group; while the median is the value which divides the group in half — in each group of MPs, half have fewer years of Parliamentary service than the median and half have more.
As the chart shows, the typical number of years service among MPs elected at each general election differs depending on whether you measure it using the mean or the median. What's more, the trends differ between the two measures.
Using the mean, the MPs elected in 1997 were the least experienced of any Parliament since 1983. But using the median, the group of MPs elected in 2001 were the least experienced.
So which is the more appropriate measure? Arguably, you need to consider both measures, because the difference between the two reflects the shape of the distribution of experience among the MPs in each group.
The following histograms show the distribution of Parliamentary experience among each cohort of MPs elected at general elections from 1983 to 2015. In each histogram the bars show the percentage of MPs whose total length of service in the House of Commons falls between the numbers shown on the X axis at the bottom of the chart.
So around 29% of MPs elected at the 2015 General Election have fewer than four years experience in the Commons — these are the MPs elected for the first time at the 2015 General Election plus some of those elected at by-elections during the last Parliament.
The difference between the mean and the median in a group is determined by the overall shape of the distribution. In more normal distributions — the more convex and bell-shaped groups — the mean and the median are closer together; while in more skewed distributions — the more concave and L-shaped groups — the mean and the median are further apart.
Generally speaking, the mean is the better measure for normally distributed data, while the median is a better way to summarise data that is skewed. Here the skewness of the distribution varies among groups of MPs, so if you had to choose just one measure of Parliamentary experience, the median is the more robust statistic.
The distribution of experience among MPs elected in 2010 and in 2015 is similar, but not identical. Both distributions are skewed. But both the mean and the median years of service are almost exactly the same in both groups. And although it is possible to identify the group of MPs newly elected in 2010 as a more experienced group in 2015 (among those with four to eight years experience) a similar sized group of newly elected MPs also joined the House in 2015, while at the same time the proportion of MPs with more than 12 years service fell.
So while many ministers now have several years experience in government, the new House of Commons is on the whole no more experienced than the last. But the precise mix of experience among elected MPs is particular to each Parliament.
This article was originally published on Second Reading, the House of Commons Library blog.
25 May 2015 15:01 GMT
When I first launched this blog I designed it around a single colour theme. Everything was rendered in black and white with a blue accent, which was present in the masthead and the links, and which became the predominant colour in all of the graphics. I now realise what a terrible idea that was for a blog that features data visualisation.
Trying to keep all of the graphics consistent with the colour scheme proved extremely limiting and was starting to make the site feel predictable and dull. More importantly, the choice of colours in a visualisation should be decided by what you are trying to reveal in the data, rather than the context in which the data is presented.
So I have tweaked the design to make the look and feel completely neutral with regard to colour, leaving me free to choose whatever colours I want or need for visualisation purposes. I have also retrospectively changed some of the earlier visualisations to introduce more variety to the visual style and to use more appropriate colours.
Lesson learned: information design should not be artificially constrained by web design.
24 May 2015 15:14 GMT
For the last nine months, along with my day-to-day work for the House of Commons Library, I have been working on a project to produce a 2015 General Election results dataset for Parliament, as well as a briefing paper summarising the results.
As you can probably imagine, Parliament needs detailed and accurate election results data as soon as possible in order to carry out its functions, for everything from allocating office space and Short Money, to reporting the gender balance of MPs, to simply finding out as quickly as possible who is coming to work as an MP for the first time.
It was an intense project that involved developing a database and a web front-end for managing the data, as well as recruiting a team of (amazing) volunteers to gather and verify it.
We started entering election results data at 4:00am on the Friday following the election, which was very exciting, although we could have done without the fire alarm that meant we had to evacuate the building for half an hour at 5:15.
This week we made our data public. The full results can be downloaded in two CSVs from Parliament's research briefings website, along with the briefing paper. You can also explore the same data interactively through a new website developed by Parliament's rapid apps team called General Elections Online, or Geo for short.
The results have been gathered directly from Returning Officers and include complete data on polling day electorates and invalid ballots. A future edition of the dataset will also include declaration times.
While we have done everything possible to ensure the figures are accurate, the data should be treated as provisional. We will correct any errors we find as quickly as possible, so always check online for the latest version of the dataset and briefing paper whenever you intend to use them.