Back in August 2013 I started to learn Go, a powerful new programming language which promised to give Club Log a performance boost thanks to its clever concurrency features. For a review of why I chose Go, you might want to read my original post.
I’m writing on 10 November, and it’s all gone as planned! Today is a significant watershed as, for the first time, Club Log is now running on GoDXCC. This means that all DXCC lookups (for new uploads and of course when interacting with the Club Log web site) are being performed by GoDXCC, and there is no longer a PHP lookup.
One of the big wins – the one I’d been hoping for – was performance. When I started to benchmark the old PHP code, I was surprised to find it can take as much as 30ms to calculate a DXCC, depending on the slashes and prefixes in question. In the Go code, even without concurrency, I was able to get this down to 1.5ms. This enhancement is mainly due to the fresh approach, because with hindsight and experience the PHP code was inherently slower. A simple win, for example, is to load and retain all of the prefixes and exceptions in a memory map (instead of using recursive SQL lookups to do the work on demand). Since GoDXCC is a persistent daemon, and exposed exclusively via a simple HTTP API, it was in this area that some really big performance wins were made.
Once I added concurrency I was able to get the average lookup time down to 300 microseconds (0.3ms), which makes GoDXCC 100x faster than its PHP predecessor. I had to add various mutual exclusivity lock-outs at this stage: GoDXCC needs to accept semaphores to reload its maps, a bit like invalidating a Memcache hash, and still carry on processing requests while this is happening. That was a new challenge for me, and in places the GoDXCC code is considerably less accessible than PHP because of these issues – but that’s a small price to pay for a hundred-fold reduction!
You might detect I am happy with Go. In fact, working with Go is a pleasure. It stretches your imagination and gives you more expressive ways to code. It is a very novel language, which feels convenient and accessible, but is capable of tackling incredibly advanced work. The way Goroutines and channels interact is simply perfect for my needs in Club Log, so now I have broken through the learning stages I am keen to press on and use Go in more places. The next thing I will tackle will be the remaining bottleneck in new uploads, the ADIF parser.