Go gotchas - WaitGroup (and struct paramaters / method receivers)

Go has been my go-to side project language for quite some time now (since before v1.0), and when I started the Matasano crypto challenges it seemed like a perfect fit for a number of reasons - it doesn’t force me to write a lot of boilerplate, is low-level enough to allow implementing your own crypto primitives and it comes with a rich standard library (I am looking at you Scala). I made my way through the first set and while solving one of the problems I wanted to run a certain function in parallel.

The simplest way of making this use all CPU cores is to run each calculation in it’s own goroutine, and the standard way tracking if all of them completed is using a sync.WaitGroup - for simplicity’s sake this code assumes we only care about side effects (printing) and do not consume the result:

package main

import (
	"fmt"
	"sync"
)

func main() {
	numbers := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
	var wg sync.WaitGroup
	for _, n := range numbers {
		wg.Add(1)
		go func(in int) {
			fmt.Printf("%d: %d\n", in, cpuIntensive(in))
			wg.Done()
		}(n)
	}
	wg.Wait()
}

// does something CPU intensive
func cpuIntensive(n int) int {
	return n * n * n
}

package main

import (
	"fmt"
	"sync"
)

func main() {
	numbers := []int{1, 10, 100, 1000}
	var wg sync.WaitGroup
	for _, n := range numbers {
		wg.Add(1)
		go runInGoroutine(n, wg)
	}
	wg.Wait()
}

func runInGoroutine(in int, wg sync.WaitGroup) {
	fmt.Printf("cpuIntensive(%d): %d\n", in, cpuIntensive(in))
	wg.Done()
}

// does something CPU intensive
func cpuIntensive(n int) int {
	return n * n * n
}

What is happening here - we only extracted a function? Go is kind enough to let us know that our program is deadlocked - but why? Our simple refactoring wasn’t correct - we changed the way the code run in goroutine uses the sync.WaitGroup variable. Previously it closed (as in closure over it, now it takes it as a parameter.

The issue is that the WaitGroup is passed by value, so each goroutine gets a copy of the WaitGroup. This means that when we call wg.Wait() we are waiting on a WaitGroup that will never be modified by child goroutines, and we will never exit the main function. The fix is simple - we pass a pointer to all goroutines, letting WaitGroup take care of concurrent modifications:

package main

import (
	"fmt"
	"sync"
)

func main() {
	numbers := []int{1, 10, 100, 1000}
	var wg sync.WaitGroup
	for _, n := range numbers {
		wg.Add(1)
		go runInGoroutine(n, &wg)
	}
	wg.Wait()
}

func runInGoroutine(in int, wg *sync.WaitGroup) {
	fmt.Printf("cpuIntensive(%d): %d\n", in, cpuIntensive(in))
	wg.Done()
}

// does something CPU intensive
func cpuIntensive(n int) int {
	return n * n * n
}