What the ��� Is Fuzzing

Go 1.18 introduced a number of changes, but the much-anticipated introduction of generics to the language stole the spotlight. Since the release, many people have been talking about generics and when to use them, but 1.18 also introduced native support for fuzzing to the go ecosystem which isn’t getting quite as much attention. Fuzzing is an interesting concept and Go the first language to support it natively, so let’s take a look at what this fuzzing thing is and what it can do for us!

Concept

Although it isn’t an entirely new concept, I have no previous experience with fuzzing test techniques. My introduction to fuzzing came with the introduction of it to Go, and the explanation from the Go blog where they explain:

“Fuzzing is a type of automated testing that continuously manipulates inputs to a program to find bugs. Go fuzzing uses coverage guidance to intelligently walk through the code being fuzzed to find and report failures to the user. Since it can reach edge cases which humans often miss, fuzz testing can be particularly valuable for finding security exploits and vulnerabilities.”

Cool! So fuzzing will generate inputs and test cases that we might not think of and execute them extremely quickly. It will write failing cases to a local test data directory so that we can read the input that triggered a failure and turn that input into a unit test.

Example

• string, []byte
• int, int8, int16, int32/rune, int64
• uint, uint8/byte, uint16, uint32, uint64
• float32, float64
• bool

Fuzz tests look and behave similar to normal Go unit tests, but with f *testing.F replacing t *testing.T as the test argument, and f.Fuzz replacing t.Run. We also need to seed the fuzz engine with some values to set a baseline. We do this via the f.Add method. The f.Add method can take an arbitrary number of arguments, but the number and types must correspond to the arguments in the f.Fuzz method. I worked through the Go blog tutorial which uses the example of a string reversal function and shows how we can incrementally improve it using the bugs found via the fuzz tests. I also tried out writing some simple fuzz tests myself to get a better understanding of the bits and pieces involved

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

func FuzzRepeat(f *testing.F) { // fuzz tests use a `Fuzz` prefix instead of `Test`
	// Seed corpus
	f.Add("a", 50)
	f.Add("Geek Space 9", 6)
	f.Add("I Love Go!", 8)

	f.Fuzz(func(t *testing.T, input string, times int) { // takes a target function and passes it fuzzed parameters
		result := Repeat(input, times)

		if result != "" && strings.Count(result, input) != times {
			t.Errorf("Invalid result, input  %s got %s", input, result)
		}
	})
}

If you execute the test like a normal Go test, it will behave as such. It will use the seed corpus as a test table and tell you if there is a failure.

We can launch our fuzz test with the command:

go test -fuzz=FuzzRepeat

We can optionally set a time limit for our fuzzing by providing the optional -fuzztime argument:

go test -fuzz=FuzzRepeat -fuzztime 30s

This is useful for integrating into CI pipelines, otherwise, it will run indefinitely or until it encounters an error.

After running our fuzz test we see the below output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

fuzz: elapsed: 0s, gathering baseline coverage: 0/31 completed
fuzz: elapsed: 0s, gathering baseline coverage: 31/31 completed, now fuzzing with 16 workers
fuzz: elapsed: 3s, execs: 527521 (175831/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 6s, execs: 991471 (154610/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 9s, execs: 1442237 (150247/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 12s, execs: 1910904 (156263/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 15s, execs: 2365211 (151409/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 18s, execs: 2827139 (153986/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 21s, execs: 3275903 (149612/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 24s, execs: 3731937 (151980/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 27s, execs: 4166144 (144763/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 30s, execs: 4646505 (160109/sec), new interesting: 0 (total: 31)
fuzz: elapsed: 30s, execs: 4646505 (0/sec), new interesting: 0 (total: 31)
PASS
ok      fuzzing 30.115s

There isn’t a lot happening for these tests, as the code we are testing is extremely simple. Let’s coerce our function into a panic and see how the fuzzer handles it. I added the following if statement to the implementation of the Repeat function:

1
2
3

	if input == "GeekSpace9" {
		panic(1)
	}

I then ran the fuzz test again but without a time limit go test -fuzz=FuzzRepeat. This time I don’t use a time limit because I want it to run until it encounters the failing test input.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

fuzz: elapsed: 1m0s, execs: 9767423 (162878/sec), new interesting: 0 (total: 27)
fuzz: elapsed: 1m3s, execs: 10247679 (160086/sec), new interesting: 0 (total: 27)
fuzz: elapsed: 1m6s, execs: 10729739 (160664/sec), new interesting: 0 (total: 27)
fuzz: minimizing 46-byte failing input file
fuzz: elapsed: 1m6s, minimizing
--- FAIL: FuzzRepeat (66.32s)
    --- FAIL: FuzzRepeat (0.00s)
        testing.go:1349: panic: %!s(int=1)
            goroutine 666736 [running]:
            runtime/debug.Stack()
                /usr/local/go/src/runtime/debug/stack.go:24 +0x90
            testing.tRunner.func1()
                /usr/local/go/src/testing/testing.go:1349 +0x1f2
            panic({0x593800, 0x5fd8f0})
                /usr/local/go/src/runtime/panic.go:838 +0x207
            github.com/kidsan/gs9_article_golang_fuzzing.Repeat(...)
                /home/kieran/test/fuzzing/main.go:7
            github.com/kidsan/gs9_article_golang_fuzzing.FuzzRepeat.func1(0xc0056b9380, {0xc000016ae6, 0xa}, 0xffffffffffffffe0)
                /home/kieran/test/fuzzing/main_test.go:15 +0x2b7
            reflect.Value.call({0x5978e0?, 0x5d0c58?, 0x13?}, {0x5c2839, 0x4}, {0xc0067ca1e0, 0x3, 0x4?})
                /usr/local/go/src/reflect/value.go:556 +0x845
            reflect.Value.Call({0x5978e0?, 0x5d0c58?, 0x514?}, {0xc0067ca1e0, 0x3, 0x4})
                /usr/local/go/src/reflect/value.go:339 +0xbf
            testing.(*F).Fuzz.func1.1(0x4eac5f?)
                /usr/local/go/src/testing/fuzz.go:337 +0x231
            testing.tRunner(0xc0056b9380, 0xc0002b58c0)
                /usr/local/go/src/testing/testing.go:1439 +0x102
            created by testing.(*F).Fuzz.func1
                /usr/local/go/src/testing/fuzz.go:324 +0x5b8
            
    
    Failing input written to testdata/fuzz/FuzzRepeat/2d310e8334aaa3e128bf0c24ef2dc15a9a7fb321e976afaa0e2d9d048d48cfda
    To re-run:
    go test -run=FuzzRepeat/2d310e8334aaa3e128bf0c24ef2dc15a9a7fb321e976afaa0e2d9d048d48cfda
FAIL
exit status 1
FAIL    github.com/kidsan/gs9_article_golang_fuzzing   66.322s

Nice! In just 1 minute and 6 seconds, the fuzzer ran the function 10729739 times and found that the input “GeekSpace9” caused a panic. It then created a file locally in testdata/fuzz/FuzzRepeat containing the following:

go test fuzz v1
string("GeekSpace9")
int(-32)

Subsequent runs of the FuzzRepeat test use this testdata as baseline coverage to catch any regressions. This basic example is available to run at this repository.

I wanted to see a more complex scenario for fuzz testing, and fortunately, the Go blog links to the Go Fuzzing Trophy Case where some real-world bugs caught with fuzzing can be reviewed. The go-zero framework also uses some fuzz tests which are a bit more complicated and were interesting to read (for example, they fuzz the number of workers in their mapreduce tests).

What’s Next?

Now that I have some foundational knowledge on fuzzing, I want to see if I can find an area where it would benefit us in the applications we are developing at Geek Space 9. Adding fuzzing tests to our pipelines to ensure the security and reliability of our microservices may help us identify unforeseen issues. It seems from the examples in the Go Fuzzing Trophy case that fuzzing is a particularly useful tool in identifying improper handling of non-utf8 encoded string values, so any exposed interface that accepts string values and performs manipulation or validation of that string will be a good candidate for fuzzing.