Inconsistency when testing network implementations with the URLProtocolStub strategy

The problem

This week I stumbled across some issues in my CI. The tests were failing “occasionally”, but by taking the time to investigate, I noticed some inconsistency.

"Xcode displaying a XCTestExpectation fulfill error"

There was a problem in URLSessionHTTPClientTests, which is an infrastructure test suite that ensures proper usage of Apple’s URLSession (it works for other network modules too, like Alamofire). The strategy used in the tests is to intercept requests by using a stubbed URLProtocol with a observerRequests property. This is because we need to assert values like the URL and HTTP method used. You can learn more about this strategy in this WWDC 2018 session and by taking some time to look at the implementation. The tests are as shown:

func test_cancelGetFromURLTask_cancelsURLRequest() {
    let receivedError = resultErrorFor(taskHandler: { $0.cancel() }) as NSError?
    XCTAssertEqual(receivedError?.code, URLError.cancelled.rawValue)
}

func test_get_makesAGetRequestWithProvidedURL() {
    let url = URL(string: "https://www.a-specific-url.com")!
    let exp = expectation(description: "Wait for request observation")

    URLProtocolStub.observeRequests { request in
        XCTAssertEqual(request.httpMethod, "GET")
        XCTAssertEqual(request.url, url)
        exp.fulfill()
    }

    makeSUT().get(from: url) { _ in }

    wait(for: [exp], timeout: 1.0)
}

The two cases test_cancelGetFromURLTask_cancelsURLRequest and test_get_makesAGetRequestWithProvidedURL were failing when running next to each other. Further investigations demonstrated that canceling a URLSessionDataTask didn’t always stop URLProtocolStub.startLoading from running, so even after the cancel test is finished, the request still initiates.

"Debug statements with timestamps"

Both requests execute after test_get_makesAGetRequestWithProvidedURL sets up its observer, so this is the reason the expectation is called twice, causing API violation — multiple calls made to -[XCTestExpectation fulfill] to show.

The are two solutions to this problem:

1. Only run the stub if the URL matches.

override class func canInit(with request: URLRequest) -> Bool {
    return request.url == stub.url
}

This way the unwanted request is blocked from running in other tests. One drawback here is that the additional URL property in the stub will lead to a lot of changes in existing code.

2. Wait for the request completion in the cancel test case

func test_cancelGetFromURLTask_cancelsURLRequest() {
	let exp = expectation(description: "Wait for request")
	URLProtocolStub.observeRequests { _ in exp.fulfill() }

	let receivedError = resultErrorFor(taskHandler: { $0.cancel() }) as NSError?
	wait(for: [exp], timeout: 1.0)

	XCTAssertEqual(receivedError?.code, URLError.cancelled.rawValue)
}

Considering the observer will be called at the end of startLoading, waiting for an expectation will also prevent the unwanted request from running in other cases. It fixes the issue, doesn’t require a lot of changes, and gives the test a nice context.

Conclusion

Even though this issue occurred from a specific framework behavior, it shows a nice example of how using expectations can make tests more consistent. We can’t control the URLProtocol internals but we can protect our codebase from its actions.

I could only solve this issue in my codebase with the assistance of the people at iOS Lead Essentials. I posted this issue on our community and got instant help! I also learned this URLProtocolStub implementation from them ✅.

References

URLProtocol docs

URLProtocolStub implementation

URLSessionHTTPClientTests.swift