Crunching numbers here. I’ve been testing the performance of running Tuist commands vs. xcodebuild. Specifically when it comes to tests. My findings have been that whenever I run tuist test I’ll get it somewhat slower than if I run xcodebuild build-for-testing and xcodebuild test-without-building. The difference is about 2%, but those seconds add up in CI, specially for larger projects!
I’m wondering if there are any best practices for using Tuist to improve performance of the test command.
There’s also the idea of build-for-testing. We use this so we can reuse builds in different CI test tasks. Is there something like that with Tuist?
Expectation
tuist test to be better or equal than xcodebuild.
Context
Tuist version: 4.43.1
Reproduction (mandatory for problems)
Time tuist test.
Time xcodebuild build-for-testing followed by xcodebuild test-without-building.
Times will be better for xcodebuild. I pressume difference is going to be barely recognizable unless is a large project.
“tuist generate” invokes “xcodebuild test” under the hood after generating the project. What numbers do you get if you run “xcodebuild test” and how do they compare to the combination of “build-without-testing”, and “test-without-building”?
It’d be great to also get a breakdown by running Tuist from the sources and checking the time breakdown with instruments.
Project generation adds time, but I’d not expect it to be that high.
I benchmarked it including the project generation part for when I used xcodebuild. So tuist test takes longer than tuist generate + xcodebuild build-for-testing + xcodebuild test-without-building. I’ll profile this further with Instruments and see what I find.
I’ll profile this further with Instruments and see what I find.
This would be great . Having a more granular breakdown will help spot where the time spent.
When the project is connected to the server, there might be some network requests to pull state for caching and selective testing, but I believe this is not your case?
tuist test is a bit different from tuist generate as we try to focus on unit test targets whereas tuist generate would pull in binaries just for external dependencies. In other words, tuist generate only hashes external dependencies, not your source code. I’d try running tuist test --no-binary-cache --no-selective-testing and compare those numbers with tuist generate --no-binary-cache --no-selective-testing + xcodebuild ....
Hashing internal targets for really large projects might add up, but the tradeoff is usually worth it if you use selective testing and binary caching.
I’ll keep testing this in the future, as we do want to try using selective testing and caching in the future. However, there’s a big limitation here for us and that’s that we can’t “build without testing” and “test without building”. This is particularly important for us and there are two specific scenarios:
Integration tests: the test scheme is built first, and then the individual tests are sharded into many multiple agents running them. We don’t want to have to build the scheme in every agent.
Unit tests: we have some flaky tests that pass on retry. So we build for testing, and then test without building. If it fails, we test without building again. If we don’t do this, then it will take significantly longer even if some of the stuff is cached in derived data.
. Having a more granular breakdown will help spot where the time spent.