Tuist Registry Initiative

Initiatives
1

This initiative’s main goal is to improve the SwiftPM resolution by implementing the Package Registry standard.

About

Why

When resolving dependencies, SwiftPM defaults to doing deep clones to resolve the correct version. Package Registry is a standard that allows the implementation of centralized registries, so that consumers only need to download source archives for versions they need instead of doing deep clones.

We plan to provide all packages in the Swift Package Index in a Tuist implementation of the Package Registry to improve the resolution times.

Goals

The goals of this initiative are:

  • Server implementation of the Package Registry. We’re mostly done there.
  • Implement convenient integration for:
    • Tuist projects using XcodeProj-based integration
    • Tuist projects using the default SwiftPM integration
    • Vanilla SwiftPM projects
    • Vanilla Xcode projects with the default SwiftPM integration
  • Contribute back to swift-package-manager and improve its performance
1 Like
2

Hey folks :wave:

I wanted to provide an update with some initial performance findings that we gathered from testing our registry with tuist/tuist and our staging server environment.

We will also soon open the registry to initial testers – either tomorrow or early next week. We’ll provide the details here and in Slack.

Measurements

Here are the numbers I gathered (note the gathered numbers are from 1-2 runs, so take them with a pinch of salt)

Without registry :
Cold run without registry and with cache (Saving cache time 10s, install dependencies 1 min 7 s - 1 min 17s): Add actions cache · tuist/tuist@7dddb0b · GitHub
Warm run without registry and with cache (restore cache 8s, ~388 MB, install dependencies 20 s): Add actions cache · tuist/tuist@7dddb0b · GitHub

With registry:
Cold run with registry and cache (install dependencies 51 s - 55 s): Add actions cache · tuist/tuist@bce21b1 · GitHub
Warm run with registry and cache (restore cache 2s, ~71 MB - install dependencies 23 s): Add actions cache · tuist/tuist@bce21b1 · GitHub

Findings

The time savings for downloading and uploading the CI cache of the .build directory are significant – around 80 %.

The resolution step itself is faster by around 20 %. We expected slightly better numbers and we believe that we can contribute back to the swift-package-manager to improve those further.

Primarily, the archives could be downloaded in parallel, whereas now they are downloaded serially. The downloads occur here. When the loop is run in parallel, the download part of the resolution is 80 % faster – down to around 2 seconds from 13. We will try to open a PR with this change. Unfortunately, once and if merged, the change would only ship with the next Swift version.

We’ll also investigate if there are other parts of the resolution that can be sped up.

2 Likes
3

Hey!

We believe the Registry is ready for initial testers. Before wider announcement and general availability, we’d like to confirm with some early adopters that the registry works as expected.

Additionally, we’d love to get numbers for how much faster the registry resolution is for you. If you could share them here or in Slack (DMs are fine, too!), that’d be very much appreciated.

To get started with the registry, you can follow the official docs.

If you have any questions or general feedback, let us know either here or in Slack.

Note that the registry will get even faster with the new release of the Swift toolchain thanks to this PR , but you should see performance improvements already as documented in the previous post.

4

Hi Marek! I’m interested in giving this a go in our project. May I ask how I would best go about benchmarking our current setup or the most important figures to capture to compare the registry change results reliably?

5

Thanks @kelvinharron for your interest!

There are two scenarios that would be interesting to test:

  • Use registry in your CI setup. I’d recommend running the flow at least twice with registry and compare that with your average resolution times without registry. If you cache the resolved .build folder across CI runs, it would be good to compare also the time that the CI takes to download the cached artifacts.
  • Locally, I’d recommend using hyperfine
  • First, set a baseline without registry by running hyperfine --prepare 'rm -rf ~/.swiftpm .build && swift package purge-cache'--warmup 1 'command-to-resolve. command-to-resolve would be tuist install if you are using the XcodeProj-based integration. Note this command needs to be run from the directory where your Package.swift is located if you are using the XcodeProj-based integration.
  • Second, compare that by setting up the registry and rerunning the same command. Make sure that packages are resolved via registry. For how to ensure that, following our docs should be sufficient.
6

I tested locally with hyperfine as described by @marekfort.

  • The benchmark is not perfect as I forgot to update Tuist in the SPM benchmark. So it used an old version of Tuist, but the registry benchmark used 4.39.1
  • I had to update Config() to Tuist() in between benchmarks
hyperfine --prepare 'rm -rf ~/.swiftpm .build && swift package purge-cache' --warmup 1 'tuist install'
Benchmark 1: tuist install
  Time (mean ± σ):     128.019 s ± 10.807 s    [User: 122.502 s, System: 34.209 s]
  Range (min … max):   112.304 s … 142.985 s    10 runs

The first approach with the installation option flag failed with:

The 'swift' command exited with error code 1 and message:
error: 'ensembles-next': unknown dependency 'ZipArchive' in target 'EnsemblesZip'; valid dependencies are: 'ZipArchive.ZipArchive', 'RNCryptor-objc' (from 'https://github.com/mentalfaculty/RNCryptor-objc'), 'ObjectiveDropboxOfficial' (from 'https://github.com/mentalfaculty/dropbox-sdk-obj-c')

I’m not sure what the error is here as ensembles-next is private so I guess it’s not in the Tuist registry. In the Package manifest the dependency is declared as „ZipArchive“ in the target „EnsemblesZip“.

I then reverted the installation option and edited the Package.swift to follow this instruction:

dependencies: [
-   .package(url: "https://github.com/pointfreeco/swift-composable-architecture", from: "0.1.0")
+   .package(id: "pointfreeco.swift-composable-architecture", from: "0.1.0")
]

Benchmark 1: tuist install
  Time (mean ± σ):     141.842 s ± 11.060 s    [User: 115.828 s, System: 33.759 s]
  Range (min … max):   127.072 s … 159.361 s    10 runs

So the registry is 13,8 sec slower in this case. I can try again, if I did something wrong.

7

Thanks @kaioelfke for taking the registry for a spin :slightly_smiling_face:

This is a bug in SwiftPM that we fixed in: Fix resolve failing when package from registry is referenced by name by fortmarek · Pull Request #8166 · swiftlang/swift-package-manager · GitHub

Our registry also fixes Package.swift from the registry – but we can’t fix the issue for private packages as they are not in the registry. You either need to:

  • Fork that dependency and use full references such as .product(name: "ZipArchive", package: "ZipArchive"),
  • Don’t use --replace-scm-with-registry and use direct registry references as you did
  • Wait for the fix to land with the next Swift toolchain

It seems like there’s no id:branch: overload for url:branch dependencies?

Yes, such an option wouldn’t make sense for a registry. Registry is only for released versions by design.

These failed with no matching version errors, but does that simply mean it’s not in the registry perhaps?

Some packages were not populated properly. So yes, this is a bug on our side to be fixed soon.

RevenueCat didn’t work I tried GitHub - RevenueCat/purchases-ios-spm: This mirror of RevenueCat’s iOS SDK is optimized for integrating via SPM.

This package is not in the SwiftPackageIndex, so it’s expected we don’t have it in our registry: PackageList/packages.json at main · SwiftPackageIndex/PackageList · GitHub

and GitHub - RevenueCat/purchases-ios: In-app purchases and subscriptions made easy. Support for iOS, watchOS, tvOS, macOS, and visionOS. This also didn’t work GitHub - openalloc/SwiftRegressor: A linear regression tool that’s flexible and easy to use

Both of these should now be in the registry.

So the registry is 13,8 sec slower in this case. I can try again, if I did something wrong.

The registry being slower is not something I’ve seen, yet, so it’s somewhat surprising. Note on the CI, it’s better to cache the resolved directory (Tuist/.build in case of Tuist integration). The time it takes to download and upload the cache with registry should definitely be less as the registry is more efficient when it comes to storing data.

If you send me your Package.swift (either here or via DMs), I can also test things out on my machine.

One other thing that would be interesting to try is to check out the branch from this PR, build for release (swift build --configuration release) and then run hyperfine again, this time around with the specific binary:

hyperfine --prepare 'rm -rf ~/.swiftpm .build && swift package purge-cache' --runs 5 --warmup 1 '/path-to-cloned-swift-package-manager/.build/release/swift-package resolve'
8

Hi @marekfort apologies for the delay!

As we mentioned offline, the command I used in the Tuist directory that works is:
hyperfine --prepare 'rm -rf ~/.swiftpm .build && swift package purge-cache' --warmup 1 'tuist install'

Results - without Registry

  Time (mean ± σ):     76.274 s ±  7.563 s    [User: 39.720 s, System: 20.860 s]
  Range (min … max):   67.811 s … 90.747 s    10 runs

Results - with Registry

  Time (mean ± σ):     101.805 s ±  9.781 s    [User: 30.174 s, System: 17.020 s]
  Range (min … max):   89.292 s … 123.899 s    10 runs

How do these numbers look?

Configuring the Registry

I configured the registry in our org as per the docs which worked great. Then I updated our Tuist object with --replace-scm-with-registry set but tuist install would fail with the following output:

The 'swift' command exited with error code 1 and message:
warning: 'newrelic-ios-agent-spm': /Users/kelvinharron/IdeaProjects/ehr-ios/Tuist/.build/checkouts/newrelic-ios-agent-spm/Package.swift:9:15: warning: 'v9' is deprecated: iOS 12.0 is the oldest supported version
 7 |     name: "NewRelic",
 8 |     platforms: [
 9 |         .iOS(.v9), .macOS(.v10_14), .tvOS(.v9), .watchOS(.v10)
   |               `- warning: 'v9' is deprecated: iOS 12.0 is the oldest supported version
10 |     ],
11 |     products: [

/Users/kelvinharron/IdeaProjects/ehr-ios/Tuist/.build/checkouts/newrelic-ios-agent-spm/Package.swift:9:44: warning: 'v9' is deprecated: tvOS 12.0 is the oldest supported version
 7 |     name: "NewRelic",
 8 |     platforms: [
 9 |         .iOS(.v9), .macOS(.v10_14), .tvOS(.v9), .watchOS(.v10)
   |                                            `- warning: 'v9' is deprecated: tvOS 12.0 is the oldest supported version
10 |     ],
11 |     products: [
error: 'Quick.Quick': unknown dependency 'Nimble' in target 'QuickTests'; valid dependencies are: 'Quick.Nimble', 'apple.swift-argument-parser', 'apple.swift-algorithms', 'Quick.swift-fakes'
error: 'Quick.Quick': unknown dependency 'Nimble' in target 'QuickLintTests'; valid dependencies are: 'Quick.Nimble', 'apple.swift-argument-parser', 'apple.swift-algorithms', 'Quick.swift-fakes'
error: 'Quick.Quick': unknown dependency 'Nimble' in target 'QuickTests'; valid dependencies are: 'Quick.Nimble', 'apple.swift-argument-parser', 'apple.swift-algorithms', 'Quick.swift-fakes'
error: 'Quick.Quick': unknown dependency 'Nimble' in target 'QuickLintTests'; valid dependencies are: 'Quick.Nimble', 'apple.swift-argument-parser', 'apple.swift-algorithms', 'Quick.swift-fakes'

Consider creating an issue using the following link: https://github.com/tuist/tuist/issues/new/choose

I removed that replace text command and moved all the dependencies from urls to ids, with the following dependencies:

.package(url: "https://github.com/realm/realm-swift", exact: "20.00.0"),
.package(url: "https://github.com/hmlongco/Factory", exact: "2.3.2"),
.package(url: "https://github.com/1024jp/GzipSwift", exact: "6.0.1"),
.package(url: "https://github.com/evgenyneu/keychain-swift", exact: "24.0.0"),
.package(url: "https://github.com/siteline/swiftui-introspect", exact: "1.2.0"),
.package(url: "https://github.com/apple/swift-algorithms", exact: "1.2.0"),
.package(url: "https://github.com/apple/swift-collections", exact: "1.1.1"),
.package(url: "https://github.com/Quick/Quick", exact: "7.6.0"),
.package(url: "https://github.com/Quick/Nimble", exact: "13.3.0"),
.package(url: "https://github.com/pendo-io/pendo-mobile-sdk", exact: "3.5.0"),
.package(url: "https://github.com/IDScanNet/IDScanIDParserIOS", exact: "1.220127.1"),
.package(url: "https://github.com/Kolos65/Mockable", exact: "0.1.3"),
.package(url: "https://github.com/newrelic/newrelic-ios-agent-spm", exact: "7.5.3"),
.package(url: "https://github.com/auth0/JWTDecode.swift", exact: "3.2.0"),
.package(url: "https://github.com/Alamofire/Alamofire.git", exact: "5.10.2"),

To get tuist install to work correctly, I needed to change the following back to URLs

.package(url: "https://github.com/realm/realm-swift", exact: "20.00.0"),
.package(url: "https://github.com/IDScanNet/IDScanIDParserIOS", exact: "1.220127.1"),
.package(url: "https://github.com/newrelic/newrelic-ios-agent-spm", exact: "7.5.3"),
.package(url: "https://github.com/auth0/JWTDecode.swift", exact: "3.2.0"),

I faced repeated failures trying to bench it after all these changes.

Benchmark 1: tuist install
Error: Command terminated with non-zero exit code 1 in benchmark iteration 2. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.

I appended --show-output to the command and given how new relic or pendo resolution would often fail in our CI runs, I think I got lucky when it did work. Let me know if you have anything else for me to try. Tuist version 4.39.0.

9

Hey @kelvinharron!

The issue with Quick is resolved, thanks for reporting that.

As for performance numbers, I’ll be doing a bigger write-up later on, but the TL;DR is that:

  • clean checkouts won’t be significantly faster with the registry – for the time being. We’ll need to dig deeper into why as we do end up downloading much less data with registry, so there are probably some performance improvements we can do in SwiftPM directly.
  • on CI, we recommend caching the directory with packages. Restoring and saving the cache is improved by more than 80 %. The absolute time saving depends on how many dependencies you have and will be most likely in the range of 10 s up to almost 2 minutes.
  • Given the much improved space efficiency, your checked out repositories will become lighter in local environments (again, depends on how many dependencies you have)

For exact numbers, see: GitHub - tuist/registry-tests

1 Like
10

Thanks Marek! I’m really keen to adopt this as you pointed out in Slack, it could help with the failing resolves from Swift Package Manager which pendo and new relic are the known culprits. What is the effort to enable registry to pick up additional repos? is it something the community can help with?

I tried it again this evening and Quick failed for version 7.6.0 but bumping it to 7.6.2 fixed it. :slight_smile: Thanks for resolving it! I’m going to update our projects to use the registry going forward as I can’t see any drawbacks of adopting it, especially with how proactive you’ve been with the changes.

For your point about caching the directory with packages, would you recommend we do that as users of Tuist Cloud?

EDIT: silly question Marek, would you recommend on post clone of the repo but before Tuist install, we run the block of code in this CI setup example?

11

Great to hear that :crossed_fingers:

The error

error: 'Quick.Quick': unknown dependency 'Nimble' in target 'QuickTests'; valid dependencies are: 'Quick.Nimble', 'apple.swift-argument-parser', 'apple.swift-algorithms', 'Quick.swift-fakes'

is actually a bug in the SwiftPM registry implementation itself. The fix for it has been merged, but not released, yet.

We do have a workaround for this issue in our registry implementation by replacing .byName references with the full .product(name: "...", package: "...") and it looks like we haven't got the replacement right for Quick 7.6.0. We'll look into that – but glad to hear Quick 7.6.2` works for you well :slightly_smiling_face:

For your point about caching the directory with packages, would you recommend we do that as users of Tuist Cloud?

The caching of packages should be done with the standard CI cache – how exactly that’s done depends on your CI provider. We will soon roll out an update of the documentation with better instructions of what to do on the CI: Add registry keychain entry on the CI for projects using the Xcode default integration by fortmarek · Pull Request #7249 · tuist/tuist · GitHub. However, I’m honestly not sure how to set up caching for the .build directory with Xcode Cloud – I’d be surprised if that wasn’t possible, but it wouldn’t be the first time I would be surprised by strong limitations of Xcode Cloud. Unfortunately, there’s not much we can do if that’s the case. If you do end up making that work, it’d be great if you could document your findings of using the Tuist Registry with Xcode Cloud at this forum (either here or as a separate post).

silly question Marek, would you recommend on post clone of the repo but before Tuist install, we run the block of code in this CI setup example?

As noted in our new docs, the keychain dance is not necessary if you use the XcodeProj-based integration.

1 Like
12

The result with SPM main (as the PR was merged) is:

hyperfine --prepare 'rm -rf ~/.swiftpm .build && swift package purge-cache' --runs 5 --warmup 1 '/Users/kaioelfke/Developer/swift-package-manager/.build/release/swift-package resolve'
Benchmark 1: /Users/kaioelfke/Developer/swift-package-manager/.build/release/swift-package resolve
  Time (mean ± σ):     149.736 s ±  8.973 s    [User: 144.384 s, System: 50.641 s]
  Range (min … max):   141.938 s … 163.422 s    5 runs

But this isn’t directly comparable as I’m in a different place with a different internet connection. I guess I’d need to run the default SPM from Xcode CLI again to compare.