Questions around Caching and Remote Projects

Troubleshooting & how to
1

Hi there! I’ve been using Tuist for its project generation features for a while, but I’ve recently gotten started with a remote project to try to leverage more velocity improvements, and I have a few questions.

Setup context: I’m working on a one-person iOS project, targeting iOS 17, using a Tuist manifest (Tuist.swift and Project.swift, the xcodeproj and xcworkspace are gitignored). The project is highly modularized (more than it needs to be at this stage), and contains a few third party dependencies including ComposableArchitecture, swift-argument-parser, swift-syntax and Tuist/Xcodeproj, almost all of which are using the Tuist registry. It has 1 app target, 1 Swift executable target that helps me create boilerplate, and plenty of its dependencies use Macros. As it’s a one-person project, I don’t have any CI running yet, but I create tickets in a project tracker and merge PRs into main once per ticket.

My questions:

  • When should I use tuist cache? The Recommended Setup part of the docs doesn’t explicitly mention invoking it. I’m currently doing it prior to each ticket when I pull main, which seems to work, but leads me to followup questions.
  • Despite using tuist cache, tuist generate takes longer than I’d expect (roughly 15 seconds per run). I ran tuist cache immediately followed by tuist generate twice, and there are some cache behaviors I don’t understand. All of these packages use the registry. I’m linking the runs in a followup comment, because the forum tells me “new users can only post 2 links”.
    • The target ComposableArchitecture in pointfreeco.swift-composable-architecture misses both times.
    • The target Sharing in pointfreeco.swift-sharing goes from being a remote hit to a miss.
    • The target DependenciesMacros in pointfreeco.swift-dependencies goes from being a local hit to a miss.
    • Most of the remaining cache misses are pointing to a private repo that I don’t expect to be cached yet, but I’d still expect to be able to get a significantly faster tuist generate time.
  • The docs for ensuring hashes are deterministic mentions “You can use the diff command to compare the projects generated by two consecutive invocations of tuist generate or across environments or runs.” I’m not sure how I would use the diff command for this - could you provide an example?

Thank you for any assistance!

2

Extra context for the staff with visibility into runs: the 2 runs I’m looking at are here and here.

3

Hi @cneville :wave:

Let me answer your questions:

The recommendation for that page assumes a CI setup. In your case, since it’s a solo-developer project with no CI, the answer depends on what you want to turn into binaries. If it’s just the dependencies, which I’d not expect to change regularly, you can warm the cache locally by doing tuist cache whenever you update dependencies.

If you’d like to turn some of your local targets into binaries, my recommendation would be that you set up CI and run tuist cache for every commit landing in main. tuist cache is incremental, so if there’s nothing to cache, the execution will be fast. Note that this requires signing up, but since it’s a one-person project, I’d expect your usage to fit within the free plan.

How are you integrating those dependencies? I’d expect those scenarios to work, so if you can put a small project together with the dependencies that are not turned into binaries I can take a look into it. It should work with private packages too. At the end of the day, the caching logic looks at the entire graph and doesn’t care whether a target comes from a SPM resolution that has been done as part of tuist install.

This is an area where we plan to invest more. But until that happens, here are some tips:

  • You might experience non-deterministic hashing, either because there’s a bug in our hashing logic, or the project has some dependencies with the environment in which it runs (e.g. an absolute path to the system).
  • You can use tuist cache --print-hashes in two different environments and builds to see if the hashes are the same. If they are not, then we need to dig deeper to see what might be causing the difference.
  • For that, a tool that you’ll find useful is xcdiff by Bloomberg. You can generate two projects in consecutive runs or across environments, and then use that tool to compare the Xcode projects side by side. If there are differences, they should arise. For example, if a project has an absolute path, they’ll be different in each environment.
4

The command for warming external dependencies only is tuist cache --external-only. I agree that in a solo project, caching only external dependencies is probably the best trade-off.

You can also compare hashes between runs directly in the Tuist dashboard as long as you are using Tuist 4.41.0.

5

Thanks so much for the detailed answers folks, it’s very appreciated! I had a bit more time to investigate and implement suggestions, so I have another wave of findings/questions.

I did a tuist generate, renamed the project, did another tuist generate, and compared the two with xcdiff. It revealed a lot of diffs, which seem to point to my dependencies sometimes being .frameworks and sometimes being .xcframeworks? Partial output attached below.

❌ FILE_REFERENCES

⚠️  Only in first (30):

  • ../../../../../.cache/tuist/Binaries/03bea48a4ef1703a3d0833486973a92f/SwiftSyntax509.xcframework
  • ../../../../../.cache/tuist/Binaries/03bea48a4ef1703a3d0833486973a92f/SwiftSyntax510.xcframework
  • ../../../../../.cache/tuist/Binaries/03bea48a4ef1703a3d0833486973a92f/SwiftSyntax600.xcframework
  • ../../../../../.cache/tuist/Binaries/15bdb4263ccf212f4b5d32e5d9d73943/Sharing.xcframework
  • ../../../../../.cache/tuist/Binaries/15bdb4263ccf212f4b5d32e5d9d73943/Sharing1.xcframework
  • ../../../../../.cache/tuist/Binaries/15bdb4263ccf212f4b5d32e5d9d73943/Sharing2.xcframework
  • ../../../../../.cache/tuist/Binaries/1688580bb60da4388103215a3b6ae510/ComposableArchitecture.xcframework
  • ../../../../../.cache/tuist/Binaries/42488e87c6d61a07ae8b5510f0d8eea9/Parsing.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftBasicFormat.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftDiagnostics.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftParser.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftParserDiagnostics.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftSyntax.xcframework
  • ../../../../../.cache/tuist/Binaries/46f917fab286010c3a0af4f204a27617/SwiftSyntaxBuilder.xcframework
  • ../../../../../.cache/tuist/Binaries/4d9320d6ab408361bcc49a0749530c62/API.xcframework
  • ../../../../../.cache/tuist/Binaries/4d9320d6ab408361bcc49a0749530c62/ActivityAPI.xcframework
  • ../../../../../.cache/tuist/Binaries/4d9320d6ab408361bcc49a0749530c62/AuthAPI.xcframework
  • ../../../../../.cache/tuist/Binaries/4d9320d6ab408361bcc49a0749530c62/EndpointRoutingAPI.xcframework
  • ../../../../../.cache/tuist/Binaries/4d9320d6ab408361bcc49a0749530c62/RoutingAPI.xcframework
  • ../../../../../.cache/tuist/Binaries/55b6e253ccbcbf915e22a74cbae9c02f/swift-composable-architecture_ComposableArchitecture.bundle
  • ../../../../../.cache/tuist/Binaries/6fd099b01212513056753015063adb14/SwiftNavigation.xcframework
  • ../../../../../.cache/tuist/Binaries/8782395cf6ed06fecf1f9f8ef3d94919/SwiftUINavigation.xcframework
  • ../../../../../.cache/tuist/Binaries/8782395cf6ed06fecf1f9f8ef3d94919/UIKitNavigation.xcframework
  • ../../../../../.cache/tuist/Binaries/8b24c7674c47514566857315065d253d/CasePathsCore.xcframework
  • ../../../../../.cache/tuist/Binaries/8d410f1fcbdcf5ffba82aef21cfd8550/swift-sharing_Sharing.bundle
  • ../../../../../.cache/tuist/Binaries/94d8e8104a02dd21fedfa063e15433d0/URLRouting.xcframework
  • ../../../../../.cache/tuist/Binaries/972d6d1cb1fa6ae1b0bd3a75f261c637/_SwiftSyntaxCShims.xcframework
  • ../../../../../.cache/tuist/Binaries/9f31fcd7c5399737597c7c0798efa617/Perception.xcframework
  • ../../../../../.cache/tuist/Binaries/c774c6217aaea34d6da1d288b184eece/DependenciesMacros.xcframework
  • ../../../../../.cache/tuist/Binaries/cf0a8aebbdeb7ab07b970e15ec847f27/CasePaths.xcframework


⚠️  Only in second (30):

  • ../../../../../.cache/tuist/Binaries/0900825ad7a2e471e956748d81e3a847/SwiftBasicFormat.xcframework
  • ../../../../../.cache/tuist/Binaries/1203b9f70f4cbdca684b5882659e2ae8/Perception.xcframework
  • ../../../../../.cache/tuist/Binaries/140ab5570ae30e209826bd7795562a97/SwiftSyntax509.xcframework
  • ../../../../../.cache/tuist/Binaries/598add337614878231ac5e12226e66bc/SwiftParser.xcframework
  • ../../../../../.cache/tuist/Binaries/5fbdc0d3d5758c763890e4b2c6c85f3c/SwiftParserDiagnostics.xcframework
  • ../../../../../.cache/tuist/Binaries/824ca30457bac52589b2e5d1e8db1a8f/SwiftDiagnostics.xcframework
  • ../../../../../.cache/tuist/Binaries/85a2928dd4f03193e343abaf29c14a2c/SwiftSyntax510.xcframework
  • ../../../../../.cache/tuist/Binaries/af9e1bea6fef3b4adddd96f92769f4e4/SwiftSyntax.xcframework
  • ../../../../../.cache/tuist/Binaries/bada902e9154fc6fa8a491feb0d12743/Sharing2.xcframework
  • ../../../../../.cache/tuist/Binaries/bfe4987c583ced26997ba286bebd1fdf/SwiftSyntax600.xcframework
  • ../../../../../.cache/tuist/Binaries/cd5fb89f3ef7e0771764438c3bf43006/SwiftSyntaxBuilder.xcframework
  • ../../../../../.cache/tuist/Binaries/cfa705663e1ea293925e3ee3709129f3/Sharing1.xcframework
  • ../../../../../.cache/tuist/Binaries/e0348d9d3fa7017a80e6544a74a3dc62/_SwiftSyntaxCShims.xcframework
  • API.framework
  • ActivityAPI.framework
  • AuthAPI.framework
  • CasePaths.framework
  • CasePathsCore.framework
  • ComposableArchitecture.framework
  • DependenciesMacros.framework
  • EndpointRoutingAPI.framework
  • Parsing.framework
  • RoutingAPI.framework
  • Sharing.framework
  • SwiftNavigation.framework
  • SwiftUINavigation.framework
  • UIKitNavigation.framework
  • URLRouting.framework
  • swift-composable-architecture_ComposableArchitecture.bundle
  • swift-sharing_Sharing.bundle


✅ BUILD_PHASES > "ActivityAPIClient" target
❌ BUILD_PHASES > "ActivityFeatureKit" target

⚠️  Only in first (1):

  • Static XCFramework Dependencies (copyFiles)


❌ BUILD_PHASES > "ActivityFeatureUI" target

⚠️  Only in first (1):

  • Static XCFramework Dependencies (copyFiles)


❌ BUILD_PHASES > "AlertsFeatureKit" target

⚠️  Only in first (1):

  • Static XCFramework Dependencies (copyFiles)


⚠️  Only in second (1):

  • Dependencies (copyFiles)

As for how I’m integrating these dependencies, the project has a Tuist.swift and a Tuist/Package.swift, the dependencies are using the registry, and there’s otherwise nothing special. I’m not overriding any Package types from the default .staticFramework, but when I tried that, I get a million Target '_' has been linked from target '_' and target '_', it is a static product so may introduce unwanted side effects. warnings from downstream/2nd-order dependencies. All of my internal targets are static.

let package = Package(
    name: "MyPodiumApp-Dependencies",
    dependencies: [
        .package(
            id: "apple.swift-argument-parser",
            from: Version(1, 5, 0)
        ),
        .package(
            id: "apple.swift-log",
            from: Version(1, 6, 2)
        ),
        .package(
            id: "pointfreeco.swift-case-paths",
            from: Version(1, 6, 0)
        ),
        .package(
            id: "pointfreeco.swift-composable-architecture",
            from: Version(1, 17, 1)
        ),
        .package(
            id: "pointfreeco.swift-dependencies",
            from: Version(1, 6, 3)
        ),
        .package(
            id: "pointfreeco.swift-identified-collections",
            from: Version(1, 1, 1)
        ),
        .package(
            id: "pointfreeco.swift-sharing",
            from: Version(2, 3, 0)
        ),
        .package(
            id: "pointfreeco.swift-tagged",
            from: Version(0, 10, 0)
        ),
        .package(
            id: "pointfreeco.swift-url-routing",
            from: Version(0, 6, 2)
        ),
        .package(
            id: "swiftlang.swift-syntax",
            from: Version(600, 0, 0)
        ),
        .package(
            id: "square.valet",
            from: Version(5, 0, 0)
        ),
        .package(
            id: "tuist.XcodeProj",
            from: Version(8, 26, 4)
        )
	]
)

To summarize, I think I have 2 issues, which might actually be the same issue: 1) unexplainable cache misses on targets such as ComposableArchitecture and 2) tuist generate taking longer than expected (15-25s on a pretty small project), generating nondeterministic projects.

1 more curiosity question re. cache --external-only: is the guidance there just dependent on the size of the project/number of internal targets? I figured might as well (try to) cache everything, even if some number of my internal targets are changing frequently.

Thanks again for your time.

6

So, I still have unexpected nondeterministic project generation, but I found something surprising on the “long generation times” front.

additionalFiles: [
    "**/*.md",
    "**/__Snapshots__/**/*",
    ".swiftlint.yml",
].map { .glob(pattern: .path($0) }

I have these additional files included in my project. Just removing the "**/__Snapshots__/**/*", file glob took my average tuist generate time down from ~15 seconds to ~3 seconds :scream:. So I guess the 2 issues mentioned above were separate after all.