Selective testing for non-generated projects

marekfort · February 19, 2025, 1:59pm

Re-running the full test suite on every CI run is wasteful and time-consuming. Wouldn’t it be great if you could run tests impacted by your changes? That’s exactly what Tuist’s selective testing aims to solve. However, until now, this feature was reserved only to project generated with Tuist.

We’re very excited to announce that this feature is now available for any Xcode project setup, be it generated or not: Selective testing for all Xcode projects

How to get started? Just add tuist to your xcodebuild and start benefiting from faster tests and soon interesting insights, such as flaky tests:

tuist xcodebuild -scheme App -destination "name=iPhone 16"

Head over to our docs to learn more: Selective testing · Develop · Guides · Tuist

Apart from tests, tuist graph now supports non-generated Xcode projects as well We will continue bringing improvements traditionally reserved to generated projects to the whole Swift ecosystem. We couldn’t be more excited about this.

Huge shout-out goes to @ajkolean who laid the foundations by introducing the XcodeGraphMapper component: Mapping XcodeProj to XcodeGraph

Do you have feedback? Let us know! Feel free to post it directly below

kaioelfke · February 20, 2025, 4:16am

Looks quite cool. I tried this for my main job. I got this unhelpful error:

We received an error that we couldn't handle:
    - Localized description: Missing or invalid file path for `PBXBuildFile`: Unknown.
    - Error: missingFilePath(name: nil)

With tuist graph I got better logs:

2025-02-20T12:11:13+0800 error dev.tuist.cli : We received an error that we couldn't handle:
    - Localized description: The operation couldn’t be completed. (Path.PathValidationError error 1.)
    - Error: invalid absolute path '${PODS_ROOT}/Target Support Files/Pods-SomeTestTargetUITests/Pods-SomeTestTargetUITests-frameworks-${CONFIGURATION}-input-files.xcfilelist'

So for our project this doesn’t work as we still have CocoaPods and there’s these variables for PODS_ROOT and CONFIGURATION. I guess this isn’t supported yet. But I wanted to share this as feedback.

ajkolean · February 20, 2025, 4:53pm

Seems like the issue stems from env vars failing to validate as an AbsolutePath.

One way to address this could be leveraging PBXBuildConfiguration.buildSettings to resolve variables dynamically before validation. Something like:

func resolveBuildVariables(in path: String, using buildSettings: [String: Any]) -> String {
    buildSettings.reduce(path) { resolvedPath, setting in
        guard let value = setting.value as? String else { return resolvedPath }
        return resolvedPath.replacingOccurrences(of: "${\(setting.key)}", with: value)
    }
}

Then in the mapper:

let pathString = try fileRef.fullPath(sourceRoot: xcodeProj.srcPathString).map {
    resolveBuildVariables(in: $0, using: buildSettings)
}

What are your thoughts, @marekfort!

marekfort · February 20, 2025, 5:17pm

Hey @kaioelfke!

Thanks a lot for the feedback. Cocoapods projects should be supported, but it’s not something we’ve tested.

@ajkolean I’ve already fixed the issue via: feat: support dynamic file list paths by fortmarek · Pull Request #132 · tuist/XcodeGraph · GitHub

We can’t replace the build settings values directly as some variables are dynamic such as $CONFIGURATION.

@kaioelfke I’ll be creating a new patch release soon that should have a fix for the issue, I’ll let you know once that’s out, so you can retest it

ajkolean · February 20, 2025, 7:07pm

Ahh yeah, good shout! xcodebuild will resolve these, but anything running outside of it (like tuist graph) won’t, which is fine since tuist graph doesn’t actually need fully qualified paths.

I think any other commands that don’t go through xcodebuild would need to resolve them themselves first (from build settings / Process.env) if an actual path is required.

marekfort · February 20, 2025, 8:11pm

I think any other commands that don’t go through xcodebuild would need to resolve them themselves first (from build settings / Process.env) if an actual path is required.

Yes, that’s true. We do try to resolve them when hashing the graph to make the hashes more accurate – on the best effort basis. We could try to resolve paths where possible when mapping as well – but some, like those containing the CONFIGURATION, are by design not statically resolvable as they do depend on the build time configuration.

kaioelfke · February 21, 2025, 10:32am

I tried 4.43.1, but it still fails with:

    - Localized description: Missing `PBXBuildFile.file` reference.
    - Error: missingFileReference

So I’ll need to run Tuist in Xcode to debug what file reference this is about as the error doesn’t tell me much.

marekfort · February 21, 2025, 10:45am

Would you mind doing that? I’ll also look into improving the error messages.

You can run the tuist graph command from Xcode to try the XcodeGraphMapper.

kaioelfke · February 24, 2025, 6:59am

The root cause was a messy Xcode pbx project file. But as written also on the Tuist website this is quite common for big old projects.

6E3957882C2A72AD00A4B537 /* (null) in Frameworks */,

It would have been helpful to get some message to search in the pbxproj for 6E3957882C2A72AD00A4B537. Then i could have fixed it without debugging Tuist.

Now I’m on to the next error:

We received an error that we couldn't handle:
    - Localized description: No platform could be inferred from target 'AdjustSignature'.
    - Error: noPlatformInferred("AdjustSignature")

This is a target from pod 'Adjust', '~> 5.0'. The target just contains a xcframework.

github.com/CocoaPods/Specs

Specs/1/7/0/AdjustSignature/3.35.2/AdjustSignature.podspec.json

79144f576

{
  "name": "AdjustSignature",
  "version": "3.35.2",
  "license": {
    "type": "Proprietary",
    "text": "Copyright (c) 2018-Present Adjust GmbH, http://www.adjust.com/."
  },
  "homepage": "https://github.com/adjust/adjust_signature_sdk",
  "authors": {
    "Adjust GmbH": "mobsec at adjust dot com"
  },
  "summary": "The Adjust Signature.",
  "source": {
    "http": "https://github.com/adjust/adjust_signature_sdk/releases/download/v3.35.2/AdjustSigSdk-iOS-tvOS-Dynamic-3.35.2.xcframework.zip"
  },
  "requires_arc": true,
  "frameworks": [
    "Foundation",
    "QuartzCore"
  ],

This file has been truncated. show original

The error is because, pbxTarget.productType is nil in let productType = pbxTarget.productType?.mapProductType()

In PBXTargetMapper.swift

func map(
        pbxTarget: PBXTarget,
        xcodeProj: XcodeProj,
        projectNativeTargets: [String: ProjectNativeTarget],
        packages: [AbsolutePath]
    ) async throws -> Target {

I naively assume this is due to it being a PBXAggregateTarget and not a PBXNativeTarget. So productType doesn’t exist as a key. But I’m not really so deep into pbx.

marekfort · February 24, 2025, 8:51am

Definitely. We’ve already merged a change that does exactly that: feat: improve error message when PBXBuildFile.file reference is not found by fortmarek · Pull Request #136 · tuist/XcodeGraph · GitHub

This will be included in the next Tuist release.

Yes, that’s quite likely. I’ll look into this.

marekfort · February 25, 2025, 5:54pm

@kaioelfke the latest Tuist release should include a fix for selective testing of projects with aggregate targets. It’d be great if you could take it for a spin!

kaioelfke · February 26, 2025, 3:54am

The fix worked. Thank you @marekfort. But there’s yet again another error:

No valid path for resource file element: Alamofire-tvOS-Alamofire

// Or

No valid path for resource file element: Alamofire-iOS-Alamofire

From debugging Tuist I figured out that:

    sourceTree = buildProductsDir
    path = "Alamofire.bundle"
    name = "Alamofire-iOS-Alamofire"

So this function will return nil and throw:


let pathString = try fileElement
            .fullPath(sourceRoot: xcodeProj.srcPathString)
            .throwing(PBXResourcesMappingError.missingFullPath(fileElement.name ?? "Unknown"))

func fullPath(sourceRoot: Path) throws -> Path? {
        switch sourceTree {
        case .absolute?:
            return path.flatMap { Path($0) }
        case .sourceRoot?:
            return path.flatMap { sourceRoot + $0 }
        case .group?:
            let groupPath: Path?

            if let group = parent {
                groupPath = try group.fullPath(sourceRoot: sourceRoot) ?? sourceRoot
            } else {
                let projectObjects = try objects()
                let isThisElementRoot = projectObjects.projects.values.first(where: { $0.mainGroup == self }) != nil
                if isThisElementRoot {
                    if let path {
                        return sourceRoot + Path(path)
                    }
                    return sourceRoot
                }

                // Fallback if parent is nil and it's not root element
                guard let group = projectObjects.groups.first(where: { $0.value.childrenReferences.contains(reference) }) else {
                    throw PBXProjError.invalidGroupPath(sourceRoot: sourceRoot, elementPath: path)
                }
                groupPath = try group.value.fullPath(sourceRoot: sourceRoot)
            }

            guard let fullGroupPath: Path = groupPath else { return nil }
            guard let filePath = self is PBXVariantGroup ? try baseVariantGroupPath() : path else { return fullGroupPath }
            return fullGroupPath + filePath
        default: // buildProductsDir case will end up here
            return nil
        }
    }

The Alamofire CocoaPods dependency is pod 'Alamofire', '~> 5.3' resolved to Alamofire (5.10.2) currently.

marekfort · February 28, 2025, 3:26pm

Hey @kaioelfke

I’m not able to reproduce when I add Alamofire with that given version to a Cocoapods project. Would you be able to create a reproducible sample?

kaioelfke · March 3, 2025, 4:36am

Good morning, @marekfort it turns out that this bug only happens when having two targets with Alamofire dependencies and each target having a different platform.

Maybe there’s a more minimal reproduction.

The attached sample is an iOS and tvOS app (and some test targets that Xcode added automatically, but I think they shouldn’t matter).

The podfile is this:

# Uncomment the next line to define a global platform for your project
# platform :ios, '9.0'

target 'AlamoSampleTwo' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks!
  platform :tvos, '18.0'

  # Pods for AlamoSampleTwo
  pod 'Alamofire', '~> 5.3'

  target 'AlamoSampleTwoTests' do
    inherit! :search_paths
    # Pods for testing
  end

  target 'AlamoSampleTwoUITests' do
    # Pods for testing
  end

end

target 'AlamoSampleTwoMobile' do
  # Comment the next line if you don't want to use dynamic frameworks
  use_frameworks!
  platform :ios, '18.0'

  # Pods for AlamoSampleTwo
  pod 'Alamofire', '~> 5.3'

  target 'AlamoSampleTwoMobileTests' do
    inherit! :search_paths
    # Pods for testing
  end

  target 'AlamoSampleTwoMobileUITests' do
    # Pods for testing
  end

end

AlamoSampleTwo.zip (497.1 KB)

tuist graph will fail with either Alamofire-iOS-Alamofire or Alamofire-tvOS-Alamofire missing depending on timing.

I think Cocoapods changes some stuff when you depend on something on different platforms.