RFC: Project Markers for Decision Tracking

[pepicrft]

Need/Problem

Teams and developers make decisions that might have an impact on their projects’ metrics. For example, changing CI providers or setting a build setting across projects. While they can see the impact through our dashboard, and eventually be notified about changes in trends (e.g. in cache effectiveness), they don’t have a mechanism to record and visualize what decisions are made and when, making it harder to establish correlations.

Motivation

I’ve noticed solutions like error tracking platforms have such primitives, often called markers or deploy markers, which are commonly used to flag a deployment or release of a new version of the software and correlate error trends to them. AppSignal, Sentry, and Datadog all offer similar features. I think this idea would be useful in the context of project health since there are many decisions that can impact the metrics we observe:

• A new version of the build system
• New hardware rolled out to developers
• A new build configuration

Detailed design

We can start simple and non-opinionated by introducing the concept of markers. A marker is an event bound to a project with a date and title associated with it.

I’d suggest that we create three CRUD interfaces for adding them:

• REST API: /api/markers
• CLI: tuist markers list/create/update/remove
• Dashboard:
  • Add a marker
  • Remove a marker
  • See the markers in the graphs

Drawbacks

One drawback I can think of is that being weakly opinionated here can lead to a mess of markers for every decision, not just the significant ones, with teams thinking that everything could impact their metrics. As a result, graphs would end up being cluttered.

We can perhaps start with a weakly opinionated model and then iterate with some opinions once we have a better understanding of the use cases. For example, in error tracking, marking a deploy is kind of an obvious one. Alternatively, we can gather feedback and come up with marker types that serve as guidance.

Alternatives

We could have a system that detects these things automatically. While this is something we could detect (for example, the first time a new version of Xcode is detected), I think automatic detection should be an extension of the implementation rather than the default. Once we have the primitive, we can add an “auto-marking” feature where we try to be smart about detection. AppSignal and others do this too with deployments since you include the commit as a piece of information when reporting errors.

Adoption strategy

I expect most developers to use this through the dashboard initially, but with the CLI interface in place and more developers using coding agents, I expect agents to eventually suggest the creation of markers through the CLI. We’ll write docs and a blog post introducing the concept so developers start getting familiar with it.

How we teach this

As mentioned above, we can put blog posts, videos, and docs out there for people to get familiar with the idea. We can also connect it with the same concept in error tracking platforms so that developers can learn incrementally from an idea they are already familiar with.

Cross-platform availability

This feature can be available across all platforms that Tuist supports. While the examples above reference Xcode, Android teams that end up using Tuist can leverage markers in exactly the same way to track decisions like Gradle version updates or changes to their build configuration.

[marekfort]

Thanks for flagging this and I agree there’s a need for more clearly seeing important environment/codebase changes in the data we’re showing.

@pepicrft

AppSignal and others do this too with deployments since you include the commit as a piece of information when reporting errors.

There’s one big difference between the deployments in platforms like AppSignal and the data we’re showing. Deployments are linear (i.e., there’s only one version of the app live at a given time) while the data we track is non-linear (with the exception of bundles) – you might have switched to Xcode xx on main, but your release or other branch might still be on an older Xcode version.

@pepicrft

We can start simple and non-opinionated by introducing the concept of markers. A marker is an event bound to a project with a date and title associated with it.

Most of the examples you posted – such as a Xcode or a Swift version – is not something flipped remotely. Instead, that’s a change usually done by a merge to main, but the API you’re proposing doesn’t tackle that. You could create a marker tied to a specific commit or a PR, but that needs some extra thinking to go into this.

@pepicrft

We could have a system that detects these things automatically. While this is something we could detect (for example, the first time a new version of Xcode is detected), I think automatic detection should be an extension of the implementation rather than the default. Once we have the primitive, we can add an “auto-marking” feature where we try to be smart about detection

I think I disagree with the approach – imho, it’s easier to start with something opinionated, in this case us building the auto-detection, understand how would teams find this feature useful, and then extract it into something more generic rather than the other way around. Otherwise, we’re risking of coming up with a suboptimal API that we might get stuck with unless we want to do breaking changes.

[pepicrft]

@marekfort

Most of the examples you posted – such as a Xcode or a Swift version – is not something flipped remotely. Instead, that’s a change usually done by a merge to main, but the API you’re proposing doesn’t tackle that. You could create a marker tied to a specific commit or a PR, but that needs some extra thinking to go into this.

Since we store VCS information, we can associate information to commits and surface that in the dashboard. For example a commit in in main that represents the merge of running CI with version of Xcode X.

@marekfort

I think I disagree with the approach – imho, it’s easier to start with something opinionated, in this case us building the auto-detection, understand how would teams find this feature useful, and then extract it into something more generic rather than the other way around. Otherwise, we’re risking of coming up with a suboptimal API that we might get stuck with unless we want to do breaking changes.

That’s fine. I think sometimes exposing an API is a bit of an invitation to try, which gives us to use-cases that they wouldn’t come up with if they didn’t play with the idea in the first place. But I’m not too strong about it, so we can start with the few cases that we know and move from there.

[marekfort]

@pepicrft

That’s fine. I think sometimes exposing an API is a bit of an invitation to try, which gives us to use-cases that they wouldn’t come up with if they didn’t play with the idea in the first place. But I’m not too strong about it, so we can start with the few cases that we know and move from there.

Yeah, that’s fair, but I think in this case, it’s better we figure out how we want to present the markers in the dashboard (and how they should work in relation to git) before it makes sense to invite others to come up with their own use cases.

@pepicrft

Since we store VCS information, we can associate information to commits and surface that in the dashboard. For example a commit in in main that represents the merge of running CI with version of Xcode X.

We can definitely do that for the markers with the automatic detection. For custom markers, I still don’t see the full picture.

If you want to add a custom marker after the commit was merged to main, then easy enough. But I assume usually, you’d want to document/create such a marker in a PR stage, before a commit on main exists. In that scenario, would you then let users create markers associated with a PR – in which case, we’d need to also know when that PR is merged.