feat(turbo): add cache flag #9348

NicholasLYang · 2024-10-29T20:40:14Z

Description

Adds a cache flag that lets you set cache permissions per cache type and per action (read or write)

Testing Instructions

Adds tests in turborepo_cache/src/config.rs for parsing and in turborepo-lib/src/opts.rs for loading and resolving

vercel · 2024-10-29T20:40:19Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
examples-nonmonorepo	❌ Failed (Inspect)			Nov 5, 2024 5:07pm

8 Skipped Deployments

Name	Status	Preview	Updated (UTC)
examples-basic-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-designsystem-docs	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-gatsby-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-kitchensink-blog	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-native-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-svelte-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-tailwind-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm
examples-vite-web	⬜️ Ignored (Inspect)	Visit Preview	Nov 5, 2024 5:07pm

chris-olszewski

I'm excited for this, biggest thing I'd like to see is some tests regarding how the new cache struct gets resolved wrt existing cache config settings.

crates/turborepo-lib/src/config/env.rs

chris-olszewski · 2024-10-31T13:34:09Z

crates/turborepo-lib/src/opts.rs

@@ -362,16 +358,37 @@ impl<'a> TryFrom<OptsInputs<'a>> for ScopeOpts {
 impl<'a> From<OptsInputs<'a>> for CacheOpts {
    fn from(inputs: OptsInputs<'a>) -> Self {
        let is_linked = turborepo_api_client::is_linked(inputs.api_auth);
-        let skip_remote = if !is_linked {
-            true
+        let mut cache = inputs.config.cache();


I think this layering logic is confusing since env vars or config settings can override CLI level cache settings:
e.g. TURBO_FORCE=1 turbo run build --force=false would result in remote:rw,local:rw cache behavior, but TURBO_REMOTE_ONLY=1 turbo run build --cache='local:rw would result in no caches are enabled.

I think it would make more sense to fold the existing cache flags into the new cache struct at each level in the config, but I'm not convinced that wouldn't be confusing. Could we get away with having the new cache struct conflict with existing cache flags at each level e.g turbo run build --remote-only --cache='...' would throw but TURBO_REMOTE_ONLY=1 turbo run build --cache='...' be fine where --cache wins out completely?

Is there a way to have a group of flags conflict with a single flag but not with each other? I wasn't sure if that's possible in clap. Ofc I can manually add that as a restriction

The layering issue is a tricky one. If the config specifies remote:rw in a file, but then we add on local:rw, do we want to do an OR or an AND for those? I could see either one working

crates/turborepo-cache/src/lib.rs

chris-olszewski

LFG

chris-olszewski · 2024-11-05T15:28:21Z

crates/turborepo-cache/src/config.rs

+    #[test_case("local:rx", Err(Error::InvalidCacheAction { c: 'x', text: "local:rx".to_string(), span: Some(SourceSpan::new(6.into(), 5.into())) }) ; "invalid action")]
+    #[test_case("file:r", Err(Error::InvalidCacheType { s: "file".to_string(), text: "file:r".to_string(), span: Some(SourceSpan::new(0.into(), 4.into())) }) ; "invalid cache type")]
+    fn test_cache_config(s: &str, expected: Result<CacheConfig, Error>) {
+        assert_eq!(CacheConfig::from_str(s), expected);


Not blocking at all, but it might be nice to snapshot the error messages, easier to update if we change the tests/logic at all. There's a JSON reporter that makes assert_json_snapshot easy to use: https://github.com/vercel/turborepo/blob/main/crates/turborepo-lib/src/engine/builder.rs#L1358

…tween existing flags and cache flag

Also fix up config loading

… env var mapping

…on't have to worry about layering

# DO NOT MERGE CONDITION 2.3 release. ### Description Documenting #9348 ### Testing Instructions 👀 --------- Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com>

turbo-orchestrator bot added created-by: turborepo labels Oct 29, 2024

vercel bot deployed to Preview – examples-nonmonorepo October 29, 2024 21:24 View deployment

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 17:09 View deployment

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 17:19 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from b9eea2e to 8fb5b84 Compare October 30, 2024 17:20

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 17:21 View deployment

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 20:34 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from bfa85f1 to 02f4bfc Compare October 30, 2024 20:38

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 20:39 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from 02f4bfc to b5ffd7a Compare October 30, 2024 20:40

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 20:40 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from b5ffd7a to 0adcd94 Compare October 30, 2024 20:42

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 20:43 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from 0adcd94 to 0ee6787 Compare October 30, 2024 20:45

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 20:46 View deployment

NicholasLYang marked this pull request as ready for review October 30, 2024 20:48

NicholasLYang requested a review from a team as a code owner October 30, 2024 20:48

NicholasLYang requested review from tknickman and chris-olszewski October 30, 2024 20:48

vercel bot deployed to Preview – examples-nonmonorepo October 30, 2024 22:25 View deployment

chris-olszewski requested changes Oct 31, 2024

View reviewed changes

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from a485d73 to a7f05c9 Compare October 31, 2024 19:54

vercel bot deployed to Preview – examples-nonmonorepo October 31, 2024 19:55 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from a7f05c9 to 1470a4d Compare October 31, 2024 19:55

vercel bot deployed to Preview – examples-nonmonorepo October 31, 2024 19:55 View deployment

vercel bot deployed to Preview – examples-nonmonorepo October 31, 2024 20:07 View deployment

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from 9c99caf to da88747 Compare November 4, 2024 18:28

NicholasLYang requested a review from chris-olszewski November 4, 2024 18:28

vercel bot deployed to Preview – examples-nonmonorepo November 4, 2024 18:28 View deployment

vercel bot deployed to Preview – examples-nonmonorepo November 4, 2024 19:04 View deployment

chris-olszewski approved these changes Nov 5, 2024

View reviewed changes

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from e565429 to ad09204 Compare November 5, 2024 16:22

vercel bot deployed to Preview – examples-nonmonorepo November 5, 2024 16:22 View deployment

turbo-orchestrator bot added the area: examples Improvements or additions to examples label Nov 5, 2024

vercel bot deployed to Preview – examples-nonmonorepo November 5, 2024 16:37 View deployment

vercel bot had a problem deploying to Preview – examples-nonmonorepo November 5, 2024 16:59 Failure

NicholasLYang added 14 commits November 5, 2024 12:06

Adding cache parser and connect to layered config

4a30dc1

Actually add logic for restricting cache actions based on cache config

3a9345d

Add comments for corresponding cache flag

0c2ef0a

Rename fs -> local

1520998

Add testing for fully resolved CacheConfig, including interactions be…

350a50e

…tween existing flags and cache flag

Refactor the cache args to be in RunArgs, not ExecutionArgs.

a805d35

Also fix up config loading

Fix bug

d01e0ad

Add snapshots

e1df778

Don't allow cache config to be sourced from config file. Also include…

fe52e61

… env var mapping

Made the new cache config incompatible with old cache config, so we d…

63798b6

…on't have to worry about layering

Fix clippy

24d30db

Snapshot testing

2331efe

Delete old snapshots

7abf036

Update integration tests

9be223f

NicholasLYang force-pushed the nicholasyang/add-cache-flag branch from 112cc6c to 9be223f Compare November 5, 2024 17:06

vercel bot had a problem deploying to Preview – examples-nonmonorepo November 5, 2024 17:07 Failure

NicholasLYang merged commit 4cfbc44 into main Nov 5, 2024
39 of 40 checks passed

NicholasLYang deleted the nicholasyang/add-cache-flag branch November 5, 2024 18:17

This was referenced Nov 5, 2024

docs: Move docker command up for clarity. #9390

Merged

Document --cache flag. #9427

Merged

anthonyshew added a commit that referenced this pull request Nov 14, 2024

Document --cache flag. (#9427)

649a096

# DO NOT MERGE CONDITION 2.3 release. ### Description Documenting #9348 ### Testing Instructions 👀 --------- Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(turbo): add cache flag #9348

feat(turbo): add cache flag #9348

NicholasLYang commented Oct 29, 2024 •

edited

Loading

vercel bot commented Oct 29, 2024 •

edited

Loading

chris-olszewski left a comment

chris-olszewski Oct 31, 2024

NicholasLYang Oct 31, 2024

NicholasLYang Oct 31, 2024

chris-olszewski left a comment

chris-olszewski Nov 5, 2024

feat(turbo): add cache flag #9348

feat(turbo): add cache flag #9348

Conversation

NicholasLYang commented Oct 29, 2024 • edited Loading

Description

Testing Instructions

vercel bot commented Oct 29, 2024 • edited Loading

chris-olszewski left a comment

Choose a reason for hiding this comment

chris-olszewski Oct 31, 2024

Choose a reason for hiding this comment

NicholasLYang Oct 31, 2024

Choose a reason for hiding this comment

NicholasLYang Oct 31, 2024

Choose a reason for hiding this comment

chris-olszewski left a comment

Choose a reason for hiding this comment

chris-olszewski Nov 5, 2024

Choose a reason for hiding this comment

NicholasLYang commented Oct 29, 2024 •

edited

Loading

vercel bot commented Oct 29, 2024 •

edited

Loading