Support Parallel Gradle Configuration Cache #2391

jGleitz · 2025-01-08T13:17:59Z

The Gradle plugin currently doesn’t support Parallel Configuration Caching, but it would be useful it it did.

When enabling org.gradle.configuration-cache.parallel, spotless tasks randomly fail with exceptions like this one:

Execution failed for task ':[subproject]:spotlessJava'.
Could not read path '[project path]/[subproject]/build/spotless-lints/spotlessKotlinGradle'.

Stack Trace

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':[subproject]:spotlessJava'.
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:38)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:209)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:166)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
        at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:85)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:459)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:376)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
Caused by: org.gradle.api.GradleException: Could not read path '[project path]/[subproject]/build/spotless-lints/spotlessKotlinGradle'.
        at org.gradle.api.internal.file.collections.PathVisitor.visitFileFailed(PathVisitor.java:106)
        at org.gradle.api.internal.file.collections.PathVisitor.visitFileFailed(PathVisitor.java:38)
        at org.gradle.api.internal.file.collections.DefaultDirectoryWalker.walkDir(DefaultDirectoryWalker.java:44)
        at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:150)
        at org.gradle.api.internal.file.collections.DirectoryFileTree.visitFrom(DirectoryFileTree.java:128)
        at org.gradle.api.internal.file.collections.DirectoryFileTree.visit(DirectoryFileTree.java:114)
        at org.gradle.api.internal.file.collections.FileTreeAdapter.visit(FileTreeAdapter.java:110)
        at org.gradle.api.internal.file.AbstractFileTree.getFiles(AbstractFileTree.java:52)
        at org.gradle.api.internal.file.collections.FileTreeAdapter.getFiles(FileTreeAdapter.java:56)
        at org.gradle.api.internal.file.AbstractFileCollection$1.addTreeContents(AbstractFileCollection.java:133)
        at org.gradle.api.internal.file.AbstractFileCollection$1.visitFileTree(AbstractFileCollection.java:138)
        at org.gradle.api.internal.file.collections.FileTreeAdapter$1.visitFileTree(FileTreeAdapter.java:125)
        at org.gradle.api.internal.file.collections.DirectoryFileTree.visitStructure(DirectoryFileTree.java:109)
        at org.gradle.api.internal.file.collections.FileTreeAdapter.visitContents(FileTreeAdapter.java:121)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.api.internal.file.CompositeFileCollection.lambda$visitContents$0(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.collections.DefaultConfigurableFileTree.visitChildren(DefaultConfigurableFileTree.java:188)
        at org.gradle.api.internal.file.CompositeFileCollection.visitContents(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.api.internal.file.CompositeFileCollection.lambda$visitContents$0(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:67)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:100)
        at org.gradle.api.internal.file.DefaultFileCollectionFactory$ResolvingFileCollection.visitChildren(DefaultFileCollectionFactory.java:306)
        at org.gradle.api.internal.file.CompositeFileCollection.visitContents(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.AbstractFileCollection.getFiles(AbstractFileCollection.java:123)
        at org.gradle.api.internal.file.SubtractingFileCollection.getIntrinsicFiles(SubtractingFileCollection.java:57)
        at org.gradle.api.internal.file.AbstractOpaqueFileCollection.iterator(AbstractOpaqueFileCollection.java:55)
        at org.gradle.execution.plan.MissingTaskDependencyDetector$1.visitCollection(MissingTaskDependencyDetector.java:72)
        at org.gradle.api.internal.file.AbstractOpaqueFileCollection.visitContents(AbstractOpaqueFileCollection.java:60)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.api.internal.file.CompositeFileCollection.lambda$visitContents$0(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:67)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:100)
        at org.gradle.api.internal.file.DefaultFileCollectionFactory$ResolvingFileCollection.visitChildren(DefaultFileCollectionFactory.java:306)
        at org.gradle.api.internal.file.CompositeFileCollection.visitContents(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.api.internal.file.CompositeFileCollection.lambda$visitContents$0(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:67)
        at org.gradle.api.internal.file.collections.UnpackingVisitor.add(UnpackingVisitor.java:92)
        at org.gradle.api.internal.file.DefaultFileCollectionFactory$ResolvingFileCollection.visitChildren(DefaultFileCollectionFactory.java:306)
        at org.gradle.api.internal.file.CompositeFileCollection.visitContents(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.api.internal.file.CompositeFileCollection.lambda$visitContents$0(CompositeFileCollection.java:113)
        at org.gradle.api.internal.tasks.PropertyFileCollection.visitChildren(PropertyFileCollection.java:48)
        at org.gradle.api.internal.file.CompositeFileCollection.visitContents(CompositeFileCollection.java:113)
        at org.gradle.api.internal.file.AbstractFileCollection.visitStructure(AbstractFileCollection.java:360)
        at org.gradle.execution.plan.MissingTaskDependencyDetector.lambda$detectMissingDependencies$2(MissingTaskDependencyDetector.java:69)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
        at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:89)
        at org.gradle.execution.plan.MissingTaskDependencyDetector.detectMissingDependencies(MissingTaskDependencyDetector.java:69)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.lambda$execute$0(LocalTaskNodeExecutor.java:39)
        at org.gradle.api.internal.tasks.execution.TaskExecution.validate(TaskExecution.java:471)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:76)
        at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:56)
        at org.gradle.internal.execution.steps.AbstractCaptureStateBeforeExecutionStep.execute(AbstractCaptureStateBeforeExecutionStep.java:64)
        at org.gradle.internal.execution.steps.AbstractCaptureStateBeforeExecutionStep.execute(AbstractCaptureStateBeforeExecutionStep.java:43)
        at org.gradle.internal.execution.steps.AbstractSkipEmptyWorkStep.executeWithNonEmptySources(AbstractSkipEmptyWorkStep.java:125)
        at org.gradle.internal.execution.steps.AbstractSkipEmptyWorkStep.execute(AbstractSkipEmptyWorkStep.java:56)
        at org.gradle.internal.execution.steps.AbstractSkipEmptyWorkStep.execute(AbstractSkipEmptyWorkStep.java:36)
        at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
        at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
        at org.gradle.internal.execution.steps.HandleStaleOutputsStep.execute(HandleStaleOutputsStep.java:75)
        at org.gradle.internal.execution.steps.HandleStaleOutputsStep.execute(HandleStaleOutputsStep.java:41)
        at org.gradle.internal.execution.steps.AssignMutableWorkspaceStep.lambda$execute$0(AssignMutableWorkspaceStep.java:35)
        at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:289)
        at org.gradle.internal.execution.steps.AssignMutableWorkspaceStep.execute(AssignMutableWorkspaceStep.java:31)
        at org.gradle.internal.execution.steps.AssignMutableWorkspaceStep.execute(AssignMutableWorkspaceStep.java:22)
        at org.gradle.internal.execution.steps.ChoosePipelineStep.execute(ChoosePipelineStep.java:40)
        at org.gradle.internal.execution.steps.ChoosePipelineStep.execute(ChoosePipelineStep.java:23)
        at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.lambda$execute$2(ExecuteWorkBuildOperationFiringStep.java:67)
        at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:67)
        at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:39)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:46)
        at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:34)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:48)
        at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:35)
        at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:61)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:127)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:116)
        at org.gradle.api.internal.tasks.execution.ProblemsTaskPathTrackingTaskExecuter.execute(ProblemsTaskPathTrackingTaskExecuter.java:40)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:209)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:166)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
        at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:85)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:459)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:376)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
Caused by: java.nio.file.NoSuchFileException: <project path>/<subproject>/build/spotless-lints/spotlessKotlinGradle
        ... 106 more

The error shows that the spotlessJava task tries to load the files of spotlessKotlinGradle. So, it looks like some state is shared across different tasks and that that state is not protected against parallel writes from the configuration cache.

Gradle version: 8.12
Spotless version: 7.0.1

The text was updated successfully, but these errors were encountered:

nedtwigg · 2025-01-08T17:22:20Z

Happy to take PRs to improve this. I do not think configuration cache is a good design, and parallel configuration cache is especially bad. The most valuable PR would be a test that reliably reproduces one of these bugs, which I expect to be nearly impossible, but I hope to be wrong!

I wish Gradle would look at Redis a little bit - it's a far more performance-sensitive application than Gradle, and Redis won its space by going in the opposite direction that Gradle is currently pointing itself.

breskeby · 2025-03-12T20:38:00Z

Can you elaborate what's wrong with the configuration cache design? Especially with taking this further and allow parallel configuration as well as isolated project support I have high hopes this will tackle the major bottle of bigger builds which usually is configuration time that does not scale in usual gradle projects.

nedtwigg · 2025-03-13T00:41:06Z

Can you elaborate what's wrong with the configuration cache design?

This ship has sailed, but I think it should store the cache in RAM for reuse within a single JVM, not on disk for reuse across multiple JVMs on a single machine. Relative to serializing on disk, storing in RAM would be faster to execute and faster to configure, and it reduces the implementation constraints on plugin authors. I went into more detail here: #644 (comment)

Last time I checked, com.gradle.plugin-publish still doesn't support configuration cache. This bifurcation of the plugin ecosystem has made Gradle feel permanently half-broken since the configuration cache feature shipped.

The long tail of subtle bugs in Java serialization and Gradle's idiosyncratic implementation of it make me fear that the Gradle ecosystem will continue to feel half broken forever.

you can't use List.of Configuration cache failure 7.0.0BETA4 / Gradle 8.12 w/ custom prettier step #2372 (comment)
deep tension between the requirements of remote build cache and configuration cache
- to support remote build cache your inputs need to not have any absolute paths in them
- to support configuration cache you have to roundtrip yourself to disk, but without any absolute paths because that would break the remote part of remote build cache
- you can't use custom equality, because build cache wisely uses the serialized representation of your keys to determine equality. but that's a huge problem if you have to restore yourself from those serialized keys
- Fix remote build cache #2298

if you serialize a list of java objects, you might activate the implicit "object graph" part of java serialization, which is non-deterministic and thus breaks both local and remote build cache

spotless/lib/src/main/java/com/diffplug/spotless/ConfigurationCacheHackList.java

Lines 70 to 78 in 519d36c

    
           for (Object obj : backingList) { 
        
           	// if write out the list on its own, we'll get java's non-deterministic object-graph serialization 
        
           	// by writing each object to raw bytes independently, we avoid this 
        
           	if (serializeToByteArrayFirst) { 
        
           		out.writeObject(LazyForwardingEquality.toBytes((Serializable) obj)); 
        
           	} else { 
        
           		out.writeObject(obj); 
        
           	} 
        
           }

parallel configuration as well as isolated project

I hope it works, and I'm happy to merge any PRs that improve Spotless' compliance with Gradle's new features! But realistically, I believe that Gradle is papercutting its ecosystem to death for relatively little benefit.

nedtwigg added the enhancement label Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Parallel Gradle Configuration Cache #2391

Support Parallel Gradle Configuration Cache #2391

jGleitz commented Jan 8, 2025

nedtwigg commented Jan 8, 2025

breskeby commented Mar 12, 2025

nedtwigg commented Mar 13, 2025

Support Parallel Gradle Configuration Cache #2391

Support Parallel Gradle Configuration Cache #2391

Comments

jGleitz commented Jan 8, 2025

nedtwigg commented Jan 8, 2025

breskeby commented Mar 12, 2025

nedtwigg commented Mar 13, 2025