Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

'ratchet' mode is very slow (plugin-maven) #1025

Closed
6 tasks
rjmunro opened this issue Dec 8, 2021 · 6 comments
Closed
6 tasks

'ratchet' mode is very slow (plugin-maven) #1025

rjmunro opened this issue Dec 8, 2021 · 6 comments

Comments

@rjmunro
Copy link

rjmunro commented Dec 8, 2021

I have 6830 java files in my maven project (found with git ls-files "*.java" | wc -l) spread across 44 modules. When I have no changes (git diff --name-only origin/develop returns nothing) and I scan in ratchet mode, it takes nearly 10 minutes to do a no-op. When I removed ratchet mode because I wanted to see how many files were still misformatted, it took less than 20 seconds, even though it had to process every java file.

I was expecting it to be a lot slower because it had to actually read and format 6830 java files, rather than just look at git and see nothing has changed.

If you are submitting a bug, please include the following:

  • summary of problem
  • gradle or maven version
  • spotless version
  • operating system and version
  • copy-paste your full Spotless configuration block(s), and a link to a public git repo that reproduces the problem if possible
  • copy-paste the full content of any console errors emitted by gradlew spotless[Apply/Check] --stacktrace

If you're just submitting a feature request or question, no need for the above.

@nedtwigg nedtwigg changed the title 'ratchet' mode is very slow 'ratchet' mode is very slow (plugin-maven) Dec 8, 2021
@nedtwigg
Copy link
Member

nedtwigg commented Dec 8, 2021

That is surprising! Here's the codepath as it exists today, but #935 is gonna change things a bit.

final List<File> files;
if (ratchetFrom.isPresent()) {
files = collectFilesFromGit(formatterFactory, ratchetFrom.get());
} else {
files = collectFilesFromFormatterFactory(formatterFactory);
}
if (filePatterns == null || filePatterns.isEmpty()) {
return files;
}
final String[] includePatterns = this.filePatterns.split(",");
final List<Pattern> compiledIncludePatterns = Arrays.stream(includePatterns)
.map(Pattern::compile)
.collect(Collectors.toList());
final Predicate<File> shouldInclude = file -> compiledIncludePatterns
.stream()
.anyMatch(filePattern -> filePattern.matcher(file.getAbsolutePath())
.matches());
return files
.stream()
.filter(shouldInclude)
.collect(toList());

private List<File> collectFilesFromGit(FormatterFactory formatterFactory, String ratchetFrom) throws MojoExecutionException {
MatchPatterns includePatterns = MatchPatterns.from(
withNormalizedFileSeparators(getIncludes(formatterFactory)));
MatchPatterns excludePatterns = MatchPatterns.from(
withNormalizedFileSeparators(getExcludes(formatterFactory)));
Iterable<String> dirtyFiles;
try {
dirtyFiles = GitRatchetMaven
.instance().getDirtyFiles(baseDir, ratchetFrom);
} catch (IOException e) {
throw new MojoExecutionException("Unable to scan file tree rooted at " + baseDir, e);
}
List<File> result = new ArrayList<>();
for (String file : withNormalizedFileSeparators(dirtyFiles)) {
if (includePatterns.matches(file, true)) {
if (!excludePatterns.matches(file, true)) {
result.add(new File(baseDir.getPath(), file));
}
}
}
return result;
}

Iterable<String> getDirtyFiles(File baseDir, String ratchetFrom) throws IOException {
Repository repository = repositoryFor(baseDir);
ObjectId sha = rootTreeShaOf(baseDir, ratchetFrom);
IndexDiff indexDiff = new IndexDiff(repository, sha, new FileTreeIterator(repository));
indexDiff.diff();
String workTreePath = repository.getWorkTree().getPath();
Path baseDirPath = Paths.get(baseDir.getPath());
Set<String> dirtyPaths = new HashSet<>(indexDiff.getChanged());
dirtyPaths.addAll(indexDiff.getAdded());
dirtyPaths.addAll(indexDiff.getConflicting());
dirtyPaths.addAll(indexDiff.getUntracked());

@nedtwigg
Copy link
Member

I believe this should be improved in plugin-maven 2.17.7. Can you report back @rjmunro?

@rjmunro
Copy link
Author

rjmunro commented Dec 24, 2021

On my project on my Intel Macbook Pro, using version 2.17.7 it still takes about 9 minutes, similar to version 2.17.6. With ratchet turned off it takes 20 seconds with either version, so it's not made a noticeable difference.

@nedtwigg
Copy link
Member

Bummer :(. I'm surprised by this, but the good news is that 2.18.0 adds an up-to-date cache which should help a lot whether ratchet is on or off. (#935)

@rjmunro
Copy link
Author

rjmunro commented Feb 9, 2022

Can this be reopened?

@nedtwigg nedtwigg reopened this Feb 9, 2022
@nedtwigg
Copy link
Member

This should be much improved in our latest plugin-gradle 6.13.0 and plugin-maven 2.30.0 thanks to

I'm going to close this again, but please reopen if it is still issue in the latest releases.

# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

No branches or pull requests

2 participants