Add SLoC (Source Lines of Code) metric to versions #11453
+794
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds a new `JSONB` column called `linecounts` to the versions table to store Source Lines of Code statistics for each crate version. The column stores language breakdown and totals as structured `JSON` data, enabling flexible schema evolution without requiring additional migrations. The database schema and test snapshots are updated accordingly to reflect this new column structure.
This introduces a new workspace crate that provides line counting functionality using `tokei`. The crate includes `LinecountStats` and `LanguageStats` data structures for storing results, along with core analysis functions for processing file contents. The implementation includes language filtering to exclude non-programming files and path filtering to skip test and example directories. Comprehensive test coverage is provided with `insta` snapshots to ensure reliable functionality. This crate provides the foundation for adding SLOC metrics to crates.io by offering a clean, testable interface for analyzing source code statistics.
This adds the `linecounts` field to both the `Version` struct and `NewVersion` builder. The field stores linecount data as `JSON`, following the established pattern for flexible schema evolution without requiring additional migrations. The `linecounts` field is `Optional` to handle existing versions that don't have this data, and will be populated for new versions during the publish process. This design ensures backward compatibility while enabling rich source code metrics for future crate versions.
This enhances the tarball processing pipeline to include SLOC analysis by adding `crates_io_linecount` dependency to the tarball processing crate and extending the `TarballInfo` struct with a `linecount_stats` field. The integration occurs seamlessly during tarball file processing, where each qualifying source file is analyzed and its statistics are accumulated. All tarball processing test snapshots are updated to include linecount data, demonstrating the feature works correctly across various crate structures. The integration preserves existing functionality while adding minimal overhead to the tarball validation and processing pipeline.
Turbo87
added
C-enhancement ✨
This modifies the publish endpoint to extract and store linecount statistics by extracting linecount data from tarball processing results and serializing the stats to `JSON` for database storage. The linecount data is then passed to the `NewVersion` builder for persistence. All publish-related test snapshots are updated to include linecount data, demonstrating that the integration works correctly across various publish scenarios. The implementation maintains backward compatibility with null linecount values for any edge cases.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces basic source code analysis for newly published versions. A new
crates_io_linecountworkspace crate uses thetokeicrate to analyze source files during the publish process. The system collects language breakdowns and line count statistics, storing them as JSON in a newlinecountscolumn on theversionstable.The analysis runs during tarball processing and excludes test directories and non-programming files. All existing functionality remains unchanged, with the new column being optional for backward compatibility.
Note that this is only the first step in a series of pull requests. The follow-up PRs will: