feat: compliant cli checker for correct variants and dependencies #121

drbh · 2025-04-14T15:23:41Z

This PR adds a new cli tool compliant that checks kernels for compliance.

$ compliant
Hugging Face kernel compliance checker

Usage: compliant <COMMAND>

Commands:
  list   List fetched repositories with build variants
  check  Check repository compliance and ABI compatibility
  help   Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

check args

$ compliant check --help
Check repository compliance and ABI compatibility

Usage: compliant check [OPTIONS] --repos <REPOS>

Options:
  -r, --repos <REPOS>            Repository IDs or names (comma-separated)
  -m, --manylinux <MANYLINUX>    Manylinux version to check against [default: manylinux_2_28]
  -p, --python-abi <PYTHON_ABI>  Python ABI version to check against [default: 3.9]
  -a, --auto-fetch               Automatically fetch repositories if not found locally
  -r, --revision <REVISION>      Revision (branch, tag, or commit hash) to use when fetching [default: main]
      --long                     Show all variants in a long format. Default is compact output
      --show-violations          Show ABI violations in the output. Default is to only show compatibility status
      --format <FORMAT>          Format of the output. Default is console [default: console] [possible values: console, json]
  -h, --help                     Print help

Usage

list all kernels (in cache and have a build variant)

$ compliant list
.
├── kernels-community/activation
├── kernels-community/deformable-detr
├── kernels-community/flash-mla
├── kernels-community/quantization
╰── 4 kernel repositories found

checking a repo

 $ kernels-community/activation
├── build: Total: 13 (CUDA: 12, ROCM: 1)
│   ├── ✓ CUDA
│   ╰── ✗ ROCM
╰── abi: compatible
    ├── ✓ manylinux_2_28
    ╰── ✓ python 3.9

with --long output for variant specific abi compatibility

$ compliant check --repos kernels-community/activation --long
├── build: Total: 13 (CUDA: 12, ROCM: 1)
│  ✓ CUDA
│    ├── torch25-cxx11-cu118-x86_64-linux
│    ├── torch25-cxx11-cu121-x86_64-linux
│    ├── torch25-cxx11-cu124-x86_64-linux
│    ├── torch25-cxx98-cu118-x86_64-linux
│    ├── torch25-cxx98-cu121-x86_64-linux
│    ├── torch25-cxx98-cu124-x86_64-linux
│    ├── torch26-cxx11-cu118-x86_64-linux
│    ├── torch26-cxx11-cu124-x86_64-linux
│    ├── torch26-cxx11-cu126-x86_64-linux
│    ├── torch26-cxx98-cu118-x86_64-linux
│    ├── torch26-cxx98-cu124-x86_64-linux
│    ╰── torch26-cxx98-cu126-x86_64-linux
│  ✗ ROCM
│    ├── torch25-cxx11-rocm5.4-x86_64-linux
│    ├── torch25-cxx11-rocm5.6-x86_64-linux
│    ├── torch25-cxx98-rocm5.4-x86_64-linux
│    ├── torch25-cxx98-rocm5.6-x86_64-linux
│    ├── torch26-cxx11-rocm5.4-x86_64-linux
│    ├── torch26-cxx11-rocm5.6-x86_64-linux
│    ╰── torch26-cxx11-rocm62-x86_64-linux
╰── abi: compatible
    ├── ✓ manylinux_2_28
    ╰── ✓ python 3.9

screenshot to show coloring

Checks

must contain all builds for a given arch (cuda/rocm)
must pass abi-check
conforms to having a build dir
add json output format

danieldk

Really nice to have this! Added a bunch of comments.

compliant/Cargo.toml

compliant/src/lib.rs

compliant/src/main.rs

danieldk

Thanks for making all the changes! Looks very good already, added some more comments on the new changes.

danieldk · 2025-05-02T19:06:36Z

kernel-compliance-check/build.rs

+    println!("cargo:warning=Fetching remote variants JSON...");
+    let url = "https://raw.githubusercontent.com/huggingface/kernel-builder/refs/heads/main/build-variants.json";
+
+    let mut remote_variants_json = String::new();
+
+    match ureq::get(url).call() {
+        Ok(resp) => {
+            match resp.into_reader().read_to_string(&mut remote_variants_json) {
+                Ok(_) => {
+                    println!(
+                        "cargo:warning=Successfully fetched remote variants ({} bytes)",
+                        remote_variants_json.len()
+                    );
+                }
+                Err(e) => {
+                    println!("cargo:warning=Error reading response body: {e}");
+                    // Instead of returning an empty JSON, provide fallback content
+                    remote_variants_json = String::from("{}");
+                }
+            }
+        }
+        Err(e) => {
+            println!("cargo:warning=Error fetching remote variants: {e}");
+            // Provide fallback content
+            remote_variants_json = String::from("{}");
+        }
+    };


This is incompatible with Nix, we cannot have network access during the build. We should make the JSON file part of the compliance checker source files. I think (but we need to check) if we add a symlink to the JSON file in e.g. src, cargo publish would replace the symlink by the contents of the file. In that way we can keep it in the current repo location, but also include it in the crate (when uploading to crates.io).

I also don't think we should be writing Rust code through a string, we don't get syntax checking, completion, etc. We should use include_str or include_bytes and then serde-parse it. The cost of doing so is going to be near-zero.

great point, I've updated with a much more simple solution to include_str of the build_variants.json at the root of this repo, and added a symlink to copy that file into src (need to confirm that publish will work too), but the build.rs is now removed and is much more clean

danieldk · 2025-05-02T19:10:45Z

kernel-compliance-check/build.rs

+    VARIANTS_CACHE.get_or_init(|| {{
+        serde_json::from_str(VARIANTS_DATA).unwrap_or_else(|_| {{
+            // Provide a fallback empty object if parsing fails
+            serde_json::json!({{}})


If the data comes from the repo, we know it's going to be in-sync and we can panic when the JSON is invalid (since it's a bug).

updated/removed in latest changes

kernel-compliance-check/src/lib.rs

danieldk · 2025-05-02T19:14:17Z

kernel-compliance-check/src/lib.rs

+            cmd.arg("--force");
+        }
+
+        println!("Using huggingface-cli to download repository");


NIT: I would print such messages to stderr. For me stdout is only for output that you may want to pipe to other processes.

good point, ive updated many of the print lines to eprintln

kernel-compliance-check/src/lib.rs

danieldk · 2025-05-02T19:22:08Z

kernel-compliance-check/src/lib.rs

+
+    // Parse object file
+    let file = object::File::parse(&*binary_data)
+        .map_err(|e| eyre::eyre!("Cannot parse object file: {}: {}", so_path.display(), e))?;


In all such cases I always use with_context in place of map_err. The original error is printed as well as part of the chain when the error is bubbled up to outside main. I think error chains are nicer than errors nested in a string (e.g. imagine the caller of this function wrapping it in a string again).

oh nice thats a great recommendation, i've updated all of the map_err to with_context, that is a very nice error improvement. Thanks!

danieldk · 2025-05-02T19:24:50Z

kernel-compliance-check/src/models.rs

+
+        /// Manylinux version to check against
+        #[arg(short, long, default_value = "manylinux_2_28")]
+        manylinux: ManylinuxVersion,


I think we should stringly-type this to avoid having to keep the enum in sync with the data from the manylinux project.

makes sense, updated in the latest commits. Thanks!

…ole formatting

…r, stringly type manylinux, respect env cache var

danieldk

🎉

drbh force-pushed the add-compliant-checker-tool branch from bcc946d to bd4f49f Compare April 14, 2025 22:52

drbh marked this pull request as ready for review April 24, 2025 00:07

danieldk reviewed Apr 24, 2025

View reviewed changes

drbh requested a review from danieldk May 1, 2025 13:30

danieldk reviewed May 2, 2025

View reviewed changes

drbh added 20 commits May 6, 2025 12:15

feat: compliant cli checker for correct variants and dependencies

96d7a7e

feat: improve output and add json format option

18aa548

fix: explore various download methods

9885490

fix: simplify fetch logic to single path

29d6dfc

fix: fetch variants from repo

2e8b496

feat: split files and simplify formatting

c64b9a8

feat: refactor logic into lib, improve error handling, types and cons…

7a2b8c8

…ole formatting

fix: prefer versioned kernel-abi-check dep

0e0e8eb

fix: improve metadata to prep for cargo publish

d06bcb2

feat: rename project and dir

a919c88

fix: split dep imports

4d625e1

feat: improve core sync logic and file structure

076d240

feat: vendor variants at build time

e57bf66

feat: prefer eyre over anyhow

d6324e4

fix: remove unused struct

8067b3d

fix: prefer peekable over manual enumeration

697c347

fix: add missing eyre dep change

fe3bfa4

feat: check hub for latest snapshot in all cases

c41ba85

fix: address all clippy lints

f0db3c7

fix: small typo cleanup

b8026b1

drbh force-pushed the add-compliant-checker-tool branch from 40d8da1 to b8026b1 Compare May 6, 2025 16:16

drbh added 2 commits May 6, 2025 12:46

fix: avoid build script, prefer error chaining, prefer print to stder…

4fc183b

…r, stringly type manylinux, respect env cache var

fix: add symlink for variant json

7c5ce43

drbh requested a review from danieldk May 6, 2025 16:49

fix: add missing eprints

6df4df4

danieldk approved these changes May 7, 2025

View reviewed changes

danieldk merged commit 4784203 into main May 7, 2025
8 checks passed

danieldk deleted the add-compliant-checker-tool branch May 7, 2025 12:43

feat: compliant cli checker for correct variants and dependencies #121

feat: compliant cli checker for correct variants and dependencies #121

Uh oh!

Conversation

drbh commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Usage

Checks

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danieldk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

drbh commented Apr 14, 2025 •

edited

Loading