Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs #126427

Oneirical · 2024-06-13T19:05:39Z

Part of #121876 and the associated Google Summer of Code project.

rustbot · 2024-06-13T19:05:47Z

rustbot has assigned @jieyouxu.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-06-13T19:05:49Z

The run-make-support library was changed

cc @jieyouxu

This PR modifies tests/run-make/. If this PR is trying to port a Makefile
run-make test to use rmake.rs, please update the
run-make port tracking issue
so we can track our progress. You can either modify the tracking issue
directly, or you can comment on the tracking issue and link this PR.

cc @jieyouxu

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

src/tools/run-make-support/src/lib.rs

fbstj · 2024-06-13T20:46:27Z

src/tools/run-make-support/src/lib.rs

+    paths.filter_map(|entry| entry.ok()).filter(|path| path.is_file()).for_each(|path| {
+        let file = fs_wrapper::open_file(path);
+        let reader = io::BufReader::new(file);
+        reader
+            .lines()
+            .filter_map(|line| line.ok())
+            .filter(|line| re.is_match(line))
+            .for_each(|_| match_count += 1);
+    });
+    match_count


I have a feeling this can be streamlined using the .count() method to remove the need to manually increment, but I don't know what the most idiomatic style would be

tests/run-make/intrinsic-unreachable/rmake.rs

tests/run-make/sepcomp-separate/Makefile

jieyouxu · 2024-06-16T22:39:14Z

src/tools/run-make-support/src/lib.rs

+/// Inside a glob pattern of files (paths), read their contents and count the
+/// number of regex matches with a given expression (re).
+#[track_caller]
+pub fn count_regex_matches_in_file_glob(re: &str, paths: &str) -> usize {
+    let re = regex::Regex::new(re).expect(format!("Regex expression {re} is not valid.").as_str());
+    let paths = glob::glob(paths).expect(format!("Glob expression {paths} is not valid.").as_str());
+    use io::BufRead;
+    paths
+        .filter_map(|entry| entry.ok())
+        .filter(|entry| entry.as_path().is_file())
+        .filter_map(|path| fs::File::open(&path).ok())
+        .map(|file| io::BufReader::new(file))
+        .flat_map(|reader| reader.lines().filter_map(|entry| entry.ok()))
+        .filter(|line| re.is_match(line))
+        .count()
+}


Discussion: a globbing API makes me very uneasy, I would strongly prefer if this was a walkdir-style API (and specialized helpers built on such an API, see src/tools/tidy's usage of walkdir-style APIs) instead, because it's not immediately clear if this is recursive, deals with "hidden" files, handles non-UTF8, among other filesystem / glob-related possible failstates and gotchas.

I have tried my hand at a reworked version of this that does not use glob - see the latest commit. Tell me what you think!

jieyouxu · 2024-06-16T22:41:25Z

tests/run-make/sepcomp-cci-copies/rmake.rs

+fn main() {
+    rustc().input("cci_lib.rs").run();
+    rustc().input("foo.rs").emit("llvm-ir").codegen_units(6).arg("-Zinline-in-all-cgus").run();
+    assert_eq!(count_regex_matches_in_file_glob(r#"define\ .*cci_fn"#, "foo.*.ll"), 2);


Discussion: cf. the support library comment, I feel like this helper API makes it almost too opaque for the test reader of what is being tested.

jieyouxu · 2024-06-16T22:42:43Z

@rustbot author

jieyouxu · 2024-06-17T16:42:50Z

Note to myself: revisit this PR @rustbot review

bors · 2024-06-19T02:43:23Z

☔ The latest upstream changes (presumably #126607) made this pull request unmergeable. Please resolve the merge conflicts.

jieyouxu · 2024-06-29T07:29:48Z

I think we should just avoid using walkdir because there's some funny windows-targets related business going down (tried debugging, no luck). We can just build it upon fs::read_dir() et al.

@rustbot author

Oneirical · 2024-07-02T15:34:44Z

Notes for myself:

Use shallow_find_files instead of walkdir.
In intrinsic-unreachable, use fs_wrapper::read_to_string instead of open_file.

Oneirical · 2024-07-05T16:16:02Z

@rustbot review

jieyouxu

Tests look good, r=me after hoisting the helper into the support lib.

jieyouxu · 2024-07-06T05:33:16Z

tests/run-make/sepcomp-separate/rmake.rs

+fn count_regex_matches_in_files_with_extension(re: &regex::Regex, ext: &str) -> usize {
+    let fetched_files = shallow_find_files(cwd(), |path| has_extension(path, ext));
+
+    let mut count = 0;
+    for file in fetched_files {
+        let content = fs_wrapper::read_to_string(file);
+        count += content.lines().filter(|line| re.is_match(&line)).count();
+    }
+
+    count
+}


Suggestion: can you hoist this into the support library and add some docs for what it does? I know I may have said to keep it localized in the test files previously, sorry!

jieyouxu · 2024-07-06T05:34:00Z

@rustbot author

Oneirical · 2024-07-08T14:17:28Z

@bors r=@jieyouxu

bors · 2024-07-08T14:17:30Z

📌 Commit f04d0c6 has been approved by jieyouxu

It is now in the queue for this repository.

…llaumeGomez Rollup of 4 pull requests Successful merges: - rust-lang#126427 (Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs) - rust-lang#127237 (Improve code of `run-make/llvm-ident`) - rust-lang#127325 (Migrate `target-cpu-native`, `target-specs` and `target-without-atomic-cas` `run-make` tests to rmake) - rust-lang#127482 (Infer async closure signature from (old-style) two-part `Fn` + `Future` bounds) Failed merges: - rust-lang#127357 (Remove `StructuredDiag`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#126427 - Oneirical:oktobertest, r=jieyouxu Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs Part of rust-lang#121876 and the associated [Google Summer of Code project](https://blog.rust-lang.org/2024/05/01/gsoc-2024-selected-projects.html).

rustbot assigned jieyouxu Jun 13, 2024

Oneirical force-pushed the oktobertest branch 2 times, most recently from 954fc4f to bf298cb Compare June 13, 2024 19:22

This comment has been minimized.

# to view

fbstj reviewed Jun 13, 2024

View reviewed changes

src/tools/run-make-support/src/lib.rs Outdated Show resolved Hide resolved

Oneirical force-pushed the oktobertest branch from bf298cb to 9d50fdc Compare June 13, 2024 20:39

This comment has been minimized.

# to view

fbstj reviewed Jun 13, 2024

View reviewed changes

Kobzol changed the title ~~Rewrite intrinsic-unreachable, sepcomp-separate, sepcomp-inlining and sepcomp-separate run-make tests to rmake.rs~~ Rewrite intrinsic-unreachable, sepcomp-cci-copies, sepcomp-inlining and sepcomp-separate run-make tests to rmake.rs Jun 13, 2024

Kobzol mentioned this pull request Jun 13, 2024

Tracking Issue for porting run-make tests to use Rust #121876

Open

Oneirical force-pushed the oktobertest branch from 9d50fdc to bce0bfe Compare June 13, 2024 21:02

This comment has been minimized.

# to view

Oneirical force-pushed the oktobertest branch from bce0bfe to 348c183 Compare June 14, 2024 14:02

jieyouxu reviewed Jun 16, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 16, 2024

Oneirical force-pushed the oktobertest branch 2 times, most recently from cbf8157 to dc989e7 Compare June 17, 2024 15:54

This comment has been minimized.

# to view

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 17, 2024

Oneirical force-pushed the oktobertest branch from dc989e7 to 91efd20 Compare June 17, 2024 16:43

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 19, 2024

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 29, 2024

Oneirical force-pushed the oktobertest branch 2 times, most recently from 4bbe6b9 to 64a7343 Compare July 5, 2024 15:06

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 5, 2024

jieyouxu approved these changes Jul 6, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 6, 2024

Oneirical force-pushed the oktobertest branch 2 times, most recently from 8e89c65 to 1c69496 Compare July 8, 2024 14:13

Oneirical added 3 commits July 8, 2024 10:13

rewrite intrinsic-unreachable to rmake

a4c72b6

rewrite sepcomp-separate to rmake

2dda1e3

rewrite sepcomp-inlining and -separate to rmake.rs

f04d0c6

Oneirical force-pushed the oktobertest branch from 1c69496 to f04d0c6 Compare July 8, 2024 14:14

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 8, 2024

GuillaumeGomez mentioned this pull request Jul 8, 2024

Rollup of 4 pull requests #127489

Merged

bors merged commit 9804cf9 into rust-lang:master Jul 8, 2024
6 checks passed

rustbot added this to the 1.81.0 milestone Jul 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs #126427

Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs #126427

Oneirical commented Jun 13, 2024

rustbot commented Jun 13, 2024

rustbot commented Jun 13, 2024

This comment has been minimized.

This comment has been minimized.

fbstj Jun 13, 2024

Oneirical Jun 13, 2024 •

edited

Loading

This comment has been minimized.

jieyouxu Jun 16, 2024 •

edited

Loading

Oneirical Jun 17, 2024 •

edited

Loading

jieyouxu Jun 16, 2024

jieyouxu commented Jun 16, 2024

This comment has been minimized.

jieyouxu commented Jun 17, 2024

bors commented Jun 19, 2024

jieyouxu commented Jun 29, 2024

Oneirical commented Jul 2, 2024

Oneirical commented Jul 5, 2024

jieyouxu left a comment

jieyouxu Jul 6, 2024

jieyouxu commented Jul 6, 2024

Oneirical commented Jul 8, 2024

bors commented Jul 8, 2024

Rewrite intrinsic-unreachable, sepcomp-cci-copies, sepcomp-inlining and sepcomp-separate run-make tests to rmake.rs #126427

Rewrite intrinsic-unreachable, sepcomp-cci-copies, sepcomp-inlining and sepcomp-separate run-make tests to rmake.rs #126427

Conversation

Oneirical commented Jun 13, 2024

rustbot commented Jun 13, 2024

rustbot commented Jun 13, 2024

This comment has been minimized.

This comment has been minimized.

fbstj Jun 13, 2024

Choose a reason for hiding this comment

Oneirical Jun 13, 2024 • edited Loading

Choose a reason for hiding this comment

This comment has been minimized.

jieyouxu Jun 16, 2024 • edited Loading

Choose a reason for hiding this comment

Oneirical Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

jieyouxu Jun 16, 2024

Choose a reason for hiding this comment

jieyouxu commented Jun 16, 2024

This comment has been minimized.

jieyouxu commented Jun 17, 2024

bors commented Jun 19, 2024

jieyouxu commented Jun 29, 2024

Oneirical commented Jul 2, 2024

Oneirical commented Jul 5, 2024

jieyouxu left a comment

Choose a reason for hiding this comment

jieyouxu Jul 6, 2024

Choose a reason for hiding this comment

jieyouxu commented Jul 6, 2024

Oneirical commented Jul 8, 2024

bors commented Jul 8, 2024

Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs #126427

Rewrite `intrinsic-unreachable`, `sepcomp-cci-copies`, `sepcomp-inlining` and `sepcomp-separate` `run-make` tests to rmake.rs #126427

Oneirical Jun 13, 2024 •

edited

Loading

jieyouxu Jun 16, 2024 •

edited

Loading

Oneirical Jun 17, 2024 •

edited

Loading