Auto merge of #10807 - dtolnay-contrib:sha256, r=weihanglo

Apply GitHub fast path even for partial hashes

### What does this PR try to resolve?

As flagged in #10079 (comment), it's not great to assume that git SHAs would always be 40-character hex strings. In the future they will be longer.

> Git is on a long-term trajectory to move to SHA256 hashes ([current status](https://lwn.net/SubscriberLink/898522/f267d0e9b4fe9983/)). I suppose when that becomes available/the default it's possible for a 40-digit hex-encoded hash not to be the full hash. Will this fail for that case?

The implementation from #10079 fails in that situation because it turns dependencies of the form `{ git = "…", rev = "[…40 hex…]" }` into fetches with a refspec `+[…40 hex…]:refs/commit/[…40 hex…]`. That only works if the 40 hex digits are the *full* long hash of your commit. If it's really a prefix ("short hash") of a 64-hex-digit SHA-256 commit hash, you'd get a failure that resembles:

```console
error: failed to get `dependency` as a dependency of package `repro v0.0.0`

Caused by:
  failed to load source for dependency `dependency`

Caused by:
  Unable to update https://github.com/rust-lang/dependency?rev=b30694b4d9b29141298870b7993e9aee10940524

Caused by:
  revspec 'b30694b4d9b29141298870b7993e9aee10940524' not found; class=Reference (4); code=NotFound (-3)
```

This PR updates the implementation so that Cargo will curl GitHub to get a resolved long commit hash *even if* the `rev` specified for the git dependency in Cargo.toml already looks like a SHA-1 long hash.

### Performance considerations

⛔ This reverses a (questionable, negligible) benefit of #10079 of skipping the curl when `rev` is a long hash and is not already present in the local clone. These curls take 200-250ms on my machine.

🟰 We retain the much larger benefit of #10079 which comes from being able to precisely fetch a single `rev`, instead of fetching all branches and tags in the upstream repo and hoping to find the rev somewhere in there. This accounts for the entire performance difference explained in the summary of that PR.

🟰 We still skip the curl when `rev` is a **long hash** of a commit that is already previously fetched.

🥳 After this PR, we also curl and hit fast path when `rev` is a **short hash** of some upstream commit. For example `{ git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9" }` would previously have done the download-all-branches-and-tags codepath because `b30694b4d9` is not a long hash. After this PR, the curl to GitHub informs us that `b30694b4d9` resolves to the long hash `b30694b4d9b29141298870b7993e9aee10940524`, and we download just that commit instead of all-branches-and-tags.

### How should we test and review this PR?

I tested with the following dependency specs, using `/path/to/target/release/cargo generate-lockfile`.

```toml
# Before: slow path (fetch all branches and tags; 70K git objects)
# After: fast path (20K git objects)
cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b2914129" }
```

```toml
# Before and after: fast path
cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b29141298870b7993e9aee10940524" }
```

```toml
# Before and after: fast path
cargo = { git = "https://github.com/rust-lang/cargo", rev = "refs/heads/rust-1.14.0" }
```

```toml
# Before and after: same error "revspec 'rust-1.14.0' not found"
# You are supposed to use `branch = "rust-1.14.0"`, this is not considered a `rev`
cargo = { git = "https://github.com/rust-lang/cargo", rev = "rust-1.14.0" }
```

I made sure these all work both with and without `rm -rf ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-*` in between each cargo invocation.

FYI `@djc`

Author: bors

src/cargo/sources/git/utils.rs

@@ -7,14 +7,15 @@ use crate::util::{network, Config, IntoUrl, MetricsCounter, Progress};
 use anyhow::{anyhow, Context as _};
 use cargo_util::{paths, ProcessBuilder};
 use curl::easy::List;
-use git2::{self, ErrorClass, ObjectType};
+use git2::{self, ErrorClass, ObjectType, Oid};
 use log::{debug, info};
 use serde::ser;
 use serde::Serialize;
 use std::env;
 use std::fmt;
 use std::path::{Path, PathBuf};
 use std::process::Command;
+use std::str;
 use std::time::{Duration, Instant};
 use url::Url;
 
@@ -759,11 +760,15 @@ pub fn fetch(
 
     // If we're fetching from GitHub, attempt GitHub's special fast path for
     // testing if we've already got an up-to-date copy of the repository
-    match github_up_to_date(repo, url, reference, config) {
-        Ok(true) => return Ok(()),
-        Ok(false) => {}
-        Err(e) => debug!("failed to check github {:?}", e),
-    }
+    let oid_to_fetch = match github_fast_path(repo, url, reference, config) {
+        Ok(FastPathRev::UpToDate) => return Ok(()),
+        Ok(FastPathRev::NeedsFetch(rev)) => Some(rev),
+        Ok(FastPathRev::Indeterminate) => None,
+        Err(e) => {
+            debug!("failed to check github {:?}", e);
+            None
+        }
+    };
 
     // We reuse repositories quite a lot, so before we go through and update the
     // repo check to see if it's a little too old and could benefit from a gc.
@@ -793,11 +798,10 @@ pub fn fetch(
         }
 
         GitReference::Rev(rev) => {
-            let is_github = || Url::parse(url).map_or(false, |url| is_github(&url));
             if rev.starts_with("refs/") {
                 refspecs.push(format!("+{0}:{0}", rev));
-            } else if is_github() && is_long_hash(rev) {
-                refspecs.push(format!("+{0}:refs/commit/{0}", rev));
+            } else if let Some(oid_to_fetch) = oid_to_fetch {
+                refspecs.push(format!("+{0}:refs/commit/{0}", oid_to_fetch));
             } else {
                 // We don't know what the rev will point to. To handle this
                 // situation we fetch all branches and tags, and then we pray
@@ -994,45 +998,79 @@ fn init(path: &Path, bare: bool) -> CargoResult<git2::Repository> {
     Ok(git2::Repository::init_opts(&path, &opts)?)
 }
 
+enum FastPathRev {
+    /// The local rev (determined by `reference.resolve(repo)`) is already up to
+    /// date with what this rev resolves to on GitHub's server.
+    UpToDate,
+    /// The following SHA must be fetched in order for the local rev to become
+    /// up to date.
+    NeedsFetch(Oid),
+    /// Don't know whether local rev is up to date. We'll fetch _all_ branches
+    /// and tags from the server and see what happens.
+    Indeterminate,
+}
+
 /// Updating the index is done pretty regularly so we want it to be as fast as
 /// possible. For registries hosted on GitHub (like the crates.io index) there's
 /// a fast path available to use [1] to tell us that there's no updates to be
 /// made.
 ///
 /// This function will attempt to hit that fast path and verify that the `oid`
-/// is actually the current branch of the repository. If `true` is returned then
-/// no update needs to be performed, but if `false` is returned then the
-/// standard update logic still needs to happen.
+/// is actually the current branch of the repository.
 ///
 /// [1]: https://developer.github.com/v3/repos/commits/#get-the-sha-1-of-a-commit-reference
 ///
 /// Note that this function should never cause an actual failure because it's
 /// just a fast path. As a result all errors are ignored in this function and we
 /// just return a `bool`. Any real errors will be reported through the normal
 /// update path above.
-fn github_up_to_date(
+fn github_fast_path(
     repo: &mut git2::Repository,
     url: &str,
     reference: &GitReference,
     config: &Config,
-) -> CargoResult<bool> {
+) -> CargoResult<FastPathRev> {
     let url = Url::parse(url)?;
     if !is_github(&url) {
-        return Ok(false);
+        return Ok(FastPathRev::Indeterminate);
     }
 
+    let local_object = reference.resolve(repo).ok();
+
     let github_branch_name = match reference {
         GitReference::Branch(branch) => branch,
         GitReference::Tag(tag) => tag,
         GitReference::DefaultBranch => "HEAD",
         GitReference::Rev(rev) => {
             if rev.starts_with("refs/") {
                 rev
-            } else if is_long_hash(rev) {
-                return Ok(reference.resolve(repo).is_ok());
+            } else if looks_like_commit_hash(rev) {
+                // `revparse_single` (used by `resolve`) is the only way to turn
+                // short hash -> long hash, but it also parses other things,
+                // like branch and tag names, which might coincidentally be
+                // valid hex.
+                //
+                // We only return early if `rev` is a prefix of the object found
+                // by `revparse_single`. Don't bother talking to GitHub in that
+                // case, since commit hashes are permanent. If a commit with the
+                // requested hash is already present in the local clone, its
+                // contents must be the same as what is on the server for that
+                // hash.
+                //
+                // If `rev` is not found locally by `revparse_single`, we'll
+                // need GitHub to resolve it and get a hash. If `rev` is found
+                // but is not a short hash of the found object, it's probably a
+                // branch and we also need to get a hash from GitHub, in case
+                // the branch has moved.
+                if let Some(local_object) = local_object {
+                    if is_short_hash_of(rev, local_object) {
+                        return Ok(FastPathRev::UpToDate);
+                    }
+                }
+                rev
             } else {
                 debug!("can't use github fast path with `rev = \"{}\"`", rev);
-                return Ok(false);
+                return Ok(FastPathRev::Indeterminate);
             }
         }
     };
@@ -1065,18 +1103,50 @@ fn github_up_to_date(
     handle.get(true)?;
     handle.url(&url)?;
     handle.useragent("cargo")?;
-    let mut headers = List::new();
-    headers.append("Accept: application/vnd.github.3.sha")?;
-    headers.append(&format!("If-None-Match: \"{}\"", reference.resolve(repo)?))?;
-    handle.http_headers(headers)?;
-    handle.perform()?;
-    Ok(handle.response_code()? == 304)
+    handle.http_headers({
+        let mut headers = List::new();
+        headers.append("Accept: application/vnd.github.3.sha")?;
+        if let Some(local_object) = local_object {
+            headers.append(&format!("If-None-Match: \"{}\"", local_object))?;
+        }
+        headers
+    })?;
+
+    let mut response_body = Vec::new();
+    let mut transfer = handle.transfer();
+    transfer.write_function(|data| {
+        response_body.extend_from_slice(data);
+        Ok(data.len())
+    })?;
+    transfer.perform()?;
+    drop(transfer); // end borrow of handle so that response_code can be called
+
+    let response_code = handle.response_code()?;
+    if response_code == 304 {
+        Ok(FastPathRev::UpToDate)
+    } else if response_code == 200 {
+        let oid_to_fetch = str::from_utf8(&response_body)?.parse::<Oid>()?;
+        Ok(FastPathRev::NeedsFetch(oid_to_fetch))
+    } else {
+        // Usually response_code == 404 if the repository does not exist, and
+        // response_code == 422 if exists but GitHub is unable to resolve the
+        // requested rev.
+        Ok(FastPathRev::Indeterminate)
+    }
 }
 
 fn is_github(url: &Url) -> bool {
     url.host_str() == Some("github.com")
 }
 
-fn is_long_hash(rev: &str) -> bool {
-    rev.len() == 40 && rev.chars().all(|ch| ch.is_ascii_hexdigit())
+fn looks_like_commit_hash(rev: &str) -> bool {
+    rev.len() >= 7 && rev.chars().all(|ch| ch.is_ascii_hexdigit())
+}
+
+fn is_short_hash_of(rev: &str, oid: Oid) -> bool {
+    let long_hash = oid.to_string();
+    match long_hash.get(..rev.len()) {
+        Some(truncated_long_hash) => truncated_long_hash.eq_ignore_ascii_case(rev),
+        None => false,
+    }
 }