cargo segfaults on NFS with git dependencies

[asomers]

On FreeBSD, running "cargo build" on any repo with a git dependency specified in Cargo.toml will segfault. It doesn't matter if the dependency is on github or uses a file:/// url.

cargo version
cargo 0.11.0 (built 2016-07-27)
cargo test
    Updating git repository `https://github.com/asomers/libc`
error: Unable to update https://github.com/asomers/libc?rev=a4721e8

To learn more, run the command again with --verbose.
fish: 'cargo test' terminated by signal SIGSEGV (Address boundary error)

Here is the backtrace from the core file:

#0  0x00000000014c5be1 in ?? ()
#1  0x00000000014da841 in ?? ()
#2  0x00000000014da636 in ?? ()
#3  0x00000008023bb685 in CRYPTO_free (str=0x80521ffa8) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/mem.c:442
#4  0x00000008023b9cb2 in ASN1_STRING_free (a=0x8052add00) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/asn1_lib.c:425
#5  0x00000008023b2bb2 in ASN1_primitive_free (pval=0x8052adcf0, it=0x8052084e8) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:244
#6  0x00000008023b28eb in asn1_item_combine_free (pval=<optimized out>, it=<optimized out>, combine=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:166
#7  0x00000008023b2aac in ASN1_template_free (pval=<optimized out>, tt=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:186
#8  0x00000008023b28eb in asn1_item_combine_free (pval=<optimized out>, it=<optimized out>, combine=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:166
#9  0x00000008023b28eb in asn1_item_combine_free (pval=<optimized out>, it=<optimized out>, combine=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:166
#10 0x00000008023b27d6 in ASN1_item_free (val=<error reading variable: Cannot access memory at address 0x1fe0>, it=0x8052084e8) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/asn1/tasn_fre.c:72
#11 0x00000008022ff793 in cleanup (a=0x8053724a0) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/x509/x509_lu.c:222
#12 0x0000000802303d30 in sk_pop_free (st=<optimized out>, func=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/stack/stack.c:327
#13 0x00000008022ff740 in X509_STORE_free (vfy=<optimized out>) at /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/x509/x509_lu.c:261
#14 0x0000000801eb4156 in SSL_CTX_free (a=0x805238380) at /usr/src/secure/lib/libssl/../../../crypto/openssl/ssl/ssl_lib.c:2142
#15 0x000000000144f77a in ?? ()
#16 0x000000000143deb1 in ?? ()
#17 0x0000000802deea46 in __cxa_finalize (dso=<optimized out>) at /usr/src/lib/libc/stdlib/atexit.c:237
#18 0x0000000802d7e04c in exit (status=101) at /usr/src/lib/libc/stdlib/exit.c:66
#19 0x00000000014a0ca0 in ?? ()
#20 0x0000000001114801 in ?? ()
#21 0x000000000109a757 in ?? ()
#22 0x00000000010a31e0 in ?? ()
#23 0x000000000109c15c in ?? ()
#24 0x0000000001098f92 in ?? ()
#25 0x00000000014ac079 in ?? ()
#26 0x00000000014b991c in ?? ()
#27 0x00000000014b98bf in ?? ()
#28 0x00000000014abafa in ?? ()
#29 0x0000000001098971 in ?? ()
#30 0x00000008017f0000 in ?? ()
#31 0x0000000000000000 in ?? ()

Using Cargo from git, the result is an assertion error instead of a segfault.

~/src/rust/cargo/target/debug/cargo version
cargo 0.13.0
~/src/rust/cargo/target/debug/cargo test
    Updating git repository `https://github.com/asomers/libc`
Assertion failed: (e->refcount.val == 0), function free_cache_object, file /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/pack.c, line 73.
fish: '~/src/rust/cargo/target/debug/c…' terminated by signal SIGABRT (Abort)

In this case, the core file contains no intelligible stack trace.

My Cargo.toml file contains the following lines. However, every other variation I tried produces the same result, as long as it involves git.

[dependencies]
libc = { git = "https://github.com/asomers/libc", rev="a4721e8"}

[alexcrichton]

Oh dear, this sounds bad!

Could you try running the test suite of git2-rs and see if there's a smaller test case exhibiting this assertion?

[asomers]

Nope. In the 0.4.4 and the git head version of git2-rs, all tests pass. I can't build the 0.4 version because of an unresolved import error. However, I did find a test failure in libgit2 that might be related.

libgit2/libgit2#3911

[alexcrichton]

Definitely sounds fishy! Perhaps you could try building a debug version of Cargo and stepping through the faulting function?

[asomers]

When I build a debug version of Cargo, gdb can't understand the core file. However, gdb has no problems with a core file made from a release version of gdb. Odd. But I did manage to get a stack trace from the debug version by setting a breakpoint at the assertion that fails. Unfortunately, the bug is a refcount mismatch in a structure deep within the bowels of libgit2. As such, it's beyond my ability to debug. I'm afraid that it's going to take a libgit2 expert.

> #0  free_cache_object (o=0x805cc7210) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/pack.c:73
> #1  0x00000000018f2411 in cache_free (cache=0x805d036f0) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/pack.c:86
> #2  0x00000000018f231c in git_packfile_free (p=0x805d03680) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/pack.c:992
> #3  0x000000000194800e in git_mwindow_put_pack (pack=0x805d03680) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/mwindow.c:128
> #4  0x00000000018e8111 in pack_backend__free (_backend=0x805ccdc00) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/odb_pack.c:533
> #5  0x00000000018e4347 in odb_free ()
> #6  0x00000000018e42c2 in git_odb_free ()
> #7  0x000000000190251c in set_odb (repo=0x805c598a0, odb=0x805a46dc0) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/repository.c:63
> #8  0x000000000190239b in git_repository__cleanup (repo=0x805c598a0) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/repository.c:117
> #9  0x00000000019025d5 in git_repository_free (repo=0x805c598a0) at /usr/home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/libgit2-sys-0.4.5/libgit2/src/repository.c:128
> #10 0x00000000018aa9d8 in git2::repo::{{impl}}::drop (self=0x7fffffff0930) at /home/somers/.cargo/registry/src/github.heygears.com-1ecc6299db9ec823/git2-0.4.4/src/repo.rs:1700
> #11 0x0000000001419afd in git2..Repository::drop.43884::hdda813d727d29064 ()
> #12 0x0000000001607bd4 in sources..git..utils..GitDatabase::drop.48956::h4fa91c402b197326 ()
> #13 0x00000000011fec80 in _$LT$sources..git..source..GitSource$LT$$u27$cfg$GT$$u20$as$u20$core..source..Source$GT$::update::hd77cec083b7cd665 ()
> #14 0x0000000001300912 in cargo::core::registry::PackageRegistry::load::_$u7b$$u7b$closure$u7d$$u7d$::he53950680ace08c8 ()
> #15 0x000000000130098d in extern$u20$$u22$rust.call$u22$$u20$fn$LP$$u5b$closure$SP$src$u2f$cargo$u2f$core$u2f$registry.rs.176.9.$u20$187.11$u20$self.$RF$$u27$static$u20$mut$u20$$RF$$u27$static$u20$mut$u20$core..registry..PackageRegistry$LT$$u27$static$GT$$C$$u20$source_id.$RF$$u27$static$u20$$RF$$u27$static$u20$core..source..SourceId$C$$u20$kind.$RF$$u27$static$u20$core..registry..Kind$u5d$$C$$u20$$LP$$RP$$RP$$u20$.$GT$$u20$std..result..Result$LT$$LP$$RP$$C$$u20$Box$LT$util..errors..CargoError$u20$$u2b$$u20$$u27$static$GT$$GT$::once_shim.40630::h933b4c9aae749c5d ()
> #16 0x00000000012ffc8d in _$LT$F$u20$as$u20$util..errors..ChainError$LT$T$GT$$GT$::chain_error::h18f7abb78a257ad7 ()
> #17 0x00000000012f18ab in cargo::core::registry::PackageRegistry::load::h6c3f46ffd916c95d ()
> #18 0x00000000012f07d6 in cargo::core::registry::PackageRegistry::ensure_loaded::hcc3656632f6462d4 ()
> #19 0x0000000001306e8b in _$LT$core..registry..PackageRegistry$LT$$u27$cfg$GT$$u20$as$u20$core..registry..Registry$GT$::query::h470044f9865772b2 ()
> #20 0x0000000001295af7 in cargo::core::resolver::Context::query::haf23c048198cfa13 ()
> #21 0x0000000001294779 in cargo::core::resolver::Context::build_deps::_$u7b$$u7b$closure$u7d$$u7d$::h38da1e9044856445 ()
> #22 0x0000000001294609 in core::ops::impls::_$LT$impl$u20$std..ops..FnOnce$LT$A$GT$$u20$for$u20$$RF$$u27$a$u20$mut$u20$F$GT$::call_once::h76547c815785738d ()
> #23 0x00000000012944eb in _$LT$std..option..Option$LT$T$GT$$GT$::map::hcda37f6e37edbfe1 ()
> #24 0x000000000129444f in _$LT$std..iter..Map$LT$I$C$$u20$F$GT$$u20$as$u20$std..iter..Iterator$GT$::next::h302db353e370b30c ()
> #25 0x0000000001294216 in _$LT$$LT$std..result..Result$LT$V$C$$u20$E$GT$$u20$as$u20$std..iter..FromIterator$LT$std..result..Result$LT$A$C$$u20$E$GT$$GT$$GT$..from_iter..Adapter$LT$Iter$C$$u20$E$GT$$u20$as$u20$std..iter..Iterator$GT$::next::h1bb44f23f6578855 ()
> #26 0x00000000012941df in _$LT$$RF$$u27$a$u20$mut$u20$I$u20$as$u20$std..iter..Iterator$GT$::next::hb4e3e9346a73cb2c ()
> #27 0x000000000129505b in _$LT$std..vec..Vec$LT$T$GT$$GT$::extend_desugared::hbde4df6c20c592f6 ()
> #28 0x000000000129412f in _$LT$std..vec..Vec$LT$T$GT$$u20$as$u20$std..iter..FromIterator$LT$T$GT$$GT$::from_iter::hc2cac91186743725 ()
> #29 0x0000000001293862 in _$LT$std..result..Result$LT$V$C$$u20$E$GT$$u20$as$u20$std..iter..FromIterator$LT$std..result..Result$LT$A$C$$u20$E$GT$$GT$$GT$::from_iter::hcdc19b1783a793ea ()
> #30 0x00000000012936f3 in core::iter::iterator::Iterator::collect::h4a876d4069282243 ()
> #31 0x000000000126ee6e in cargo::core::resolver::Context::build_deps::h99af12331a1d2add ()
> #32 0x00000000012626bd in cargo::core::resolver::activate::h1e69af3f16829146 ()
> #33 0x0000000001259439 in cargo::core::resolver::activate_deps_loop::h0ae3648f1012c7f2 ()
> #34 0x0000000001256a51 in cargo::core::resolver::resolve::h2c90c965d4960b85 ()
> #35 0x000000000134c3f8 in cargo::ops::resolve::resolve_with_previous::h70e55ec8e1e41554 ()
> #36 0x0000000001331207 in cargo::ops::resolve::resolve_ws::hfc62e85c7c05d850 ()
> #37 0x0000000001349c39 in cargo::ops::cargo_compile::resolve_dependencies::hcbb8cdbc75083d0f ()
> #38 0x0000000001346e7a in cargo::ops::cargo_compile::compile_ws::h127ee0c267563315 ()
> #39 0x000000000134542e in cargo::ops::cargo_compile::compile::h14faf6e2d9d049be ()
> #40 0x000000000158e1d3 in cargo::ops::cargo_test::compile_tests::h225dac33512adcf6 ()
> #41 0x000000000158d59c in cargo::ops::cargo_test::run_tests::h0c5f717fdcacdf65 ()
> #42 0x00000000011a0bf0 in cargo::test::execute (options=Options = {...}, config=0x7fffffffe6c8) at src/bin/test.rs:130
> #43 0x000000000119a87a in cargo::call_main_without_stdin<cargo::test::Options,()> (exec=0x119fd30 <cargo::test::execute>, config=0x7fffffffe6c8, usage="...", args=&[collections::string::String](len: 2) = {...}, options_first=true) at src/cargo/lib.rs:71
> #47 0x00000000010cd227 in cargo::execute_main_without_stdin::_$u7b$$u7b$closure$u7d$$u7d$::hf7886269c7efcc62 () at src/libcore/cmp.rs:509
> #48 0x00000000010c8839 in cargo::process::_$u7b$$u7b$closure$u7d$$u7d$::hd21227d3eb2334d2 () at src/libcore/result.rs:29
> #49 0x00000000010c808a in cargo::process<(),closure> (callback=closure = {...}) at src/cargo/lib.rs:79
> #50 0x00000000010c8010 in cargo::execute_main_without_stdin<cargo::Flags,()> (exec=0x10dae70 <cargo::execute>, options_first=true, usage="...",) at src/cargo/lib.rs:57
> #51 0x00000000010c7d5e in cargo::main () at src/bin/cargo.rs:71
> #52 0x0000000001b31429 in std::panicking::try::call::hbbf4746cba890ca7 ()
> #53 0x0000000001b3eccc in __rust_try ()
> #54 0x0000000001b3ec6f in __rust_maybe_catch_panic ()
> #55 0x0000000001b30eaa in std::rt::lang_start::hbcefdc316c2fbd45 ()
> #56 0x00000000010dd0ea in main () at src/bin/cargo.rs:220

[alexcrichton]

We are falling a bit behind on libgit2's latest version at this point, so this could very well be a bug that's been fixed in master. Unfortunately it won't be trivial to update, so it may be difficult to test out

[asomers]

FYI, libgit2 just fixed an issue that may be related: libgit2/libgit2#3931

[alexcrichton]

@asomers I've opened #3174 which updates libgit2 to include that commit you've referenced. Want to give it a spin after it merges?