Pass --depth 1 to git

[domenkozar]

Some git repositories can be really big (i.e. amazonka), does stack need anything else from the git history besides the latest revision? This would save bandwidth and time fetching git dependencies.

[mihaimaruseac]

It seems that we are already doing this?

stack/src/Stack/Upgrade.hs

Line 203 in 7574e33

let args = [ "clone", repo , "stack", "--depth", "1", "--recursive", "--branch", branch]

[mgsloan]

@mihaimaruseac That is the invocation for stack upgrading from its own source repo. I believe that @domenkozar is referring to the invocations for fetching dependencies from git.

IIRC the reason depth wasn't set to 1 was in order to have sharing of git objects when updating the git dependencies - I believe at some point it would try to reuse the existing repo instead of recloning. Looks like now it simply reclones each time, though, so indeed --depth 1 could be used here -

stack/src/Stack/PackageLocation.hs

Line 229 in 2c8cdfb

RepoGit -> cloneAndExtract "git" ["--recursive"] ["--git-dir=.git", "reset", "--hard"]

[mihaimaruseac]

Makes sense then. i'll do a PR asap in that case.

[mihaimaruseac]

Oh, but that file has been dropped from master

[mihaimaruseac]

If I understand

stack/subs/pantry/src/Pantry/Types.hs

Lines 1318 to 1331 in 7574e33

	github = withObjectWarnings "PLArchive:github" $ \o -> do
	GitHubRepo ghRepo <- o ..: "github"
	commit <- o ..: "commit"
	let archiveLocation = ALUrl $ T.concat
	[ "https://github.com/"
	, ghRepo
	, "/archive/"
	, commit
	, ".tar.gz"
	]
	archiveHash <- o ..:? "sha256"
	archiveSize <- o ..:? "size"
	os <- optionalSubdirs o
	pure $ pure $ NE.map (\(archiveSubdir, pm) -> PLIArchive Archive {..} pm) (osToPms os)

correctly then it's now cloning at a specified commit.

Guess I'll just have to trace it when I get some more free time and see how it behaves.

[domenkozar]

That is using the tarball without git, which is a good alternative. But using git could have a similar performance, if --depth 1 was added a few lines above.

[dbaynard]

I believe there was a discussion with @snoyberg and @sol about a new way to proceed here (I think it involved caching somewhere).

[snoyberg]

With the new way we do things with Pantry (checking out the repo and caching it in the SQLite store), skipping the history clone would be great. The relevant code for doing the cloning is here:

stack/subs/pantry/src/Pantry/Repo.hs

Lines 74 to 99 in dbe1572

	let (commandName, resetArgs, submoduleArgs, archiveArgs) =
	case repoType' of
	RepoGit ->
	( "git"
	, ["reset", "--hard", T.unpack commit]
	, Just ["submodule", "update", "--init", "--recursive"]
	, ["archive", "-o", tarball, "HEAD"]
	)
	RepoHg ->
	( "hg"
	, ["update", "-C", T.unpack commit]
	, Nothing
	, ["archive", tarball, "-X", ".hg_archival.txt"]
	)
	let runCommand args = void $ proc commandName args readProcess_
	logInfo $ "Cloning " <> display commit <> " from " <> display url
	runCommand ("clone" : [T.unpack url, suffix])
	-- On Windows 10, an upstream issue with the `git clone` command means that
	-- command clears, but does not then restore, the
	-- ENABLE_VIRTUAL_TERMINAL_PROCESSING flag for native terminals. The
	-- folowing hack re-enables the lost ANSI-capability.
	when osIsWindows $ void $ liftIO $ hSupportsANSIWithoutEmulation stdout
	created <- doesDirectoryExist dir
	unless created $ throwIO $ FailedToCloneRepo repo

The problem is that, to my recollection, some older versions of Git do not support cloning at a specific commit. Therefore, cloning at depth 1 would prevent us from then resetting to the commit we want.

[domenkozar]

I see, it cloning revision even requires special feature on the server: https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

I guess the only thing that can be done then is to make depth an option, so user can hand-pick it.

[dbaynard]

Yes, a server feature, though we can pick sensible defaults?

[snoyberg]

How about a setting in the extra-deps itself saying whether to turn on --depth 1 or not? Then we can bikeshed on whether on or off is the correct default.

[snoyberg]

Moved to wishlist https://github.com/commercialhaskell/stack/wiki/Wishlist