git lfs support

[Xarlot]

Does libgit2sharp support working with repo that contains github lfs binding. If so, where i can find any info how to use it?

[shiftkey]

Git LFS is implemented via smudge/clean filters - it's not supported out-of-the-box but is possible to do.

[Xarlot]

Thanks for answer.

It seems that GitHub client works that way. It would be nice to have the support of the git lfs out of the box because the approach with smudge/clean filters looks very complex.

[whoisj]

It seems that GitHub client works that way. It would be nice to have the support of the git lfs out of the box because the approach with smudge/clean filters looks very complex.

I pushed a change today which should make it easier, but not "out of the box".

[ejsmith]

Any progress on this? Would love to have this supported.

[whoisj]

Any progress on this? Would love to have this supported.

It is rather trivial to support if Git+Bash are on the box. Remember that Git's filters and hooks are both assumed to be Borne Again Shell Scripts (BASH), so there's no real way around having Bash on the box. Assuming it's a Linux box, you can be assured it is there.

[zezba9000]

What can I do to get this working?
Anyone know of a workaround or what code needs to be changed to make this work?
I started making a GUI tool thinking this would just work....

[whoisj]

What can I do to get this working?

Just register your callback handler with GlobalSettings (https://github.com/libgit2/libgit2sharp/blob/master/LibGit2Sharp/GlobalSettings.cs#L208).

Then use that callback to send the appropriate commands to Bash.

[zezba9000]

Thanks but what git-lfs methods do I invoke on Clean, Complete, Create, etc?
They're being invoked but I'm not sure how this correlates to "git-lfs add" etc?
The docs in this area on the libgit2 site don't have any use cases...

[zezba9000]

So far I'm doing stuff like this without any luck:

using (var process = new Process())
            {
                process.StartInfo.FileName = "cmd";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.CreateNoWindow = false;
                process.StartInfo.UseShellExecute = false;
                process.Start();

                process.StandardInput.WriteLine("git-lfs clean \"" + path + "\"");
                process.StandardInput.Flush();

                process.StandardInput.Close();
                process.WaitForExit();
            }

What commands are "appropriate" and do I still need to call the base virtual method "base.Clean(...)"?

[zezba9000]

So I have it working on small files where "Clean" only needs to be called once.
If it needs to be called multiple times "git-lfs status" doesn't show the file as staged.

Here is the code (any ideas?)

protected override void Clean(string path, string root, Stream input, Stream output)
        {
            //base.Clean(path, root, input, output);

            using (var process = new Process())
            {
                process.StartInfo.FileName = "git-lfs";
                process.StartInfo.Arguments = "clean";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.RedirectStandardOutput = true;
                process.StartInfo.CreateNoWindow = true;
                process.StartInfo.UseShellExecute = false;
                process.Start();

                input.CopyTo(process.StandardInput.BaseStream);
                input.Flush();
                process.StandardInput.Flush();
                process.StandardInput.Close();
                input.Close();

                process.WaitForExit();

                process.StandardOutput.BaseStream.CopyTo(output);
                process.StandardOutput.BaseStream.Flush();
                output.Flush();
                output.Close();
                process.StandardOutput.Close();
            }
        }

        protected override void Complete(string path, string root, Stream output)
        {
            output.Close();
            //base.Complete(path, root, output);
        }

[whoisj]

More likely what you should be doing is passing the clean of filter text to Git\bin\sh.exe and sending the stream to its stdin while collecting the results from stdout.

[zezba9000]

Ok have it commiting to lfs correctly via the code below.

However when I call "var blob = RepoUserControl.repo.ObjectDatabase.CreateBlob(item.FilePath);" its invoking "clean" which corrupts git-lfs as this is just used for viewing a file not commiting.

public class LFSFilter : Filter
    {
        private Process process;

        public LFSFilter(string name, IEnumerable<FilterAttributeEntry> attributes) : base(name, attributes)
        {
        }

        protected override void Clean(string path, string root, Stream input, Stream output)
        {
            try
            {
                // write all file data to stdin
                input.CopyTo(process.StandardInput.BaseStream);
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Complete(string path, string root, Stream output)
        {
            try
            {
                // finalize stdin and wait for git-lfs to finish
                process.StandardInput.Flush();
                process.StandardInput.Close();
                process.WaitForExit();

                // write git-lfs pointer for 'clean' to git or file data for 'smudge' to working copy
                process.StandardOutput.BaseStream.CopyTo(output);
                output.Flush();
                output.Close();

                process.Dispose();
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Create(string path, string root, FilterMode mode)
        {
            try
            {
                // launch git-lfs
                process = new Process();
                process.StartInfo.FileName = "git-lfs";
                process.StartInfo.Arguments = mode == FilterMode.Clean ? "clean" : "smudge";
                process.StartInfo.WorkingDirectory = RepoUserControl.repoPath;
                process.StartInfo.RedirectStandardInput = true;
                process.StartInfo.RedirectStandardOutput = true;
                process.StartInfo.CreateNoWindow = true;
                process.StartInfo.UseShellExecute = false;
                process.Start();
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }

        protected override void Initialize()
        {
            base.Initialize();
        }

        protected override void Smudge(string path, string root, Stream input, Stream output)
        {
            try
            {
                // write git-lfs pointer to stdin
                input.CopyTo(process.StandardInput.BaseStream);
            }
            catch (Exception e)
            {
                MessageBox.Show(e.Message);
            }
        }
    }

[zezba9000]

Here is my filter so far. It works in most cases (checkouts, merges, etc) BUT for some reason when cloning a repo some objects get there file data replaced with lfs pointers: https://github.com/zezba9000/Git-Game-GUI/blob/master/GitGameGUI/RepoUserControl.xaml.cs#L22

Why or why didn't git just expose Git-Core as a lib people could interface with... major design flaw with git.

[ethomson]

Was the filter invoked for those objects? If so, was git-lfs called for those objects? If so, did it succeed or fail? What was its output?

[zezba9000]

Yes its called for all of my objects under lfs. I get no exceptions or errors that I can see.
From what I can tell so far is maybe "Smudge" is being called as a "Clean" as the Output string is the working file and so I'm writing the pointer information to what should be file data.

So in short maybe "mode = Clean" when it should be "mode = Smudge". Maybe this is totally off, I'm very confused as to why this is happening.

One more important note! when I call "git-lfs smudge" and finish with "process.StandardInput.Close();" it doesn't trigger git-lfs to quit like it does with when I call "git-lfs clean" is called.

This is being testing with GitLabs running on Ubuntu 16.04.

[zezba9000]

Another note is this is being tested with 3 jpg images at around 300kb.
Some resolve fine in a clone while one doesn't (and has its pointer as its data). So I feel like i'm doing something a little wrong.

[zezba9000]

Ok so lfs clean and smudge seem to be working.
But using "RepoUserControl.repo.Network.Push" isn't pushing lfs files (this has been my cause of confusion).

Using the bash on windows pushes just fine BUT when using libgit2sharp to push it pushes the lfs pointer and NOT the binary data. Thus corrupting the data by changing it to its text ptr on the remote server and in turn cloning will pull that corrupted lfs file. (or maybe its just smudge fails as the pointer doesn't reference any data as it didn't get pushed).

Are there other hooks or something? If the bash "git push" is hooking "git-lfs push" then what do we need to do to get this working in libgit2? I think this is the last thing to get lfs working in libgit2sharp all my other tests seem to be working so far.

Example:
https://github.com/zezba9000/Git-Game-GUI/blob/master/GitGameGUI/ChangesUserControl.xaml.cs#L405

[ethomson]

But using "RepoUserControl.repo.Network.Push" isn't pushing lfs files (this has been my cause of confusion).

Nor should it.

If the bash "git push" is hooking "git-lfs push" then what do we need to do to get this working in libgit2?

You need to either:

a) Execute git-lfs push when you do a push, or better:
b) Run the pre-push hook before you do the push.

[zezba9000]

Nor should it.

Ya my bad, forgot to update that comment.

You need to either:

a) Execute git-lfs push when you do a push, or better:
b) Run the pre-push hook before you do the push.

Ok thanks, i'll take a look later today.
That would be in PushOptions correct?

[ethomson]

You need to either:

a) Execute git-lfs push when you do a push, or better:
b) Run the pre-push hook before you do the push.

Ok thanks, i'll take a look later today.
That would be in PushOptions correct?

No - LibGit2Sharp will not execute this for you. You'll likely need to run it in the context of bash (probably the bash that is included with Git for Windows):

Process.Start(@"C:\Program Files\Git\bin\bash.exe", myRepo.Info.Path + @"\hooks\pre-push");

[zezba9000]

@ethomson Hey so after I get "git lfs pre-push" working. I see there is also "git lfs pointer". Do you know what gitlib2sharp situation I would need to use this?

I want to make sure iv'e handled all "Low level commands" listed in git-lfs to feel confident using it in my GUI at work etc.

[ethomson]

No - I'm afraid that I do not know what that does.

@technoweenie is this a command that is needed for normal usage?

[zezba9000]

Cool so pushing now works via this method: https://github.com/zezba9000/Git-Game-GUI/blob/master/GitGameGUI/ChangesUserControl.xaml.cs#L409

Just hope i'm not missing a condition with "git lfs pointer".

[technoweenie]

Nah, it's more of a utility command: https://github.com/github/git-lfs/blob/master/docs/man/git-lfs-pointer.1.ronn

[zezba9000]

Awesome tnx!

[ethomson]

Great! Thanks @technoweenie, and @whoisj for the info! Closing this.

[tedlofgren]

One more important note! when I call "git-lfs smudge" and finish with "process.StandardInput.Close();" it doesn't trigger git-lfs to quit like it does with when I call "git-lfs clean" is called.

I got smudge to exit nicely just as clean do.

It seems that you have to consume the StandardOutput before waiting to exit.

Like so

protected override void Complete(string path, string root, Stream output)
    ...
    process.StandardInput.Flush();
    process.StandardInput.Close();
    process.StandardOutput.BaseStream.CopyTo(output);
    process.WaitForExit();

Thank you @zezba9000 - this made supporting lfs in my tool real easy.

[orrindeng]

Just remind someone else who is using this way to support lfs.

When you use git-lfs pre-push origin to push the objects to pushed, please check the source/destination of the commit sha.

In my case, the Destination is my local sha, and the Source is the remote one. That takes my whole day. :(

And you need to write the local at the first, with remote followed. See https://github.com/git-lfs/git-lfs/blob/main/docs/man/git-lfs-pre-push.adoc

LibGit2Sharp Version: 0.26.2

[dedmen]

Thanks for the snippet @zezba9000 It does work nicely, but it is very slow due to having to launch a new Process for every file.
It is much more efficient to use the git lfs filter-process.
I posted an example here https://gist.github.com/dedmen/ab740ad9ebfde0403e8223480bef91ae

It only needs to launch one process (per repository) and processes all files with it. That runs much faster, especially after doing a pull where it goes over all the files with a quick clean.

[zezba9000]

@dedmen This is much newer code base FYI: https://github.com/reignstudios/Git-It-GUI
Now with Github Desktop though there has been less of reason to continue the project.

[lucianm]

Thanks for the snippet @zezba9000 It does work nicely, but it is very slow due to having to launch a new Process for every file. It is much more efficient to use the git lfs filter-process. I posted an example here https://gist.github.com/dedmen/ab740ad9ebfde0403e8223480bef91ae

It only needs to launch one process (per repository) and processes all files with it. That runs much faster, especially after doing a pull where it goes over all the files with a quick clean.

Hi @dedmen, thank you so much, this was exactly what I was looking for, I managed to integrate (maybe as of now, still a bit hackish) your Gist in git-tfs, which unfortunately lacks git-LFS (or I think, any kind of filtering) awareness. I tried it out after setting up a TFS repository containing also binaries which in the git repo cloned from it, are tracked by LFS (via the .gitattributes) and your LFSFilter is nicely running "Clean" on tracked binaries, before libgit2sahrp which is used by git-tfs creates a git commit.

Unfortunately, it's another story with binaries commited in git which git-tfs should then send to TFS. Running the command which will do that would checkout the commit to be transferred to TFS into a temporary "TFS workspace", which is just a folder containing the changes from the commit, but not a diff, but as their actual content, to let TFS know about it and do its own "diff magic" if applicable. Here I would have expected the LFSFilter to "smudge", but it actually "cleaned", therefore git LFS pointer files have been checked in to TFS, instead of the actual blobs.
Might there be still something incomplete with the implemetation from the gist, or do I have to further investigate what git-tfs foes in this scenario?
Maybe I would need to add more console traces and turn on some environment variables for debugging libgit2(sharp) to understand what'S going on...

[dedmen]

I'm running my code in a git-SVN sync tool.
I'm pretty sure my checkout works fine, but I also do a SVN checkout right after which would overwrite the files if they were bad.
So I can't confidently say that my checkout is working.
Are you calling the PostCheckout() method from my snippet? It's probably not easy to see how to call it. I'll come back to this on Monday and post a sample on how to use that.

The filter code, just does what it gets told to do by libgit2. So there shouldn't be an issue in the filter itself. But maybe the post checkout is needed.

[lucianm]

I guess you're right, I traced all method calls of your filter and only Initialize, Create, Clean and Complete are being called in general, when I fetch / pull data from TFS, also it's the only occasion I've seen Smudge being called, I would have expected to see it when checking in to TFS (so in the other direction). None of PrePush/PostCheckout/PostCommit have been called, I too suspected that could be the issue, you seem to confirm by emphasizing PostCheckout needs to be called...

[lucianm]

I figured it out how to make it work when checking in to TFS as well, it was just changing one line in the git-tfs code. I will fork git-tfs and publish my changes, which I would eventually PR upstream, if it's ok with you to also incorporate your Gist, @dedmen ?

[dedmen]

I didn't realize I forgot to put a license onto it.
Yeah do whatever with it!