Add (simple) git support

[koppor]

As user, I would like to have a seamless integration in git. For "straight-forward" cases, JabRef should do "magic".

Implementation drivers

• In case of an error, the file should be in the state before JabRef started to do some wizard thing

Opening a library

1. When opening a library (.bib file), JabRef checks whether the .bib file is contained in a git repository. If not in a git repository, no further action is done. Otherwise it is tagged as "versioned" (BibDatabaseContext)
2. JabRef exetues a "git pull" to update the .bib file
3. If error: notification of error, revert file to previous state
4. JabRef opens the updated file

Saving of a library

No action taken (to save number of commits)

Saving a library

1. JabRef saves the .bib file
2. A git commit is created. git commit message "Automatic update via JabRef".
3. git pull --rebase is executed.
4. In case of an error, revert to commit created at step 2 and try git pull
5. In case of an error, revert to commit created at step 2 and ouput a notification
6. Then, git push
7. In case of an error, output error
8. Output "Successfully pushed"

---

Advanced implementation

A user might want to trigger pushing

• Reuse button "synchronize" of shared database. This button has the same functionality as "Saving a library"
• Let user configure frequency of push. At this frequence, on "Saving a library", a commit and the steps of "Saving a library" are done.

[koppor]

At #8431 a user described his issues with current JabRef not having any git support.

[lbenicio]

Hi, my colleague and I are students from the University of São Paulo in Brazil, and we were asked to help in the open-source community. We would like to take a look into this issue and try to implement it. Could we have it assigned to me? Do you have any information to share that would be nice to know beforehand?

[koppor]

Dear Leonardo! Thank you for choosing JabRef! I reserved the issue for you.

Please note that this is a big project.

I shared information for behavior in the issue description. You can start investigating current behavior at SaveDatabaseAction.java.

[koppor]

Here our general newcomer text (which is also information to share)

As a general advice for newcomers: check out Contributing for a start. Also, guidelines for setting up a local workspace is worth having a look at.

Feel free to ask here at GitHub, if you have any issue related questions. If you have questions about how to setup your workspace use JabRef's Gitter chat. Try to open a (draft) pull-request early on, so that people can see you are working on the issue and so that they can see the direction the pull request is heading towards. This way, you will likely receive valuable feedback.

[koppor]

@lbenicio Any updates? Any UML diagrams? Concepts? UI sketches? Initial code?

[lbenicio]

Hi, sorry for the lack of updates.

From now, the idea for the simple implementation seems simple. Here what we have:

1. Verify .git
2. call git sync method
3. issue the commands
4. on failure, open a UI.

We were struggling with the point of external user interaction with the git to understand if it changes any behavior.

Now we are looking into how to open the UI with the changes.

From what I understand, we should present a solution proposal and then follow up with the code, right?

We were scheming the solution, but we did not have a diagram. We are on 2 people working 4h/week/each on the project, so the progress may be a little slow?

But what concerns me is that we did not figure what the complexity of the project yet. It looks like direct implementation.

Should we “reserve” an issue just when we get a proper solution?

[koppor]

Hi, sorry for the lack of updates.

No worries.

From now, the idea for the simple implementation seems simple. Here what we have:

1. Verify .git
2. call git sync method
3. issue the commands
4. on failure, open a UI.

Sounds OK. No need for a real UI. We have "dialogService". Example:

    dialogService.notify(Localization.lang("Issue on GitHub successfully reported."));

We were struggling with the point of external user interaction with the git to understand if it changes any behavior.

I don't know, which of these steps outlined above are unclear:

1. JabRef saves the .bib file
2. A git commit is created. git commit message "Automatic update via JabRef".
3. git pull --rebase is executed.
4. In case of an error, revert to commit created at step 2 and try git pull
5. In case of an error, revert to commit created at step 2 and ouput a notification
6. Then, git push
7. In case of an error, output error
8. Output "Successfully pushed"

From what I understand, we should present a solution proposal and then follow up with the code, right?

In Software Engineering, one uses diagrams etc to communicate. UML sequence diagrams are a good thing to learn. -- You can also write code and submit a work-in-progress pull request to JabRef. To discuss things.

We were scheming the solution, but we did not have a diagram. We are on 2 people working 4h/week/each on the project, so the progress may be a little slow?

I don't know about your final date. Slow progress is very OK. Better 4h "deep work" hours than 8 non-focussed ones. I would suggest that you come back with intermediate results after each working block. We can open a separate skype group for you if you want.

I would also like to share the thought that this is a work educating you to become a better programmer and a better software architect. One needs 10.000 hours to become an expert. Thus, each hour invested here brings you closer to the goal becoming an expert.

In October, there is #hacktoberfest. I would recommend to use that opportunity to level-up your coding skills even more.

But what concerns me is that we did not figure what the complexity of the project yet. It looks like direct implementation.

The project is flagged as "large", because it is not straight-forward code. One needs to understand the current saving code and where to plug-in. It could be that the final "productive" code is 50 lines of code. Nevertheless, there will be 500 to 1000 lines of code for testing (JUnit tests, ...).

Should we “reserve” an issue just when we get a proper solution?

It is very OK to reserve an issue. We take back the reservation if the solution is implemented :) or the student resigns. We have enough other issues where other persons can work on.

[lbenicio]

Great, thank you the help and very detailed attention! Much appreciated!

In Software Engineering, one uses diagrams etc to communicate. UML sequence diagrams are a good thing to learn. -- You can also write code and submit a work-in-progress pull request to JabRef. To discuss things.

We will try to submit a work in progress solution/diagram to exemplify the solution as fast as possible.

The project is flagged as "large", because it is not straight-forward code. One needs to understand the current saving code and where to plug-in. It could be that the final "productive" code is 50 lines of code. Nevertheless, there will be 500 to 1000 lines of code for testing (JUnit tests, ...).

perfect, I believe we are on the right track.

I will take the opportunity to introduce my partner @marcelojunior1

[marcelojunior1]

Hi @koppor , I'm Marcelo and Leonardo's partner in our class. Thank you for your support and we hope to be able to contribute to JabRef.

[koppor]

@marcelojunior1 also welcome on bord!

I think, the place in the code is /src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java#L257

The are already test cases for SaveDatabaseAction. You can add git-based tests.

For another project, I needed to create a git repository within JUnit tests. Maybe, the code is an inspiration for your tests, too? -- https://github.com/koppor/heylogs/blob/34c84ebb101b310bd0d715c669ab38e6d0300e65/heylogs-api/src/test/java/internal/heylogs/GitDiffRuleTest.java#L120

[lbenicio]

Great! The help is much appreciated, thank you!

We intended to update the git handler class and use it to manage this kind of git repositories as well.

About our progress: we are just finishing our solution proposal

Yes, the tests are really the complexity here, and this will help a lot.

[lbenicio]

After a long wait, here is our solution proposal:

We should start writing unit tests next week.

Maybe we add our code to an “intermediate” pull request, so you can track our progress

[koppor]

Thank you for sharing the UML flow chart.

I have following comments:

• Name the .bib file "bib file" - "bibbi" seems something non-English.
• I don't see how "commit bibbi file" can work without any saving (I think, the file should be saved before any interaction with a git remote. Thus, "saveDatabase()" should be called, and then the git "magic" handled
• My proposal with "git rebase" was an advanced thing. Maybe drop that completely and just to a "git pull" with merge strategy.
• In case there is merge conflict at pulling, reset to the local version - and just output a warning error that the user needs to resolve the conflicts for themselves.

Code hint: On org.jabref.logic.git.GitHandler you see some git magic. I think, that class cannot be used directly: It creates a git repository on start. Either you modify it - or your start with your own class. Your own class could limit the the life cycle of the GitRepository when doing a save - and then throw away that handle (try-with-resources).

Notes

1. you need to suspend the file change listener. Otherwise, JabRef will start nagging that the file changed on disk (during your git operations)
2. you need to make JabRef aware of the remote changes: The file needs to be loaded again in JabRef. Search for the usage of org.jabref.gui.collab.DatabaseChangesResolverDialog.

---

Advanced: The SLR functionality of JabRef (included in org.jabref.logic.git.GitHandler) has support to handle patches (SlrGitHandler.java#L95). I think, this cannot be directly used, but I have not thought about it.

The core idea should be to find the modified entries by the citation key (AKA BibTeX key) locally and remotly. And then displaying a merge entries dialog if the bibtex key is remote and locally the same, but different content. -- You can call the org.jabref.gui.collab.DatabaseChangesResolverDialog for that.

One could even reuse that dialog:

Put in a flow:

• save the library
• pull changes
• conflicts exist: Revert changes
• start file-changed-on-disk listener
• put state of remote on disk
• JabRef will trigger the DatabaseChangesResolverDialog

[koppor]

Screenshot of the Merge Entries dialog (as of today)

[lbenicio]

Wow, great, thanks!

I don't see how "commit bibbi file" can work without any saving (I think, the file should be saved before any interaction with a git remote. Thus, "saveDatabase()" should be called, and then the git "magic" handled

Totally forgot that. You are correct. We should wait for file save before calling for git save.

you need to make JabRef aware of the remote changes: The file needs to be loaded again in JabRef. Search for the usage of org.jabref.gui.collab.DatabaseChangesResolverDialog.

Forgot that too. Totally makes sense to update JabRef internal database with the updated changes from git pull.

I will update our flow chart to reflect the changes and post here.

[koppor]

Thank you! - As a side comment: Please ensure that gitk is running on your machine. Then you can view the state of the repository easily. Tutorial at: https://lostechies.com/joshuaflanagan/2010/09/03/use-gitk-to-understand-git/.

[lbenicio]

So, this is the final idea. The user should be on a loop to fix conflicts until the user cancels the action, and we summarize to the normal flow and finish the git save method.

From this, we believe we should have an implementation and test strategy (which would be bigger then the code itself)

[marcelojunior1]

Sorry, I accidentally remove my assignment. :(

[koppor]

Regarding the diagram. I thought, it was an UML flow chart, now you are using another syntax. We computer scientists are picky on the syntax of things. Please choose an existing thing (and do not re-invent something else). Example: Casing OUTPUT and output. I think, the extra casing is not necessary. -- Round circles versus boxes versus boxes with rounded corners versus a yellow box. -- Stick with plain rectangles please.

I put the comments straight, I hope, this is OK

• Do not put method names inside the flow chart. On that level, you are not sure which method your code will be in. Moreover, saveGitDatabase() is a method, you will implement. The flowchart suggests that some git checking happens afterwards. This is strange. Just remove saveGitDatabase()
• Flow charts do not have the parallel semantics. Thus, either change the diagram syntax or think whether there is something wrong. I would just continue at "revert to the local version". The dialog itself tells the user that something was wrong. Two different things happening confuse users
• There is no "stop file-changed-ondisk listener" in the diagram.
• No arrow from "JabREf will trigger ..." to "merge conflicts". The flow ends here
• No arrow from "output successful message" to "save database". Because: The flow is started by the user triggering the save. It is NOT an endless loop.

[lbenicio]

Ok, great! Yes, we decided not to use a proper diagram format, it was supposed to a sketch sorry, it was not a good idea.

We got some progress, and managed to create the commit, we should submit a “preview” PR, so you can track our progress soon.

[koppor]

Ok, great! Yes, we decided not to use a proper diagram format, it was supposed to a sketch sorry, it was not a good idea.

10.000 hours to become an expert. Drawing one diagram and getting feedback makes you a better diagrammer. Diagrams are useful for software developers and architects. Keep practicing!

Note that drawing and discussing a diagram takes shorter time than implementing everything. This is the main benefit of diagrams.

We got some progress, and managed to create the commit, we should submit a “preview” PR, so you can track our progress soon.

Nice. Looking forward!

[marcelojunior1]

Note that drawing and discussing a diagram takes shorter time than implementing everything. This is the main benefit of diagrams.

Yes, we agree! We were trying to do push/commit operations using ssh, but it was very complicated to import and configure org.eclipse.jgit.ssh.apache package. Now it's working and we can think more about the diagram.

[lbenicio]

We just submitted a pull request, so everybody can track our progress and provide suggestions.

#10422

[lbenicio]

Hi @koppor

Writing to update you about our progress. We almost finished the implementation and are moving to tests. We got the preferences' menu working and the git ssh, and user/pass login working as well as the login retrieval from env variables and preferences menu.

Next sprint, we are moving to unit tests (if nothing unexpected appears)

Our ETA would be 4 weeks to finish everything. I hope we can make it (since our semester would end).

[koppor]

Thank you for the update. It would be nice if you reopened the PR or commented if the time budget is spend on your side. No need to implement GPG signing. Just get the "basic" functionality running properly. Since this touches a core of JabRef, clean and working code is more important than features.

Especially, the user experience has to be great ^^. One can check how other tools do it. At https://www.reddit.com/r/LaTeX/comments/1179mdx/texstudio_git_integration_for_easy_committing/ there is a good explanation for TeXStuido.

Side note: I would have relied on https only, but OK for me that you worked on SSH, too. 😅

[koppor]

For requirements refinement, please read on related work, e.g., TeXStudio git integration: https://www.reddit.com/j9az1u9?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=2

[felicitasmaia]

We are planning to work on this project as part of a university lecture. Therefore, we would appreciate an update on its progress and any relevant information or tasks we could contribute to. Thanks!

[koppor]

We are planning to work on this project as part of a university lecture. Therefore, we would appreciate an update on its progress and any relevant information or tasks we could contribute to. Thanks!

@felicitasmaia Nice to hear. The contributors stopped working on it. Maybe, you start from scratch? Alternatively, you can read into the diff at https://github.com/JabRef/jabref/pull/10586/files. -- My gut feeling is, however, that the whole SSH handling is a bit too much. Our SLR feature also worked without that IMHO.

[priyanshu16095]

Is this free to take?
/assign-me

[github-actions]

👋 Hey @priyanshu16095, thank you for your interest in this issue! 🎉

We're excited to have you on board. Start by exploring our Contributing guidelines, and don't forget to check out our workspace setup guidelines to get started smoothly.

In case you encounter failing tests during development, please check our developer FAQs!

Having any questions or issues? Feel free to ask here on GitHub. Need help setting up your local workspace? Join the conversation on JabRef's Gitter chat. And don't hesitate to open a (draft) pull request early on to show the direction it is heading towards. This way, you will receive valuable feedback.

Happy coding! 🚀

⏳ Please note, you will be automatically unassigned if the issue isn't closed within 90 days (by 04 May 2025). A maintainer can also add the "📌 Pinned"" label to prevent automatic unassignment.

[priyanshu16095]

/unassign-me

[github-actions]

👋 Hey @priyanshu16095, you've been automatically unassigned from this issue due to inactivity.

Note

If you'd like to be re-assigned, just leave another comment or ask a maintainer to assign you again.
If you're still actively working on the issue, let us know by commenting, and we can pin it to prevent automatic unassignment.

[SihasA]

Hi, we would like to propose the following to solve this solution:

1. When a library is opened check whether it is being version controlled via functions in GitHandler.java provided in JabRef
2. Syncronise with remote repo as per preferences- automaticly/ manually. Current sync state of version controlled files are to be displayed appropriately.

3. Add a tab to preferences that handles "git preferences" managing optional auto push and sync. The exact preferences are to be further discussed and streamlined.

4. Introduce a merge conflict resolution mode that introduces a GitHub style resolver with a side by side of the file, showcasing additions and deletions.

5. We were also thinking of allowing users to access git pull/ commit/push commands and functionality by introducing either an individual field or a git tab under files.

Please let us know any thoughts on the following ideas and if this solves the proposed issue well enough.

[koppor]

Thank you for the initial proposal. I very much miss options to choose from.

Did you hear about the "Shared SQL Database" feature? --> https://docs.jabref.org/collaborative-work/sqldatabase

While reading your pseudo-flow-chart, I thought, integration with a git remote repo is something very similar. The "only" difference is that the library is saved locally, too.

• Option A is now to implement a "Shared SQL Database" with "git" read as "Shared SQL Database" and add synchronization to a loal file.
• Option B is to handle the synchroniaztion differently (as you described), but use the ideas of the "Shared SQL Database" feature. There, the thing of concurrent modifications is also thought of
• Option C is to implement everything new and ignore the "Shared SQL Database" knowledge at all.

I currently think, your proposal is doing Option C. I would ask to consider option B, please.

---

1. When a library is opened check whether it is being version controlled via functions in GitHandler.java provided in JabRef

Seems to be OK... It should be a property in org.jabref.model.database.BibDatabaseContext. Similar to bibDatabaseContext.getDatabasePath();. See handling in org.jabref.gui.LibraryTab#updateTabTitle for unsaved libraries.

2. Syncronise with remote repo as per preferences- automaticly/ manually. Current sync state of version controlled files are to be displayed appropriately.

Yes

No. Use the tab title. See handling for unsaved libraries:

JabRef stores in the tab title information on saved an unsaved. This is a property of the "state". The git sync state is also such a state. Thus, it should also be in the title.

You can use some unicode characters to display the state - to make the title short.

Proposal: First work on "Sync on save" - and then think of manual sync. - Otherwise, there is too much to think of.

NO SVN SUPPORT - git only (!)

There should be no "notification" in the sense of org.jabref.logic.util.NotificationService#notify. Maybe rework this using the tab title and the ida of "sync-on-save"

3. Add a tab to preferences that handles "git preferences" managing optional auto push and sync. The exact preferences are to be further discussed and streamlined.

Add it to "General" as there is already a section "Saving"

You can also move "Saving" to a new tab "Saving" in the preference and add a sub section "Git" on that.

Please, do not mix up "commit" and "push". Please read some basic git tutorials to get the difference between "commit" and "push". Also learn about the staging area of git and the difference between "local" and "remote" repository. -- These terms should be used.

Use "JabRef auto check-in" - it is based on A) what TeXStudio uses and B) how git writes "check-in":

4. Introduce a merge conflict resolution mode that introduces a GitHub style resolver with a side by side of the file, showcasing additions and deletions.

Here, two concepts are mixed:

1. Library diff
2. Conflict resolution

I very much like the concept of "Library diff". Very neat. This is a "viewer" for org.jabref.logic.bibtex.comparator.BibDatabaseDiff. This can be done as sub project (somehow) independent of the git support. -- The viewer could be extended with the functionality to pick and choose. Keep addition, keep deletion etc. However, the tool should merge as much as possible. Exmaple: I add entry number 999. The other user modifies entry 555. Then, the update of 555 should be in without any intervention of my side. Only if I modify 555 too, then there should be a dialog asking me which data I want to use. This should be done using the [Merge Entries Dialog](https://docs.jabref.org/finding-sorting-and-cleaning-entries/mergeentries dialog).

However, as first step, please use the existing functionality org.jabref.gui.collab.DatabaseChangesResolverDialog.

Add test cases on BibTeX for things that can happen. E.g.,

• two parties modify the same BibEntry,
• one party adds an entry, another one removes it, but they are very far away and the can be merged without conflict.

5. We were also thinking of allowing users to access git pull/ commit/push commands and functionality by introducing either an individual field or a git tab under files.

This is OK. For the manual mode (see above). -- Should be 3 to hours of work.

Please think more in the BibTeX domain. How to uniquly identify a BibEntry, different kind of "merge conflicts". org.jabref.logic.bibtex.comparator.BibDatabaseDiff#compareEntries is already a good start. See also the test cases.

Please let us know any thoughts on the following ideas and if this solves the proposed issue well enough.

This is a good start, but much more work ahead. Maybe, you can split up and have persons working on test cases.

I did not see any back reference to comments in this issue. However, I think, the really important was to pointer to the existing database conflict resolution dialog.

Please note that there was an existing try at #10586. That was OKish - However, do not force the user to configure the git repository. You can assume an existing one and a proper git configuration (username + email). -- I just remembered the PR, because it had much code because of SSH - and it could be that this is necessary, but I am not sure.

As advanced step, you can allow for creation of a local git repository and configuration of the remote. However, this could be too much effort (also w.rt. maintenance).

---

Please also think about test cases: Automatic interacton with another git repository. I assume, it has to be made available as sub module in JabRef's code?

Code hints (for another project): https://github.com/koppor/heylogs/blob/34c84ebb101b310bd0d715c669ab38e6d0300e65/heylogs-api/src/test/java/internal/heylogs/GitDiffRuleTest.java#L120

---

Related Work:

TeXStudio

Configuration:

The two yellow options should be offered by JabRef, too.

The "to undo before last saved operation" is very interesting.

This means: As user, I have git in place. I close JabRef. I open JabRef. I press "Undo". Normally, there would be no Undo information. However, with "git", there is undo possible.

Manual possibilities:

[koppor]

One can experience the current file-based conflict handling as follows:

1. Open "Chocolate.bib" in JabRef. (The file is located in src\test\resources\testbib).
2. Open "Chocolate.bib" in "Notepad++"
3. Delete the BibEntry "Keen_2001".
4. See in JabRef
   
5. Click on "Review Changes" opens a dialog showing kind of a diff between the local file and the in-memory state in JabRef:
   

[koppor]

Notes on discussion:

• On push: If it does not work: pull. If these are conflicts: DO NOT force the user to resolve the conflicts. Let the user postpone the merge conflict resolution (and let him continue to work)
• "git commit --ammend" vs. "git commit" (maybe in the case non-auto-sync to auto-sync) -- this is simlar to choose MariaDB or Postgres as remote SQL database (the first one uses manual sync - and the last one uses auto sync)
• Show branch in status (tab)
• (some more internal notes exist)

[SihasA]

Refining our requirements we have agreed on the following four working packages

1. Detect whether the library is under git version control and pull from repo
2. Show the state of synchronization
3. User preferences of auto sync functionality ( Happy Days Git integration - write only, no merge conflict integration)
4. Merge Conflict Resolution (to be integrated into 3)

We will follow the suggested guidelines about working packages 1 & 2, such as showcasing the sync state with the file name, using Unicode characters.

For Working Package 3:

• We shall go through org.jabref.model.database.BibDatabaseContext to execute on-save functionality.
• With regards to implementing preferences- we shall proceed with adding preferences to the current 'Saving' under the "General" tab as suggested. If the segment gets cluttered we shall opt for a new "Saving" tab.

Working Package 4:

• We shall make a better formulated and more thorough proposal for you in two weeks and see if this adheres to the requirements of the problem and the standards of JabRef.

We aim to have a draft PR for working package 1 by Friday.

[LinusDietz]

Thanks! I think if you work in parallel, you should be able to tackle WP1 and WP2 until Friday.

[SihasA]

Alright perfect! Will start on both ASAP

[koppor]

WP4 is the largest one.

1. Smooth merge of non-conflicting changes in JabRef
2. Smooth user interaction in case conflicting changes in JabRef

If an update contains both non-conflicting and conflicting updates, only the conflicting changes need to be present to the user. the non-conflicting ones should just be merged. Weaving this into JabRef is the challenge of this project. Everything else is just the preparation of this.