Huggingface GGUF Editor

[CISC]

The Huggingface GGUF Editor

🎉 Check out my latest project 🌍✨

A powerful editor designed specifically for editing GGUF metadata and downloading the result directly from any Huggingface repository you have access to (you must # for access to gated or private ones).

With its user-friendly design, you can effortlessly edit any GGUF metadata through the GGUF Editor hosted on Huggingface Spaces! 🌍✨🎉

Here are some basic usage examples:

• Missing Pre-Tokenizer: If you notice that there's missing/incorrect metadata like the crucial Pre-Tokenizer name, Fill-in-Middle tokens, or any other metadata based feature, edit/create GGUF metadata in a flash! 📝✨🔗

• Chat Template Editing: Suppose you want to fix an issue with the chat template(s) by updating them directly in the GGUF before downloading it, go ahead! Edit the existing template(s) or even add new ones for RAG, Tools, etc... 🗣️👥

• Downloading Edited Files: Once you've made your edits using the GGUF Editor, download your edited file directly from the Huggingface repository, no need to learn how to install and use complicated scripts! 🚀✨

[CISC]

I have just added support for easily editing individual tokenizer.ggml.scores/token_type entries with quick and easy lookup of every token, making this the most advanced GGUF Editor currently available. :)

This comes in real handy when models are released with incorrect metadata just like the recent Yi-Coder models that had incorrect token_type/score for the <|im_start|> token (as well as incorrect EOS token), there are still GGUFs with wrong metadata available...

[CISC]

Unfortunately there was an oversight that could cause unaligned tensors, this should now be fixed.

If you experienced nonsensical results from an edited GGUF, please grab a new copy, sorry for the inconvenience! :(

[ghchris2021]

Thanks for the FOSS! It looks nice / useful.
I haven't spent a lot of time looking over the details but one thing comes to mind which might help the UX if it's possible so maybe it is something interesting to discuss or consider as a feature.

GGUFs are usually monolithic up to some size then they're usually but not necessarily split into various usually somewhat equally-large size chunks of up to several/many gigabytes each.

I haven't verified how the file encoding / format is but I assume maybe the metadata is right at the beginning of the first file or at least is a small section somewhere in the group of files.

So if you want to make any kind of metadata change you're by definition modifying a tiny tiny fraction of the typical GGUF file / file-set total.

So it seems easy to inspect / modify that tiny piece online in the HF space UI, but the result is (as I interpret what little I know / see so far) having to download at least a large "chunk" of the (if split) GGUF or the entire single file to obtain a copy of the tiny metadata changes.

If that's true then it might be nice to have SOME way(s) to modify the metadata but any combination of :

A: Generate / download instructions or a script to use the standard llama.cpp local-copy GGUF metadata editor CLI to edit / replace the metadata according to the selected change(s). That way one has only a few lines of code / instruction to download and can apply the desired change without downloading gigabytes of unnecessary GGUF data, too.

B: Generate / download some kind of "binary patch" file / script that can (somehow) just take the binary differences between the original GGUF and the modified one and encode those so one can apply them with any standard (there are plenty of standard / common file patching tools e.g. for unix but also cross platform) patching tool or if there's any reasonable use case for it a custom patch script (which I'd avoid in favor of an already sufficient purpose-built standard patch tool). Then one downloads a tiny set of instructions / patch file and has a straightforward way to apply the differences to a GGUF you already have downloaded via some easy / quick means.

C: Maybe it is already possible but if so it is unknown to me and maybe others so perhaps discussing / commenting on the right techniques would help many users. So let's say there is a possible work flow to instead of downloading the modified GGUF it is actually modified inside or saved to one's OWN (or writable) HF repo / space / whatever; maybe that's where you put a copy of the original GGUF to be modified and you basically want to just make a "small delta" to it to create a new GIT version or whatever that is 99.999% the same except for the metadata block. Ok so then to efficiently download THAT from your modified HF repo / space / whatever where you have a modified copy it would be ideal to use an efficient file transfer / synchronization protocol like rsync or such which will only copy the changed portion (approximately) and realize quickly it does not have to download the rest of the unchanged N GBys. That would involve some kind of efficient binary delta (or git LFS commit condent delta...?) aware transfer protocol or software. IDK how "git lfs pull" or whatever actually might be used optionally or defaults to work in terms of efficiently pulling small updates to large binary files but IF there is a git lfs based way it probably bears promotion / discussion since I may not be the only nescient one. And if HF CLI / UI supports some other "scp", "rsync" or other means to actually efficiently copy small updates of binary large files that'd be great to know about also.
So basically a nice post / blog post / FAQ / whatever might be very useful to many if there
are optimum techniques to "fork" / "clone" some upstream GGUF (from some accessible source repo someone has), copy that into one's own repo without necessary download / upload to your local PC, inspect / edit GGUF metadata and save it to one's HF repo or whatever, possibly re-split the file (see below) then download either the modified original file or the deltas relative to the upstream GGUF highly efficiently SOMEHOW from one's own HF or wherever one has the changes available / saved.

D: It's possibly relevant to the editing workflow but also possibly relevant much more widely to the way ggufs are made / published. So if the metadata "section/chunk" is tiny and the rest of the data is huge then it's nice to be able to edit / send / update it efficiently independently. GGUF already gained the ability to support "splitting" with variable amounts of "data" in each "piece/chunk" file. So if that's so then why not GENERATE a gguf that is SPLIT so that for example somefile-00001-of-00099.gguf contains essentially ONLY the metadata (and maybe other highly relevant small header etc. content) and is tiny. Then the OTHER GGUF split files would get the various pieces of the model data which would be large and highly less likely to need to be edited / updated in small granularity.
IDK of any reason why you / anyone could not intentionally generate "metadata-separate" (or similar effect) GGUFs so you can easily download updates to that one and still have it work with NN other chunks of GGUF file split data which themselves do not (typically) change / get edited in small parts.
So if you're providing an easy way for people to "play with" modified metadata in GGUFs then maybe encoding it (GGUFs) into some "metadata update friendly" split format could be user friendly for
these reasons? Of course I could see promoting the idea to the llama.cpp GGUF encoding / processing / medatata editing tools as well as maybe a nice option / default.

E: Of course I guess it's possible to take a HF space's "software" and if it's open source etc. fork / reuse it etc. So maybe your GGUF editor GUI is already able to be derived-from but anyway the thought crosses my mind that if it is using some simple-ish gradio UI on top of some python or whatever script then maybe there is some nice option to reuse 70% of that or whatever and make a very analogous metadata editor UI people can download / use locally to augment / replace the llama.cpp metadata editing / inspection CLI as a UI option. But I assume that'd turn into a github project or maybe llama.cpp pull request or something to extend the metadata / gguf editing / composition UIs in whatever ways. But maybe something to think about if the user friendly UI UX could be enhanced for local use cases also. I'm not personally proposing a fork or project just sharing a concept FWIW.

[CISC]

I haven't verified how the file encoding / format is but I assume maybe the metadata is right at the beginning of the first file or at least is a small section somewhere in the group of files.

Correct.

So if you want to make any kind of metadata change you're by definition modifying a tiny tiny fraction of the typical GGUF file / file-set total.

Also correct. :)

A: Generate / download instructions or a script to use the standard llama.cpp local-copy GGUF metadata editor CLI to edit / replace the metadata according to the selected change(s). That way one has only a few lines of code / instruction to download and can apply the desired change without downloading gigabytes of unnecessary GGUF data, too.

To some extent this would be possible, but unfortunately the scripts are not quite as capable as the GGUF Editor yet (mainly to keep the command-line clutter down).

Additionally it might be worth keeping in mind that the GGUF Editor is primarily meant to save you the trouble of downloading a "broken" GGUF in the first place, allowing you to do changes to a GGUF before downloading it. You can in most instances even create permalinks to download an edited GGUF (however this is a bit of a hidden feature at the moment as it will only work on small changes due to URI limitations).

B: Generate / download some kind of "binary patch" file / script that can (somehow) just take the binary differences between the original GGUF and the modified one and encode those so one can apply them with any standard (there are plenty of standard / common file patching tools e.g. for unix but also cross platform) patching tool or if there's any reasonable use case for it a custom patch script (which I'd avoid in favor of an already sufficient purpose-built standard patch tool). Then one downloads a tiny set of instructions / patch file and has a straightforward way to apply the differences to a GGUF you already have downloaded via some easy / quick means.

Not a bad idea, however I feel this may put a higher technical bar on the process than should be necessary, esp. if going fully cross-platform. Just downloading a file is something anyone can do (esp. if using the permalink feature).

C: [... GIT ...]

Supporting push/clone is on the TODO, however this is currently restricted by storage capacity. Also HF is currently working on improving upload/download/storage, which means we may be able to do some neat tricks in the future.

D: [... GGUF splits ...]

If more uploaders start using the --no-tensor-first-split options (ie, first split file only contains metadata) this would be much less of an issue, then you can just edit the tiny first split and be done. Doing so post-download is not so simple as you would need to modify all the related files. Making a tool that takes a small metadata-only GGUF and adjusts the other GGUF(s) is certainly feasible, but the potential for major screw-ups if not applied correctly by the user is significant (hard to make fool-proof).

E: Of course I guess it's possible to take a HF space's "software" and if it's open source etc. fork / reuse it etc. So maybe your GGUF editor GUI is already able to be derived-from but anyway the thought crosses my mind that if it is using some simple-ish gradio UI on top of some python or whatever script then maybe there is some nice option to reuse 70% of that or whatever and make a very analogous metadata editor UI people can download / use locally to augment / replace the llama.cpp metadata editing / inspection CLI as a UI option. But I assume that'd turn into a github project or maybe llama.cpp pull request or something to extend the metadata / gguf editing / composition UIs in whatever ways. But maybe something to think about if the user friendly UI UX could be enhanced for local use cases also. I'm not personally proposing a fork or project just sharing a concept FWIW.

As it's MIT licensed, anyone is free to do pretty much what they want with it, however the GGUF Editor can be modified to work on local files if run locally, quite easily so if keeping the Gradio UI, I might consider making this a run-time option in the future.

[gogo2464]

Hello. I need a gguf decompiler. Does this tools do what I need please?

If not, where is the reference of gguf file format? I could write my decompiler.

If not existing, I still could read:
https://github.com/ggerganov/llama.cpp/blob/5bbc7362cb93265f4c853fd89800a6255cc26985/ggml/src/gguf.cpp#L705

[mofosyne]

I think you are specifically looking to unpack the individual compressed weights? If I understand there is a different packing algorithm for each type as the read method in GPU is different for each type. You would need to read the code directly as there isn't any real specific documentation on the unpacking method.

Have you checked Tensor Encoding Schemes and examples/gguf/gguf.cpp?

[gogo2464]

yes I am looking for unpacking individual weight.

I have checked the gguf.cpp file but I did not checked https://github.com/ggerganov/llama.cpp/wiki/Tensor-Encoding-Schemes yet.

[gogo2464]

@mofosyne sorry for the delay! I forgot to send my message. Could I mind to first:

• document the unpacking
• code the unpacking
• merge in llamacpp

[gogo2464]

Oh I can just read https://github.com/ggerganov/llama.cpp/wiki/Tensor-Encoding-Schemes#how-are-these-tensors-packed !

Can I open an issue please?

[mofosyne]

If it's documentation, then maybe chuck your issue on improving tensor packing documentation proposal to https://github.com/ggerganov/llama.cpp/issues/new?template=040-refactor.yml

[Dampfinchen]

Hey, would it possible to make a local version of that for already downloaded local GGUFs? I know you can use llama.cpp specific scripts, but this tool would be way more intuitive.

[CISC]

Hey, would it possible to make a local version of that for already downloaded local GGUFs? I know you can use llama.cpp specific scripts, but this tool would be way more intuitive.

You're in luck, I've had this on my TODO a while now, and I finally implemented it today. :)

It allows you to clone the repo, then run app.py with a local path parameter which will let you browse your local GGUFs in that path.

I will release it soon (and post here about it), just waiting for next gradio release which will fix a breaking bug.