Building applications with LLMs through composability, in Kotlin

Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper

Develop DL models using Pytorch and Hugging Face

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

This project shows how to derive the total number of training tokens from a large text dataset from 🤗 datasets with Apache Beam and Dataflow.

Python script for manipulating the existing tokenizer.

Use custom tokenizers in spacy-transformers

Use Huggingface Transformer and Tokenizers as Tensorflow Reusable SavedModels

Small library that provides functions to tokenize a string into an array of words with or without punctuation

[Unofficial] Simple .NET wrapper of HuggingFace Tokenizers library

Package to align tokens from different tokenizations.

Multi-arch bindings for @huggingface/tokenizers.

Megatron-LM/GPT-NeoX compatible Text Encoder with 🤗Transformers AutoTokenizer.

A graphical user interface for the Elasticsearch Analyze API

Visualize some important concepts related to LLM architectures.

A high-performance tokenizer built to rival GPT-4, trained on the C4 dataset.

Byte Pair Encoding (BPE) tokenizer tailored for the Turkish language

Fine tuning pre-trained transformer models in TensorFlow and in PyTorch for question answering

ML Model designed to learn compositional structure of LEGO assemblies