Skip to content

Feat: Add token counting #47

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

Feat: Add token counting #47

wants to merge 19 commits into from

Conversation

Angelchev
Copy link
Member

@Angelchev Angelchev commented Jan 3, 2024

Closes: #41

Overview

  • Add command to count tokens from a visual selection.
  • Add python script to count tokens with OpenAI's tiktoken.
  • Add utils.vim to launch python scripts more easily.
  • Add automatic venv setup for Neovim with lazy.nvim package manager via build.lua.
    • Create a python virtual environment automatically for Linux or Windows (Untested).
    • Install dependencies from requirements.txt automatically.
  • Add Neovim animated notification integration with nvim-notify.

Todo

  • Add python test for new script
  • Check for tiktoken dependency in Python script, and throw error if not present.
  • Investigate automatically setting up python venv and installing requirements for vim 8.
  • Update README.md to document optionally setting up python venv.
  • Update vimdocs:
    • New command and availability dependant on tiktoken
    • Integration of nvim-notify

Testing

  • Test automatic setup for Neovim with lazy.nvim.
    • Linux
    • Mac OSX
    • Windows

Screenshots

Neovim Animated Notifications via nvim-notify

Screenshot_20240103_233935

Automatic venv setup for Neovim with lazy.nvim

Screenshot_20240103_234057

@Angelchev Angelchev added documentation Improvements or additions to documentation enhancement New feature or request labels Jan 3, 2024
@Angelchev Angelchev self-assigned this Jan 3, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feat: Add tokens counter
1 participant