Skip to content

llama : refactor get / set state + remove redundant kv cache API #1143

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 1 commit into from
Apr 24, 2023

Conversation

ggerganov
Copy link
Member

@ggerganov ggerganov commented Apr 23, 2023

  • Normalize the code style
  • Move the definitions at the correct place in llama.cpp
  • Retire llama_get_kv_cache(), llama_get_kv_cache_size() and llama_set_kv_cache()

Not sure how to test this - maybe we need to add an example, or extend main with store/load state functionality

@ggerganov ggerganov added the refactoring Refactoring label Apr 23, 2023
@ggerganov ggerganov requested a review from xaedes April 23, 2023 16:08
Copy link
Collaborator

@xaedes xaedes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good to me, I love the readability improvements. The save_load script from #730 (comment) works as well.

I have converted this to an example with cmake already, will pull request it.

@ggerganov ggerganov merged commit c4fe84f into master Apr 24, 2023
@ggerganov ggerganov deleted the refactor-state branch April 24, 2023 04:40
@ejones
Copy link
Collaborator

ejones commented Apr 25, 2023

If it's helpful, I put up a take on save/load state in main in #1169 (mostly due to my impatience with 65B on the chat-13B prompt, hah).

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
refactoring Refactoring
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants