Issue while trying to use

[mbbajra]

Ref : examples/configs/llm/hf_pipeline_llama2

I am getting following error while trying to run
pip install accelerate transformers==4.33.1 sentencepiece --upgrade
.....

 warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:265:21
          |
      265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
          |
          = note: `#[warn(unused_mut)]` on by default
      
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:282:21
          |
      282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
      
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
          |
      200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
          |                                                           ----^^^^^^^
          |                                                           |
          |                                                           help: remove this `mut`
      
      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default
      
      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted
     
...

  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

I heard it might have been because of latest version of rustc so I tried RUSTUP_TOOLCHAIN=1.72.1

However, I discovered that it needs latest version of rustc.

[Pouyanpi]

Hi @mbbajra,

This issue doesn't seem to be related to NeMo Guardrails, but rather to the transformers library.

I attempted to reproduce the error locally but was unable to do so. The issue could be due to various factors, and one immediate check could be following.

Here are the versions I'm using:

Python version: 3.11.8 (main, Mar 18 2024, 13:25:18) [Clang 15.0.0 (clang-1500.3.9.4)]
setuptools version: 65.5.0
pip version: 24.0

You might want to compare these with your own. If your pip or setuptools versions are older, upgrading them might resolve the issue (try it in a virtual env):

pip install --upgrade pip setuptools

If the problem persists, I recommend checking this related issue on the transformers GitHub page: huggingface/transformers#2831