Any way to change context limit?

[shadowdoggie]

Is there any setting in any of the scripts to change the context limit? :)

Thanks in advance!

[Khalilbz]

Still have the same problem as you, but Here is a simple thing I did, that may help
I have changed the parameter -n to -n 1048 the context is longer by almost ~50% but still not able to generate long text, then changed it to -n 4096 and got almost the same length as -n 1048

Info:
CPU: 4 cores
RAM: 16G
Model: 7B
RAM used during generation is 4.6G

256 Limit (Almost 8 lines)

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.
Here is a list of 100 sentences in the context of IT 1. IT is important is terms of technology 2. IT is important in terms of technology. 3. Information Technology is important in terms of technology. 4. Information Technology is important in terms of technology. 5. Information Technology is important in terms of technology. 6. Information Technology is important in terms of technology. 7. Information Technology is important in terms of technology. 8. Information Technology is important in terms of technology. 9. Information Technology is important in terms of technology. 10. Information Technology is important in terms of technology. 11. Information Technology is important in terms of technology. 12. Information Technology is important in terms of technology. 13. Information Technology is important in terms of technology. 14. Information Technology is important in terms of technology. 15. Information Technology is important in terms of technology. 16. Information Technology is important in terms of technology. 17. Information Technology is important in terms of technology. 18. Information Technology is important in terms of technology. 19. Information Technology is important in terms of technology. 20. Information Technology is important in terms of technology. 21. Information Technology is important in terms of technology. 22. Information Technology is important

main: mem per token = 14434244 bytes
main:     load time =  2390.74 ms
main:   sample time =   156.97 ms
main:  predict time = 58222.04 ms / 205.01 ms per token
main:    total time = 61601.12 ms

1048 Limit (Almost 12 lines)

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.
Here is a list of 100 sentences in the context of IT 1. IT is important is terms of technology 2. IT is very important in terms of technology 3. IT is important in terms of technology. 4. IT is very important in terms of technology 5. IT is important in terms of technology 6. IT is very important in terms of technology 7. IT is important in terms of technology 8. IT is very important in terms of technology 9. IT is important in terms of technology 10. IT is very important in terms of technology 11. IT is important in terms of technology 12. IT is very important in terms of technology 13. IT is important in terms of technology 14. IT is very important in terms of technology 15. IT is important in terms of technology 16. IT is very important in terms of technology 17. IT is important in terms of technology 18. IT is very important in terms of technology 19. IT is important in terms of technology 20. IT is very important in terms of technology 21. IT is important in terms of technology 22. IT is very important in terms of technology 23. IT is important in terms of technology 24. IT is very important in terms of technology 25. IT is important in terms of technology 26. IT is very important in terms of technology 27. IT is important in terms of technology 28. IT is very important in terms of technology 29. IT is important in terms of technology 30. IT is very important in terms of technology 31. IT is important in terms of technology 32. IT is very important in terms of technology 33. IT is important in terms of technology 34. IT is very important in terms of technology 35. IT is important in terms of technology 36. IT is very important in terms of technology 37. IT is important in terms of technology 38. IT is very important in terms of technology 39. IT is important in terms of technology 40. IT is very important in terms of technology 41. IT is important in terms of technology 42. IT is very important in terms of technology 43. IT is important in terms of technology 44. IT is very important in terms

main: mem per token = 14434244 bytes
main:     load time =  2677.93 ms
main:   sample time =   287.36 ms
main:  predict time = 108363.11 ms / 212.06 ms per token
main:    total time = 112125.02 ms

---

I have tried with the Model 13B, it is so slow, uses about 8G of RAM, and output was about 9 Lines with -n 1048, and outputs about 12 Lines with -n 4048

[eous]

#71

[bitRAKE]

@eous Nice work, not a single crash since the patch.