Refactor tool of creating pretrain dataset

[gongel]

PR types

Function optimization

PR changes

Others

Description

• Refactor tool of creating pretrain dataset, using AutoTokenizer

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 52.92%. Comparing base (7072406) to head (17e2655).
Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9454      +/-   ##
===========================================
+ Coverage    52.82%   52.92%   +0.10%     
===========================================
  Files          677      677              
  Lines       107941   107941              
===========================================
+ Hits         57018    57127     +109     
+ Misses       50923    50814     -109     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests