Regression Test Pipeline

[Abhinay1997]

• Add English text normlaization.
• Add WER calculations.
• Compare and check norm outputs from Python implementation
• Add WER to the regression tests once Memory and Latency Regression Tests #99 is merged

[Abhinay1997]

Code is still messy. Needs cleanup once the normalization starts working.

[Abhinay1997]

Running tests locally. Adding more unit tests for the new normalization code.

[atiorh]

@Abhinay1997 As discussed offline, here are the last few items before we can merge this one:

• Removing example test result JSONs
• Removing the Fraction implementation altogether
• Adding AudioEncoder latency measurements to LatencyStats

We discussed the following as nice to haves (could defer to a future PR):

• Writing the following static attributes to the resulting JSON:

"static_attributes": 
- whisperkit_version (string)
- os (string)
- encoder_compute_units (string)
- decoder_compute_units (string)
- enable_word_timestamps (boolean)
- enable_eager_decoding (boolean)
- enable_vad (boolean)
- silence_threshold 
- is_low_power_mode (boolean)
- is_stream_simulated (boolean)

• Periodically (e.g. 1 minute) measuring the following stats in a separate thread and writing the timeseries results to the final JSON:

"system_measurements":
- thermal_state (string)
- device_temperature (int)
- memory_total_available_gb (float)
- memory_total_used_gb (float)
- memory_app_allocated_gb (float)
- memory_app_used_gb (float)
- memory_swap_used_gb (float)
- battery_level (float)
- disk_total_space_gb (float)
- disk_free_space_gb (float)

[Abhinay1997]

@atiorh made the changes except for the AudioEncoder latency stats. Need to add a callback for that. Discussing with Zack on this.

[ZachNagengast]

Added a comprehensive benchmarking automation script to this PR, ready to merge with passing tests 👍 Check out the BENCHMARKS.md for details on how to test locally.

More info:
This is now a fully automated system to run benchmarks on any connected device in parallel that runs full checks on every model for short form and long form audio. It is using fastlane now via the makefile such that running the command make benchmark-devices will start the script and it will run through a list of models and output the results to be processed in an upload_folder. We also are building a hugginface space that will process and display the data in a convenient way, more on that soon.