GGML_USE_METAL=1 fails on iPadOS running iOS 15/16 (and 17 perhaps)

[ronyfadel]

Error is:

ggml_metal_init: load pipeline error: Error Domain=AGXMetalA12 Code=3 "Encountered unlowered function call to air.simd_max.f32" UserInfo={NSLocalizedDescription=Encountered unlowered function call to air.simd_max.f32}

If I remove calls to simd_max, it still fails with:

ggml_metal_init: load pipeline error: Error Domain=AGXMetalA12 Code=3 "Encountered unlowered function call to air.simd_sum.f32" UserInfo={NSLocalizedDescription=Encountered unlowered function call to air.simd_sum.f32}

etc.. i.e. all simd func calls are failing.

[ronyfadel]

@ggerganov what do you think?

This has been mentioned here as well: ggml-org/llama.cpp#2564 (cc @jhen0409)

[jhen0409]

Related ggml-org/llama.cpp#2550

The A-chip models A12Z and before not supported some SIMD operations like simd_max and simd_sum we used in Metal. To fix this, need some works to write new kernels or patch.

I was thinking to do this but not very urgency, and now I think it not worthy to do because it may make affect for future works and neglected maintenance, so I choice to do #1293 for disable Metal in unsupported devices.

[ronyfadel]

@bobqianic how is this completed?

Can you please link the related PR or issue?

[bobqianic]

The A-chip models A12Z and before not supported some SIMD operations like simd_max and simd_sum we used in Metal. To fix this, need some works to write new kernels or patch.

#1293 was merged last week. Therefore, even when using GGML_USE_METAL=1 during compilation, the -ng flag can prevent device failures. It's worth noting that the Metal performance on Apple2 to Apple6 family devices is relatively low. Running on ANE might result in faster transcription speeds. If you prefer to keep this issue open, I'm happy to do that for you.

#1270 (comment)

[sindresorhus]

This limitation is not documented anywhere, so it's impossible to know that it doesn't work on some devices without scouring the issue tracker.

I think Whisper should simply detect the unsupported devices and not activate GPU mode even if instructed to on those devices.

Whisper could use https://developer.apple.com/documentation/metal/mtldevice/3143473-supportsfamily?language=objc to detect the supported GPUs.

[ronyfadel]

@sindresorhus agreed. At the moment it's too easy for this to blow up in the developer's (and the user's) face.

[ggerganov]

Yup, I'll fix this now.

Can somebody remind me which was the earliest apple family that we support?
Should be Apple7 and above, correct?

[bobqianic]

Should be Apple7 and above, correct?

Yes.

[ggerganov]

#1547 should disable GPU seamlessly if Apple7 family is not supported by the device