[FEATURE] Utilize Apple Neural Engine

[rbpasker]

I'm wondering if there is a plan to deploy on ANE

https://machinelearning.apple.com/research/neural-engine-transformers

This year at WWDC 2022, Apple is making available an open-source reference PyTorch implementation of the Transformer architecture, giving developers worldwide a way to seamlessly deploy their state-of-the-art Transformer models on Apple devices.
This implementation is specifically optimized for the Apple Neural Engine (ANE), the energy-efficient and high-throughput engine for ML inference on Apple silicon. It will help developers minimize the impact of their ML inference workloads on app memory, app responsiveness, and device battery life. Increasing the adoption of on-device ML deployment will also benefit user privacy, since data for inference workloads remains on-device, not on the server.

[dblock]

[Catch All Triage - Attendees 1, 2, 3, 4, 5]

Thanks for opening this.

[zane-neo]

@rbpasker ml-commons leverage djl: https://github.com/deepjavalibrary/djl to accomplish the model inference, it seems djl is the correct place to create this issue.