Priorities?

[artyom-beilis]

DLPrimitives already gives promising results... But I'm really wondering what to prioritize:

1. Add more useful operators (dropout, upscale, lstm, prelu, mse-loss etc) to make DLPrimitives fully featured?
2. Try to improve existing OpenCL frameworks like Caffe (or PlaidML) by using DLPrimitives core operations?
3. Start working on pytorch OpenCL backend - that is huge undertaking?

...

Full thoughts there: http://blog.dlprimitives.org/post/4

[alex3s]

@artyom-beilis this is really a wonderful project. I don't know how to encourage you into continuing developing it, but the war between Nvidia and AMD for global monopoly on DL reminds me of Web Browser wars between Netscape and Internet Eplore in 2000's.
By now nobody is using either of them, it only led to decreased performance and ultimately browsers like Firefox and Chrome, etc. were born. DLPrimitives can be our future Firefox of Machine Learning!
Even Elon Musk is developing his own chip, and it Pytorch is too Nvidia oriented (i.e. use Cuda.() to call Rocm haha). It has to run on something softwarewise.
What I'm missing in DLPrimitives besides performance is caching of the compile kernels in binary form on the sdd.
For example I want to train an ensemble of models, check various hyperparameter ranges etc. It seems like starting a small model takes long compilation time. Is DLPrimitives caching these kernels?
Why performance is important? It makes one card perform like two cards. Waiting time is important.
Thank you again for developing this project! All the best!

[artyom-beilis]

the war between Nvidia and AMD for global monopoly on DL reminds me of Web Browser wars between Netscape and Internet Eplore in 2000's.

There is no war. AMD are irrelevant there as btw Intel.

They try to follow nvidia with hip so customers can migrate easy but they forget that it us better to invest in open technologies in long term.

and it Pytorch is too Nvidia oriented

Actually they and TF have support of potential alternative but no one implemented anything that really works till now

For example there is Vulkan backend but it is way way too limited

Is DLPrimitives caching these kernels?

Not yet. BTW nvidia opencl driver does it automatically.

But it is on todo list

[alex3s]

You are right: Nvidia has a monopoly already. I have recently seen deepfaceswap implementation which offered either Cuda or DirectX12 exe files for all other cards. Is DirectX12 a viable way to implement deep learning on Windows10, or is it just for inference?

[artyom-beilis]

Is DirectX12 a viable way to implement deep learning on Windows10, or is it just for inference?

I have no experience with it. I'm not really Windows user. Also Direct3D is yet another tool that targets one platform. I heard about it it may be viable for training as well but don't expect cuda/cudnn-level performance.

[artyom-beilis]

Started working on pytorch integration: http://blog.dlprimitives.org/post/5

At this point alexnet inference over pytorch opencl works with virtually identical performance to dlprimitives microframework.

To be continued...

[artyom-beilis]

Tried to integrate Intel's onednn: http://blog.dlprimitives.org/post/9
wasted effort.

[artyom-beilis]

What I'm missing in DLPrimitives besides performance is caching of the compile kernels in binary form on the sdd.

Added in latest version.

It was very annoying for AMD and Intel GPUs to build kernels each time. So I added cache support

[artyom-beilis]

See: https://github.com/artyom-beilis/dlprimitives/blob/master/docs/kernel_cache.md