Enum for memory modes

[timmoon10]

We currently specify memory modes for the Memory class with an unsigned int:

Elemental/include/El/core/Memory/decl.hpp

Line 62 in ae00b5b

void SetMode(unsigned int mode);

Things are messy since the same number means different things on different devices (mode 1 means pinned memory on CPU and CUB memory pool on GPU). This was not a good design (mea culpa). Imagine an idyllic future with something like:

Matrix<float,Device::CPU> A;
A.MemoryMode(); // Returns memory_mode_type::default
A.SetMemoryMode(memory_mode_type::pinned_memory);

This would require making simultaneous changes in LBANN.

[ndryden]

I have no substantial comment, other than to say that this should be done. We may want to be even more general, but this seems like a good step.