Also simplified the implementation of all matrix types.

This avoids an assert failure when assigning a value to an empty array.

This is necessary to fix build with GCC 10, other branches need to be
rebased ASAP.

To/matrices

See merge request !59