This can handle non-const -> const constructions and assignments. Due to
the automatically generated copy- and move-constructors and assignment
operators, some SFINAE tricks and little code duplication are needed to
make it work.