Thanks for the question. You would actually need to select another date hierarchy field like "Month Start Date" in order to format the field into MMMM yy format. The "Month" date hierarchy field returns an integer value rather than a date, and as a result cannot be formatted via the date part formatter.
The below wiki article outlines which calculated fields are returned as dates and which are returned as integers (to give you a better idea of which fields you should be building your hierarchies from):
Hopefully this information helps. Please let me know otherwise. Have a great day!