Having thought about it a bit more since my tweet when you first suggested the idea, here are my musings:
This would leave you beholden to which ever mapping/tile provider that browser chose to use. Each of the mapping providers differ in quality and in the types (e.g. not all include satellite photos) of tiles they provide and how often they are updated. This means that you will get differing look and feel between browers which isn’t great and adds to testing.
I also wonder what this would do to the Map provider market, I would guess that Chrome would default to Google Maps, IE/Edge would opt for Bing Maps and Safari for Apple Maps. Leaving Firefox and Opera as wildcards, would Google or Microsoft try and entice them use their platforms as they do with default search providers? Would it be configurable, like search providers? What would this do to the likes of OpenStreetmap?
Also to make use of Google Maps tiles you need a API key for “commercial” use iirc which would make life tricky as it would mean the browser teams would need to come to an arrangement with Google who are probably the current “default” map provider for many.
The video tag was a little different because it basically replaced flash which had a whole raft of reasons (mainly security) for needing to die and we still had/have the issue of which browsers support what codecs (and the licensing issues round those)