Would authorising your handset actually be a weakness? If someone has access to your handset then they already have access to your code generator or SMS anyway.
And if someone has access to my desktop then they are either easily traced (who was in the house at this time) or there are other, easier ways to steal from me (like taking my watch and all my belongings in the same room as the desktop).
The attack vector for most people is people accessing their account remotely.
From a more cynical point of view - if someone in Russia hacks your Amazon, you'll blame Amazon*. If someone compromises your account in person - a family member, a thief, a co-worker, a friend - you're going to blame yourself or the thief, not Amazon.
*For the niave user at least.