The issue is that once you removed the amp page, the canonical from the amp URL is inherently no longer there. As others are saying a rewrite rule in your .htaccesa file to remove /amp from the end of the URL via 301 redirect is the way to go.
RewriteRule ^/?(.*)/amp/?$ /$1 [R=301,L]
That should do the trick as the amp plugin adds