I don't believe you should *always* work with number pairs. If the design works in Retina with a 1px border, but a 2px border is too heavy, use a 1px border. When you scale it down it will stay as 1px - if applied as a style - which will be fine.
As Jonathan says, it all comes down to how an app feels overall. Scaled down, blurry images do create the feeling that you're in a cheap, rushed app - whether you actually consciously notice that or not.
Sometimes designers DO go too far with these myopic analyses of pixels, but then I've seen a fair number of coders spend hours refining a piece of code for no real reason other than exactness. Exactly the same.