It depends what they're doing. Tweeting on your behalf - yes, you would see that and know where it came from. Sending DMs? There's no source parameter - so you wouldn't know which application was compromised. Following or unfollowing - again, you'd have no idea.