for me it's always a low frequency vibration that gets into my recordings, but depending on your voice you can probably eq that out.
other than that it's often doors slamming, loud tvs, etc. I got so fed up that I had to build a recording booth.
no matter how you angle a mic, the room will still reflect and direct the sound into the mic one way or another, but facing the mic away will at least reduce it.