When you are giving to your partner, you can't help but be more involved, present, and attuned. Being attuned is a very important part of a secure attachment. It's what "good enough" parents do for their children much of the time. A good enough parent is aware of the child's moods and needs, and verbalizes what s/he sees in the child (e.g., "You seem a little sad, did something happen at school today?") This is how children learn the language of feelings.
Many adults never learned to be attuned to their own feelings, body sensations, and thoughts as they were growing up. There can be many reasons for that. One that I see a fair amount is that the child was busy taking care of a parent (emotionally, if not physically) in order to feel (or be) safe at home.
How do you feel safe with your partner? How do you let down your guard and tolerate feeling vulnerable as you are building trust and intimacy? Sometimes it seems easier just to have an affair, especially if you're "just friends" with someone you're not married to and you begin to share your innermost thoughts and feelings with him/her. Over time, you may share less with your partner and more with the friend. The balance of intimacy is shifting outside the marriage.
Try giving more at home and less away from home. Remember, you're the one giving either way.