I would agree with the others that has posted here already. Welcome, and your husband is definitely not being honest with you. Especially since he was never a "private" person with you. Regardless if he was/is a private person he gave up the rights to said privacy by having a minimum an inappropriate relationship and at most a full blown affair that you just haven't caught him in yet. He should have NOTHING to hide.
It gets down to setting boundaries on what you are and are not willing to put up/live with. For example, you could tell him that you want all his passwords, and for him to delete her number, and never see her or communicate with her in anyway. 100% No Contact. If you refuse to do these things, then I will have no choice but to assume that you are continuing this inappropriate relationship. Since you have already proven yourself untrustworthy. I'm not willing stay in a marriage with a third party, so you will need to find somewhere else to live.
The key is sticking to your guns. Trust your gut. I would bet you a paycheck he is still communicating with her, and it's likely a full blown physical affair, from what you have said on here. You just haven't caught him. He has only admitted to things you have caught him on and can prove.
Male BS, D-day July 2015, trying to stay out of the dark.....