It is a very real possibility. I would suggest trying to explain it to your husband. Whether he can take it in and synthesize it, remains to be seen. But I could not connect to my loving feelings (or really any feelings) for my husband after his one stupid lapse I shared earlier. At one point when we were talking, I looked at him and said something very similar to what you wrote.
Beyond just worrying that his growth would take far longer than my patience, I worried that one day, years after everyone, including he and I, thought everything was great, I'd discover him lying about something relatively inconsequential and I would just be done. No fuss, no muss, just done. That is how it can sometimes be for me. I will fight tooth and nail to rebuild a relationship (friendship, relative, whatever...) but if later they do even a tiny bit of that same behavior, I am done. A switch is flipped and I am just stop caring or wanting to lift a finger to try. Luckily, my husband has seen this happen firsthand, so he believed me and realized that if he lied about anything again, he was likely coming home to his stuff in neat, labeled boxes. Because I wouldn't even be mad at that point, just finished. Hopefully, your husband will be able to :get it too" though he seems a little slow on the uptake. Still a bit too mired in the noise in his own head to listen to yours...
BS - Female
Married 27 years, one adult child DD May 2016 “When we are no longer able to change a situation, we are challenged to change ourselves.” - V Frankl