I am sure that every cross dresser will have this dilemma and battle within, of if he is doing right thing or should he stop. It is very complicated to explain it to anyone and you will not find a single soul who would understand you.
You will be scared to come out and dress as girl, but you will also not be able to stop.
In my initial days I was so confused and I had started to wish I was a girl. I would speak to guys and act like a tough guy, I even longed company of girls and admired beautiful girls around me, but in privacy I always wished I was a girl.
However I was never attracted to guys and not even attracted to them right now. But I will write some more on that later.
I am certain that most of the time it will be a tug of war between mind and heart. There is no way out once you are in your panties. The only choice is to come out in open as a cross dresser or be a secret one. Most of us never manage to get enough courage to become a full time sissy.