I said no, but only once

My first time was with a friend of the family, a guy I had fancied for a long time. He was much older than me, in his late 30s. I had just turned 20 and we were making out in my dorm and I told him I didn't want to have sex with him yet (but didn't tell him I was a virgin because at the time I found this embarrassing). We proceeded to make out on my bed, but a few moments later he started to have intercourse with me and I didn't tell him no again. I think he was caught up in the heat of the moment, I could have, should have said no again as I didn't want to have sex, but my first no should have been followed up with a question or some kind of communication on his part before starting to have intercourse. After we were done, he noted rather unemotively "Oh, you're all trembling."