My child's father threatened to her?(pls read details)?

My child's father was only around briefly after she was born(he was 14 at the time, I was 15), he has never had any custodial rights, nor has he/his family ever paid me any child support all 7 years of her life. Anyway, she is 7 now, and a few days ago he came by to visit her and I let him. Yesterday I was taking a nap and he was in the living room playing with her. He toldher he was going to the store and to come with him, and she said she had to ask mommy first. He asked her why and she told him that he was a stranger and mommy told her never to go anywhere with strangers. He kept inisting and she said that mommy was the boss. He told her that if she kept being rude he was going to her. I sometimes my daughter(I am not against it), but only if she is REALLY acting up and since I raised her alone, I'm the only one that has the right to do it. She woke me up and he gave me a long lecture about how he should have the right to punish her since he's her father, I told him that he's never taken any responsibility so it's not his right. I made him leave b/c he would not stop. Was that an OK thing to do or is he right?