Letting my daughter go..?

I have been divorced for two years now. I have been involved with someone for quite some time now. This man wants to take my daughter, along with his daughters, on a sledding get away which is going to be across the state. My ex-husband states that if I let her go that he will call the police on this man as he insists that this man was the reason for our divorce. Our divorce papers do not state that I need permission to take our daughter across the state line. I know my daughter is safe in this man's hands, and she became very upset when her Dad said she couldn't go. Her father is not active in her life. He tells her he will see her and not show up. I called the police dept to let them know of the situation and that I gave her my permission to go on the trip. We both have joint custody, but I have resedential custody. It 's obvious that my ex is doing this out of hate for the other man and is not thinking of her feelings. Have I done the wrong thing by letting her go?