Your child is the most important person in the world to you. Despite that, the person you hate the most is the only other person who loves your child as much as you do. You have a divorce, but that doesn't mean you get to walk away from the person you despise.
To you, your ex is toxic, but to your child, he's the epitome of everything a good father should be. You can't justify trying to remove him from your child's life, even though you can't stand to be around him. What can you do to make this easier on yourself?
Practicing patience with your custody arrangements and ex is something you may need to do for some time. It will come easier over time, since hurt feelings go away and the pain of a divorce dulls. When your ex does something that annoys you but has no bearing on the relationship between him and his child, it's possibly best to leave it alone and to ignore his behavior. The only exception is if you're being harassed or suffering because of the negative behaviors.
The next step is to minimize contact with your ex-husband as much as possible. If you need to drop off your child, stay in the vehicle and watch him walk inside. If you need to pick him up, wait for him to come to the vehicle. Unless your ex has something he needs to talk to you about that has to do with your child, there is no reason to continue contact that hurts you.
Finally, establish strong boundaries. Your ex has crossed boundaries and upset you, so consider talking to him about what you can do to avoid those conflicts in the future.
By establishing boundaries, being patient and reducing contact, you can get through this difficult time without negatively affecting your child's relationship with his father.