My estranged husband hasnt made a visit to my son for 4 months -?

The court says I have to make him available every fortnight - he picks him up from nursery and his mum drops him back on Sunday at my parents. This is getting unbearable - I cant plan things and my son who is 4 is getting upset about the uncertainty. What can I do - he was abusive and this is just part of his wanting to control everything.