Withholding Payment Under JCT and NEC Contacts

Provided that a party to a contract issues a compliant withholding notice setting out the ground or grounds and the sums attributable to each ground then the party may legitimately withhold.
So what can you do to defend that position?