Late Payment of Commercial Debts

Excuse 2: We don't pay penalties

Applicability

All, but this usually only comes up on first penalty

Customer does not believe penalties apply to them
See excuse 1, penalties are a statutory part of all commercial contracts
Customer knows penalties are in the contract but believes some point of principle that they should not pay them
This is no different to considering any other aspect of the contract does not apply - such as paying for goods, etc. They presumably think paying within agreed terms is also optional.
Customer does not believe they have an obligation to adhere to the agreement they have made
Whether we charge penalties or not should not be an issue - it only applies if the customer breaks the contract and pays late. Customer probably has a fundamental misunderstanding that they don't have to do what was agreed, but that it is fair to do whatever they can get away with and feels aggrieved that they failed to get away with paying late when they usually do (with other suppliers).

Cancelling charges the first time can help to resolve this, but only sensible if customer is educated by the experience and will pay on time in future.
Having explained that the penalties do apply and got the customer to understand this, if they still think as some principle that they should not pay, then you could perhaps ask which other aspects of the contract they feel should not apply either. Ultimately, a customer that does not understand the basic concept of a contract like this is not a customer you want as they are likely to be constant trouble.
May be worth stopping penalties on their account as sooner or later they will leave because of some other misunderstanding or point of principle, at which point a large bill for penalty charges can be issued and pursued via small claims court without difficulty.