I know married couples cannot use condoms to prevent pregnancy, but what if one of them is infertile, or the condom is used strictly to restrict the spread of venereal disease?

The marital act is not intended solely for the procreation of children; it is also intended for the unity of the couple, and contraception hinders the full union of the couple. Even an infertile couple must leave their relations open to the possibility of new life, unlikely though new life may be. Rachel, Hannah, and Elizabeth are all examples of women in scripture who were believed infertile and yet conceived children. As for the prevention of venereal disease: Though moral theologians have discussed the possibility of licit means of preventing the transmission of venereal disease in married couples, the ordinary teaching of the Church is that couples who are not able to safely engage in marital relations should entirely abstain from engaging in them.