I keep getting white dots on my HDRI-lit renders

The problem comes when the HDRI has areas of high contrast next to one another. The easiest way to avoid such problems is to use FP Blur in the Image Editor to blur out areas of such high contrast and that way eliminate problems with the lighting of your image. If you need to be able to see the HDRI in your picture, you can clone it and not apply a blur to the cloned version and use that as the reflection map.