Want to quickly add or subtract from a selection?
You could go up to the option bar and select the "add" or "subtract" options.
Try this, hold down the shift key to add to any selection tool. Hold down the alt/option key to subtract from any selection. You'll be saving so much time you'll be able to enjoy that 2nd cup of coffee!