What additional key do I hold down with the freeform tool to remove from a selection?

I use the <shift> key to add to a selection, but how do I remove an area from a selection?

Have you tried clicking the selection again while still on Shift?