Could you confirm you have selected the option to minimize the Debut window when that is covering the area you are recording. To do that you may open Debut, go to Options > Record > Screen capture and thick the box Minimize Debut window when recording if it is covering the recording selection. You can also select the option to disable preview when recording in screen capture.
In case the issue remains, you may want to contact the NCH Software support team for further assistance.