I have never even seen a Mac, let alone used one.
Am I interpreting your query correctly? You are trying to screen-capture video and record the broadcast sound along with it? You get no sound, but when you record the webcam+microphone it works OK?
Have you been into your Mac's sound settings and set the default recording device? Then choose that device in Debut's recording settings, there should be a drop-down box in the area where you select microphone/speakers.
If you can't resolve your problem I suggest you contact NCH support - even though you may not have paid extra for support, they will respond, just not as quickly.