Thanks for the solution.
I figured out what's wrong. Somehow the lossless export the audio as 6 channels, while it was actually 2 only. I tested with short portion of the export that if I manually switch the channel to 2, it'll be fine.
The solution, however, allows me not to spend time on export again and just put the audio in.
It was a computer game video trying to be "lossless quality", thus the high bitrate. hehhe. It caused so much trouble though.
-------
Avidemux doesn't let me to just replace the audio track. So I ended up using the new lossless export with 2 channels anyway. Thanks for your help.