It seems the resolution limit of the vertical does not allow a 4k vertical video to be exported.
-
I then used regedit to edit the custom resolution entry to create a vertical 4k resolution, bypassing the program's limit when entering value. It worked to a certain, which it can export vertical 4K but using only CPU encoding.... super slow LOL.