There's no difference here between 10.36, 10.75 and 10.84 for this function. Masking works just as shown in the image above.
The checkerboard area seen in your PV image is not what a JPG would produce. A transparent PNG would display that way, and could be overlaid on track 2.
In version 10.36 here, there are mask shape choices...
But that may be due to A/B testing. Each install has subtle differences.
If you have not plotted an outline of the bird, then masking will be of little use. Chromakeying is a better solution.
So it's unclear what's going on. Be sure that your video driver and Direct-X are up to date... https://www.nch.com.au/kb/10265.html