Hello,
To change the paper size in Pixillion, add your JPG images, select PDF as output format and set them to be combined. Click "Encoder Settings" to set the paper size and other alignment settings. A description of those options can be found in the user's manual at help.nchsoftware.com/help/en/pixillion/win/imageencodesettings_pdf.html
If you keep having issues, please enter a detailed support ticket at http://www.nch.com.au/support/index.html
Make sure to include your registration code and all the files you're using.