What would be the original video format that you are converting, the issue is that you cannot load the converted videos on your PS3? So as by default, PS3 plays MP4 files but only with the codec MPEG-4 AVC H. 264 /Xvid/DivX and audio codec of AAC. Other MP4 video files are unsupported like MPEG-4 ASP.
You can check the supported formats here: https://manuals.playstation.net/document/en/ps3/current/video/filetypes.html
In this case, try selecting the File Options to set the encoder of the file.