IVM cannot directly transfer between VoIP and conventional telephone lines. To do this there needs to be some form of VoIP to PSTN gateway, such as a FXO Adapter or a VoIP gateway provider.
Sometimes IVM dials too quickly for the PBX or your phone company. In that case you may need to insert a pause (',') or longer pauses (',,,') using commas after the flash character ('!') and after the dialed digits.
Another possible problem with transfers that use a hook flash is that different countries and different phone systems can require different flash times. Check your PBX flash time or ask your telephone company (if you are not using a PBX).
You may also want to reach out to NCH tech team and see if they can help: https://www.nch.com.au/support/index.html