Hello,
For the registration issue, what they would actually do is sent you a link to download the version of the program that is compatible with the license you have, so that you can install from there and register successfully.
You can find all NCH programs older version installers on this link: http://www.oldversiondownload.com/ , you can search there but I think its best for you to wait on a reply from NCH they will probably send you the correct link right away so you do not have look around the website above.
About the memory limit, I guess it could be talking about the actual PC memory, try opening the program and going to your task manager and see if Debut is consuming a lot of memory you can have both open and try and record to see if the memory usage goes up. If it does you would be best reaching out to NCH for technical assistance.