Hi supraestereo,
Your current bottleneck will be your upload speed since you are uploading a complete copy of your transmission to each of your 20 listeners. It is hard to give you an exact number in regard to what bandwidth you will need to support a specific number of listeners, but I would suggest using your current setup as a guide. If you are starting to have issues at 20 listeners with a 1.1MB/s connection then you are likely to need at least a 6MB/s connection to support 100 simultaneous users.
Another option you have is to reduce the quality of your audio stream. I'm not sure what kind of content you are streaming, but reducing the bitrate of the transmission will allow you to reach more users. The actual drop in audible quality will depend a bit on the content, but it may be an acceptable compromise for you.
Regards,
Ben