Looking at the Samba vfs_recycle man page I can't see any support for limiting the recycle bin size. There are file size limits there, although I'm not sure if they're adjustable within Webmin. If not, the config file is /etc/samaba/smb.conf and you can use the pre-installed 'nano' to edit it (it's a commandline text editor which is somewhat like Notepad).

So bottom line, I don't think that what you want is possible OOTB.

However, this is Linux, so (pretty much) anything is possible if you really want it! :)

Digging in a bit deeper, I note that it's possible to adjust the timestamps when moving to the recycle bin. There is also a tool called tmpreaper which will clean up files based on timestamp. That still doesn't meet your requirements re filesize, but using these bits, together with a function to get file tree size, we could create a (daily?) cron job to check the size of the .recycle directory and if it's over a certain size, remove files older than X days. Then recheck the file tree size and if it's still not the desired size, reduce the age filter by a day and clean again. Rinse and repeat until the .recycle directory is smaller than the desired size. So to put all this together, here is that basis of a script (perhaps save it as /usr/local/bin/recycle-cleanup?!):

#!/bin/bash -e
KEEP_DAYS=61 # default time to keep; in days + 1
KEEP_SIZE=512 # default file tree size to keep; in MB
RECYCLEBINS="/srv/storage/docs " # space separated list of recyclebin locations
treesize() {
    # return the disk space size (in MB) used by the filetree
    du -cms "${@}" | grep total | tr '[[:space:]]' '+' | cut -d+ -f1
}

for recyclebin in ${RECYCLEBINS}; do
    while [[ "$(treesize ${recyclebin})" -gt "${KEEP_SIZE}" ]]; do
        KEEP_DAYS=$((KEEP_DAYS-1))
        tmpreaper --verbose ${KEEP_DAYS}d ${recyclebin}
    done
done

Please note that I haven't tested this at all so it may requires some more tweaking. Plus I haven't described how we might set this up as a cron job. However, I've already spent more time on this than I ideally should have. So please let me know if you want to proceed and I'm happy to work with you to get this working as desired.