by Mary Cooch.
You say you are using 1.9 Can you confirm that the student is not already enrolled in the course and that they are not enrolled sitewide via "site admin>users>permissions>assign system roles" or even assigned in a category and that your default role is "authenticated user "and NOT "student"? in user policies? Any of these options would allow them to access the course without the enrolment key and would stop your group enrolment working.