The grandparents in this situation could go to court for grandparent visitation. It is VERY minimal though, and not always is it granted.
If after your husband adopts this child, why not try and work out something between the grandparents to allow them contact with the child. Whether it be visiting every so often, or using them as a babysitter for a night out for your hubby and yourself. Alot of times, if you are reasonable with feelings and allow a relationship between the parties; the courts do not need to get involved.
Pursue the adoption, and go from there. If the grandparents are adament on getting visitation rights, go with the flow. But it's getting back to where the parents have the last say in whom they're children spend time with, and at what times. (the way it should be) If you do not refuse contact with the grandparents, you should be alright all the way around.
Good luck!