You gotta tell him the truth, whether he likes it or not.
The truth: You have a bed. Sleep in it. I love you. But that does not mean you can sleep in my bed.
Also, even infants should not sleep with adults. Many, many such SIDS cases every year where parents accidentally smother a kid. Not a good idea to ever practice.