If he punishes you for being sad over not being able to physically near him you need to invest yourself in someone else.

With that said there are no easy answers. You just need to be honest with each other and determine if this relationship gives you what you need or if yu are able to live with what it can not.

Good luck.