His_little one, I don't know exactly what your respective work and life schedules are, but one thing that stuck out to me here was not so much the amount of time between when you see each other (which is understandable, given the geographic distance), but the amount of time that passes with you not talking to one another.
I think it makes sense to go a number of days without talking if it's for something like a "boys' weekend," but going that long consistently could be a real obstacle to your developing relationship.
During the long-distance phase of our relationship, Sir and I talked on the phone daily, even if it was just several short conversations in between "this and that." I realize you two are brand new together and that may be a bit much, but it sounds like the lack of communication for 3-4 days is driving you crazy. It would drive me crazy, too.
It seemed like my Master and I both had a need to keep in close contact with each other, even when our relationship was very new. I ended up getting a discounted international calling plan so it wouldn't be too expensive for us to talk regularly (since I was in the U.S. and he was in Canada.)
Anyway, as I said, I don't know the reasons why you two don't communicate more frequently, but I'm guessing it's not the distance. I'm sure if he lived closer, you would most likely see each other more often, but since he doesn't, I guess the best advice I could give would be to make it a priority to talk more often, if you can.
I just quoted this paragraph because it underscored what I was thinking.