I've got bad knees. I didn't kneel on pencils, but I have kneeled on concrete for hours a day. I did this as part of my job, and did it for years. So though there might not be long term damage if you do it once or twice a month, maybe less. Though if you do it excessively it could end up causing damage in ways you hadn't considered.

Enjoy, just don't over do it.