I can't say my dad has always enjoyed being a pastor. He didn't want to do it either. More wanted to be the traveling type if anything. It's been hard on him over the years. Pastors can often be under-appreciated. You really stay with one group of people and really get to know them, so things like, if they leave or get offended can hit hard. My dad's focus has always been on sharing the "uncompromising truth of the bible over people-pleasing" though.

Counseling people can be tough too, as that's another role of a pastor, at least in the non-denominational part of the Church. I'm not at all familiar with the more traditional churches.