Yup... you need to allow a total stranger (the Realtor) and other total strangers to come into your house. While you're NOT there.
They will be more inclined to linger. They will be more inclined to discuss what they like about your house. They will be more inclined to raise objections... and the Realtor may be able to offer insight.
Some sellers are their own worst enemy, even though their intentions are good. They want to go around with the buyers and explain the nuances of the home. Usually this makes the buyers uneasy. You don't want your buyer to be uneasy when they are supposed to be in a buying mode.
As a Realtor, we need our buyers to be comfortable and open with us during these showings. Most people are too polite to give 100% totally honest feedback with an owner within earshot.
And don't give us the "I'll get out of your way." routine. The buyers are STILL hesitant. They are aware they are "putting you out" by being there, so they tend to rush.
The best scenario is that you are not there when they arrive, and you are not there when they leave.
I know it sounds a little weird, but it really is the best way.