It could well be a problem in the future as a manhole is usually placed at a change of direction. Change of direction means slowing down and like traffic going around a sharp corner a build up occurs and then usually clears without a problem.
If a blockage occurs the only way in (apart from rodding from previous/next manhole covers) is to dig up the concrete, unblock then make good again. Not a five minute job! A sort of guess would be at least a week of waiting for a builder, then 2-3 days of work from digging to drying out of concrete around a newly placed manhole cover, and then an inspection by the local building control. Most drain blockages are cleared with jet washers by drain companies and it's a reasonably painless task.
If the driveway was laid many years ago and there's no sign of drain covers being lifted then it probably means it's never been a problem. If the drain covers have been lifted recently (you can tell from scratch marks on the covers, dug out moss, scraping etc) then it means either someone was looking for something (e.g. mains water or electric cable) or maintenance.
Generally though, drains do not cause much of a problem as blockages will occur within the first few months of use.
If you're uncomfortable about it, ask your house surveyor. He'll probably just cover himself with some meaningless get out clause ("No problem at the time of inspection but there could be an issue in the future") but on the other hand he might be much more helpful!
