Agreed. Almost everything interesting is located outside of the "points of interests" anyway, as those are all just generic locations that mostly just contain vendor trash, with a few exceptions here and there. Quests, witcher diagrams and other unique stuff is usually found elsewhere.
On my first playthrough I went semi-completionist, keeping the question marks off but visiting each notice board and doing every quest that I bumped into, and I think that was the most enjoyable way of playing the game. Now I've been doing a completionist playthrough with the question marks on while waiting for the expansions, and the enjoyment varies on different areas. Velen just feels too "gamey", as it's so tightly packed with completely generic content with all those random treasure chests lying around. It's somewhat salvaged by the fact that some of the bandit camps contain some of the best early-game encounters, but it's still far from ideal. Novigrad is OK, with only a handful of PoI's that are spread around the map quite sparsely, and mostly in remote places that make sense. And then there's Skellige, which has around 60 (!) smuggler's caches alone, all of them being exactly the same: around three barrels with chests attached to them floating in the water, guarded by a horde of sirens or occasionally drowners that are literally killed by pressing just one button (just keep firing the crossbow and Geralt will kill everything eventually), containing the worst vendor trash in the game in form of weapons and armor that weigh a ton and will encumber you immediately (not like it matters, you can go as much over the weight limit as you want to and it still sail around just like before) but are worth thousands of crowns when brought to a shopkeeper. Going through all this shit completely fucks up the pacing of the game and takes a lot more time than the entire main quest storyline on Skellige.
I think it would've been better if the treasure stuff didn't even have any kind of a marker attached to it. They're all visible from a distance anyway, just leave them there for the player to find by himself. Some other points of interest, like Abandoned Sites or certain Places of Power (i.e. those that are in plain sight and not in some underground cavern), are something that you could probably pick up from a notice board, though, as you think those would be common knowledge anyway.
.
