I think in an ideal world, things should be ask you described, link based on how good he contents being linked to is and not avoiding linking for fear of Google but sadly, if you link to a site that is on the wrong site of Google, you will be dragged down to the dogs
I am speaking from experience