Firstly you have misrepresented what I said, I didn't say 'irrelevant links really cause harm' I said an off topic link form an authority 'could' possibly do more harm than good.
My reasoning is this.:~
Authorities know their stuff. Government departments are tightly focused, an off topic link therefore could confuse the search engines into theming your site wrongly. Nothing more, nothing less. I was talking about theming sites and authority links, not off topic linking in general. Two completely different things, as low authority sites have low trust levels and as such probably can't carry much contextual power.
Hope that has made my position a little clearer
