I have Lily Allen's album on my computer still, I just can't be bothered deleting it.
Wheatus' album is on there even though I only like their cover of A Little Respect
Chas N'*Dave - I once bought a Chas N Dave greatest hits album for like two quid and it came with an extra full album on a bonus disc (all in all about 70 songs...) and I'm ashamed to admit I listen to it far more than I should...
The Distillers - Yes I was a retarded teenager once and yet two albums are still in my library...
I don't really think there's anything else. I have a few more CD's I'm none too proud of but I ignore their existence.
As for other music on my computer I don't think there's anything else I should feel guilty about. I still stand by the fact Elton John has a few damn good songs. My music taste is pretty good all in all.