Drop in backlinks. Obviously when not aggressively acquiring new backlinks
to your site from day to day, existing backlinks may even deteriorate in value.
The referring site may disappear, be changed, lose relevance or lose PR itself,
reducing the PR that is passed onto you.