CJam, 6273

{49f^245b16777213%}

XOR each character with 49, reduce the resulting string via x, y ↦ 245x + y, and take the residue modulo 16,777,213 (the largest 24-bit prime).

Scoring

$ cat hash.cjam
qN% {49f^245b16777213%} %N*N
$ all=$(wc -l < british-english-huge.txt)
$ unique=$(cjam hash.cjam < british-english-huge.txt | sort | uniq -u | wc -l)
$ echo $[all - unique]
6273