Fixing lead table with duplicate leads

OK... I've explained to my client the importance of using the duplicate check feature of the list loader, but inevitably, they failed to use it...
So, the question is, is there a /simple/ method to deobfuscate this situation? There are duplicate leads in the lead list - however, the complicating factor is that all the duplicate leads have different lead_id's (I assume this is used as the primary key on the table).
At the point at which it was brought up to me, there are currently 54,000+ leads in the table, a large chunk of which are duplicates. I've tried using a variety of methods to fix this, including:
SELECT DISTINCT (and various permutations thereof) SQL statements
sed/awk scripting (the major result of which was the intake of mass quantities of alcohol)
grep & regexing (see item #2 for result)
This situation has me utterly perplexed, and I'm afraid that my PHP ability falls far short of being able to fix this issue. My coding ability, as a whole, falls short of doing this outside of COBOL.
Any ideas, anyone ?
So, the question is, is there a /simple/ method to deobfuscate this situation? There are duplicate leads in the lead list - however, the complicating factor is that all the duplicate leads have different lead_id's (I assume this is used as the primary key on the table).
At the point at which it was brought up to me, there are currently 54,000+ leads in the table, a large chunk of which are duplicates. I've tried using a variety of methods to fix this, including:
SELECT DISTINCT (and various permutations thereof) SQL statements
sed/awk scripting (the major result of which was the intake of mass quantities of alcohol)
grep & regexing (see item #2 for result)
This situation has me utterly perplexed, and I'm afraid that my PHP ability falls far short of being able to fix this issue. My coding ability, as a whole, falls short of doing this outside of COBOL.
Any ideas, anyone ?