Remove Duplicate Lines in a file

[nmuntz]

Hi,

Can somebody point me in the right direction on what would be some of the most efficient ways of removing duplicate lines in a file? I probably won't be loading files bigger than 5 mb, but still I would like to do it as efficiently as possible.

Thanks a lot in advance !