Remove duplicate lines using awk

[Alef00]

Hi,

nice script, thanks.

Sometimes after downloading videos with autogenerated subs you get many duplicate lines.
To remove them I just added an AWK one-liner to the SED command like this:

sed -r -e 's/^\xef\xbb\xbf//'
-e 's/\r//'
-e 's/^[0-9]$//'
-e '/^[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} --> [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}$/d'
-e 's/^\s$//'
-e '/^$/d;s/<[^>]*>//g' "$1" | awk '!NF || NR > rec[$0]; {rec[$0] = NR+2}'

This now removes duplicate lines only if the duplicate item follow s immediately immediately follow one another.

Bye