[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: uniq Bug

From: Ryan Helinski
Subject: Re: uniq Bug
Date: Tue, 27 Jun 2006 21:22:41 -0400
User-agent: Thunderbird (Windows/20060516)

Thank you, Bob, especially for the URL.

Eric Blake has straightened me out, and this topic doesn't require any further discussion, except making this discrepancy more clear in the man page as it is clear in the Texinfo page so others don't run into the same problem.

Thanks again for all your help,


Bob Proulx wrote:
Eric Blake wrote:
Not sure if this has already been discovered, but I found a problem with uniq. If I sat down and looked a the code, I could probably see how to fix it. It seems to always occur with very large unsorted streams (files).

Below are the commands I ran to exploit the bug (which I originally thought was my error). Sorting the stream before removing duplicate lines is inconsistent with just removing duplicate lines:
Thanks for the report.  However, uniq only works on sorted streams.
By definition, uniq only looks at consecutive lines, to see if they
are identical.

Strictly speaking the input does not need to be sorted.  You are right
in saying that it only works on adjacent lines.  This is what the
manual has to say about it.

  info coreutils uniq

     The input need not be sorted, but repeated input lines are detected
  only if they are adjacent.  If you want to discard non-adjacent
  duplicate lines, perhaps you want to use `sort -u'.

If the file is not sorted, then the same line might appear twice.  And
changing this would make slow uniq down (either requiring more
memory or more time to keep a list of all previously seen unique lines),
not to mention violating POSIX.

It is left up to the user to sort the file, or not, as desired.

Here is what the standards say:


  Repeated lines in the input shall not be detected if they are not

I enjoyed the dry humor of this part:

      The sort utility can be used to cause repeated lines to be
      adjacent in the input file.



reply via email to

[Prev in Thread] Current Thread [Next in Thread]