bug#8067: sort fails to sort completely, due to "similar" keys.

[Bob Harris]

On Feb 17, 2011, at 4:30 PM, Eric Blake wrote:

> On 02/17/2011 01:46 PM, Bob Harris wrote:
> > Howdy,
> >
> > (note: I know I should give you version information with this, but  
> > (1) I
> > am not sure that this message will be read by anyone, and (2) I think
> > the problem probably transcends versions.  If I get a response and  
> > the
> > actual version is important, I will take the time to find it.)
>
> Thanks for the report, and you are correct that your issue transcends
> versions.  However, if you use coreutils 8.6 or newer (the latest is
> 8.10), then the new --debug option would have helped you.
>
> > I have a file of genomic short sequence info in which it so happens  
> > that
> > two of my sort key values are similar.  The two keys are
> >    HWI-ST407_110127_0082_A80L25ABXX:5:2:11746:46371#0/1
> >    HWI-ST407_110127_0082_A80L25ABXX:5:21:17464:6371#0/1
> > As you can see, these are identical if one removes the colons.
>
> Which sounds like exactly what sort does when you are sorting in the
> en_US.UTF-8 locale.
>
> > I have tried several different options but none seem to work.  -d  
> > seems
> > to be the default, and it has the behavior indicated above.  -n fails
> > completely.  -g also fails.  Reading the man page, I don't see any  
> > other
> > options to control the comparison function.
>
> Then you missed this part (in the sort man page, which is in turn
> generated from 'sort --help'):
>
> *** WARNING ***
> The locale specified by the environment affects sort order.
> Set LC_ALL=C to get the traditional sort order that uses
> native byte values.
>
> > I understand *why* -d considers these two keys equal.  What I don't
> > understand is why there is no option that says "order them
> > lexicographically".
>
> That option is your set of locale-specific environment variables.  Why
> it's not an explicit option is due to historical accident (that's the
> way POSIX specified it).  Maybe GNU sort should add a
> --collate-locale=... option as an extension that overrides LC_ALL, but
> that seems a bit like bloat, and doesn't buy much over using the
> standardized means of choosing collation sequencing.
>
> > Is there a hidden sort option that will do what I need?
>
> Yep - try 'LC_ALL=C sort ...' to see the difference.
>
> > I'm pretty sure I'm not the first person to run into this problem.
>
> You're not.  It's a FAQ:
>
> http://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021
>
> --
> Eric Blake   address@hidden    +1-801-349-2682
> Libvirt virtualization library http://libvirt.org


Thanks Eric, for the informative reply, and the FAQ link.

That makes sense (in the sense that I can see how to correct it).   
Currently my LC_ALL is set to nothing.  Some additional googling after  
sending my previous message revealed that this is also affected by  
LANG (my default is en_US.UTF-8 and LANG=C gives the desired result).   
I was in the process of investigating what else I might break my  
fiddling with LANG when your message arrived.

So I'll investigate LC_ALL instead, and see if there are potentially  
any negative side effects, so I'll (hopefully) know what trade off I  
am making (if any).

Thanks again,
Bob H

P.S.  You're right, I missed the warning in the man page.  I was  
diligently looking through the options for one that would do what I  
needed, and didn't realize there were other descriptive notes below  
the options.