--- Begin Message ---
|
Subject: |
[PATCH] grep: grouping of patterns including back reference |
|
Date: |
Sat, 03 Nov 2018 20:41:46 +0900 |
Hi,
When grep uses regex, it splits a pattern with multiple lines by
newline character. Compilation and executution run for each fragment.
That causes slowdown. By this change, each fragment is divided into
groups by whether the fragment includes back reference in a pattern or
not. a frgment which includes back reference constitutes group, and all
frgments which include back reference also constitute a group.
This change extremely speeds-up following case.
- before
$ yes 00000000000000000000000000000000000000000x | head -10 >in
$ time -p env LC_ALL=C src/grep -f pat in
real 1.94
user 1.70
sys 0.24
$ yes 00000000000000000000000000000000000000000x | head -100 >in
$ time -p env LC_ALL=C src/grep -f pat in
real 13.04
user 12.78
sys 0.25
- after
$ yes 00000000000000000000000000000000000000000x | head -10 >in
$ time -p env LC_ALL=C src/grep -f pat in
real 0.75
user 0.56
sys 0.17
$ yes 00000000000000000000000000000000000000000x | head -100 >in
$ time -p env LC_ALL=C src/grep -f pat in
real 0.74
user 0.57
sys 0.16
Thanks,
Norihiro
0001-grep-grouping-of-patterns-including-back-reference.patch
Description: Text document
--- End Message ---
--- Begin Message ---
|
Subject: |
Re: bug#33249: [PATCH] grep: grouping of patterns including back reference |
|
Date: |
Sun, 22 Dec 2019 16:57:12 -0800 |
|
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 |
On 11/3/18 9:25 PM, Norihiro Tanaka wrote:
> $ seq -f '%040g' 0 9999 | sed '1s/$/\\(0\\)\\1/' >pat
Thanks for the test case and sorry about the delay. And thanks for spotting the
speedup opportunity. I found a few problems with the proposed patch, though:
> + if (keys[1] == '\\')
> + keys++;
This mishandles the case where the input byte sequence contains '\', '\', '1'
where the first '\' is the last byte of a multibyte character. Such a byte
sequence can contain backreferences but the function will say it doesn't.
> + if (backref && prev < p)
> + {
> + len = p - prev;
> + buf = xrealloc (buf, (buflen + len) * sizeof *buf);
> + memcpy (buf + buflen, p, len);
> + buflen += len;
> + }
This seems to have three problems. First, the memcpy copies from P, but it
should copy from PREV. Second, this code assigns to LEN, which breaks a later
use of LEN. Third, if there are many patterns with backreferences, repeated use
of realloc will have O(N**2) behavior.
> + for (size_t i = 0; i < dc->pcount; i++)
> + dc->patterns[i + 1] = dc->patterns[i];
This copies dc->patterns[0] to the later values in that array, when a memmove
was intended.
So, after installing the patch, I immediately installed the attached patch,
which should address the abovementioned issues.
Thanks again. You did the hard work - I merely proofread it.
0001-grep-fix-some-bugs-in-pattern-grouping-speedup.patch
Description: Text Data
--- End Message ---