Fwd: Multiple sqlworkers

[Thomas Bereknyei]

[forgot to cc the mailing list]
Ole,

Running your example with sqlite gives me the same result. I suspect
sqlite enforces a bit more serialization when it comes to the results,
while something like postgres allows multiple queries in flight. This
example gives me numbers like 187, 201, 193, 190....

seq 100 | parallel --sqlmaster postgresql:///tom/tabl 'sleep 1; echo {}'
(
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl & wait ) | wc -l

So perhaps with postgres the queries don't do quite the same thing? Or
its "too fast" and the other queries in flight are given the same row?


On Sat, Jun 6, 2020 at 4:03 PM Ole Tange <ole@tange.dk> wrote:
>
> On Mon, Apr 6, 2020 at 5:22 AM Thomas Bereknyei <tomberek@gmail.com> wrote:
>
> > Running some experiments with the sqlmaster/sqlworker system shows
> > that running multiple independent workers results in many jobs being
> > done multiple times. This is noted in the manual as
> >
> > "If you have more than one --sqlworker jobs may be run more than once."
> >
> > In simple testing, this is not an occasional thing, but results in
> > nearly all jobs being performed N times.
>
> So I tested:
>
> seq 100 | parallel --sqlmaster sqlite3:///fisk/fisk 'sleep 4;echo {}'
> (
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> wait
> )  | wc -l
>
> This gives 100..103 on my machine. Thus 0..3 jobs were run twice.
>
> So I cannot reproduce the error you see.
>
> That is not to say that you have not found a bug, but I cannot
> reproduce it on my system.
>
>
> /Ole