[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Technical problems [was: Re: My view on bootstrappable effort (Re: Check
Technical problems [was: Re: My view on bootstrappable effort (Re: Checkpoint)]
Wed, 29 Aug 2018 14:04:23 +0200
On 2018-08-29 10:26, Arne Babenhauserheide wrote:
Amirouche Boubekki <address@hidden> writes:
On 2018-08-28 19:06, Amirouche Boubekki wrote:
a) ordered key-value stores are such a primitive
Sometime people call them "sorted". Ordered is required
to have more assumptions about how the data is structured.
Because keys are ordered, you know that "similar" keys
will be "collocated". That what makes _range_ scan possible
(in opposition to full scan). range scan will return a slice
of key that start with a given prefix. By construction (at insertion
time) it might be all the keys that represent a given document.
There is various pattern you can implement with ordered key value
store the one I know and experimented with are for instance
space filling curve. But to build a leaderboard for instance,
you can implement a probabilistic data structure inside the ordered
key value store.
If you remove the oredered, you end up with something like gdbm.
In that situation there is no "extra structure" that you can take
advantage to speed up queries. It always have to be a full scan.
Or single key get.
b) priority and age-based distributed task queue is another
Indeed, it's a distributed peer to peer network (in the sens every node
can do anything). But it's a trusted network. Untrusted peers can not
access the task queue directly.
How do you prioritize?
That's the question! I will start another topic with that question.
What’s your source of truth?
The database, that is FoundationDB in the distributed setup.
Does truth have to be defined centrally or does it center around each
Yes truth is defined centrally, but with fault tolerance. Every peer (or
has a heartbeat to check every minute that every role is fulfilled.
It's fulfilled if a lock is taken in FDB. If it's not there (or expired)
there is a
race for taking on that role. FDB is atomic and consistent. So only one
will take a given role at a given moment. Until the peer goes down
they will fulfill their duty for the role they have taken.
Like keeping the task queue up-to-date.
In practice, the role is fulfilled by a fiber of a peer.
(hopefully with eventual consistency within society)?
When it comes to FDB, it's consistent. Every peer see the same version
of the truth.
I’m asking because the minimum truth you need for any communication
system is (spam? item).
Like I said earlier, this is a controlled environment. Some kind of peer
architecture for scaling centralized systems.
thanks for the link.
PS: sorry for keeping this short.
Thanks for replying.
Amirouche ~ amz3 ~ http://www.hyperdev.fr