Block store performance over the network

[Ludovic Courtès]

Hello!

The performance of raw block writes over the network (here, over an
SSHFS mount) is similar to that over a local file system [0]:

--8<---------------cut here---------------start------------->8---
$ dd if=/dev/zero bs=1024 count=1024 | chop-archiver --no-smart-store -S 
gdbm_block_store -f /net/fdn/,,t -i uuid_block_indexer -A 
chop-archiver: warning: 0: trailing garbage in block-indexer
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 76.5329 s, 13.7 kB/s
tree_indexer:uuid_block_fetcher:uuid_index_handle:64::0c9a5519-dcab-45bd-86de-ee7f2f64296e

$ rm /net/fdn/,,t 

$ dd if=/dev/zero bs=1024 count=1024 | chop-archiver --no-smart-store -S 
tdb_block_store -f /net/fdn/,,t -i uuid_block_indexer -A 
chop-archiver: warning: 0: trailing garbage in block-indexer
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 36.5865 s, 28.7 kB/s
tree_indexer:uuid_block_fetcher:uuid_index_handle:64::7533b6ea-a244-4b2b-b529-1ca5581e4dba

$ rm /net/fdn/,,t 

$ dd if=/dev/zero bs=1024 count=1024 | chop-archiver --no-smart-store -S 
fs_block_store -f /net/fdn/,,t -i uuid_block_indexer -A 
chop-archiver: warning: 0: trailing garbage in block-indexer
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 73.1324 s, 14.3 kB/s
tree_indexer:uuid_block_fetcher:uuid_index_handle:64::95da227b-0f60-4b4c-9afd-9db291e1871c

$ rm -rf /net/fdn/,,t 

$ dd if=/dev/zero bs=1024 count=1024 of=/net/fdn/,,t
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 12.6362 s, 83.0 kB/s
--8<---------------cut here---------------end--------------->8---

Clearly, there is room for optimization. ;-)

The main goal should be to reduce the number of round trips.  Instead of
doing typically 2 round trips per block (1 for chop_store_block_exists,
as done by the smart block store, and 1 for chop_store_write_block), I
think we need to be able to batch these requests.

So we would have a chop_store_blocks_exist and chop_store_write_blocks
(note the plural), which would allow the indexer to submit a whole bunch
of blocks at once.

Then we need a block store back-end that can actually take advantage of
this.  Of the existing ones, only the Sun RPC one could do that, by
changing its WRITE_BLOCK RPC into a WRITE_BLOCKS one, etc.

To make things more convenient, we’ll need to have a protocol over RSH,
and so we’ll need something akin to ‘rsync --server’ or
‘git-receive-pack’, which would run on the receiving side.

I’ll probably start by changing the interfaces to allow that, then
change the Sun RPC store and server, and finally look at implementing
the RSH-based store.

Feedback welcome!

Thanks,
Ludo’.

[0] https://lists.gnu.org/archive/html/libchop-devel/2010-06/msg00000.html