[GNUnet-SVN] r10650 - in gnunet: . src/transport

[gnunet]

Author: grothoff
Date: 2010-03-18 15:21:35 +0100 (Thu, 18 Mar 2010)
New Revision: 10650

Modified:
   gnunet/TODO
   gnunet/src/transport/transport_api_address_lookup.c
Log:
update

Modified: gnunet/TODO
===================================================================
--- gnunet/TODO 2010-03-18 13:07:21 UTC (rev 10649)
+++ gnunet/TODO 2010-03-18 14:21:35 UTC (rev 10650)
@@ -22,104 +22,21 @@
     transport cost changes
   - disconnect notifications from TCP not always as nice as we might want
     them to be (but how can we do this cleanly given that UDP is 
connectionless?)
-  - hostlist testcase shows that (randomly) we jump to an address that
-    seems to be a plugin function *after* the plugin has been unloaded
-    (directly from scheduler); check that *all* plugin tasks are cancelled
-    after plugin is unloaded! (need better testcases for that...)
   - implement transport API to pretty-print transport address 
     + transport_api extension (API extension!)
     + service-transport extension (protocol extension)
-  - implement gnunet-transport (transport configurator / tester)
   - instantly filter addresses from *other* peers that 
     are *equal* to our own address + port (i.e., localhost:2086).  We 
     no longer filter those for outgoing (helps with loopback testing
     and keeps the code clean), but we should filter strictly *impossible*
     incoming addresses!  This is for efficiency, not correctness.
-  - UPnP-based IP detection
-    (Note: build library always, build service when libxml2/etc. are available)
-  - We currently are happy to take any address told to us in a WELCOME
-    to our set of addresses; we should have some minimal threshold-based
-    scheme, limiting both the total number of addresses that we accept 
-    this way as well as requiring multiple confirmations; also, we
-    should possibly try to confirm that the given address works for
-    us ourselves (loopback-style) before adding it to the list
-    [SECURITY issue]
-    + we may be able to simplify WELCOME messages (no need to add 
-      addresses there anymore, but may help to learn them there anyway...).
-    + we probably want some kind of voting/counting for learning IP addresses
-      (maybe including IP addresses in ads proportional to how often others
-       report them? we at least need some protection against >64k HELLOs!),
-    + provide a way to give the user a list of "learned" IP addresses and
-      a way to easily "veto" addresses off the list!
-      => If MiM attacker uses vetoed address, blacklist the specific IP for
-         the presumed neighbour!
   - [./transport/gnunet-service-transport.c:173]: (style) struct or union 
member 'TransportPlugin::rebuild' is never used
   - [./transport/plugin_transport_tcp.c:391]: (style) struct or union member 
'Plugin::address_update_task' is never used
-* FS:
-  - [./fs/gnunet-service-fs.c:208]: (style) struct or union member 
'LocalGetContext::results_bf_size' is never used
-  - [./fs/gnunet-service-fs.c:501]: (style) struct or union member 
'PendingRequest::used_pids_size' is never used
-  - [./fs/gnunet-service-fs.c:654]: (style) struct or union member 
'ConnectedPeer::last_client_replies' is never used
-  - [./fs/gnunet-service-fs.c:669]: (style) struct or union member 
'ConnectedPeer::avg_delay' is never used
-  - [./fs/gnunet-service-fs.c:675]: (style) struct or union member 
'ConnectedPeer::avg_priority' is never used
-  - [./fs/gnunet-service-fs.c:688]: (style) struct or union member 
'ConnectedPeer::pending_requests' is never used
-  - [./fs/gnunet-service-fs.c:694]: (style) struct or union member 
'ConnectedPeer::last_p2p_replies_woff' is never used
-  - [./fs/gnunet-service-fs.c:700]: (style) struct or union member 
'ConnectedPeer::last_client_replies_woff' is never used
 * CORE:
   - SET_KEY is triggered every 2.5 minutes (after session is up); should
     just do PING (not SET_KEY + PING)
   - "Forcing disconnect of XXX due to inactivity" -- can happen every few MS!?
     (disconnect does not really succeed, or what?)
-* TOPOLOGY:
-  - needs more testing (especially F2F topology)
-  - needs to re-try connecting after disconnect (currently, it
-    initially triggers a connection request, but if that connection
-    fails / goes down, it does not retry in a timely fashion;
-    cause seems to be the 'blacklist_after_attempt' being set to 1h,
-    which is rather long -- and should probably be adjusted based on
-    the number of connections / known peers)
-  - If the topology daemon crashes, peers that were put on the
-    blacklist with transport will never be removed from it (until
-    transport service dies); we should use the blacklist notification
-    API to learn about the exact set of blacklisted peers at all times
-    (FIXME: the transport_api implementation of blacklisting
-     also does not work nicely for this since it won't let us know about
-     disconnect-reconnect events and the implicit whitelisting
-     that might happen here; that's not so bad since we will
-     re-blacklist on pre-connect attempts anyway, so this is 
-     a minor issue).
-  - the code uses the term 'blacklist' for both peers that are forbidden
-    to connect (i.e. F2F mode) as well as peers that we currently
-    won't try to actively connect to ourselves (since we just tried);
-    This is confusing.  We need two distinct terms.
-  - move code to use hash table instead of linked list
-  - instead of periodically discarding blacklisted entries,
-    simply add task that is triggered at the right time (earlier free,
-    more balanced load)
-  - check if new HELLO learned is different from old HELLO
-    before resetting entire state!
-* UTIL:
-  - trust: need *fast* way to check/update trust in peers
-           (async peerinfo would not be right)
-  - scheduler should change OS process priority based on task priority;   
-    should make better use of task priorities in general
-  - only connect() sockets that are ready (select()) [Nils]
-    [On W32, we need to select after calling socket before
-     doing connect etc.]
-* STATISTICS:
-  - synchronous/asynchronous API (& implementation) is not nice;
-    => provide notification-based API
-  - does not seem to work with timeouts (especially if service is not running)
-* DATACACHE:
-  - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs 
satisfied)
-* ARM:
-  - need to get rid of synchronous API for service starts (cause all kinds of 
problems)
-    [=> eliminate for need to tell ARM about service starts most of the time!] 
[Safey]
-  - better tracking of which config changes actually need to cause process 
restarts by ARM.
-  - listen for requests to discover dependencies between services (and avoid
-    having to explicitly program start requests)
-  - better crash management (attach debugging support, capture and analyze
-    debug output, detect random vs. deterministic crashes)
-  - shutdown sequence?
 * HELLO:
   - need function to test "equivalency" of HELLOs; use in topology!
 * FS:
@@ -127,17 +44,14 @@
     + active reply route caching design & implementation of service,
       gap extension!
   - gnunet-publish cannot be aborted using CTRL-C
-  - gnunet-publish segfaults if file does not exist
   - on some systems, keyword search does not find locally published content
     (need testcase of command-line tools!)
   - 2-peer download is still too slow (why?)
-* Module features to implement:
   - advanced FS API parts
     + namespaces: fundamental namespace API
     + search: SBlocks, KSBlocks, probes, notify FS-service of known results
     + collection
     + location URIs (publish, search, download)
-    + persistence support (publish, unindex, search, download)
     + indexing: index-failure-cleanup
     + download: management/bounding of parallel downloads (for recursive dl.)
     + datastore reservation (publishing)
@@ -147,8 +61,7 @@
     + gnunet-download (many options)
     + gnunet-directory (man page, options)
     + gnunet-pseudonym (all of it)
-    + gnunet-service-fs (remove failing on-demand blocks, stats, hot-path 
routing, load-based routing, nitpicks)
-    + datastore: do active migration support here?   
+    + gnunet-service-fs (remove failing on-demand blocks, hot-path routing, 
load-based routing, nitpicks)
   - implement adv. FS testcases 
     + getopt API
     + insert: sblocks, loc uris
@@ -156,30 +69,20 @@
     + search: sblocks, skblocks, loc uris
     + namespaces
     + collection
-  - implement FS performance tests (needs TESTING)
+  - implement FS performance tests
     + insert
     + download
     + search
     + unindex
-  - non-anonymous FS service (needs DHT)
-    + basic DHT integration
-    + CS-DHT-functions (DHT-put of LOC)
-    + P2P-functions (DHT-get)
-  - setup (RC-pre0)
-    + default generation
-    + need to settle basic design; do we want to keep guile?
-  - testing (RC-pre0)
-    + modify configuration to allow controlling connections for non-local 
starts
-    + testbed creation with topology (needs working F2F topology) [Nate]
-    + testbed with churn [Nate]
-    + implement library for distributed testing [Nate]
-    + implement testcases for distributed testing [Nate]
-    + test basic peer re-configure [Nate]
-    + test topology creation [Nate]
-    + test churn generation [Nate]
-* TESTING:
-  - consider changing API for peer-group termination to 
-    call continuation when done
+  - [./fs/gnunet-service-fs.c:208]: (style) struct or union member 
'LocalGetContext::results_bf_size' is never used
+  - [./fs/gnunet-service-fs.c:501]: (style) struct or union member 
'PendingRequest::used_pids_size' is never used
+  - [./fs/gnunet-service-fs.c:654]: (style) struct or union member 
'ConnectedPeer::last_client_replies' is never used
+  - [./fs/gnunet-service-fs.c:669]: (style) struct or union member 
'ConnectedPeer::avg_delay' is never used
+  - [./fs/gnunet-service-fs.c:675]: (style) struct or union member 
'ConnectedPeer::avg_priority' is never used
+  - [./fs/gnunet-service-fs.c:688]: (style) struct or union member 
'ConnectedPeer::pending_requests' is never used
+  - [./fs/gnunet-service-fs.c:694]: (style) struct or union member 
'ConnectedPeer::last_p2p_replies_woff' is never used
+  - [./fs/gnunet-service-fs.c:700]: (style) struct or union member 
'ConnectedPeer::last_client_replies_woff' is never used
+
 * WWW:
   - Get IPv6 hooked up
   - change DNS
@@ -203,6 +106,53 @@
 * HOSTLIST:
   - 'server' uses 'GNUNET_PEERINFO_iterate', should probably switch to 
notification API
     (for more instant / up-to-date hostlists at lower cost) [OPTIMIZATION]
+* STATISTICS:
+  - does not seem to work with timeouts (especially if service is not running)
+* UTIL:
+  - trust: need *fast* way to check/update trust in peers
+           (async peerinfo would not be right)
+  - scheduler should change OS process priority based on task priority;   
+    should make better use of task priorities in general
+  - only connect() sockets that are ready (select()) [Nils]
+    [On W32, we need to select after calling socket before
+     doing connect etc.]
+* DATACACHE:
+  - add stats (# bytes available, # bytes used, # PUTs, # GETs, # GETs 
satisfied)
+* FS:
+  - persistence support (publish, unindex, search, download)
+  - non-anonymous FS service (needs DHT)
+    + basic DHT integration
+    + CS-DHT-functions (DHT-put of LOC)
+    + P2P-functions (DHT-get)
+  - active migration support (in fs or in datastore?)
+* TOPOLOGY:
+  - needs more testing (especially F2F topology)
+  - needs to re-try connecting after disconnect (currently, it
+    initially triggers a connection request, but if that connection
+    fails / goes down, it does not retry in a timely fashion;
+    cause seems to be the 'blacklist_after_attempt' being set to 1h,
+    which is rather long -- and should probably be adjusted based on
+    the number of connections / known peers)
+  - If the topology daemon crashes, peers that were put on the
+    blacklist with transport will never be removed from it (until
+    transport service dies); we should use the blacklist notification
+    API to learn about the exact set of blacklisted peers at all times
+    (FIXME: the transport_api implementation of blacklisting
+     also does not work nicely for this since it won't let us know about
+     disconnect-reconnect events and the implicit whitelisting
+     that might happen here; that's not so bad since we will
+     re-blacklist on pre-connect attempts anyway, so this is 
+     a minor issue).
+  - the code uses the term 'blacklist' for both peers that are forbidden
+    to connect (i.e. F2F mode) as well as peers that we currently
+    won't try to actively connect to ourselves (since we just tried);
+    This is confusing.  We need two distinct terms.
+  - move code to use hash table instead of linked list
+  - instead of periodically discarding blacklisted entries,
+    simply add task that is triggered at the right time (earlier free,
+    more balanced load)
+  - check if new HELLO learned is different from old HELLO
+    before resetting entire state!
 
 0.9.0pre2:
 * Module features to implement:
@@ -220,13 +170,34 @@
     + Fragmentation library 
     + actual plugin
   - HTTP backend (RC-pre2)
+* ARM:
+  - need to get rid of synchronous API for service starts (cause all kinds of 
problems)
+    [=> eliminate for need to tell ARM about service starts most of the time!] 
[Safey]
+  - better tracking of which config changes actually need to cause process 
restarts by ARM.
+  - listen for requests to discover dependencies between services (and avoid
+    having to explicitly program start requests)
+  - better crash management (attach debugging support, capture and analyze
+    debug output, detect random vs. deterministic crashes)
+  - shutdown sequence?
 
+
 0.9.0pre3:
 * GUIs to implement:
   - qt (RC-pre3)
     + see discussions @ FISL about integration with event loop!
 * Determine RC bugs and fix those!
+* TESTING:
+  - modify configuration to allow controlling connections for non-local starts
+  - testbed creation with topology (needs working F2F topology) [Nate]
+  - testbed with churn [Nate]
+  - implement testcases for distributed testing [Nate]
+  - test basic peer re-configure [Nate]
+  - test topology creation [Nate]
+  - test churn generation [Nate]
+  - consider changing API for peer-group termination to 
+    call continuation when done
 
+
 0.9.0:
 * Documentation
   - update man pages
@@ -253,14 +224,33 @@
 
 
 0.9.x:
-* Plugins to implement:
+* TRANSPORT:
   - SMTP transport backend
   - HTTPS transport backend
     + improved HTTPS support in MHD
     + actual plugin
-* Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on 
gnunet-devs in 9/2009)
-* Allow checking of presence of search results and/or content via command-line 
tools
-  (add options to gnunet-search / gnunet-download to limit search to local 
peer)
+  - Implement method of learning our external addresses from
+    other peers; need some kind of threshold-based
+    scheme, limiting both the total number of addresses that we accept 
+    this way as well as requiring multiple confirmations; also, we
+    should possibly try to confirm that the given address works for
+    us ourselves (loopback-style) before adding it to the list
+    + we may be able to simplify WELCOME messages (no need to add 
+      addresses there anymore, but may help to learn them there anyway...).
+    + we probably want some kind of voting/counting for learning IP addresses
+      (maybe including IP addresses in ads proportional to how often others
+       report them? we at least need some protection against >64k HELLOs!),
+    + provide a way to give the user a list of "learned" IP addresses and
+      a way to easily "veto" addresses off the list!
+      => If MiM attacker uses vetoed address, blacklist the specific IP for
+         the presumed neighbour!
+  - implement gnunet-transport (transport configurator / tester)
+  - UPnP-based IP detection
+    (Note: build library always, build service when libxml2/etc. are available)
+* FS:
+  - Remove KBlocks in gnunet-unindex (see discussion with Kenneth Almquist on 
gnunet-devs in 9/2009)
+  - Allow checking of presence of search results and/or content via 
command-line tools
+    (add options to gnunet-search / gnunet-download to limit search to local 
peer)
 * STATISTICS:
   - should use BIO instead of mmap
 * PEERINFO:

Modified: gnunet/src/transport/transport_api_address_lookup.c
===================================================================
--- gnunet/src/transport/transport_api_address_lookup.c 2010-03-18 13:07:21 UTC 
(rev 10649)
+++ gnunet/src/transport/transport_api_address_lookup.c 2010-03-18 14:21:35 UTC 
(rev 10650)
@@ -70,10 +70,26 @@
     }
 }
 
+/**
+ * Convert a binary address into a human readable address.
+ *
+ * @param sched scheduler to use
+ * @param cfg configuration to use
+ * @param address address to convert (binary format)
+ * @param addressLen number of bytes in address
+ * @param numeric should (IP) addresses be displayed in numeric form 
+ *                (otherwise do reverse DNS lookup)
+ * @param nameTrans name of the transport to which the address belongs
+ * @param timeout how long is the lookup allowed to take at most
+ * @param aluc function to call with the results
+ * @param aluc_cls closure for aluc
+ */
 void
 GNUNET_TRANSPORT_address_lookup (struct GNUNET_SCHEDULER_Handle *sched,
-                                 const struct GNUNET_CONFIGURATION_Handle
-                                 *cfg, const char *address, size_t addressLen,
+                                 const struct GNUNET_CONFIGURATION_Handle  
*cfg, 
+                                const char *address, 
+                                size_t addressLen,
+                                int numeric,
                                  const char *nameTrans,
                                  struct GNUNET_TIME_Relative timeout,
                                  GNUNET_TRANSPORT_AddressLookUpCallback aluc,