Skip to content

Add enhanced bootstrap mechanism - #39

Closed
fonic wants to merge 2 commits into
jech:masterfrom
fonic:bootstrap
Closed

Add enhanced bootstrap mechanism#39
fonic wants to merge 2 commits into
jech:masterfrom
fonic:bootstrap

Conversation

@fonic

@fonic fonic commented May 13, 2018

Copy link
Copy Markdown
Contributor

This adds an enhanced bootstrapping mechanism based on bittorrent/libbtdht. My goal was to create a mechanism that is configurable/flexible, fast (while not overloading the network) and implementable with minimal changes to existing code. I think this goal is reached quite well.

Additions:

  • adds data structures bootstrap and bootstrap_node
  • adds bootstrap state logic (failed, disabled, complete, enabled, running), adds events to notify application of state changes
  • adds DHT_BOOTSTRAP_... defines to configure bootstrap behavior
  • adds bootstrap timer bootstrap_time used by dht_periodic()
  • adds bootstrap_periodic(), which does the main work and is called by dht_periodic() while bootstrapping
  • adds helper functions bootstrap_update_timer(), bootstrap_switch_state(), random_bucket() and dump_bootstrap_nodes()
  • adds dht_add_bootstrap_node() to add bootstrap nodes to internal lists
  • adds dht_enable_bootstrap() to enable/disable bootstrapping
  • adds dht_bootstrap_state() to query current bootstrap state

Changes:

  • dht_init() initializes bootstrap data structures
  • dht_uninit() frees allocated bootstrap resources
  • new_node() will not ping dubious nodes while address family is bootstrapping
  • neighbourhood_maintenance() will not do anything while address family is bootstrapping
  • bucket_maintenance() will not do anything while address family is bootstrapping and will not consider both address families for find_node while bootstrapping
  • dump_tables() will list bootstrap nodes using dump_bootstrap_nodes()

Bootstrap mechanism:

  • bootstrap is completely separate for IPv4 and IPv6
  • bootstrap nodes are added to internal lists using dht_add_bootstrap_node()
  • bootstrap is enabled using dht_enable_bootstrap(), switches state to 'enabled' and sets bootstrap timer to 'now'
  • dht_periodic() calls bootstrap_periodic() based on bootstrap timer
  • bootstrap_periodic() adds bootstrap nodes from internal list to routing table using fake ids (myid, first bit flipped, last 4 bytes randomized), switches state to 'running' and updates bootstrap timer
  • bootstrap_periodic() is called periodically from now on:
    • buckets and nodes are processed randomly
    • if a node is dubious, a ping message is sent
    • if a node is good and more dubious nodes can be handled, a find_node message is sent
    • processing ends if limits defined by DHT_BOOTSTRAP_... constants are hit or all nodes have been processed
  • bootstrapping ends when the defined target of good nodes is reached, when bootstrapping failed or when bootstrapping is disabled
  • since bootstrapping is separate for each address family, it is possible e.g. that IPv4 bootstrapping is complete while IPv6 bootstrapping is still running

Command line for testing using well-known bootstrap nodes:
dht-example <port> 67.215.246.10 6881 82.221.103.244 6881 2001:41d0:c:5ac:5::1 6881 87.98.162.88 6881 212.129.33.59 6881 174.129.43.152 6881 2001:41d0:2:599c::5d71:2184 25401 104.168.32.188 25401

Bootstrapping both IPv4 and IPv6 with default settings should take about 45-60s.

NOTE:
This is based on my PR #37 to make use of the enhanced debug output features.

Screenshot:
screenshot_20180524_093151

@fonic fonic changed the title Enhanced bootstrap mechanism Add enhanced bootstrap mechanism May 15, 2018
@fonic fonic mentioned this pull request May 15, 2018
fonic added 2 commits May 25, 2018 14:54
Extend debug message logging / debug output:
- add support for message types (debug, info, warning, error)
- introduce logging macros to easily log messages (debugf, infof,
  warnf, errorf)
- log messages to user-defined callback instead of file so that
  application may be decide how to handle debug output
- add string conversion functions that may be used in printf-style
  function calls (address family, IP protocol version, bucket/node
  ID, socket address, etc.)
- add DHT_LOG_ defines to control certain features
- add function to retrieve statistics (buckets, good nodes,
  dubious nodes, total nodes)
- modify dump_bucket() and dht_dump_tables() to output tables
- migrate all debugf calls to make use of the new capabilities
- add additional debug output
- modify dht-example to make use of the new capabilities
Add enhanced bootstrap mechanism:
- add data structures 'bootstrap' and 'bootstrap_node'
- add bootstrap states (failed, disabled, complete, enabled, running)
- add defines 'DHT_BOOTSTRAP_...'
- add bootstrap timer 'bootstrap_time'
- add 'bootstrap_periodic()', 'bootstrap_switch_state()', 'bootstrap_
  _update_timer()' and 'random_bucket()'
- add 'dht_add_bootstrap_node()', 'dht_enable_bootstrap()' and 'dht_
  bootstrap_state()'
- modify 'dht_init()' to initialize and 'dht_uninit()' to free boot-
  strap resources
- modify 'dht_periodic()', 'new_node()', 'neighbourhood_maintenance()'
  and 'bucket_maintenance()'
- add 'dump_bootstrap_nodes()' and modify 'dump_tables()' to list
  bootstrap nodes
- modify dht-example to make use of this feature
@jech

jech commented Jun 30, 2018

Copy link
Copy Markdown
Owner

Rejected, sorry.

The bootrstrap mechanism should live in the client, not in the library. If you cannot implement your favourite bootstrap mechanism in the client, this means that the interfaces provided by the library are deficient, and you should concentrate on designing the right interfaces.

What is more, the bootstrap mechanism you propose is rather hackish, and introduces a number of weird dependencies to the code (adding nodes with a fake id? Pinging nodes from new_node?).

Finally, your patch makes a number of gratuitious stylistic changes (adding an overly-complex logging mechanism, replacing conditionals with switches, and even replacing spaces with tabs), which makes it next to impossible to review.

@jech jech closed this Jun 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants