BlkExc - Improved peer selection

[benbierens]

I would like to discuss and design our way to better peer selection in the blockexchange engine. I think we can apply a similar pattern as for IndexingStrategy. We can make an abstraction that hides the peer-selection-strategy from the engine code, so that we can later slot in implementations that optimize for speed vs peer-reliability vs price.

What are the inputs required?

• Block being downloaded
• Currently connected peers
  Per connected peer we might want to know:
• Measure of reliability (did they respond to previous messages?)
• Measure of speed (duration of previous block downloads?)
• Price for block (can be omitted for now)

The biggest puzzles are acceptable ways of detecting reliability, speed, and deciding how to remember faulty/unreliable peers. At the moment, we just disconnect from them. But if discovery returns them again, we will reconnect.

Please put your insights here.

[Bulat-Ziganshin]

A stranger's note: for me, it seems that different users may have very different opinions about the best strategy, and moreover - their personal choice of strategy may be super-important for them (it may directly affect their earnings).

If my assumptions are true, it may be a good place to allow external customization, such as a loadable Python script. Not necessarily to implement it ASAP, but you may choose to implement a very simple strategy now, assuming that later you will support external customization and users will soon implement all sorts of strategies and make a "marketplace" of them.

A middle point is to drop an idea of external scripts, but expect that users will drop Nim code with a strategy they've chosen into program sources and compile their own executable. In this case, you just need an extensible API, providing forward and backward compatibility between a strategy code and the main program

[benbierens]

I completely agree. This is one very important reason why we should have an abstraction between the engine and the strategy. I suppose initially we will implement in-code our own "PriceOptimizeStrategy" and "SpeedOptimizeStrategy" or similar. But the abstraction should be such that it could support "ExternalPythonScriptStrategy".

A very important question follows: What are the inputs and outputs of such an abstraction? What information can it see, and what does the output look like?

[Bulat-Ziganshin]

Making a decision early is a waterfall approach, which works great when you have enough evidence. In an agile way, you make quick-and-dirty decisions and change them as you gather more information. If I got this right, changing this decision affects only a few points (one call place plus a few strategies), so the price of changes would be low.

ATM, you have a few strategies in mind. Can you sketch them for yourself, and make a list of input data they need? In the same way, decide which data they should return so you can implement the engine code. So, that will be your API, version 1.

Then, make a quick-and-dirty strategy engine:

1. make it a Nim class. This will allow you to add later more events to it, like onLoad and onTimer. In particular, you may want to send data, necessary for a strategy, from different places of the main code.
2. if possible, implement in the base class a simple strategy, which is enough for low load. E.g. "send all blocks", "choose a random peer", things like that
3. make a separate directory for strategies of this kind, and put each strategy in a separate source file
4. implement a config/cmdline parameter choosing a strategy from the fixed set
5. create the strategy object at the program start and call its methods, so it can keep an internal state between calls

At this point, you will have a basic plugin engine. Then

1. help strategies to self-configure by passing the contents of the appropriate config-file section to their onLoad event
2. make each strategy register itself at the program start with a name, and implement a universal code that chooses from all registered strategies using this name
3. modify build/link scripts to include in the program all .nim files from the strategy directory

At this point, one can add a 3rd-party strategy to the program just by dropping its Nim file into the strategies dir, and adding necessary parameters to the config file. And the next stage is making an extensible API, which means that server v2 will be compatible with clients v3 and vice versa, using an approach similar to Protobuf.

The biggest puzzles are acceptable ways of detecting reliability, speed, and deciding how to remember faulty/unreliable peers

Can you move the "puzzle" parts into the strategies? My experience with an extensible program tells me that a program community enjoys solving "artistic" problems — i.e., ones that don't have a fixed solution. The point is that "engineering" problems have only one solution, while for "artistic" ones, everyone can make his variant and compete with other's solutions. And humans enjoy to self-express and compete :)

Such plugins with a simple API may also be low-hanging fruit for people interested in contributing to the program without learning its entire codebase, e.g. students, hobby programmers...

PS:

• How does an open-source programmer create an OS?
• He creates a project at GitHub and calls for contributors.
  :)