Dynamic shards

[ivakegg]

We need a mechanism such that when the size of the data in a day's worth of shards gets too large, then ingest will automatically start using the next set of shards. For example if the number of shards is currently 10, then once some configurable threshold is met then shards 10-19 will be used instead of 0-9.

[ivakegg]

The idea is to be able to configure the ShardIdFactory with classes that implement some interface which can be used to override the shard id for an event. The method needs to be passed in the event (of course), the shard id that the factory would have calculated, and the max number of shards for the event.