-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bitswap Retrieval, Sync version #86
Conversation
add context key based blockstore and routing implementations that limit results for a bitswap session to the retrieval in question
@@ -156,6 +160,8 @@ type RetrievalEvent interface { | |||
// StorageProviderId returns the peer ID of the storage provider if this | |||
// retrieval was requested via peer ID | |||
StorageProviderId() peer.ID | |||
// Protocol | |||
Protocols() []multicodec.Code |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We've gone plural for Protocols but singular for StorageProviderId. Some events (Candidates*
) return an empty peerID for a call to StorageProviderId()
because they have a plural form that makes sense for them, StorageProviderIds()
. Aren't we going to end up in a situation where the majority of events have a single protocol in the slice returned from this? How many will it make sense to have the plural form?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
arguably, the right thing here is to just return retrieval candidates, IMHO
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but actually, I think storage provider id indicates that an event is associated with a specific storage provider. protocols indicates all the protocols it applies to. Honestly? I'm not sure what the right thing is.
CandidateSplitter: NewProtocolSplitter([]multicodec.Code{multicodec.TransportGraphsyncFilecoinv1, multicodec.TransportBitswap}), | ||
CandidateRetrievers: candidateRetrievers, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
still not a fan of how these two arrays have to match up for this to work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and, what's going to happen when our candidateRetrievers
only has graphsync but the splitter is considering both? Do we need a NullRetriever()
to take its place or something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no it's setup properly to just ignore
CoordinationKind: types.RaceCoordination, | ||
CandidateSplitter: NewProtocolSplitter([]multicodec.Code{multicodec.TransportGraphsyncFilecoinv1, multicodec.TransportBitswap}), | ||
CandidateRetrievers: candidateRetrievers, | ||
CoordinationKind: types.RaceCoordination, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is going to get weird if/when we have both strategies using the same linksystem.
The CAR(s) shouldn't double-write if both bitswap and graphsync both fetch the same block and try and put it, but we ought to test that this works cleanly and we don't end up with weird CARs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea I know we'll fix in the streaming PR
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
Co-authored-by: Rod Vagg <[email protected]>
cleanup errors on the multiblockstore
to be safe, make the changes to counts in bitswap retriever atomic
"github.com/stretchr/testify/require" | ||
) | ||
|
||
func TestBitswapRetriever(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some nice tricks in here to make the test less verbose 👌
Co-authored-by: Rod Vagg <[email protected]>
…sterify fix: improve flakes, make test runs faster
fetches from two seperate peers over bitswap, had to change the provider
ed8b56d
to
827fcd7
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #86 +/- ##
==========================================
+ Coverage 50.09% 59.27% +9.18%
==========================================
Files 36 42 +6
Lines 2703 3202 +499
==========================================
+ Hits 1354 1898 +544
+ Misses 1288 1204 -84
- Partials 61 100 +39
|
g.seq++ | ||
p, err := p2ptestutil.RandTestBogusIdentity() | ||
if err != nil { | ||
panic("FIXME") // TODO change signature |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's going on here, are we tied to a particular API for this struct or is this just an annoying TODO to get to later because it's too cumbersome for now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
copied from go-bitswap # oy
httpServer, err := httpserver.NewHttpServer(cctx.Context, address, port) | ||
|
||
// create a lassie instance | ||
lassie, err := lassie.NewLassie(cctx.Context, lassie.WithProviderTimeout(20*time.Second)) | ||
if err != nil { | ||
return err | ||
} | ||
|
||
httpServer, err := httpserver.NewHttpServer(cctx.Context, lassie, address, port) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
if cfg.Host == nil { | ||
var err error | ||
cfg.Host, err = internal.InitHost(ctx, multiaddr.StringCast("/ip4/0.0.0.0/tcp/6746")) | ||
if err != nil { | ||
return nil, err | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Co-authored-by: Rod Vagg <[email protected]>
various minor fixes to optimize behavior
Goals
Get bitswap retrieval working
Implementation
more information coming (to be delivered sync for now)