How should read and write operations be used?

[ablagoev]

At the moment there are two exported structures for read/write operations:

pub struct ReadOperation {
    pub object_name: String,
    /// flags are set by calling LIBRADOS_OPERATION_NOFLAG |
    /// LIBRADOS_OPERATION_BALANCE_READS
    /// all the other flags are documented in rados.rs
    pub flags: u32,
    read_op_handle: rados_read_op_t,
}

pub struct WriteOperation {
    pub object_name: String,
    /// flags are set by calling LIBRADOS_OPERATION_NOFLAG |
    /// LIBRADOS_OPERATION_ORDER_READS_WRITES
    /// all the other flags are documented in rados.rs
    pub flags: u32,
    pub mtime: time_t,
    write_op_handle: rados_write_op_t,
}

read_op_handle and write_op_handle are, however, private and there aren't any methods to create the structs in the crate itself. So how are read/write operations supposed to be used, or this is something like a work in progress and they are not usable yet?

Thank you.

[cholcombe973]

So it depends on what you're trying to do. Are you looking to write to rados objects synchronously? If so then the api calls you're looking for are this: rados_object_write and rados_object_read. https://github.com/ceph/ceph-rust/blob/master/src/ceph.rs#L843. If you're looking to write to rbd's ( block device's) then you'll want to look at the https://github.com/cholcombe973/ceph-rbd library.

The ReadOperation and WriteOperation structs were created to mirror ceph's api where you can stack up a bunch of requests and have them all filled at once. http://docs.ceph.com/docs/master/rados/api/librados/#c.rados_read_op_operate. I'll see if i can provide some more examples about how to use this. I realize the api is weird but that's because the underlying C api is weird. Every C library I've worked with seems to approach async/sync operations slightly differently.

[cholcombe973]

If it helps I have a decent example of how to use the sync api here: https://github.com/cholcombe973/preserve/blob/master/src/backend/ceph.rs

[ablagoev]

Hey,

Thank you for taking the time to answer my question and for maintaining the crate. It is much appreciated.

I wasn't clear enough, so I will try to explain further. I understand the read/write operations on objects and the crate does its job flawlessly in this regard.

There is however a feature in librados, for batching multiple operations (read or write) into a single atomic operation:

http://docs.ceph.com/docs/jewel/rados/api/librados/#librados_8h_1a36fb813c7253dd8af8ff063bc6101412

CEPH_RADOS_API rados_read_op_t rados_create_read_op(void)
Create a new rados_read_op_t write operation. This will store all actions to be performed atomically. You must call rados_release_read_op when you are finished with it (after it completes, or you decide not to send it in the first place).

This way you can combine a single read with a getxattrs and some other options into a single operation, instead of sending separate commands to the ceph backend.

Looking through the source code of this crate there is a method rados_perform_read_operations:

pub fn rados_perform_read_operations(&self, read_op: ReadOperation) -> RadosResult<()>

Which as you can see takes a ReadOperation structure. So the Rust wrapper appears to be exposing this part of the C API through the rados_perform_read_operations method and ReadOperation structure.

There is however no way to initialize a valid ReadOperation when using this crate, because the read_op_handle field is private and I don't see any methods to initialize the ReadOperation. So it seems to me like maybe a bug, or a work in progress, or it is me not having a complete understanding on how to work with the crate.

I hope I managed to explain it better this time.
Again thank you for your help and time. :)

Edit

I realize you also pointed to the rados_read_op_operate C API in librados, which is exactly what I am looking for too.

[cholcombe973]

Yeah you're right. It looks like I partially implemented this api. Would you be interested in taking that up? If not I can probably finish it.

[cholcombe973]

@ablagoev would you like to add on to my branch master...ceph:multi-operation? I'm not sure why but it looks like I started it, maybe got distracted or whatever and never finished it.

[ablagoev]

Hey,

Yes, absolutely. I lack the experience, though, so I'll probably need some guidance and help with polishing. I think I'll manage to work on this and have something ready in a week or so.

In the meantime, since this is also something I am interested in (sorry for sidetracking). Have you worked on, or thought about the wrapper to the async methods and what approach would you use there? As far as I know the Futures crate in Rust is the goto implementation for asynchronous computations and it plays well with other libraries. Would you consider an approach, which wraps the async methods from librados around it?

Again, thank you for taking the time.

[cholcombe973]

Awesome! Yeah I'd be happy to give you any help you want polishing it.

As for async io I hadn't thought about it in a long time. I recently figured out how to get async working more or less for libnfs and now that I look at the async librados calls again I have some ideas about how this could work. I still kinda struggle with the Futures crate and I've had trouble figuring out how C and Rust futures can play nice together. Maybe I'll see if I can mock something up in the next week and then get back to you on that? If you have ideas about how it could work I'm all for it. My vague initial idea would be to do something similar to what I did for libnfs. It kinda looks like librados drives the async behavior but I'd need to take a closer look at it.

[ablagoev]

That would be great! Thank you.

I am also willing to put in the work with the async API, but my understanding in dealing with C from Rust and passing around void pointers is non existant at the moment. If I have the time, I might experiment a bit too. :)

[cholcombe973]

Yeah we've all been there with the C to Rust interface being opaque. It looks like librados gives us an out just like libnfs did with allowing us to pass a void pointer around. That allows us to pass in boxed user FnOnce functions. I think the rados_aio_is_complete() and rados_aio_is_safe() are the functions that we poll on. I have a vague idea of how this async stuff might work.