Can't pipe stdout/stderr to the same pipe for a child process

[steveklabnik]

Issue by alexcrichton
Monday Nov 04, 2013 at 23:24 GMT

For earlier discussion, see rust-lang/rust#10269

This issue was labelled with: A-io in the Rust repository

---

This is more of a problem with the rt::io::process bindings rather than the std::run bindings (the latter needs a larger overhaul in general). The rt::io::process bindings currently don't expose the ability to have something like 2>&1 and then you capture the output of &1 (redirecting stderr to stdout and then reading stdout).

A possible solution is to add another enum variant to StdioContainer named along the lines of PipeTo(int) which means that the output of this stdio slot should get redirected to the pipe which was defined at slot int. Perhaps an IoError should be raised if it doesn't reference a CreatePipe slot, perhaps a failure.

[nerdrew]

Is there a way to do this? Is this going to get addressed ever?

[alexcrichton]

This is possible through FromRawFd and FromRawHandle for std::process, so closing.

[peterhuene]

The docs for FromRawFd claim:

This function consumes ownership of the specified file descriptor. The returned object will take responsibility for closing it when the object goes out of scope.

If that's true, how can you create two different Stdio objects to redirect the child process stdout and stderr using from_raw_fd if both own the descriptor?

That is to say:

Command::new("ls")
    .arg("-l")
    .stdout(unsafe{ Stdio::from_raw_fd(x) })
    .stderr(unsafe{ Stdio::from_raw_fd(x) })
    .spawn()

Will that double close file descriptor x?

Edit: looking at the code, it looks like Stdio's implementation of FromRawFd treats the descriptor as "not owned", so is it correct to assume this is a non-issue despite the documentation?

[sfackler]

That does seem reasonable, though we should update the docs on Stdio to note that fact.

If it did close the file descriptors, you could use the libc dup function to keep things from breaking.

[oconnor663]

A Stdio object does close its file descriptor when it drops. That's because it contains a FileDesc object, and because FileDesc implements Drop. @sfackler is right that if you want to use the same descriptor in two places, you need to use dup. (The standard library actually exposes dup in a roundabout way, via File::from_raw_fd and the File::try_clone method.)

[peterhuene]

@oconnor663 Thanks for the correction! I failed to see the Drop for FileDesc which is where the close of the descriptor occurs (for some reason I thought a RawFd was being stored and not a FileDesc).

Explicitly duplicating the descriptor seems like an easy, if platform-specific, approach to making this work properly. However, it would be nice if we could do something like .stderr(Stdio::redirect_to_stdout()) and have the stdlib handle the platform-specific details behind the scenes.

[oconnor663]

@peterhuene for what it's worth there is a stderr_to_stdout method in my duct crate :-D

[WGH-]

Hey, maybe reopen this? As @peterhuene said, .stderr(Stdio::redirect_to_stdout()) would be very useful. The hacky way of duplicating file descriptors is not particularly obvious.

For example, Python has this - https://docs.python.org/3/library/subprocess.html#subprocess.STDOUT - and it's pretty neat.

[oconnor663]

@WGH- if you want to set up pipes yourself, another option is to use the os_pipe crate. That's what duct uses internally, and the pipes implement Into<std::process::Stdio>.

[WGH-]

@oconnor663 that's a good suggestion, but it still doesn't solve the main problem.

You cannot use child.stdout.take() because child.stdout is initialized only when you use Stdio::piped().

You can't wrap the other end of the pipe with tokio::fs::File, because it doesn't work well with non-blocking fds, returning WouldBlock error instead of waiting properly.

You can't construct tokio::process::ChildStdout from a raw fd, either.

Unless I'm missing something, your only option seems to be to basically duplicate code from here, which is pretty low-level and far from portable:
https://github.com/tokio-rs/tokio/blob/4c4699be00bf4aee6218f84ec606e043286a8c2a/tokio/src/process/unix/mod.rs#L140-L231

[WGH-]

Err, sorry, I kind of conflated this with tokio problem. Right, in "vanilla" Rust it's indeed relatively easy to just wrap the other end of the pipe and go from there.

[ofek]

Is this possible yet?

[Aavtic]

For what it's worth. I found a way to get the stdout and stderr of any command on unix systems using some pipe magic.

Here is what I did:
In Rust process::Command crate you can do:

    let command = r#"bash -c "python3 main.py 2>&1""#;
    let mut proc = Command::new("bash")
        .arg("-c")
        .arg(command)
        .stdout(Stdio::piped())
        .spawn()
        .expect("Failed to execute command");

    let mut stdout = proc.stdout.take().unwrap();
    //let mut stderr = proc.stdout.take().unwrap();

    let mut buffer = String::new();

    let _ = stdout.read_to_string(&mut buffer);

It pipes bash into bash and then inside that we should specify out command and along with the stderr redirection 2>&1.

[zachs18]

On nightly you can use the anonymous_pipe feature to do this, since std::io::PipeWriter has try_clone, and can be converted into Stdio: https://doc.rust-lang.org/nightly/std/io/fn.pipe.html rust-lang/rust#127154

#![feature(anonymous_pipe)]

fn main() {
    let (reader, writer) = std::io::pipe().unwrap();
    let child_stdout = writer.try_clone().unwrap();
    let child_stderr = writer;
    // some command that outputs to its stdout and stderr
    let command = std::process::Command::new("bash")
        .args(["-c", r#"echo stdout && echo stderr >&2"#])
        .stdout(child_stdout)
        .stderr(child_stderr)
        .spawn()
        .unwrap();
    let contents = std::io::read_to_string(reader).unwrap();
    dbg!(contents); // "stdout\nstderr\n"
}