Linux pipes and open3
Three articles on how
open3 is implemented in Linux, and how it is properly used
Open3 is a common name for a function that spawns a process and
substitutes its standard input, output and error streams with the pipes
connected to the original process. This way you may interactively feed the
child with data, and read the processed data back.
The problem with open3 is that its power renders it more dangerous, and
its usage—more error-prone. As the famous Spider-Man quote shares, "with
great power comes great responsibility", and Linux inter-process communication
is not an exception. So here comes the effective usage advice number 1.
Avoid open3 unless necessary
For the purpose of clarity, throughout the post, standard input, output, and
error file handlers are called "standard filehandlers"; the process that
spawns, and the process that is spawned are the "parent", and the
"child" respectively.
The first advice about a safe open3 usage is... to avoid its usage.
Languages that are popular in Linux world usually come with a handful of tools
for process spawning, such as:
Shell redirection instead of pipes
Instead of using open3 in your program, you may try to connect process inputs and outputs with Bash redirection utilities, invoking the shell via standard functions to spawn processes.
For instance, you may write system("find | gzip >archive.gz") in Perl to
spawn a process that archives recursive directory listing into a file, instead
of passing the data through the parent process. You may use a usual
redirection into just a text file as well, if that's what you really need.
However, I wouldn't use intermediate files if they're only to dump information
temporarily for an immediate processing by something else, like this:
I consider it a bad stlye, let alone that it requires extra time on disk
access and synchronization, while it may be totally unnecessary. The
arguments to the commands passed that way will also undergo the interpretation
by the shell, so you'll need to escape your arguments, a painful and an
error-prone process.
If you need the intermediate result saved in the files for debugging purposes,
you stil may dump it in debug mode, saving cycles when your project is
compiled for production usage.
- system — spawn the process,
inheriting parent's standard file handlers;
- fork+exec — ditto, but, possibly, with additional customizations
you learned about in the previous
post, for instance (such
as tuning environment or playing with file handlers);
- popen—open only one pipe to the
process, either for reading or writing (note that popen also suffers from
some of the typical misuses of pipes).
- shell redirection untilities—Bash has a number of measures to take
control over how to output and redirect input and output of the process. A
section in one of my older posts is devoted to pipes in Bash. What could
be helpful is the ability of using shell interpreter when spawning a process
with such functions as
system in Perl.
For instance, you may write system("find | gzip >archive.gz") in Perl to
spawn a process that archives recursive directory listing into a file, instead
of passing the data through the parent process (see side note). You may use a
usual redirection into file as well, if that's what you really need.
If one of these tools fulfills your needs, then go for it! You can replace it
with open3 at any time anyway—unlike houses, software is relatively easy
to rebuild.
Where open3 should be used
However, if you'll find the alternatives listed above inefficient, or not
powerful enough, you may opt our for using open3. I'll try to enumerate
the cases where I would certainly advise using open3.
What is select?
The select function (man page) is a
common name for a function to wait on file handlers. Assume you develop a
task scheduler that connects to several machines via sockets; it should wait
for the first socket to have the result in to schedule the next task to that
specific machine (since others are obviously busy). How do you accomplish
that?
Of course, there is a faster and more easy-to-use alternative than trying a
non-blocking read on each socket once per several milliseconds. It is using
the select() function (or one of its companions, such as poll() or
kpoll) to perform a blocking wait on all of them. This function
will return the filehandler list of those that have the data readily available
in them—as soon as there will be at least one such descriptor!
You may find a plenty of manuals on how to use select; later in this post
you'll see an example.
- several consumers of one source—if you are given a source of data
that should be redirected to several consumers (such as some data that should
be both printed onto the terminal and saved into an archived log file for
history keeping reasons), you should connect a pipe to its output and error
streams, and when you read some amount of data from there (for instance, when
you read a line), send it to all the consumers you have. In some cases
you can do it with Bash (not with sh), but the code with open3 should
look much more clear.
- aggregation of several providers—if you maintain several
data sources, which all should be aggregated into a common place (a text file,
for instance) that has some concurrency issues, you might benefit from
open3. You may spawn the processes, and then select() (see
sidenote) their output streams, thus making writing of the data to the
aggregator thread-safe.
- interactive data exchange with a continuously-running process—you
can't avoid open3 if you spawn a process that responds to your input with
some text to stdout. A lot of axillary processes that are launched thousands
of times, have the interface of reading from stdin, and writing to stdout
(such as SAT
solvers), and
you most likely don't have the time to write or read anything from the
disk.
One of the examples of the application of these patterns (more specifically, a
combination of the first and the second) is transparent logging—it's a
blatant crime to use anything but open3 for this. By transparent logging
I mean combining logs from the child processes into a common logging sink in
the parent one. Usually it's done automatically: the system call just
routes standard output of the child to that of parent. However, assume you
spawn several concurrent processes, and unless you prefix each of them with a
unique identifier, you'll get lost in the log quickly.
This may be solved by opening these processes with open3, and attaching
prefixes to their output lines before printing them. Note also that this way
you may control severity of logs: for instance, you might want treat standard
error and output streams from the children differently, and that can not be
achieved with a simple popen call.
Synchronous vs. asynchronous
In this post we will mostly study issues with a synchronous processing of data
exchange with the child process. Being synchronous means that nothing else
but the interaction with the process is performed while the child is alive.
In other words, there is only one thread in the parent.
Some notes on asynchronous processing will be given at the end of this post.
Still, I consider synchronous experience extremely valuable, as it helps to
shape the vision what a good asynchronous interaction via open3 should
be.
Issues with open3
So far I've been telling that open3 is not straightforward to use, but
what causes thee complexity? Why could there be the "Dead Locks" referenced
in the title of this post? I had to learn this in the hard way, tacking
cumbersome bugs in our project, and
here's what it was about.
Following the pattern "aggregation of several providers", I spawned the
process that wrote to both stdout and stderr with the intent to combine them
both in the archived log file. The code (simplified) looked like this:
I expected to get a file that has all the lines from the stdout of the child,
and prefixed lines from the stderr of the child afterwards. However,
sometimes the application just deadlocked!
A quick strace demonstrated that the parent hung up on read from
child's stdout, and the child hung up... on write to its stderr! How
could that be?
Remember that in the first post about
pipes, I listed the
limited capacity as one of the properties of the pipes. I stressed it on
purpose, because it plays its role right here, when you try to use open3.
The pipe that connected the parent to the child was full, and the child wanted
to write an error message there. While the parent was still reading from the
output pipe, because the child was still running, and its pipes were not
closed! That's the open3 Dead Lock.
Of course, this could happen with any pair of pipes here. Assume the process
just echoes every character it takes on the input to the output (of course,
real programs will be doing a useful transformations, but for clarity we may
assume it as identical). We want to feed it twice as much characters as the
pipe's capacity.
You might think that this won't strike you unless you're dealing with
extremely long inputs and outputs. Sorry to disappoint you, but, quoting the
man 7 pipe:
In Linux versions before 2.6.11, the capacity of a pipe was the same as the
system page size (e.g., 4096 bytes on i386). Since Linux 2.6.11, the pipe
capacity is 65536 bytes.
It's not much, though, in certain cases, it's big enough to let badly written
programs work. On a larger scale, we definitely need a generic,
limit-agnostic solution.
How to prevent the dead lock
It's relatively simple to devise a generic rule of mitigating the effect of
such a limitation. To avoid deadlocks with open3, you should clear
each of the output pipes (and fill the input pipe) as soon as possible, and do not put a dependency between clearing a pipe and waiting for another pipe. So
we need to watch closely to all the file handlers (up to 3) which we plan to
read/write data from/to—and it's important that you wait for all of them at once!
If you read the sidenote above, you already know
that we could use select for this.
Using select to react to input promptly
So, a potential way to fix the program above is to write something like this:
This program, however, only outlines the approach to tackling deadlocks with
open3. It is still prone to deadlocks, though less than the original one.
Assume that the child started to print a line to its stdout, but haven't
finished it, because it got a debugging interrupt. Having not finishing printing
the line, it spits 100 Kb of debugging information to stderr with the intent
to continue printing to stdout the normal output. However, the
parent is still blocked in the out.readline() call, waiting for the line
termination character to appear there, and the child gets blocked at the write
to stderr, because the err pipe is full, and no one's going to remove
data from it. Deadlock again. (You may play with this deadlock by
open3-ing this sample
program).
The issue here is that we still do not "remove data from pipes as soon as
possible". For that, we need nonblocking reads, more low-level than
those of the readline()- and scanf-like functions.
Using nonblocking reads and your own buffers
The problem with nonblocking low-level reads, as Capt. Obvious notes, is that
they are low-level. We can't read more or less structured data from them.
Assume that we want to read a number (a number of seconds to wait before
launching the starship, for instance) from the child's stdout. If that
debugging interrupt described above is triggered just in the middle of
printing 1000, our nonblocking read will read 10 (before turning to
reading from stderr), and act accordingly, launching the multi-billion-dollar
ship prematurely. From the child's viewpoint, however, doing so is totally
legitimate, since it printed 1000 and debugging information to the
different output channels (stdout and stderr), and if the reader confuses
these numbers, it's its problem.
Do not use strings as buffers (like I do here)
In the samples below we used "strings" as buffers. However, strings in the modern scripting languages (including those used here, as it's Ruby) consist of multi-byte characters with variable length (see Joel's post on Unicode), and not with one-byte symbols. On the other hand, in some less modern and "less scripting" languages, strings can not contain zero bytes, as they would be treated as the end of the string.
Therefore, "strings" are going to misbehave if chosen as a buffer for an abstract byte stream. I used them for simplicity, and for the sake of demonstration of open3 usage; in real programs, however, you should not use them.
Therefore, we need to handle these situations accordingly, adding another
level of indirection between the child and the parent. We will store the data
we read from pipes in the intermediate storage; we could name it a
buffer. In fact, we are going to re-implement buffered read.
In the next example we will implement a linewise-open3, a subroutine that
invokes user-defined callbacks at reading a complete line from stdin or
stderr. You could play with it more, introducing scanf-like behavior
(instead of these regexps you'll see). However, we have two more issues to
discuss before getting to the code.
Reactive interaction with both input and output data
The select with callbacks works well for reading output with the input
pipe closed at all. What should we do to if we want to write something to the
input pipe of the child based what's being read from it?
To talk interactively with the child, you'll most likely need an asynchronous
processing. However, there is one pattern which allows the exchange of data
through both the input and the output in the synchronous mode. We already
agreed that we will invoke user-defined callbacks after reading lines from
stdin and stdout. However, we didn't use the return value of these callbacks
in any way! The idea about this arises immediately:
If the user-defined callbacks to stdout and stderr lines return a non-empty
string, we feed this string to stdin of the child as soon as possible. To get more control over the child, we treat NULL return value from a callback as a sign to close the standard input. We will also make our wrapper get a string
as an input and feed it at the very beginning, as it is what could trigger the
outputting of the data in the first place.
This still sounds not powerful enough, but you still have aplenty of
asynchronous options, such as interrupting
pselect with a specific signal, or
setting up a separate thread for feeding data into the input pipe. We will
omit these options in this blog post.
Watching for the child termination
As we're implementing a synchronous open3 wrapper, the natural assumption
would be that at return from the wrapper the child should be dead and reaped.
This way we can also return its exit status to the parent.
As we discussed in the previous
post, process
termination does not depend on the status of the pipes, so we should watch for
it independently.
What we actually want is a version of select that watches for
filehandlers and for the child's termination. If you know such a version
of select (which I don't), make a comment, please. For now, let's search
for another solution.
Such functionality could be simulated with a special wrapper thread that
only waits for the child (with wait),
and sends signal at its termination. This signal would interrupt select,
and we would handle that.
In our project I implemented a simpler
solution. It is based on the assumption that the more time the child is
running, the less it's likely to terminate during the next second. So we
can use timely wakeups to check for a process status (implemented as a
non-null timeout to select), and
increase the wait period with the course of time. Having the upper boundary
for that period is a good idea as well.
Note that if a process has terminated, and the pipes are still open, we assume
that the last nonblocking read will fetch us all the data we're interested in,
and we may close the pipes. This may not be true, but in such specific cases
you'll need specific solutions anyway.
Linewise-open3 code
Let's sum up what we're up to. Here's a listing of an open3 wrapper that
prints the string supplied into stdin of the child, and then invokes one of
two user-specified callbacks when a complete, \n-terminated line is read
from stdin or stdout respectively. These callbacks may return more data to
put into the stdin of the process. The execution terminates when the child is
terminated. The wrapper returns the return code of the process (everything
else is assumed to be done by callbacks).
I tested this program on random echo. You may also view the complete listing for open3 usage, and test it either with echo or with sshfs installed on your system.
Note also, that in our project we have also developed an open3 wrapper for Perl; you can view it here. It is less capable (without interactiveness), but it's live and working.
Notes on asynchronous open3 usage
In some cases you may trade the complexity for efficiency. The problem with
the dead lock above was that we had a single thread designated to read from
several pipes. Instead of introducing a buffering system, we may just spawn
several threads, and attach each of them to its own endpoint. The burden of
waking up the thread that actually has something to process doesn't vanish,
it just gets imposed on the OS scheduler, which should contain fewer bugs.
Or, to a language runtime scheduler (if it employs green threads, as in Ruby).
This may sound like an easier solution, especially in the multicore era, where
developers cheer upon every possibility to develop a multithreaded program that
makes the high-end hardware stall less. To me, it's a bit of an overkill for
simple tasks (if the can be expressed in the terms of the interface discussed
above). And in the context we used open3 in our project, too many
competing threads had already started to become a problem.
Perhaps, I'll study the asynchronous option in other posts.
Conclusion
This post concludes what I initially planned for the series of blog posts of
how to work with pipes in Linux. Another topic emerged, on the asynchronous
interaction with an open3-driven process, but I don't know if I will write
about it.
We have observed what are pipes, how they are used for inter-process
communication, what options we have for spawning processes with pipes attached
to them, and how it may be achieved in modern Linux scripting languages.
However, we mainly focused on open3, studying details of its
implementation, and the use-cases it's the most efficient in. We studied its
quirks and how to avoid the traps set up among the joint of pipes we have to
deal with.
I have learned this all spending a couple of days messing with processes that
magically deadlocked without any obvious reasons, and with nontrivial
multithreaded debugging. I hope these posts will help you when you will be
working with pipes, so that you'll avoid my mistakes.
