Re: encrypting files

From: Ivan Voras (
Date: 05/17/04

Date: Mon, 17 May 2004 22:12:29 +0200

Mok-Kong Shen wrote:

> Douglas A. Gwyn wrote:
> [snip]
>> Standard input isn't necessarily attached to a keyboard.
>> On many systems, it is possible to redirect standard
>> input and output to other data objects, and some of them
>> support "pipes" which connect standard output of one task
>> to standard input of another.

In modern operating system (well, most of them, _including_ Windows), it
is all related to a concept of "file handles". In various circumstances,
  file handles refer to many things: ordinary files, network sockets or,
what's of interest here, "pipes". Pipes are just what it says - a method
of connecting two pieces of code, which enables data to be sent between
them. (I use the terms "pipe" and "file handle" interchangably, because
from a program's (and programmer's) point of view, they can be used that

I'll try and illustrate it on unix: when a program starts, it is
automatically assigned three "pipes": stdin, stdout and stderr. Usually
(except in some bizzare cases), the pipes are unidirectional: stdin can
be read (by the executing program), stdout and stderr can be written to.
The interesting thing is where the other endpoints of the pipes are
connected. Processes (programs) are only started by other processes
(except the "init" which is the root process started by boot loader).
When a process is created, the "parent" specifies its
stdin/stdout/stderror pipes - an example of such thing is the unix
shell: when a shell is started from a network server (e.g. ssh/telnet),
it is assigned pipes so that they refer to the network connection. When
the shell writes to its stdout, it is actually writing to a network
socket. When the shell starts a process it *usually* just hands it down
its own pipes, so they can do the same. But, to make it more convenient,
there are some exceptions to that. For example:

$ ./program < myfile

This leaves stdout and stderr as is, but opens the file "myfile" and
passes the handle of the file to the process started by "./program" as
the stdin. Similary:

$ ./program > otherfile

This one leaves stdin as is (e.g. connected to a user session), creates
a file "otherfile" and passes its handle as stdout, and finally *leaves
stderr as is*. Some shells can redirect stderr with >& or something
similar. Combinations are possible:

$ ./program < file1 > file2 >& file3

This example redirects all pipes: stdin, stdout and stderr to respective
files. Many combinations are possible, including "intercepting" a
program's output (by a pair of pipes), encrypting it, and then sending
it to a network socket (which is what ssh does).

>> The gamut of code values that can be generated by keystrokes
>> varies widely from system to system. Some systems may
>> support an "escape mechanism" that lets one compose any
>> arbitrary code, e.g. the input sequence SO,d,9,SI might be
>> mapped into the single byte value hexadecimal D9.

This is handled by a concept of "terminal emulator". Terminal emulators
can exist at different levels, but their purpose is translation from
user-input data into a pipe like the above examples, and from
program-generated data into something the user sees. A typical example
are terminal codes. If you create a file containing only one byte - a
chr(7), and try to view it with cat, you will not get anything on the
screen, but instead you will hear a terminal "beep" sound. If you are
using ssh, this is what happens when you start "cat beepfile":

- cat openes the file, reads the contents (1 byte in this case), outputs
it to its stdout
- ssh takes the byte, "does something to it", writes resulting data to
network socket
- the data travels through the kernel, network, arrives to the
destination computer, where it is handed to the other ssh program via a
network socket
- ssh decrypts the data, and writes it to the screen. It is at this last
step that the process doing the actual writing (a tty console driver, or
putty, or some other ssh client), decides that the byte "means" beep,
and not a visual character. In the same fasion, an "escape sequence" of
bytes may be translated as "set the foreground color to red" or anything
else. What I'm trying to illustrate is that the software that displays
the data decides its meaning at the end point, and there is no data
transformation in the pipes :).

> I would understand 'standard' input to refer to the 'normal'
> manual input by the users (piping a character stream is
> virtually the same, the essence being a character stream).
> Otherwise, why call it 'standard' (why that term)? In Fortran,

I think the term comes from the fact that programs have to have someting
to read from/write to, so there it is - a standard set of IO handles.

Now, from my experience, Windows no longer (at least NT 4.0+) has the
inbuilt support for "text files", and that of course includes
stdin/stdout/stderr. Instead, the runtime libraries and the Console
program do the translation. For example: the runtime library (usually
msvcrt.dll, sort of a windows libc equivalent) detects if the
stdout/stderr are connected to the Console program, and then translates
\n into \r\n. You can move binary data either between programs, or
between programs and files without such translations. (Proof: various
inetd clones for Windows, one of which I wrote myself, which couldn't be
possible otherwise - Windows' pipes are binary safe).

(Albeit, this leaves the possibility that the runtime library is stupid
enough to do the translation always...)

Hope this helps somebody :)

C isn't that hard: void (*(*f[])())() defines f as an array of
unspecified size, of pointers to functions that return pointers to
functions that return void.

Relevant Pages

  • Re: coloring stdout and stderr
    ... everything printed to stdout shows up in green and everything printed to ... stderr becomes red. ... I tried pipes, but without much success, as ... output to a terminal only line buffering is used (i.e. ...
  • Re: coloring stdout and stderr
    ... I want that everything printed to stdout shows up in green and everything printed to stderr becomes red. ... I tried pipes, but without much success, as syncing with the color escapes soon became a nightmare. ... After each call to getline, n contains the size of the malloced buffer, not the lenght of the read line. ...
  • self-tailing output checker
    ... certain user-defined patterns in addition to watching exit status. ... control where output goes via standard redirection, ... it would cause the program which uses it to create two pipes and fork. ... One side of the fork would connect stdout and stderr to the write ends ...
  • Re: Execute an arbitrary program as a child process?
    ... # execand pthreads to start a process which will execute the ... Use pipeto create pipes that will replace stdin, stdout, ... you are the parent and pid is the child ...
  • Re: How to read data from a bash pipe -- thank you
    ... >> Actually this has not anything to do with cin (standard input stream, ... >> usually the keyboard). ... The question was about pipes, ...