Interprocess Communication (IPC) in Linux
Interprocess Communication (IPC) enables independent processes to exchange data and synchronize their execution. Linux provides several IPC mechanisms, including pipes, FIFOs, message queues, shared memory, semaphores, and sockets.
Learning Objectives
- Understand why IPC is required.
- Learn how independent processes communicate.
- Differentiate communication and synchronization.
- Understand IPC mechanisms provided by Linux.
- Prepare for programming with pipes and other IPC techniques.
1. What is Interprocess Communication (IPC)?
Interprocess Communication (IPC) is a collection of mechanisms that allow separate processes to exchange information and coordinate their execution. Unlike threads, which share the same address space, separate processes have independent virtual address spaces. Therefore, they require operating-system-provided communication mechanisms.
2. Why is IPC Needed?
Many applications consist of multiple cooperating processes. Examples include:
- A shell communicating with child processes.
- A web server communicating with worker processes.
- A database exchanging requests with client applications.
- Logging services receiving messages from multiple programs.
Without IPC, these independent processes cannot directly exchange data.
3. Why Can't Processes Share Variables Directly?
Each process executes within its own virtual address space. Variables stored in one process are normally inaccessible to another process.
Although the layouts appear similar, each process has its own independent memory space.
4. Communication vs Synchronization
| Communication | Synchronization |
|---|---|
| Transfers data between processes. | Coordinates execution order. |
| Moves information. | Controls when operations occur. |
| Example: Pipe | Example: Semaphore |
5. Common IPC Mechanisms
| Mechanism | Typical Use |
|---|---|
| Pipe | Parent-child communication |
| FIFO (Named Pipe) | Unrelated processes |
| Message Queue | Structured message passing |
| Shared Memory | High-speed data sharing |
| Semaphore | Synchronization |
| Socket | Local or network communication |
6. IPC Communication Models
IPC mechanisms generally follow one of two models.
- Message Passing
- Shared Memory
7. Message Passing
In message passing, processes exchange discrete messages through the operating system. The operating system manages message delivery.
8. Shared Memory
In shared memory communication, the operating system creates a memory region that multiple processes can access. Processes read and write the shared region directly.
Because multiple processes may access the same memory concurrently, additional synchronization mechanisms are typically required.
9. Characteristics of IPC
- Processes remain independent.
- Communication occurs through operating-system mechanisms.
- Some IPC methods copy data, while others share memory.
- Synchronization may be required to prevent inconsistent access.
- Different IPC mechanisms are appropriate for different workloads.
10. Interview Questions
- What is Interprocess Communication?
- Why can't two independent processes share ordinary variables?
- What is the difference between communication and synchronization?
- Name six IPC mechanisms available in Linux.
- What are the two primary IPC communication models?
- When is shared memory generally preferred over message passing?
11. Anonymous Pipes
An anonymous pipe is one of the simplest IPC mechanisms provided by UNIX and Linux. It creates a unidirectional communication channel between related processes, typically a parent process and its child.
Anonymous pipes normally exist only while the communicating processes are running.
12. pipe() System Call
A pipe is created using the pipe() system call.
On success, the kernel creates a pipe and returns two file descriptors.
13. File Descriptors Returned by pipe()
| Descriptor | Purpose |
|---|---|
| fd[0] | Read end of the pipe |
| fd[1] | Write end of the pipe |
Data written to fd[1] can be read from
fd[0].
14. Pipe Buffer
The kernel maintains an internal buffer for every pipe. When one process writes data to the pipe, the data is stored in this buffer until another process reads it.
15. Parent–Child Communication
A common pattern is:
- Create a pipe.
- Create a child process using
fork(). - Parent writes data.
- Child reads data.
16. Closing Unused Pipe Ends
Each process should close the end of the pipe that it does not use.
| Process | Close |
|---|---|
| Parent (Writer) | Close fd[0] |
| Child (Reader) | Close fd[1] |
Closing unused descriptors avoids resource leaks and allows the kernel to correctly detect end-of-file (EOF).
17. Parent Writes, Child Reads
18. Pipe Execution Flow
19. Characteristics of Anonymous Pipes
- Communication is unidirectional.
- Primarily used between related processes.
- Implemented by the kernel.
- Data flows using file descriptors.
- Temporary communication mechanism.
20. Interview Questions
- What does
pipe()return? - Which file descriptor is used for reading?
- Which file descriptor is used for writing?
- Why should unused pipe ends be closed?
- Why are anonymous pipes normally used between related processes?
- Where is pipe data temporarily stored?
- Why is
fork()commonly used with pipes?
21. Blocking Behavior of Pipes
Pipes are blocking by default. When reading or writing through a pipe, a process may block depending on the availability of data or buffer space.
| Operation | Behavior |
|---|---|
| Read from empty pipe | Blocks until data becomes available or EOF is detected. |
| Write to full pipe | Blocks until sufficient buffer space becomes available. |
22. End-of-File (EOF) on Pipes
A process reading from a pipe receives an end-of-file indication when all file descriptors referring to the write end of the pipe have been closed and no more data remains in the pipe buffer.
23. Why Must the Write End Be Closed?
If at least one write descriptor remains open, the kernel assumes that additional data may still be written. As a result, a reader waiting for more data continues to block instead of receiving EOF.
24. Pipe Buffering
Every pipe contains a kernel-managed buffer. Data written by one process is stored temporarily until another process reads it.
The exact buffer capacity is implementation dependent and should not be assumed by applications.
25. Bidirectional Communication
A single anonymous pipe transfers data in only one direction. To communicate in both directions between two related processes, two pipes are typically created.
26. Two-Pipe Communication
One pipe carries requests from the parent to the child, while the second pipe carries responses from the child back to the parent.
27. dup()
The dup() system call creates a duplicate of an existing file descriptor.
The duplicated descriptor refers to the same open file description as the original descriptor.
28. dup2()
The dup2() system call duplicates one file descriptor onto another specified descriptor.
If newfd is already open, it is closed automatically before
being reused.
29. Redirecting Standard Output
The shell redirects standard input and output by combining
dup2() with file descriptors.
After the redirection, output written to stdout is sent to
the file.
30. Pipes and Shell Redirection
Shell pipelines use pipes together with file descriptor duplication. For example:
The shell creates a pipe, forks child processes, and uses
dup2() so that the standard output of ls
becomes the standard input of wc.
31. Key Points
- Pipes block by default when reading from an empty pipe or writing to a full pipe.
- EOF is observed only after all write ends of the pipe have been closed and buffered data has been consumed.
- Two pipes are commonly used for bidirectional communication.
dup()duplicates a file descriptor.dup2()duplicates a file descriptor onto a specified descriptor.- Shell redirection and pipelines rely on
dup2().
32. Interview Questions
- Why does reading from an empty pipe block?
- When does a pipe reader receive EOF?
- Why are two pipes required for bidirectional communication?
- What is the difference between
dup()anddup2()? - How does the shell implement
ls | wc? - Why is
dup2()widely used for I/O redirection?
33. popen()
The popen() library function creates a pipe, starts a new process to execute a shell command, and returns a standard I/O stream connected to that process.
The type argument determines the direction of data flow.
| Type | Meaning |
|---|---|
| "r" | Read the command's standard output. |
| "w" | Write to the command's standard input. |
34. Reading from a Command
The program executes the date command and reads its output
through the returned stream.
35. pclose()
After communication is complete, the stream returned by
popen() must be closed using
pclose().
The return value contains the termination status of the command executed
by popen().
36. popen() Execution Flow
37. Named Pipes (FIFO)
A Named Pipe, also called a FIFO (First In, First Out), is a special file in the filesystem that allows unrelated processes to communicate. Unlike an anonymous pipe, a FIFO has a pathname and can be opened by different processes.
38. mkfifo()
A FIFO can be created programmatically using mkfifo().
The pathname specifies the FIFO file, and
mode specifies its permissions.
39. Linux FIFO Commands
Create a FIFO from the command line:
Display the FIFO:
The file type is displayed as p, indicating a named pipe.
40. FIFO Communication Example
The communicating processes do not need to have a parent-child relationship.
41. Anonymous Pipe vs Named Pipe
| Anonymous Pipe | Named Pipe (FIFO) |
|---|---|
| No pathname | Filesystem pathname |
| Usually parent-child communication | Unrelated processes can communicate |
Created with pipe() |
Created with mkfifo() |
| Temporary | Persists until removed from the filesystem |
42. Key Points
popen()executes a command and connects it to a pipe.pclose()closes the pipe stream and obtains the command's termination status.- Named pipes (FIFOs) are represented as special files in the filesystem.
mkfifo()creates a FIFO.- FIFOs allow communication between unrelated processes.
43. Interview Questions
- What is the purpose of
popen()? - What is the difference between
popen()andsystem()? - Why should
pclose()be used instead offclose()? - What is a FIFO?
- How does a FIFO differ from an anonymous pipe?
- Which system call creates a FIFO?
- Can unrelated processes communicate through a FIFO?
44. File Descriptor Passing
UNIX systems allow one process to pass an open file descriptor to another process using UNIX domain sockets. Instead of reopening the same file, the receiving process gains access to the already opened file descriptor.
This technique is commonly used by system services that need to delegate access to already opened resources.
45. socketpair()
The socketpair() system call creates two connected UNIX-domain sockets. Unlike a pipe, both ends support communication in both directions.
46. socketpair() Communication
Each socket can both read and write data.
This makes socketpair() useful for full-duplex
communication between related processes.
47. Anonymous Pipe vs socketpair()
| Anonymous Pipe | socketpair() |
|---|---|
| Unidirectional | Bidirectional (Full Duplex) |
| Read end + Write end | Both descriptors support read/write |
| Created using pipe() | Created using socketpair() |
| Simple stream communication | Supports advanced socket operations |
48. Introduction to Message Queues
A Message Queue is an IPC mechanism that allows processes to exchange discrete messages. Unlike pipes, individual message boundaries are preserved.
49. Characteristics of Message Queues
- Messages remain separate.
- Processes communicate asynchronously.
- Multiple messages may exist in the queue simultaneously.
- Operating system manages the queue.
50. POSIX Message Queues
POSIX defines a standardized message queue API. Typical functions include:
- mq_open()
- mq_send()
- mq_receive()
- mq_close()
- mq_unlink()
These functions allow applications to create, use, and remove message queues.
51. System V Message Queues
Older UNIX systems provide the System V message queue interface. Important functions include:
- msgget()
- msgsnd()
- msgrcv()
- msgctl()
Many existing UNIX and Linux applications still use this interface.
52. POSIX vs System V Message Queues
| POSIX | System V |
|---|---|
| Newer standardized API | Traditional UNIX API |
| Named queues | Key-based queues |
| mq_send() | msgsnd() |
| mq_receive() | msgrcv() |
53. Key Points
- UNIX can pass file descriptors between processes using UNIX domain sockets.
socketpair()creates two connected sockets.socketpair()supports bidirectional communication.- Message queues preserve message boundaries.
- Linux supports both POSIX and System V message queues.
54. Interview Questions
- Why is file descriptor passing useful?
- What is the difference between a pipe and
socketpair()? - Why are message queues preferred over pipes in some applications?
- Name two POSIX message queue functions.
- Name two System V message queue functions.
- What is the primary difference between POSIX and System V message queues?
55. POSIX Message Queue Programming
POSIX Message Queues provide a standardized interface for exchanging messages between processes. Unlike pipes, each message remains an independent unit.
56. mq_open()
A message queue is opened or created using mq_open().
| Parameter | Description |
|---|---|
| name | Queue name (begins with '/') |
| oflag | Open flags (O_RDONLY, O_WRONLY, O_RDWR, O_CREAT) |
| mode | Permissions when creating the queue |
| attr | Queue attributes (optional) |
57. Queue Attributes
| Field | Description |
|---|---|
| mq_flags | Queue behavior flags |
| mq_maxmsg | Maximum number of messages |
| mq_msgsize | Maximum size of each message |
| mq_curmsgs | Current number of queued messages |
58. mq_send()
The message is copied into the queue. The last argument specifies the message priority.
59. mq_receive()
The oldest highest-priority available message is removed from the queue and copied into the supplied buffer.
60. mq_close()
This closes the queue descriptor for the current process. The queue itself continues to exist until it is removed.
61. mq_unlink()
The named queue is removed from the system namespace. Existing processes that already have the queue open may continue using it until they close it.
62. Example Flow
63. Advantages of POSIX Message Queues
- Messages preserve their boundaries.
- Supports message priorities.
- Processes can communicate asynchronously.
- Standardized POSIX interface.
- Suitable for producer-consumer applications.
64. Limitations
- Kernel memory usage increases with queued messages.
- Message size limits apply.
- Applications must handle queue-full conditions.
- Generally slower than shared memory for transferring large amounts of data.
65. Interview Questions
- What is the purpose of
mq_open()? - What information is stored in
struct mq_attr? - How does
mq_send()differ from writing to a pipe? - What happens when
mq_close()is called? - Why is
mq_unlink()required? - What are the advantages of message priorities?
- When would a message queue be preferred over a pipe?
66. System V Message Queue Programming
System V Message Queues provide a traditional UNIX IPC mechanism for exchanging structured messages between processes. Unlike POSIX queues, System V queues are identified using IPC keys.
67. Message Structure
Each message begins with a message type followed by application data.
| Field | Purpose |
|---|---|
| mtype | Positive message type identifier |
| mtext | Application data |
68. msgget()
The msgget() function creates a new message queue or opens an existing one.
The key identifies the queue, while
msgflg specifies creation flags and permissions.
69. msgsnd()
Messages are placed into the queue using msgsnd().
The message type is obtained from the mtype member of the
message structure.
70. msgrcv()
Messages are retrieved using msgrcv().
The caller can request a specific message type or receive messages based
on the selection rules defined for the msgtyp parameter.
71. msgctl()
The msgctl() function performs control operations on a message queue.
The IPC_RMID command removes the message queue from the
system.
72. Execution Flow
73. POSIX vs System V Programming
| POSIX | System V |
|---|---|
| mq_open() | msgget() |
| mq_send() | msgsnd() |
| mq_receive() | msgrcv() |
| mq_unlink() | msgctl(IPC_RMID) |
| Named queues | Key-based queues |
| Descriptor returned | Queue identifier returned |
74. Advantages of System V Message Queues
- Widely supported on UNIX and Linux systems.
- Messages retain their boundaries.
- Supports message selection using message types.
- Useful for asynchronous communication.
- Well suited for legacy UNIX applications.
75. Interview Questions
- What is the purpose of
msgget()? - Why does every System V message begin with
mtype? - What is the role of
msgsnd()? - How does
msgrcv()select messages? - Which function removes a System V message queue?
- What is the difference between a queue descriptor and a queue identifier?
- Compare POSIX and System V Message Queues.
76. Shared Memory
Shared Memory is an IPC mechanism that allows multiple processes to access the same physical memory region simultaneously. Unlike pipes and message queues, data is not copied between processes. Instead, all participating processes access the same shared memory segment.
77. Why is Shared Memory the Fastest IPC?
After a shared memory segment has been created and mapped into each process, the processes can read and write directly to the shared region. No additional kernel-mediated data copying is required for each data transfer.
| IPC Mechanism | Data Transfer |
|---|---|
| Pipe | Copied through kernel buffer |
| Message Queue | Copied through kernel queue |
| Shared Memory | Direct access to shared region |
78. shmget()
A shared memory segment is created or accessed using shmget().
| Parameter | Description |
|---|---|
| key | IPC key identifying the shared memory segment |
| size | Size of the shared memory segment in bytes |
| shmflg | Creation flags and access permissions |
79. shmat()
The shmat() function attaches a shared memory segment to the address space of the calling process.
80. Reading and Writing Shared Memory
Once attached, the returned pointer behaves like a normal memory pointer.
Changes made by one attached process become visible to other attached processes.
81. shmdt()
When a process no longer requires access to the shared memory segment, it should detach it.
Detaching removes the mapping from the process's address space but does not remove the shared memory segment itself.
82. shmctl()
Administrative operations on a shared memory segment are performed using shmctl().
Using IPC_RMID marks the shared memory segment for removal.
83. Shared Memory Lifecycle
84. Why Synchronization is Required
Multiple processes may access the same shared memory segment simultaneously. Without synchronization, concurrent updates can result in inconsistent or corrupted data.
Shared memory is commonly combined with synchronization mechanisms such as semaphores or mutexes designed for interprocess use.
85. Advantages and Limitations
| Advantages | Limitations |
|---|---|
| Very high performance | Requires synchronization |
| No repeated data copying | More complex programming |
| Suitable for large data | Manual resource management |
| Efficient for frequent communication | Potential race conditions |
86. Interview Questions
- Why is shared memory considered the fastest IPC mechanism?
- What is the purpose of
shmget()? - What does
shmat()do? - What is the difference between
shmat()andshmdt()? - When should
shmctl(IPC_RMID)be called? - Why are semaphores commonly used with shared memory?
- What happens if multiple processes write to shared memory simultaneously without synchronization?
87. What is a Semaphore?
A Semaphore is a synchronization mechanism used to control access to shared resources. Unlike shared memory, a semaphore does not transfer data. Instead, it coordinates when processes or threads may access shared resources.
88. Why are Semaphores Needed?
When multiple processes access shared memory simultaneously, inconsistent updates may occur. Semaphores ensure that only the permitted number of processes enter a critical section.
89. Binary Semaphore
A binary semaphore has only two states.
| Value | Meaning |
|---|---|
| 1 | Resource available |
| 0 | Resource occupied |
A binary semaphore is commonly used to provide mutual exclusion for a single shared resource.
90. Counting Semaphore
A counting semaphore can represent multiple identical resources.
Each successful acquisition decreases the semaphore count, and each release increases it.
91. System V Semaphore Functions
| Function | Purpose |
|---|---|
| semget() | Create or access semaphore set |
| semop() | Perform semaphore operations |
| semctl() | Control or remove semaphore set |
92. semget()
Creates a semaphore set or obtains access to an existing one.
93. semop()
Performs one or more atomic operations on semaphores in a semaphore set. Typical operations decrement (wait) or increment (signal) a semaphore.
94. semctl()
Performs administrative operations such as retrieving information, setting values, or marking a semaphore set for removal.
95. POSIX Semaphore Functions
| Function | Purpose |
|---|---|
| sem_init() | Initialize semaphore |
| sem_wait() | Wait (decrement) |
| sem_post() | Signal (increment) |
| sem_destroy() | Destroy semaphore |
96. Producer–Consumer Using Semaphores
Semaphores synchronize the producer and consumer so that the producer does not write into a full buffer and the consumer does not read from an empty buffer.
97. Advantages of Semaphores
- Prevent race conditions.
- Provide synchronization between processes and threads.
- Suitable for producer–consumer problems.
- Support both binary and counting synchronization.
- Widely supported by UNIX and Linux systems.
98. Interview Questions
- What is the purpose of a semaphore?
- How does a semaphore differ from shared memory?
- What is the difference between a binary semaphore and a counting semaphore?
- Name the three primary System V semaphore functions.
- What are the purposes of
sem_wait()andsem_post()? - Why are semaphores commonly used with shared memory?
- How do semaphores solve the Producer–Consumer problem?
99. POSIX Shared Memory
POSIX Shared Memory provides a standardized method for sharing memory between processes using named shared memory objects. Unlike System V shared memory, POSIX shared memory objects are accessed using file-descriptor-like interfaces.
100. shm_open()
A POSIX shared memory object is created or opened using shm_open().
The returned file descriptor is typically resized using
ftruncate() before mapping it into memory.
101. mmap()
The mmap() function maps a file or shared memory object into the virtual address space of a process.
102. munmap()
When the mapped region is no longer required, it should be unmapped using munmap().
103. shm_unlink()
A POSIX shared memory object is removed using shm_unlink().
Removal unlinks the shared memory object's name. Existing mappings may remain usable until all references are released.
104. Memory-Mapped Files
Memory mapping allows a file to be accessed through memory rather than explicit read() and write() operations. Applications access the mapped region using ordinary memory operations.
105. UNIX Domain Sockets
UNIX Domain Sockets enable communication between processes on the same system. They provide socket-based communication without using a network.
| Feature | Description |
|---|---|
| Communication | Same host only |
| Addressing | Filesystem pathname or abstract namespace (Linux-specific) |
| Supports | Stream and datagram communication |
| Can pass file descriptors | Yes |
106. IPC Mechanism Comparison
| IPC | Speed | Related Processes | Data Copy |
|---|---|---|---|
| Pipe | Medium | Usually Yes | Kernel Buffer |
| FIFO | Medium | No | Kernel Buffer |
| Message Queue | Medium | No | Kernel Queue |
| Shared Memory | Very High | No | No Repeated Copy |
| Semaphore | N/A | Synchronization | No Data Transfer |
| UNIX Domain Socket | High | No | Socket Buffer |
107. IPC Best Practices
- Select the IPC mechanism appropriate for the application's communication pattern.
- Close or release IPC resources when they are no longer required.
- Use synchronization mechanisms when multiple processes access shared memory.
- Check the return values of all IPC-related system calls and library functions.
- Remove unused IPC objects to avoid resource leaks.
- Avoid assuming implementation-specific limits such as buffer sizes.
108. Comprehensive Interview Questions
- What is Interprocess Communication?
- Why is shared memory considered the fastest IPC mechanism?
- What is the difference between a pipe and a FIFO?
- Compare POSIX and System V Message Queues.
- When should shared memory be preferred over pipes?
- Why are semaphores required with shared memory?
- What is the purpose of mmap()?
- What is the difference between shm_open() and shmget()?
- Why are UNIX domain sockets faster than network sockets on the same machine?
- Which IPC mechanism would you choose for a high-performance database and why?
109. IPC Chapter Summary
- IPC enables independent processes to communicate and synchronize.
- Pipes provide simple communication between related processes.
- FIFOs extend pipe communication to unrelated processes.
- Message Queues preserve message boundaries.
- Shared Memory provides the highest data-transfer performance.
- Semaphores coordinate access to shared resources.
- Memory mapping enables efficient access to files and shared memory.
- UNIX Domain Sockets provide efficient local interprocess communication.
- Choosing the appropriate IPC mechanism depends on communication requirements, performance needs, and synchronization requirements.
Concept Mastery Assessment — Module 20
Verify your conceptual understanding with 12 targeted technical assessment questions. Instant feedback and explanations are provided for every choice.