Advanced I/O in UNIX/Linux
Advanced I/O extends the basic file I/O interfaces by providing additional capabilities for efficient data transfer, file descriptor management, synchronization, memory mapping, and monitoring multiple input/output sources. These interfaces enable applications to build scalable and efficient systems.
Learning Objectives
- Understand advanced UNIX I/O concepts.
- Learn blocking and non-blocking I/O.
- Understand file descriptor manipulation.
- Learn advanced data transfer techniques.
- Prepare for multiplexed I/O and memory mapping.
1. What is Advanced I/O?
Advanced I/O refers to the collection of interfaces that extend the basic input/output operations provided by the operating system. These interfaces allow applications to manage file descriptors, configure I/O behavior, synchronize file data, monitor multiple I/O sources, and improve performance.
2. Why is Advanced I/O Needed?
- Improve I/O efficiency.
- Support concurrent input and output.
- Reduce unnecessary waiting.
- Manage file descriptors.
- Provide random access to file data.
- Improve scalability of applications.
3. Advanced I/O Components
| Component | Purpose |
|---|---|
| File Descriptor Management | Manipulate existing file descriptors. |
| Synchronization | Ensure data reaches stable storage. |
| Memory Mapping | Access files through virtual memory. |
| Scatter/Gather I/O | Transfer multiple memory buffers. |
| I/O Multiplexing | Monitor multiple descriptors. |
4. Blocking I/O
In blocking I/O, a function call waits until the requested operation can be completed before returning control to the application. The calling thread remains blocked during this period.
5. Non-Blocking I/O
In non-blocking I/O, an operation returns immediately if it cannot make progress without waiting. The application can perform other work and retry the operation later.
6. Blocking vs Non-Blocking I/O
| Blocking I/O | Non-Blocking I/O |
|---|---|
| Waits until operation completes. | Returns immediately if the operation cannot proceed. |
| Simple programming model. | Requires the application to handle retry logic. |
| Thread may remain idle while waiting. | Application can perform other work while waiting. |
7. High-Level Advanced I/O Workflow
8. When Advanced I/O is Used
- Network servers.
- Database systems.
- High-performance applications.
- Event-driven software.
- Real-time monitoring tools.
- Large file processing applications.
9. APIs Covered in This Chapter
| API | Purpose |
|---|---|
fcntl() |
Manipulate file descriptor properties. |
dup() |
Duplicate a file descriptor. |
dup2() |
Duplicate to a specified descriptor. |
sync() |
Schedule file system data for writing. |
fsync() |
Synchronize a file's data and required metadata. |
fdatasync() |
Synchronize file data. |
ioctl() |
Perform device-specific control operations. |
readv() |
Scatter read into multiple buffers. |
writev() |
Gather write from multiple buffers. |
mmap() |
Map a file into memory. |
munmap() |
Remove a memory mapping. |
select() |
Monitor multiple file descriptors. |
poll() |
Monitor multiple file descriptors using arrays. |
10. Interview Questions
- What is meant by Advanced I/O?
- Why is Advanced I/O required?
- Differentiate blocking and non-blocking I/O.
- What are the advantages of non-blocking I/O?
- Name the major APIs covered under Advanced I/O.
- What is I/O multiplexing?
- Why is memory mapping considered an advanced I/O technique?
- Which API is used to duplicate file descriptors?
- Which API is used to monitor multiple file descriptors?
- What is the role of synchronization functions such as
fsync()?
11. fcntl()
The fcntl() system call performs various operations on an existing file descriptor. It can retrieve or modify descriptor flags, file status flags, file locking information, and several other file descriptor properties.
12. Why is fcntl() Needed?
- Retrieve descriptor information.
- Modify file status flags.
- Enable non-blocking I/O.
- Duplicate file descriptors.
- Configure descriptor behavior.
- Support file locking operations.
13. Common fcntl() Commands
| Command | Purpose |
|---|---|
F_GETFD |
Get file descriptor flags. |
F_SETFD |
Set file descriptor flags. |
F_GETFL |
Get file status flags. |
F_SETFL |
Set file status flags. |
F_DUPFD |
Create a duplicate file descriptor. |
14. File Descriptor Flags
File descriptor flags describe properties associated with a specific file descriptor. These flags are maintained separately for each descriptor, even if multiple descriptors refer to the same open file description.
| Flag | Description |
|---|---|
FD_CLOEXEC |
Automatically closes the descriptor during a successful exec operation. |
15. File Status Flags
File status flags describe how an open file is accessed and how I/O operations behave. These flags belong to the open file description and are shared by file descriptors that reference the same open file description.
| Flag | Description |
|---|---|
O_APPEND |
Append data to the end of the file. |
O_NONBLOCK |
Enable non-blocking operations. |
O_SYNC |
Perform synchronized writes. |
O_ASYNC |
Enable asynchronous notification where supported. |
16. F_GETFL
The F_GETFL command returns the current file status flags
associated with an open file description.
17. F_SETFL
The F_SETFL command modifies selected file status flags of
an open file description.
A common use is enabling or disabling non-blocking mode.
18. Example: Non-Blocking File Descriptor
19. fcntl() Workflow
20. Interview Questions
- What is the purpose of
fcntl()? - What is the difference between file descriptor flags and file status flags?
- What does
F_GETFLreturn? - What is the purpose of
F_SETFL? - Why should existing flags be read before calling
F_SETFL? - What is
O_NONBLOCK? - What is the purpose of
FD_CLOEXEC? - Which command retrieves descriptor flags?
- Which command retrieves status flags?
- Can multiple file descriptors share the same file status flags? Explain.
21. File Descriptor Duplication
File descriptor duplication creates a new file descriptor that refers to the same open file description as an existing descriptor. Both descriptors can be used interchangeably for input or output operations.
22. dup()
The dup() system call creates a duplicate of an existing file descriptor. The kernel returns the lowest-numbered unused file descriptor.
Returns the new file descriptor on success.
Returns -1 on failure.
23. How dup() Works
Both file descriptors reference the same open file description.
24. Resources Shared by Duplicated Descriptors
| Resource | Shared? |
|---|---|
| File Offset | Yes |
| File Status Flags | Yes |
| Open File Description | Yes |
| File Descriptor Flags | No |
25. Example: Using dup()
26. dup2()
The dup2() system call duplicates one file descriptor onto a specified file descriptor number. If the target descriptor is already open, it is closed before being reused.
27. Example: Redirecting Standard Output
28. dup() vs dup2()
| dup() | dup2() |
|---|---|
| Kernel chooses the new descriptor. | Application specifies the new descriptor. |
| Always returns the lowest available descriptor. | Uses the descriptor supplied by the caller. |
| Cannot replace an existing descriptor. | Can replace an existing descriptor. |
| Commonly used for simple duplication. | Commonly used for I/O redirection. |
29. dup2() Workflow
30. Interview Questions
- What is file descriptor duplication?
- What is the purpose of
dup()? - What is the purpose of
dup2()? - What resources are shared between duplicated file descriptors?
- Which properties are not shared by duplicated descriptors?
- When does
dup2()close the target descriptor? - Why is
dup2()commonly used for I/O redirection? - What happens if
oldfdis not a valid descriptor? - When would you use
dup()instead ofdup2()? - How do duplicated file descriptors affect the current file offset?
31. File Synchronization
Modern operating systems use buffers and caches to improve file I/O performance. As a result, a successful write operation does not necessarily mean that the data has already been stored on the physical storage device. Synchronization functions request that buffered data be transferred to stable storage.
32. Why is Synchronization Required?
- Reduce the risk of data loss after a system failure.
- Improve file system consistency.
- Ensure important application data reaches stable storage.
- Support reliable database and logging applications.
- Provide controlled synchronization of buffered writes.
33. sync()
The sync() function schedules all modified file system buffers to be written to their underlying storage devices. It operates on the entire system rather than on a single file.
The function does not return a status value.
34. fsync()
The fsync() function requests that all modified data associated with a specific file descriptor be transferred to stable storage. The synchronization includes file data and the metadata required to retrieve that data.
Returns 0 on success.
Returns -1 on failure.
35. fdatasync()
The fdatasync() function synchronizes file data and only the metadata required for subsequent data retrieval. Metadata unrelated to reading the file contents may not be synchronized.
36. Comparison of Synchronization Functions
| Function | Scope | Data | Metadata |
|---|---|---|---|
sync() |
Entire system | Yes | Yes |
fsync() |
Single file descriptor | Yes | Required metadata |
fdatasync() |
Single file descriptor | Yes | Only metadata required for data access |
37. Example: Using fsync()
38. Synchronization Workflow
39. Choosing the Appropriate Synchronization Function
| Scenario | Recommended Function |
|---|---|
| Synchronize all pending file system writes. | sync() |
| Synchronize one important file. | fsync() |
| Synchronize data while minimizing metadata updates. | fdatasync() |
40. Interview Questions
- Why are synchronization functions necessary?
- What is the purpose of
sync()? - How does
fsync()differ fromsync()? - What is the difference between
fsync()andfdatasync()? - When should
fdatasync()be preferred? - Does a successful
write()guarantee that data is stored on disk? - Which synchronization function affects the entire system?
- What type of metadata does
fdatasync()synchronize? - Why is
fsync()commonly used in database systems? - What happens if synchronization is not performed before an unexpected system failure?
41. ioctl()
The ioctl() system call performs device-specific control operations that are not provided by the standard read and write interfaces. The operation performed depends on the request code supplied by the application and the capabilities implemented by the device driver.
42. Why is ioctl() Required?
- Configure device-specific parameters.
- Query device status.
- Control terminal behavior.
- Manage communication devices.
- Perform operations beyond read() and write().
- Access driver-specific functionality.
43. ioctl() Parameters
| Parameter | Description |
|---|---|
fd |
Open file descriptor representing the device. |
request |
Device-specific operation code. |
| Optional Argument | Additional data required by the selected request. |
44. ioctl() Workflow
45. Common Uses of ioctl()
| Application Area | Example Purpose |
|---|---|
| Terminal Devices | Read or modify terminal settings. |
| Storage Devices | Obtain device information. |
| Network Interfaces | Configure interface properties. |
| Serial Communication | Configure communication parameters. |
| Graphics Devices | Access device-specific controls. |
46. Example: Query Terminal Size
47. Return Value of ioctl()
On success, ioctl() generally returns
0, although some request codes may define different
successful return values.
On failure, it returns -1 and sets
errno to indicate the reason for the error.
48. Advantages and Limitations of ioctl()
| Advantages | Limitations |
|---|---|
| Supports specialized device operations. | Behavior depends on the device driver. |
| Provides flexible device control. | Portable applications cannot assume identical request codes. |
| Extends device functionality. | Requires knowledge of driver-specific interfaces. |
49. ioctl() Execution Flow
50. Interview Questions
- What is the purpose of
ioctl()? - Why can't
read()andwrite()perform all device operations? - What does the
requestparameter represent? - Why is
ioctl()considered device-specific? - Name some common devices that use
ioctl(). - How does
ioctl()differ fromfcntl()? - What happens if an unsupported request code is passed?
- Why should portable applications avoid depending on device-specific request codes?
- How can
ioctl()be used with terminal devices? - What does
ioctl()return on failure?
51. Scatter/Gather I/O
Scatter/Gather I/O allows a single input/output operation to transfer data between a file descriptor and multiple memory buffers. Instead of performing several separate read() or write() operations, multiple buffers are processed using a single system call.
52. Why Scatter/Gather I/O?
- Reduce the number of system calls.
- Improve I/O efficiency.
- Transfer multiple buffers in one operation.
- Avoid unnecessary data copying.
- Simplify handling of structured data.
53. struct iovec
Both readv() and writev() use an array of
struct iovec structures to describe multiple memory
buffers.
| Member | Description |
|---|---|
iov_base |
Address of the memory buffer. |
iov_len |
Length of the buffer in bytes. |
54. readv()
The readv() system call reads data from a file
descriptor and distributes it across multiple buffers described by an
array of iovec structures.
55. writev()
The writev() system call gathers data from multiple buffers and writes it to a file descriptor using a single system call.
56. Example: readv()
57. Example: writev()
58. readv() vs writev()
| readv() | writev() |
|---|---|
| Reads into multiple buffers. | Writes from multiple buffers. |
| Scatter I/O. | Gather I/O. |
| One system call. | One system call. |
| Distributes incoming data. | Combines outgoing data. |
59. Scatter/Gather Workflow
60. Interview Questions
- What is Scatter/Gather I/O?
- What is the purpose of
struct iovec? - How does
readv()differ fromread()? - How does
writev()differ fromwrite()? - What information is stored in
iov_base? - What information is stored in
iov_len? - Why can
readv()andwritev()improve performance? - What is meant by Scatter I/O?
- What is meant by Gather I/O?
- Which header file defines
struct iovec?
61. Memory-Mapped I/O
Memory-mapped I/O allows a file or device to be mapped directly into a
process's virtual address space.
Instead of transferring data using repeated read() or
write() system calls, the application accesses the mapped
memory as though it were ordinary program memory.
62. Why Use Memory Mapping?
- Reduce the number of read() and write() system calls.
- Allow direct memory access to file contents.
- Simplify random access to large files.
- Improve performance for certain workloads.
- Enable shared memory between processes.
63. mmap()
The mmap() system call creates a mapping between a file or device and a region of the process's virtual address space.
On success, mmap() returns the starting address of the
mapped region.
On failure, it returns MAP_FAILED.
64. mmap() Parameters
| Parameter | Description |
|---|---|
addr |
Preferred starting virtual address (or NULL for automatic selection). |
length |
Number of bytes to map. |
prot |
Memory protection flags. |
flags |
Mapping type and options. |
fd |
File descriptor of the mapped object. |
offset |
Starting file offset for the mapping. |
65. Memory Protection Flags
| Flag | Description |
|---|---|
PROT_READ |
Pages may be read. |
PROT_WRITE |
Pages may be modified. |
PROT_EXEC |
Pages may contain executable code. |
PROT_NONE |
No access is permitted. |
66. Mapping Flags
| Flag | Description |
|---|---|
MAP_SHARED |
Modifications are visible to other processes mapping the same object and may be written back to the underlying file. |
MAP_PRIVATE |
Creates a private copy-on-write mapping. Modifications are not written back to the underlying file. |
67. Example: Reading a File Using mmap()
68. munmap()
The munmap() system call removes a previously created memory mapping from the process's virtual address space.
Returns 0 on success.
Returns -1 on failure.
69. Memory Mapping Workflow
70. Interview Questions
- What is memory-mapped I/O?
- How does
mmap()differ fromread()? - What does
MAP_SHAREDmean? - What is a copy-on-write mapping?
- What is the purpose of
munmap()? - What are the commonly used memory protection flags?
- Why must a file usually be opened before calling
mmap()? - What value indicates that
mmap()has failed? - When is memory-mapped I/O advantageous?
- What happens if a mapped region is not unmapped before process termination?
71. I/O Multiplexing
I/O multiplexing allows a single process to monitor multiple file descriptors and determine which descriptors are ready for input, output, or exceptional conditions. Instead of blocking on one descriptor, the process can efficiently wait for activity on many descriptors.
72. select()
The select() system call monitors multiple file descriptors and blocks until one or more descriptors become ready or a timeout expires.
73. fd_set Macros
| Macro | Purpose |
|---|---|
FD_ZERO() |
Initialize an empty descriptor set. |
FD_SET() |
Add a descriptor to the set. |
FD_CLR() |
Remove a descriptor from the set. |
FD_ISSET() |
Test whether a descriptor is ready. |
74. poll()
The poll() system call monitors multiple file
descriptors using an array of struct pollfd structures.
Unlike select(), it does not require descriptor bit sets.
75. struct pollfd
| Member | Description |
|---|---|
fd |
File descriptor to monitor. |
events |
Requested events. |
revents |
Events that actually occurred. |
76. Example: Waiting for Keyboard Input
77. select() vs poll()
| select() | poll() |
|---|---|
Uses fd_set bit sets. |
Uses an array of pollfd structures. |
Requires nfds (highest descriptor + 1). |
Uses the number of array elements. |
| Descriptor sets are modified by the kernel. | Event results are stored in revents. |
| Suitable for moderate numbers of descriptors. | More convenient for larger descriptor sets. |
78. I/O Multiplexing Workflow
80. Key Takeaways
fcntl()configures descriptor and file status flags.dup()anddup2()duplicate file descriptors.fsync()andfdatasync()synchronize buffered data.ioctl()performs device-specific control operations.readv()andwritev()support scatter/gather I/O.mmap()maps files into a process's virtual address space.munmap()removes an existing memory mapping.select()andpoll()monitor multiple file descriptors.
81. Advanced Interview Questions
- Explain the difference between blocking I/O and I/O multiplexing.
- When would you choose
poll()instead ofselect()? - How do duplicated file descriptors share file status information?
- Why is
fdatasync()sometimes preferred overfsync()? - Explain the advantages of scatter/gather I/O.
- What is the purpose of
MAP_PRIVATE? - How does
mmap()reduce the need for repeatedread()calls? - Why is
ioctl()considered less portable? - Describe the workflow of an event-driven server using
select(). - Differentiate
fcntl(),ioctl(), anddup2().
Concept Mastery Assessment — Module 26
Verify your conceptual understanding with 12 targeted technical assessment questions. Instant feedback and explanations are provided for every choice.