| 1 | <?xml version="1.0" encoding="ISO-8859-1"?>
 | 
|---|
| 2 | <!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
 | 
|---|
| 3 |   "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
 | 
|---|
| 4 |   <!ENTITY % general-entities SYSTEM "../../general.ent">
 | 
|---|
| 5 |   %general-entities;
 | 
|---|
| 6 | ]>
 | 
|---|
| 7 | 
 | 
|---|
| 8 | <sect1 id="ch-scripts-udev">
 | 
|---|
| 9 |   <?dbhtml filename="udev.html"?>
 | 
|---|
| 10 | 
 | 
|---|
| 11 |   <title>Device and Module Handling on a CLFS System</title>
 | 
|---|
| 12 | 
 | 
|---|
| 13 |   <indexterm zone="ch-scripts-udev">
 | 
|---|
| 14 |     <primary sortas="a-Udev">Udev</primary>
 | 
|---|
| 15 |     <secondary>usage</secondary>
 | 
|---|
| 16 |   </indexterm>
 | 
|---|
| 17 | 
 | 
|---|
| 18 |   <para>In <xref linkend="chapter-building-system"/>, we installed the Udev
 | 
|---|
| 19 |   package. Before we go into the details regarding how this works,
 | 
|---|
| 20 |   a brief history of previous methods of handling devices is in
 | 
|---|
| 21 |   order.</para>
 | 
|---|
| 22 | 
 | 
|---|
| 23 |   <para>Linux systems in general traditionally use a static device creation
 | 
|---|
| 24 |   method, whereby a great many device nodes are created under <filename
 | 
|---|
| 25 |   class="directory">/dev</filename> (sometimes literally thousands of nodes),
 | 
|---|
| 26 |   regardless of whether the corresponding hardware devices actually exist. This
 | 
|---|
| 27 |   is typically done via a <command>MAKEDEV</command> script, which contains a
 | 
|---|
| 28 |   number of calls to the <command>mknod</command> program with the relevant
 | 
|---|
| 29 |   major and minor device numbers for every possible device that might exist in
 | 
|---|
| 30 |   the world.</para>
 | 
|---|
| 31 | 
 | 
|---|
| 32 |   <para>Using the Udev method, only those devices which are detected by the
 | 
|---|
| 33 |   kernel get device nodes created for them. Because these device nodes will be
 | 
|---|
| 34 |   created each time the system boots, they will be stored on a <systemitem
 | 
|---|
| 35 |   class="filesystem">tmpfs</systemitem> file system (a virtual file system that
 | 
|---|
| 36 |   resides entirely in system memory). Device nodes do not require much space, so
 | 
|---|
| 37 |   the memory that is used is negligible.</para>
 | 
|---|
| 38 | 
 | 
|---|
| 39 |   <sect2>
 | 
|---|
| 40 |     <title>History</title>
 | 
|---|
| 41 | 
 | 
|---|
| 42 |     <para>In February 2000, a new filesystem called <systemitem
 | 
|---|
| 43 |     class="filesystem">devfs</systemitem> was merged into the 2.3.46 kernel
 | 
|---|
| 44 |     and was made available during the 2.4 series of stable kernels. Although
 | 
|---|
| 45 |     it was present in the kernel source itself, this method of creating devices
 | 
|---|
| 46 |     dynamically never received overwhelming support from the core kernel
 | 
|---|
| 47 |     developers.</para>
 | 
|---|
| 48 | 
 | 
|---|
| 49 |     <para>The main problem with the approach adopted by <systemitem
 | 
|---|
| 50 |     class="filesystem">devfs</systemitem> was the way it handled device
 | 
|---|
| 51 |     detection, creation, and naming. The latter issue, that of device node
 | 
|---|
| 52 |     naming, was perhaps the most critical. It is generally accepted that if
 | 
|---|
| 53 |     device names are allowed to be configurable, then the device naming policy
 | 
|---|
| 54 |     should be up to a system administrator, not imposed on them by any
 | 
|---|
| 55 |     particular developer(s). The <systemitem
 | 
|---|
| 56 |     class="filesystem">devfs</systemitem> file system also suffers from race
 | 
|---|
| 57 |     conditions that are inherent in its design and cannot be fixed without a
 | 
|---|
| 58 |     substantial revision to the kernel. It has also been marked as deprecated
 | 
|---|
| 59 |     due to a lack of recent maintenance.</para>
 | 
|---|
| 60 | 
 | 
|---|
| 61 |     <para>With the development of the unstable 2.5 kernel tree, later released
 | 
|---|
| 62 |     as the 2.6 series of stable kernels, a new virtual filesystem called
 | 
|---|
| 63 |     <systemitem class="filesystem">sysfs</systemitem> came to be. The job of
 | 
|---|
| 64 |     <systemitem class="filesystem">sysfs</systemitem> is to export a view of
 | 
|---|
| 65 |     the system's hardware configuration to userspace processes. With this
 | 
|---|
| 66 |     userspace-visible representation, the possibility of seeing a userspace
 | 
|---|
| 67 |     replacement for <systemitem class="filesystem">devfs</systemitem> became
 | 
|---|
| 68 |     much more realistic.</para>
 | 
|---|
| 69 | 
 | 
|---|
| 70 |   </sect2>
 | 
|---|
| 71 | 
 | 
|---|
| 72 |   <sect2>
 | 
|---|
| 73 |     <title>Udev Implementation</title>
 | 
|---|
| 74 | 
 | 
|---|
| 75 |     <sect3>
 | 
|---|
| 76 |       <title>Sysfs</title>
 | 
|---|
| 77 | 
 | 
|---|
| 78 |       <para>The <systemitem class="filesystem">sysfs</systemitem> filesystem was
 | 
|---|
| 79 |       mentioned briefly above. One may wonder how <systemitem
 | 
|---|
| 80 |       class="filesystem">sysfs</systemitem> knows about the devices present on
 | 
|---|
| 81 |       a system and what device numbers should be used for them. Drivers that
 | 
|---|
| 82 |       have been compiled into the kernel directly register their objects with
 | 
|---|
| 83 |       <systemitem class="filesystem">sysfs</systemitem> as they are detected by
 | 
|---|
| 84 |       the kernel. For drivers compiled as modules, this registration will happen
 | 
|---|
| 85 |       when the module is loaded. Once the <systemitem
 | 
|---|
| 86 |       class="filesystem">sysfs</systemitem> filesystem is mounted (on <filename
 | 
|---|
| 87 |       class="directory">/sys</filename>), data which the built-in drivers
 | 
|---|
| 88 |       registered with <systemitem class="filesystem">sysfs</systemitem> are
 | 
|---|
| 89 |       available to userspace processes and to <command>udevd</command> for device
 | 
|---|
| 90 |       node creation.</para>
 | 
|---|
| 91 | 
 | 
|---|
| 92 |     </sect3>
 | 
|---|
| 93 | 
 | 
|---|
| 94 |     <sect3>
 | 
|---|
| 95 |       <title>Udev Bootscript</title>
 | 
|---|
| 96 | 
 | 
|---|
| 97 |       <para>The <command>S10udev</command> initscript takes care of creating
 | 
|---|
| 98 |       device nodes when Linux is booted. The script unsets the uevent handler
 | 
|---|
| 99 |       from the default of <command>/sbin/hotplug</command>.  This is done
 | 
|---|
| 100 |       because the kernel no longer needs to call out to an external binary.
 | 
|---|
| 101 |       Instead <command>udevd</command> will listen on a netlink socket for
 | 
|---|
| 102 |       uevents that the kernel raises. Next, the bootscript copies any static
 | 
|---|
| 103 |       device nodes that exist in <filename
 | 
|---|
| 104 |       class="directory">/lib/udev/devices</filename> to <filename
 | 
|---|
| 105 |       class="directory">/dev</filename>. This is necessary because some devices,
 | 
|---|
| 106 |       directories, and symlinks are needed before the dynamic device handling
 | 
|---|
| 107 |       processes are available during the early stages of booting a system.
 | 
|---|
| 108 |       Creating static device nodes in <filename
 | 
|---|
| 109 |       class="directory">/lib/udev/devices</filename> also provides an easy
 | 
|---|
| 110 |       workaround for devices that are not supported by the dynamic device
 | 
|---|
| 111 |       handling infrastructure. The bootscript then starts the Udev daemon,
 | 
|---|
| 112 |       <command>udevd</command>, which will act on any uevents it receives.
 | 
|---|
| 113 |       Finally, the bootscript forces the kernel to replay uevents for any
 | 
|---|
| 114 |       devices that have already been registered and then waits for
 | 
|---|
| 115 |       <command>udevd</command> to handle them.</para>
 | 
|---|
| 116 | 
 | 
|---|
| 117 |     </sect3>
 | 
|---|
| 118 | 
 | 
|---|
| 119 |     <sect3>
 | 
|---|
| 120 |       <title>Device Node Creation</title>
 | 
|---|
| 121 | 
 | 
|---|
| 122 |       <para>To obtain the right major and minor number for a device, Udev relies
 | 
|---|
| 123 |       on the information provided by <systemitem
 | 
|---|
| 124 |       class="filesystem">sysfs</systemitem> in <filename
 | 
|---|
| 125 |       class="directory">/sys</filename>.  For example,
 | 
|---|
| 126 |       <filename>/sys/class/tty/vcs/dev</filename> contains the string
 | 
|---|
| 127 |       <quote>7:0</quote>. This string is used by <command>udevd</command>
 | 
|---|
| 128 |       to create a device node with major number <emphasis>7</emphasis> and minor
 | 
|---|
| 129 |       <emphasis>0</emphasis>. The names and permissions of the nodes created
 | 
|---|
| 130 |       under the <filename class="directory">/dev</filename> directory are
 | 
|---|
| 131 |       determined by rules specified in the files within the <filename
 | 
|---|
| 132 |       class="directory">/etc/udev/rules.d/</filename> directory. These are
 | 
|---|
| 133 |       numbered in a similar fashion to the CLFS-Bootscripts package. If
 | 
|---|
| 134 |       <command>udevd</command> can't find a rule for the device it is creating,
 | 
|---|
| 135 |       it will default permissions to <emphasis>660</emphasis> and ownership to
 | 
|---|
| 136 |       <emphasis>root:root</emphasis>. Documentation on the syntax of the Udev
 | 
|---|
| 137 |       rules configuration files is available in
 | 
|---|
| 138 |       <filename>/usr/share/doc/udev-&udev-version;/index.html</filename></para>
 | 
|---|
| 139 | 
 | 
|---|
| 140 |     </sect3>
 | 
|---|
| 141 | 
 | 
|---|
| 142 |     <sect3>
 | 
|---|
| 143 |       <title>Module Loading</title>
 | 
|---|
| 144 | 
 | 
|---|
| 145 |       <para>Device drivers compiled as modules may have aliases built into them.
 | 
|---|
| 146 |       Aliases are visible in the output of the <command>modinfo</command>
 | 
|---|
| 147 |       program and are usually related to the bus-specific identifiers of devices
 | 
|---|
| 148 |       supported by a module. For example, the <emphasis>snd-fm801</emphasis>
 | 
|---|
| 149 |       driver supports PCI devices with vendor ID 0x1319 and device ID 0x0801,
 | 
|---|
| 150 |       and has an alias of <quote>pci:v00001319d00000801sv*sd*bc04sc01i*</quote>.
 | 
|---|
| 151 |       For most devices, the bus driver exports the alias of the driver that
 | 
|---|
| 152 |       would handle the device via <systemitem
 | 
|---|
| 153 |       class="filesystem">sysfs</systemitem>. E.g., the
 | 
|---|
| 154 |       <filename>/sys/bus/pci/devices/0000:00:0d.0/modalias</filename> file
 | 
|---|
| 155 |       might contain the string
 | 
|---|
| 156 |       <quote>pci:v00001319d00000801sv00001319sd00001319bc04sc01i00</quote>.
 | 
|---|
| 157 |       The default rules provided by Udev will cause <command>udevd</command>
 | 
|---|
| 158 |       to call out to <command>/sbin/modprobe</command> with the contents of the
 | 
|---|
| 159 |       <envar>MODALIAS</envar> uevent environment variable (that should be the
 | 
|---|
| 160 |       same as the contents of the <filename>modalias</filename> file in sysfs),
 | 
|---|
| 161 |       thus loading all modules whose aliases match this string after wildcard
 | 
|---|
| 162 |       expansion.</para>
 | 
|---|
| 163 | 
 | 
|---|
| 164 |       <para>In this example, this means that, in addition to
 | 
|---|
| 165 |       <emphasis>snd-fm801</emphasis>, the obsolete (and unwanted)
 | 
|---|
| 166 |       <emphasis>forte</emphasis> driver will be loaded if it is
 | 
|---|
| 167 |       available. See below for ways in which the loading of unwanted drivers can
 | 
|---|
| 168 |       be prevented.</para>
 | 
|---|
| 169 | 
 | 
|---|
| 170 |       <para>The kernel itself is also able to load modules for network
 | 
|---|
| 171 |       protocols, filesystems and NLS support on demand.</para>
 | 
|---|
| 172 | 
 | 
|---|
| 173 |     </sect3>
 | 
|---|
| 174 | 
 | 
|---|
| 175 |     <sect3>
 | 
|---|
| 176 |       <title>Handling Hotpluggable/Dynamic Devices</title>
 | 
|---|
| 177 | 
 | 
|---|
| 178 |       <para>When you plug in a device, such as a Universal Serial Bus (USB) MP3
 | 
|---|
| 179 |       player, the kernel recognizes that the device is now connected and
 | 
|---|
| 180 |       generates a uevent. This uevent is then handled by
 | 
|---|
| 181 |       <command>udevd</command> as described above.</para>
 | 
|---|
| 182 | 
 | 
|---|
| 183 |     </sect3>
 | 
|---|
| 184 | 
 | 
|---|
| 185 |   </sect2>
 | 
|---|
| 186 | 
 | 
|---|
| 187 |   <sect2>
 | 
|---|
| 188 |     <title>Problems with Loading Modules and Creating Devices</title>
 | 
|---|
| 189 | 
 | 
|---|
| 190 |     <para>There are a few possible problems when it comes to automatically
 | 
|---|
| 191 |     creating device nodes.</para>
 | 
|---|
| 192 | 
 | 
|---|
| 193 |     <sect3>
 | 
|---|
| 194 |       <title>A kernel module is not loaded automatically</title>
 | 
|---|
| 195 | 
 | 
|---|
| 196 |       <para>Udev will only load a module if it has a bus-specific alias and the
 | 
|---|
| 197 |       bus driver properly exports the necessary aliases to <systemitem
 | 
|---|
| 198 |       class="filesystem">sysfs</systemitem>. In other cases, one should
 | 
|---|
| 199 |       arrange module loading by other means. With Linux-&linux-version;, Udev is
 | 
|---|
| 200 |       known to load properly-written drivers for INPUT, IDE, PCI, USB, SCSI,
 | 
|---|
| 201 |       SERIO and FireWire devices.</para>
 | 
|---|
| 202 | 
 | 
|---|
| 203 |       <para>To determine if the device driver you require has the necessary
 | 
|---|
| 204 |       support for Udev, run <command>modinfo</command> with the module name as
 | 
|---|
| 205 |       the argument.  Now try locating the device directory under
 | 
|---|
| 206 |       <filename class="directory">/sys/bus</filename> and check whether there is
 | 
|---|
| 207 |       a <filename>modalias</filename> file there.</para>
 | 
|---|
| 208 | 
 | 
|---|
| 209 |       <para>If the <filename>modalias</filename> file exists in <systemitem
 | 
|---|
| 210 |       class="filesystem">sysfs</systemitem>, the driver supports the device and
 | 
|---|
| 211 |       can talk to it directly, but doesn't have the alias, it is a bug in the
 | 
|---|
| 212 |       driver. Load the driver without the help from Udev and expect the issue
 | 
|---|
| 213 |       to be fixed later.</para>
 | 
|---|
| 214 | 
 | 
|---|
| 215 |       <para>If there is no <filename>modalias</filename> file in the relevant
 | 
|---|
| 216 |       directory under <filename class="directory">/sys/bus</filename>, this
 | 
|---|
| 217 |       means that the kernel developers have not yet added modalias support to
 | 
|---|
| 218 |       this bus type. With Linux-&linux-version;, this is the case with ISA
 | 
|---|
| 219 |       busses. Expect this issue to be fixed in later kernel versions.</para>
 | 
|---|
| 220 | 
 | 
|---|
| 221 |       <para>Udev is not intended to load <quote>wrapper</quote> drivers such as
 | 
|---|
| 222 |       <emphasis>snd-pcm-oss</emphasis> and non-hardware drivers such as
 | 
|---|
| 223 |       <emphasis>loop</emphasis> at all.</para>
 | 
|---|
| 224 | 
 | 
|---|
| 225 |     </sect3>
 | 
|---|
| 226 | 
 | 
|---|
| 227 |     <sect3>
 | 
|---|
| 228 |       <title>A kernel module is not loaded automatically, and Udev is not
 | 
|---|
| 229 |       intended to load it</title>
 | 
|---|
| 230 | 
 | 
|---|
| 231 |       <para>If the <quote>wrapper</quote> module only enhances the functionality
 | 
|---|
| 232 |       provided by some other module (e.g., <emphasis>snd-pcm-oss</emphasis>
 | 
|---|
| 233 |       enhances the functionality of <emphasis>snd-pcm</emphasis> by making the
 | 
|---|
| 234 |       sound cards available to OSS applications), configure
 | 
|---|
| 235 |       <command>modprobe</command> to load the wrapper after Udev loads the
 | 
|---|
| 236 |       wrapped module. To do this, add an <quote>install</quote> line in
 | 
|---|
| 237 |       <filename>/etc/modprobe.conf</filename>. For example:</para>
 | 
|---|
| 238 | 
 | 
|---|
| 239 | <screen role="nodump"><literal>install snd-pcm /sbin/modprobe -i snd-pcm ; \
 | 
|---|
| 240 |     /sbin/modprobe snd-pcm-oss ; true</literal></screen>
 | 
|---|
| 241 | 
 | 
|---|
| 242 |       <para>If the module in question is not a wrapper and is useful by itself,
 | 
|---|
| 243 |       configure the <command>S05modules</command> bootscript to load this
 | 
|---|
| 244 |       module on system boot. To do this, add the module name to the
 | 
|---|
| 245 |       <filename>/etc/sysconfig/modules</filename> file on a separate line.
 | 
|---|
| 246 |       This works for wrapper modules too, but is suboptimal in that case.</para>
 | 
|---|
| 247 | 
 | 
|---|
| 248 |     </sect3>
 | 
|---|
| 249 | 
 | 
|---|
| 250 |     <sect3>
 | 
|---|
| 251 |       <title>Udev loads some unwanted module</title>
 | 
|---|
| 252 | 
 | 
|---|
| 253 |       <para>Either don't build the module, or blacklist it in
 | 
|---|
| 254 |       <filename>/etc/modprobe.conf</filename> file as done with the
 | 
|---|
| 255 |       <emphasis>forte</emphasis> module in the example below:</para>
 | 
|---|
| 256 | 
 | 
|---|
| 257 | <screen role="nodump"><literal>blacklist forte</literal></screen>
 | 
|---|
| 258 | 
 | 
|---|
| 259 |       <para>Blacklisted modules can still be loaded manually with the
 | 
|---|
| 260 |       explicit <command>modprobe</command> command.</para>
 | 
|---|
| 261 | 
 | 
|---|
| 262 |     </sect3>
 | 
|---|
| 263 | 
 | 
|---|
| 264 |     <sect3>
 | 
|---|
| 265 |       <title>Udev creates a device incorrectly, or makes a wrong symlink</title>
 | 
|---|
| 266 | 
 | 
|---|
| 267 |       <para>This usually happens if a rule unexpectedly matches a device. For
 | 
|---|
| 268 |       example, a poorly-writen rule can match both a SCSI disk (as desired)
 | 
|---|
| 269 |       and the corresponding SCSI generic device (incorrectly) by vendor.
 | 
|---|
| 270 |       Find the offending rule and make it more specific, with the help of
 | 
|---|
| 271 |       <command>udevadm info</command>.</para>
 | 
|---|
| 272 | 
 | 
|---|
| 273 |     </sect3>
 | 
|---|
| 274 | 
 | 
|---|
| 275 |     <sect3>
 | 
|---|
| 276 |       <title>Udev rule works unreliably</title>
 | 
|---|
| 277 | 
 | 
|---|
| 278 |       <para>This may be another manifestation of the previous problem. If not,
 | 
|---|
| 279 |       and your rule uses <systemitem class="filesystem">sysfs</systemitem>
 | 
|---|
| 280 |       attributes, it may be a kernel timing issue, to be fixed in later kernels.
 | 
|---|
| 281 |       For now, you can work around it by creating a rule that waits for the used
 | 
|---|
| 282 |       <systemitem class="filesystem">sysfs</systemitem> attribute and appending
 | 
|---|
| 283 |       it to the <filename>/etc/udev/rules.d/10-wait_for_sysfs.rules</filename>
 | 
|---|
| 284 |       file. Please notify the CLFS Development list if you do so and it
 | 
|---|
| 285 |       helps.</para>
 | 
|---|
| 286 | 
 | 
|---|
| 287 |     </sect3>
 | 
|---|
| 288 | 
 | 
|---|
| 289 |     <sect3>
 | 
|---|
| 290 |       <title>Udev does not create a device</title>
 | 
|---|
| 291 | 
 | 
|---|
| 292 |       <para>Further text assumes that the driver is built statically into the
 | 
|---|
| 293 |       kernel or already loaded as a module, and that you have already checked
 | 
|---|
| 294 |       that Udev doesn't create a misnamed device.</para>
 | 
|---|
| 295 | 
 | 
|---|
| 296 |       <para>Udev has no information needed to create a device node if a kernel
 | 
|---|
| 297 |       driver does not export its data to <systemitem
 | 
|---|
| 298 |       class="filesystem">sysfs</systemitem>.
 | 
|---|
| 299 |       This is most common with third party drivers from outside the kernel
 | 
|---|
| 300 |       tree. Create a static device node in
 | 
|---|
| 301 |       <filename>/lib/udev/devices</filename> with the appropriate major/minor
 | 
|---|
| 302 |       numbers (see the file <filename>devices.txt</filename> inside the kernel
 | 
|---|
| 303 |       documentation or the documentation provided by the third party driver
 | 
|---|
| 304 |       vendor). The static device node will be copied to
 | 
|---|
| 305 |       <filename class="directory">/dev</filename> by the
 | 
|---|
| 306 |       <command>S10udev</command> bootscript.</para>
 | 
|---|
| 307 | 
 | 
|---|
| 308 |     </sect3>
 | 
|---|
| 309 | 
 | 
|---|
| 310 |     <sect3>
 | 
|---|
| 311 |       <title>Device naming order changes randomly after rebooting</title>
 | 
|---|
| 312 | 
 | 
|---|
| 313 |       <para>This is due to the fact that Udev, by design, handles uevents and
 | 
|---|
| 314 |       loads modules in parallel, and thus in an unpredictable order. This will
 | 
|---|
| 315 |       never be <quote>fixed</quote>. You should not rely upon the kernel device
 | 
|---|
| 316 |       names being stable. Instead, create your own rules that make symlinks with
 | 
|---|
| 317 |       stable names based on some stable attributes of the device, such as a
 | 
|---|
| 318 |       serial number or the output of various *_id utilities installed by Udev.
 | 
|---|
| 319 |       See <xref linkend="ch-scripts-symlinks"/> and
 | 
|---|
| 320 |       <xref linkend="ch-scripts-network"/> for examples.</para>
 | 
|---|
| 321 | 
 | 
|---|
| 322 |     </sect3>
 | 
|---|
| 323 | 
 | 
|---|
| 324 |   </sect2>
 | 
|---|
| 325 | 
 | 
|---|
| 326 |   <sect2>
 | 
|---|
| 327 |     <title>Useful Reading</title>
 | 
|---|
| 328 | 
 | 
|---|
| 329 |     <para>Additional helpful documentation is available at the following
 | 
|---|
| 330 |     sites:</para>
 | 
|---|
| 331 | 
 | 
|---|
| 332 |     <itemizedlist>
 | 
|---|
| 333 | 
 | 
|---|
| 334 |       <listitem>
 | 
|---|
| 335 |         <para remap="verbatim">A Userspace Implementation of <systemitem class="filesystem">devfs</systemitem>
 | 
|---|
| 336 |         <ulink url="http://www.kroah.com/linux/talks/ols_2003_udev_paper/Reprint-Kroah-Hartman-OLS2003.pdf"/></para>
 | 
|---|
| 337 |       </listitem>
 | 
|---|
| 338 | 
 | 
|---|
| 339 |       <listitem>
 | 
|---|
| 340 |         <para remap="verbatim">The <systemitem class="filesystem">sysfs</systemitem> Filesystem
 | 
|---|
| 341 |         <ulink url="http://www.kernel.org/pub/linux/kernel/people/mochel/doc/papers/ols-2005/mochel.pdf"/></para>
 | 
|---|
| 342 |       </listitem>
 | 
|---|
| 343 | 
 | 
|---|
| 344 |     </itemizedlist>
 | 
|---|
| 345 | 
 | 
|---|
| 346 |   </sect2>
 | 
|---|
| 347 | 
 | 
|---|
| 348 | </sect1>
 | 
|---|