Chapter 6. Package Editing Policy

Table of Contents

Introduction Section
Installation Section
Explanation Section
Configuration Section
Contents Section
About Indexing
Measurements when using multiple processors
Limiting available processors on large machines
Measuring packages which use ninja
Measuring packages which use rust

Each package in the book is written as a complete XML document using the Docbook specification. This Chapter describes how a BLFS page should be laid out.

A template of a typical package page is found in the BLFS BOOK repository under templates/template.xml. This chapter expands on the template, but generally changes needed for formatting purposes are reflected in the template more frequently than this guide.

The start of the page is the XML header. The page fits into the overall book as a sect1 element. In the header, there should be a reference to the general.ent file which includes entities that are used in multiple applications. This includes the version numbers of all applications, including the current one. Note that the reference to general.ent includes the relative address of the file. This will be at the root of the BLFS book's source tree and is normally two levels up from the application files.

Included in the header are the six entities that you see below. These are custom entities for each application and are included with each application even if the entity is only used once. This is done for consistency between applications.

If an HTTP or FTP URL is not available, leave the entity in place with a one-space string (" ") for the missing location.

Units provided for the download sizes should be kilobytes (1024 bytes) or megabytes (1024 kilobytes). Entries less than a 1000 kilobytes should be specified as whole numbers (e.g., 320 KB); larger sizes should be accurate to one decimal (e.g., 6.9 MB). Build sizes should be rounded to the nearest megabyte and displayed as whole numbers (e.g., 38 MB). SBU entries should be rounded to one decimal. If the value is less than 0.1 SBU, it should be listed as less than 0.1 SBU. Very long build times (greater than 10 SBUs) should be listed as integer values.

File sizes should be measured with the ls -l filename command. Build sizes should be measured with a df -k / command before the package is unpacked and after the package is installed (with any build/tmp directories still in place). Note that this assumes that the package is built and installed on the same filesystem. Adjust the procedure as necessary for your setup. The difference between these measurements is the build size. Note that the build size should not reflect the size of the downloaded package tarball. Build times should be measured with the procedure documented at http://www.linuxfromscratch.org/~bdubbs/about.html.

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE sect1 PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
   "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" [
  <!ENTITY % general-entities SYSTEM "../../general.ent">
  %general-entities;

<!ENTITY fam-download-http "http://gd.tuwien.ac.at/opsys/linux/gentoo/distfiles/fam-&fam-version;.tar.gz">
<!ENTITY fam-download-ftp  "ftp://oss.sgi.com/projects/fam/download/stable/fam-&fam-version;.tar.gz">
<!ENTITY fam-md5sum        "1bf3ae6c0c58d3201afc97c6a4834e39">
<!ENTITY fam-size          "301 KB">
<!ENTITY fam-buildsize     "7.7 MB">
<!ENTITY fam-time          "0.3 SBU">
]>

The application is delimited by <sect1> tags. The id attribute should be unique in the BLFS book and similar to the application name. The xreflabel is the name that will show up in the table of contents.

The <sect1info> structure is for Subversion to automatically tag the date and editor who made the most recent change to the file. It is also used to list the date of modification as a footnote at the bottom of each page.

The <?dbhtml filename="fam.html"?> line is an XSL processing instruction that tells the rendering engine where to place the output of the current page.

The required <title> section specifies the actual name of the section at the top of the application's page. This should normally be the same as the xreflabel. All titles should be in Title Case.

The <indexterm> tag is used for indexing the package. The zone attribute must match the sect1 id attribute value. The sortas attribute in the <primary> tag must start by "a-" followed by the application name in uppercase. The string enclosed by <primary> tags must be in the same case as the package name.

Note the use of entities in the XML. This allows the package version to be updated as entities without having to change the code.

Finally, note the use of whitespace. This is somewhat flexible, but should be used to make the XML source easily readable for maintainability purposes. Note that only spaces should be used for whitespace, never use tabs. Although it is not always possible, try to keep the line lengths to 80 characters or less.

      <sect1 id="fam" xreflabel="FAM-&fam-version;">
<?dbhtml filename="fam.html"?>

  <sect1info>
    <othername>$LastChangedBy: ken $</othername>
    <date>$Date: 2018-01-20 17:53:35 -0600 (Sat, 20 Jan 2018) $</date>
  </sect1info>

  <title>FAM-&fam-version;</title>

  <indexterm zone="fam">
    <primary sortas="a-FAM">FAM</primary>
  </indexterm>

The content of the application is normally divided into five areas: Introduction, Installation, Command Explanations, Configuration, and Contents. In some cases, the Command Explanations and/or Configuration sections are not required.

Introduction Section

The Introduction Section starts with a description of the application. This description is generally short, but for less well known applications, multiple paragraphs summarizing the application's purpose are appropriate.

  <sect2 role="package">
    <title>Introduction to FAM</title>

    <para>The <application>FAM</application> package 
    contains a File Alteration Monitor which is useful for notifying applications of
    changes to the file system.</para>

After the description, package locations and statistics are presented. The entities defined in the header should be used here.

        <bridgehead renderas="sect3">Package Information</bridgehead>
    <itemizedlist spacing='compact'>
      <listitem>
        <para>Download (HTTP): <ulink url="&fam-download-http;"/></para>
      </listitem>
      <listitem>
        <para>Download (FTP): <ulink url="&fam-download-ftp;"/></para>
      </listitem>
      <listitem>
        <para>Download MD5 sum: &fam-md5sum;</para>
      </listitem>
      <listitem>
        <para>Download size: &fam-size;</para>
      </listitem>
      <listitem>
        <para>Estimated disk space required: &fam-buildsize;</para>
      </listitem>
      <listitem>
        <para>Estimated build time: &fam-time;</para>
      </listitem>
    </itemizedlist>

Following the statistics, list any additional downloads needed. These files are usually patches. If no patches and/or special downloads are required, this section should be omitted.

        <bridgehead renderas="sect3">Additional Downloads</bridgehead>
    <itemizedlist spacing='compact'>
      <listitem>
        <para>Dnotify patch (Recommended): <ulink url="&patch-root;/fam-&fam-version;-dnotify-1.patch"/></para>
      </listitem>
    </itemizedlist>

Next comes dependencies. This is one of the most important parts of each application. Dependencies should be listed as Required or Optional. In many cases both are needed, and in some cases Recommended dependencies can be added. Required dependencies are those packages that are needed to build the application. Since the book assumes that the packages are being built from an LFS system, no LFS packages should be listed. If a package and a required dependency has, in turn, a required dependency in common, the second level dependency should not be listed. However, if a required dependency and the application share an optional dependency, that optional dependency should be listed.

If a package will take advantage of another package after it is built (run-time dependency), it should not be listed as a dependency, however a note may be appropriate. Use the Required, Recommended, and Optional sections as appropriate for the package.

        <bridgehead renderas="sect3">FAM dependencies</bridgehead>
    <bridgehead renderas="sect4">Required</bridgehead>
    <para><xref linkend="portmap"/></para>

    <bridgehead renderas="sect4">Recommended</bridgehead>
    <para><xref linkend="some-IDREF"/></para>

    <bridgehead renderas="sect4">Optional</bridgehead>
    <para><xref linkend="some-IDREF"/></para>

  </sect2>

Installation Section

The Installation Section provides the literal instructions needed to build the application. The instructions should be inside a <screen><userinput> construct and each instruction (except the last) should terminate with a double ampersand (&&) to facilitate cut and paste capability for the user. Note that the <screen><userinput> tags need to be on the same line as the first and last instruction with no leading spaces to render properly. Also note that you should keep lines in this (or any other) <screen><userinput> construct to 71 or fewer characters. Any more than this will not render properly in the PDF format.

You'll notice two entries below about any test suite the package may offer. Use one or the other, or in some instances, different instructions to run the package's test suite.

  <sect2 role="installation">
    <title>Installation of FAM</title>

    <para>Install <application>FAM</application> by running the
    following commands:</para>

<screen><userinput>patch -Np1 -i ../fam-&fam-version;-dnotify-1.patch &amp;&amp;
chmod 755 configure &amp;&amp;
autoreconf -f -i &amp;&amp;
./configure --prefix=/usr --sysconfdir=/etc &amp;&amp;
make</userinput></screen>

    <para>This package does not come with a test suite.</para>
    <para>To test the results, issue: <command>make check</command>.</para>

    <para>Now, as the <systemitem class="username">root</systemitem> user:</para>

<screen role="root"><userinput>make install</userinput></screen>

  </sect2>

Explanation Section

The Explanation Section elaborates on unusual commands and switches. If the instructions are common commands such as configure && make && make install, this section can be omitted. On the other hand, this section is the perfect place to explain why a command or parameter is necessary. If the Installation Section does not include a particular optional command or parameter, an entry can be made here giving the user information about the alternative or supplemental option. Since many packages have a large number of options, this explanation is normally reserved for optional instructions the editor feels are particularly important. For rendering purposes, parameters actually used in the Installation Section must use <parameter> tags. For optional switches, <option> tags should be used.

  <sect2 role="commands">
    <title>Command Explanations</title>

    <para><parameter>--sysconfdir=/etc</parameter>: This
    setting is used to set the configuration file location properly.</para>

    <para><command>patch -Np1 -i ../fam-&fam-version;-dnotify-1.patch</command>: 
    This patch enables <application>FAM</application> to use 
    the Linux kernel dnotify mechanism to inform the calling process of
    file modifications, rather than polling the file system for
    modifications.</para>

    <para><command>chmod 755 configure</command>:
    <command>configure</command> is set to read-only and
    <command>autoreconf</command> will fail if the 
    permissions aren't changed.</para>

    <para><command>autoreconf -f -i</command>: The autotools need
    rebuilding because the dnotify patch affects <filename>configure.ac</filename>
    and <filename>Makefile.am</filename>.</para>

  </sect2>

Configuration Section

This section describes any actions the user needs to take to configure the application. If no configuration is required, then it can be omitted. Note that in some cases lines in <screen> sections can be very long. Sometimes this cannot be avoided.

  <sect2 role="configuration">
    <title>Configuring FAM</title>

    <sect3 id="fam-config">
      <title>Config Files</title>

      <para><filename>/etc/rpc</filename>,
        <filename>/etc/fam.conf</filename>,
        <filename>/etc/inetd.conf</filename>, and
        <filename>/etc/xinetd.d/fam</filename> or
        <filename>/etc/xinetd.conf</filename>
      </para>

      <indexterm zone="fam fam-config">
        <primary sortas="e-etc-rpc">/etc/rpc</primary>
      </indexterm>
      <indexterm zone="fam fam-config">
        <primary sortas="e-etc-fam.conf">/etc/fam.conf</primary>
      </indexterm>
      <indexterm zone="fam fam-config">
        <primary sortas="e-etc-inetd.conf">/etc/inetd.conf</primary>
      </indexterm>
      <indexterm zone="fam fam-config">
        <primary sortas="e-etc-xinetd.conf">/etc/xinetd.conf</primary>
      </indexterm>
      <indexterm zone="fam fam-config">
        <primary sortas="e-etc-xinetd.d-sgi_fam">/etc/xinetd.d/sgi_fam</primary>
      </indexterm>
    </sect3>

    <sect3>
      <title>Configuration Information</title>

      <para>Configuring the file alteration monitor.</para>

      <para>If you use <application>inetd</application>, add the <application>FAM</application> 
      entry to <filename>/etc/inetd.conf</filename> with the following command:</para>

<screen role="root"><userinput>echo "sgi_fam/1-2 stream  rpc/tcp wait root /usr/sbin famd fam" \
    &gt;&gt; /etc/inetd.conf</userinput></screen>

      <para>If you use <application>xinetd</application>, add an entry to
      <filename>/etc/xinetd.conf</filename> with the following command (be
      sure the "nogroup" group exists):</para>

<screen role="root"><userinput>cat &gt;&gt; /etc/xinetd.conf &lt;&lt; "EOF"
<literal># description: FAM - file alteration monitor
    service sgi_fam
    {
        type            = RPC UNLISTED
        socket_type     = stream
        user            = root
        group           = nogroup
        server          = /usr/sbin/famd
        wait            = yes
        protocol        = tcp
        rpc_version     = 2
        rpc_number      = 391002
    }</literal>
EOF</userinput></screen>
    </sect3>

    <sect3 id="fam-boot">
      <title>Boot Scripts</title>

      <para>If you do not have an <command>inetd</command> daemon installed and have 
      no wish to install one, you can also start <command>famd</command> during 
      system startup by installing the <filename>/etc/rc.d/init.d/fam</filename>
      init script included in the
      <xref linkend="bootscripts"/> package.</para>

<screen role="root"><userinput>make install-fam</userinput></screen>

      <indexterm zone="fam fam-boot">
        <primary sortas="f-fam">fam</primary>
      </indexterm>

    </sect3>
  </sect2>

Contents Section

The final section describes the contents of the application.

  <sect2>
    <title>Contents</title>
    <segmentedlist>
      <segtitle>Installed Program</segtitle>
      <segtitle>Installed Library</segtitle>
      <segtitle>Installed Directories</segtitle>

      <seglistitem>
        <seg>famd</seg>
        <seg>libfam.[so,a]</seg>
        <seg>None</seg>
      </seglistitem>
    </segmentedlist>

    <variablelist>
      <bridgehead renderas="sect3">Short Descriptions</bridgehead>
      <?dbfo list-presentation="list"?>
      <?dbhtml list-presentation="table"?>

    <varlistentry id="famd">
      <term><command>famd</command></term>
      <listitem>
        <para>is the file alteration monitor daemon.</para>
        <indexterm zone="fam famd">
          <primary sortas="b-famd">famd</primary>
        </indexterm>
      </listitem>
    </varlistentry>

    <varlistentry id="libfam">
      <term><filename class='libraryfile'>libfam.[so,a]</filename></term>
      <listitem>
        <para>contains functions that support the file allocation monitor.</para>
        <indexterm zone="fam libfam">
          <primary sortas="c-libfam">libfam.[so,a]</primary>
        </indexterm>
        </listitem>
      </varlistentry>
    </variablelist>

  </sect2>
</sect1>

About Indexing

As you can see above, there are several places where indexing tags are placed into the page. The tagging has the following format:

<indexterm zone="id-start id-end">
  <primary sortas="X-sortterm">indexed-item</primary>
</indexterm>

The zone attribute has two values. The first will always be to the IDREF in the first <sect1> of the page. The first a- entry for packages will only have one IDREF. The other entries will have two IDREFS. The first will still be the page IDREF and the second will be an IDREF for the tag defining the start of the discussion of the item. The index will have two links for these references: one to the package and one to the specific section discussing the item.

The 'X' value in the sortas attribute has a specific meaning for the BLFS XSL stylesheets. It refers to the section of the index where the entry will be placed. The values are:

Table 6.1. Index Sections

Label Rendered in
a Packages
b Programs
c Libraries
d Kernel Configuration
e Configuration Files
f Bootscripts
g Others

The value after the dash in the sortas attribute is how the index item will be sorted within its section. It is not case sensitive. Do not abbreviate the text for the <primary> field in a misguided attempt to keep it within one line of your term : the text is what will be shown at the left of longindex.html, where items containing '...' as part of their directories will look very strange.

For configuration files in section 'e' the prefix to the file name has to be in a specific format. Mostly, this means the full directory path, but with '/' replaced by '-', e.g. e-etc-sgml.conf for /etc/sgml.conf. However, for a user's dot files use e-AA e.g. for ~/.pangorc use e-AA.pangorc. For variables prefixed by a dollar sign, use 'e-A.' e.g. 'e-A.exp_library-expect.rc' for $exp_library/expect.rc. Do not use the 'e-A' form without the dot - that will display correctly, but it will collate after the ~/ entries instead of before them.

Section 'g' is typically used for perl modules, in the Foo::Bar format for two-word names (not Foo-Bar), and can also be used for anything else which does not fit elsewhere but needs to be indexed, e.g. page names where there are no packages, or other types of files which have been listed in a package's list of installed files. For examples, look at the bottom of longindex.html.

Measurements when using multiple processors

For most packages, timings using a single processor (make -j1) are preferred. But many modern packages now take a very long time to build. Most users building BLFS on a modern machine will have 4 cores, so using make -j4 is perfectly acceptable for timing these packages. In one or two places make -j8 has been used, do not do that without good reason.

The rest of this section covers temporarily restricting the number of processors used on machines with more than 4 cores when using modern build systems (ninja and rust).

Limiting available processors on large machines

These modern build systems are designed to build quickly, using all the online processors (the sysconf libc _SC_NPROCESSORS_ONLN function). In linux it is common to constrain processes on large systems to a subset of the available CPUs using either taskset, cpugroups, or numactl. For example, using taskset the current term's PID ($$ in bash) can be restricted to CPUs 4-7 (typically two real cores and two multithread siblings, x86 linux will show them all as cores) using:

taskset -pc 4-7 $$

Unfortunately there is no corresponding widely-available (cross-OS) function for determining how many processors a task can use if it has been restricted like this. This has meant that packages using ninja or rust have spawned enough jobs for the whole physical machine (N+2 in ninja, N in rust). That unnecessarily slows the build down because execution is switching between all the jobs that are ready to run.

Measuring packages which use ninja

For ninja the situation can mostly be solved by passing a -jN switch e.g. -j4 to only use 4 CPUs. In a machine with only 4 CPUs, ninja would run 6 jobs, but tests show that timing differences between 4 and 6 ninja jobs are minimal in that situation. Therefore one approach on a machine with many processors is to not restrict the available cores, but to instead pass -j4 just like conventional 'make'.

Unfortunately, some packages such as qtwebengine embed ninja (it actually runs it from 'make') - it will use a system ninja if one is available, but it does not pass any -j switch, so that all online cores will be taken into account. But LFS now has a patched version of ninja, using the environment variable NINJAJOBS - this allows a many-cored machine to produce timings using:

NINJABOBS=4 make

(that is for qtwebengine), or for regular ninja packages:

NINJAJOBS=4 ninja

Measuring packages which use rust

For packages which are (partially) written in rust, the rust build system (cargo) has similarly used all online processors. This is most noticeable when compiling c++ parts of the package, or when rust is running tests, plain rust code often only compiles one job at a time.

In practice, rust uses the num_cpus crate to find out how many CPUs are available. This has now (from v1.5.0) been altered so that on linux it should correctly report the number of usable processors. Unfortunately, the versions of the sub-modules ('crates') used by a particular release of a package are set in stone. At the time of writing, only rustc-1.16.0 and firefox-56 in the book use rust, and only rustc uses num_cpus, fixed at v1.4.0.

Rust packages where the package or rustc uses num_cpus < v1.5.0

If building or using that version of rustc, and similarly for any other rust programs which use old versions of num_cpus, the only reliable way to measure a 4-core build on a machine with more than 4 processors is to take CPUs offline until only 4 are left.

Unless a process is constrained to a subset of the available CPUs, Linux will rotate processes across them. This is to even wear on the cores and to help with cooling. When limiting which processors are to be used, if a machine uses multithreading (intel i7 or above, and all modern AMD CPUs with more than 4 cores) it is best to use the appropriate pairs of CPUs - so 0,1 2,3 etc (on an 8-core i7, using logical cores 0,2,4,6 will run all physical cores at up to 100%, with full production of heat). If you are using any approach to limit the number of available cores you should use different cores from time to time to spread the usage.

The root user can do this by writing to sysfs. The following example from a 12-core (6 real cores, 6 siblings) machine will take cores 4 to 11 offline. To re-enable them, echo 1 to the files :

for i in $(seq 4 11); do
 echo 0 | tee /sys/devices/system/cpu/cpu$i/online
done

If you wish you can use 'sudo tee' here, it is omitted from the example as a reminder that only root should be doing this because it will restrict what else you can do at the same time.

Note

If you take CPU 0 (the boot CPU) offline, you run the risk of bringing the machine down. You should find that there is not an online file for cpu0 in sysfs. Because cpu1 is its sibling, it makes no sense to take that offline. You should keep a different pair of cores online whenever you do this, so that the load is spread across all the remaining CPUs, and you should bring them all back online whenever you can.

Measuring rust where rustc and the package use num_cpus ≥ v1.5.0

Here (i.e. building with rustc-1.21.0 or later, and the package either does not build num_cpus, or uses a new-enough version), use taskset to restrict the term where you are building to 4 cores.

Last updated by ken on 2018-01-20 17:53:35 -0600