Wrappers

From RockWiki

The Flist Wrapper

The ROCK Linux build scripts auto-detect which files are created by a package build and use this information to create the package file-list. There are three techniques which can be used to auto-detect this file list (which one is used can be selected in the build configuration as part of the "expert options"):

flist-wrapper

The flist-wrapper is the recommended way of doing it. It's an "elf preload library". This is a library which is loaded _before_ the c-library and so can overwrite functions of the c library. The flist-wrapper lib is overwriting all functions which can be used to create files and is writing the filenames to a log file before running the original c library function. This log file is used to create the package file list after the build has been completed. The problem with the flist-wrapper lib is that it doesn't work with statically built binaries or with non glibc c libraries. In such environments (e.g. when building a dietlibc based system), other techniques - such as those described below - must be used for creating the file list.

strace

This one is rather easy: The whole build is running in a process-tree which is monitored by strace. The log file created by strace is later used to create the package file list. However: this only works on architectures which have an strace command - which is not the case for all architectures the Linux kernel and glibc have been ported to.

find

This one is rather ugly (but sometimes the only alternative): When the build is started, a 'timestamp-file' is created. After the build has been completed, the find program is used to find all files which are newer than the 'timestamp-file'. This is really slow and has the disadvantage that nothing else can create or modify files, except the build process, while the build is running. This should only be used if no other system for creating the file list can be used.

install-wrapper

It is planed to also implement a flist-creation system using the install-wrapper (see below). The install-wrapper is called whenever a build process is using the commands install, cp, mv or ln. However: Some builds are also installing files using other commands and they would not show up in the file-list created by the install-wrapper.

For the package, target or architecture maintainer it shouldn't be necessary to know how the flist wrapper works internally. However: it can't hurt - so here are a few additional internals for the flist wrapper library:

The flist wrapper library (as all the other wrappers) is created by the Build-Tools script and is stored in the build/*/tools.* directory. It is configured using two environment variables which are set in the Build-Pkg script:

FLWRAPPER_WLOG

The file which contains the names of the file the build process is writing to. This is src.*/fl_wrapper.wlog in native or cross builds and src.*/R.src/fl_wrapper.wlog in chroot builds.

FLWRAPPER_RLOG

The flist wrapper library also logs all files the build process is reading from or is executing. This information is used to generate the package build dependencies in a reference build. You can read the results of this analysis in scripts/dep_db.txt or in the package *.cache files. This log file is named src.*/fl_wrapper.rlog (or in chroot builds src.*/R.src/fl_wrapper.rlog).

The sources for the flist wrapper library can be found at misc/tools-source/fl_wrapper.c. This C source file is auto-generated by misc/tools-source/fl_wrapper.c.sh.

The Install Wrapper

The Build-Tools script creates some files like install, cp, mv and ln in build/*/tools.*/wrapper. This directory is added to the begin of the PATH environment variable in the parse-config script and so this programs are used instead of the standard tools when one of this commands is used during the build.

This wrapper files are just symbolic links to the install_wrapper program which is modifying the parameter list based on the INSTALL_WRAPPER_FILTER variable and then running the original program from the standard bin directories.

The INSTALL_WRAPPER_FILTER variable is initialized to an empty text in scripts/parse-config and can be extended in the package, target, architecture or misc parse-config files and the package *.conf file using the var_append and var_insert functions.

The INSTALL_WRAPPER_FILTER variable must contain a shell command which is reading the destination filenames for install and the other commands from input and writing the new filenames to the output. Multiple commands might be concatenated using the pipe character.

So, e.g. if a package installs a usr/share/man/man1/ls.1 (which creates a shared-file conflict with coreutils), it's possible to add something like

var_append INSTALL_WRAPPER_FILTER "|" \
    'sed "s,share/man/man1/ls.1\$,share/man/man1/ls_x.1,"'

to the package *.conf file and the file will be installed using the new name ls_x.1.

Use this wrapper with care! It's possible to break a lot with it because all install, cp, mv and ln commands during the build are wrapped with it - not only those executed in make install.

The install wrapper is writing a log file containing the old and new parameter lists so it's easier do debug problems. This log file is named src.*/install_wrapper.log (or in chroot builds src.*/R.src/install_wrapper.log). You might want to run Build-Pkg with the option noclearsrc if you want the log file (and the entire src.* directory) be kept for debugging reasons.

The source code for the install wrapper shell script can be found at misc/tools-source/install_wrapper.sh.

The Command Wrapper

The Build-Tools script creates a cmd_wrapper application and some shell scripts like cc, gcc, gcc-2 and gcc-3 in build/*/tools.*/wrapper. This directory is added to the begin of the PATH environment variable in theparse-config script and so this programs are used instead of the standard tools when one of this commands is used during the build.

The cmd_wrapper program is simply running the 'real' applications from the standard system bin or sbin directories with a modified parameter list. A set of environment variables (CMD_WRAPPER_*) is used to configure cmd_wrapper and define how the parameter list should be modified:

CMD_WRAPPER_LOGFILE

The log file for debugging problems with cmd_wrapper. Usually this is cmd_wrapper.log in the src.* directory.

CMD_WRAPPER_BYPASS

If this variable is set to 1, the cmd_wrapper is neither modifying the parameters nor running any other wrappers.

CMD_WRAPPER_OTHERS and CMD_WRAPPER_APPEND_PO

The CMD_WRAPPER_OTHERS variable contains a colon-separated list of other wrapper programs which should be executed before doing any modification in the parameter list. This other wrappers do get the program name passed as first parameter and and all parameters passed to cmd_wrapper as further parameters. The cmd_wrapper expects the other wrapper to run it's 1st parameter and so again execute the cmd_wrapper.

The CMD_WRAPPER_APPEND_PO variable contains a list of parameters which should be added to the parameter list before the other wrappers are started. This is used e.g. for the dietlibc subsystem to call the compiler with the 'diet' program transparently for the package makefiles.

CMD_WRAPPER_INSERT and CMD_WRAPPER_APPEND

The value of CMD_WRAPPER_INSERT is inserted at the begin of the parameter list, the value of CMD_WRAPPER_APPEND is appended to the end of the parameter list.

CMD_WRAPPER_REMOVE

The parameters listed in this variable are removed from the parameter list. It's possible to pass shell-wildcards (like -O*) in this variable. Parameters which are added by CMD_WRAPPER_INSERT or CMD_WRAPPER_APPEND are not removed - only parameters passed on the command line.

CMD_WRAPPER_FILTER

The shell command specified in this variable (multiple shell-commands separated by a pipe-character) is executed and passed the so far modified parameter list (parameters separated by a newline-char - i.e. one parameter per line). The output of this command (one parameter per line) is finally passed as parameter list the the real application. This can be used for complex modifications of the parameter list.

This CMD_WRAPPER_* variables are never set directly by the build scripts - they are set by the wrapper shell scripts based on other *_WRAPPER_* variables. E.g. the 'gcc-3' wrapper script is setting CMD_WRAPPER_* to the merged values of CC_WRAPPER_*, GCC_WRAPPER_* and GCC3_WRAPPER_*.

So if a parameter should be added every time a c-compiler is used, the parameter should be added to CC_WRAPPER_APPEND. But if the parameter is gcc-3 specific, GCC3_WRAPPER_APPEND is used instead.

The cmd_wrapper application is built directly by the Build-Tools script, but the separate wrappers are created by wrappers.in files which may be placed everywhere in package/, misc/ and target/<current-target>/. See package/base/gcc3/wrappers.in for a simple example of such a wrappers.in file.

The source code for the cmd_wrapper application can be found at misc/tools-source/cmd_wrapper.c.