buildroot: remove all default guest packages to make image tiny

Author: cirosantilli

README.adoc

@@ -98,7 +98,7 @@ see this: https://askubuntu.com/questions/496549/error-you-must-put-some-source-
 
 It does not work if you just download the `.zip` from GitHub because we use link:.gitmodules[Git submodules], you must clone this repo. `./configure` then fetches only the required submodules for you.
 
-After QEMU opens up, you can start playing with the kernel modules:
+QEMU opens up and you can start playing with the kernel modules inside the simulated system:
 
 ....
 insmod /hello.ko
@@ -216,12 +216,18 @@ and then rebuild the kernel modules and re-run to see it take effect:
 
 Congratulations, you are now officially a kernel module hacker!
 
-We use `./build-buildroot` here because kernel modules go into the root filesystem, and it is Buildroot that generates our root filesystem. The kernel modules are inside a Buildroot package.
+We use `./build-buildroot` because the kernel modules go inside the root filesystem, and it is Buildroot that generates and updates our root filesystem. The kernel modules are link:packages/kernel_modules[inside a Buildroot package].
 
-`--kernel-modules` is required even if files were modified as explained at: <<rebuild-buildroot-packages>>.
+`--kernel-modules` is required for the rebuild even though files were modified as explained at: <<rebuild-buildroot-packages>>.
 
 The reboot after rebuild is annoying. We don't have a perfect solution for it yet, but there are some ideas cooking at: <<gem5-restore-new-script>>.
 
+Using <<KVM>> can speed boot up however if your host and guest have the same arch, e.g. on an `x86_64` host:
+
+....
+./run --kvm
+....
+
 Not satisfied with kernel modules? OK then, let's hack up the <<linux-kernel-entry-point,entry point of the>> Linux kernel itself.
 
 Open the file:
@@ -1405,7 +1411,11 @@ The number of cores is modified as explained at: <<number-of-cores>>
 `taskset` from the util-linux package sets the initial core affinity of a program:
 
 ....
-taskset -c 1,1 /sched_getaffinity.out
+./build-buildroot \
+  --buildroot-config 'BR2_PACKAGE_UTIL_LINUX=y' \
+  --buildroot-config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
+;
+./run --eval-busybox 'taskset -c 1,1 /sched_getaffinity.out'
 ....
 
 output:
@@ -1710,7 +1720,13 @@ But TODO I don't think you can see where you are in the kernel source code and l
 
 Step debug userland processes to understand how they are talking to the kernel.
 
-Guest:
+First build `gdbserver` into the root filesystem:
+
+....
+./build-buildroot --buildroot-config 'BR2_PACKAGE_GDB=y'
+....
+
+Then on guest:
 
 ....
 /gdbserver.sh /myinsmod.out /hello.ko
@@ -2341,10 +2357,17 @@ Files that contain device trees have the `.dtb` extension when compiled, and `.d
 
 You can convert between those formats with:
 
+....
+"$(./getvar host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
+"$(./getvar host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts
+....
+
+Buildroot builds the tool due to `BR2_PACKAGE_HOST_DTC=y`.
+
+On Ubuntu 18.04, the package is named:
+
 ....
 sudo apt-get install device-tree-compiler
-dtc -I dtb -O dts -o a.dts a.dtb
-dtc -I dts -O dtb -o a.dtb a.dts
 ....
 
 See also: https://stackoverflow.com/questions/14000736/tool-to-visualize-the-device-tree-file-dtb-used-by-the-linux-kernel/39931834#39931834
@@ -2405,13 +2428,25 @@ then `dtc a.dts` gives:
 };
 ....
 
-=== Get device tree from running kernel
+=== Get device tree from a running kernel
 
 https://unix.stackexchange.com/questions/265890/is-it-possible-to-get-the-information-for-a-device-tree-using-sys-of-a-running/330926#330926
 
 This is specially interesting because QEMU and gem5 are capable of generating DTBs that match the selected machine depending on dynamic command line parameters for some types of machines.
 
-QEMU's `-M virt` for example, which we use by default for `aarch64`, boots just fine without the `-dtb` option:
+So observing the device tree from the guest allows to easily see what the emulator has generated.
+
+Compile the `dtc` tool into the root filesystem:
+
+....
+./build-buildroot \
+  --arch aarch64 \
+  --buildroot-config 'BR2_PACKAGE_DTC=y' \
+  --buildroot-config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
+;
+....
+
+`-M virt` for example, which we use by default for `aarch64`, boots just fine without the `-dtb` option:
 
 ....
 ./run --arch aarch64
@@ -5514,6 +5549,14 @@ Each `enable` under the `events/` tree enables a certain set of functions, the h
 
 TODO: can you get function arguments? https://stackoverflow.com/questions/27608752/does-ftrace-allow-capture-of-system-call-arguments-to-the-linux-kernel-or-only
 
+===== trace-cmd
+
+TODO example:
+
+....
+./build-buildroot --buildroot-config 'BR2_PACKAGE_TRACE_CMD=y'
+....
+
 ==== Kprobes
 
 kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-)
@@ -7843,7 +7886,13 @@ OK, this is why we used gem5 in the first place, performance measurements!
 
 Let's see how many cycles https://en.wikipedia.org/wiki/Dhrystone[Dhrystone], which Buildroot provides, takes for a few different input parameters.
 
-A flexible setup is demonstrated at:
+First build Dhrystone into the root filesystem:
+
+....
+./build-buildroot --buildroot-config 'BR2_PACKAGE_DHRYSTONE=y'
+....
+
+Then, a flexible setup is demonstrated at:
 
 ....
 ./gem5-bench-dhrystone
@@ -9219,20 +9268,11 @@ First, see if you can't get away without actually adding a new package, for exam
 * if you have a standalone C file with no dependencies besides the C standard library to be compiled with GCC, just add a new file under link:packages/kernel_modules/user[] and you are done
 * if you have a dependency on a library, first check if Buildroot doesn't have a package for it already with `ls buildroot/package`. If yes, just enable that package as explained at: <<custom-buildroot-configs>>
 
-If none of those methods are flexible enough for you, create a new package as follows:
+If none of those methods are flexible enough for you, you can just fork or hack up link:packages/sample_package[] the sample package to do what you want.
 
-* use link:packages/sample_package[] as a starting point
-* fork this repository, and modify that package to do what you want
-* read the comments on that package to get an idea of how to start
-* check the main manual for more complicated things: https://buildroot.org/downloads/manual/manual.html
-* don't forget to rebuild with:
-+
-....
-./build-buildroot -- sample_package-reconfigure
-./run --eval-busybox '/sample_package.out'
-....
-+
-if you make any changes to that package after the initial build: <<rebuild-buildroot-packages>>
+For how to use that package, see: <<packages-directory>>.
+
+Then iterate trying to do what you want and reading the manual until it works: https://buildroot.org/downloads/manual/manual.html
 
 === BR2_TARGET_ROOTFS_EXT2_SIZE
 
@@ -9490,7 +9530,9 @@ xdg-open graph-size.pdf
 
 Our philosophy is:
 
-* keep the root filesystem as tiny as possible to make prebuilts small. It is easy to add new packages once you have the toolchain.
+* keep the root filesystem as tiny as possible to make <<prebuilt>> small: only add BusyBox to have a small interactive system.
++
+It is easy to add new packages once you have the toolchain, and if you don't there are infinitely many packages to cover and we can't cover them all.
 * enable every feature possible on the toolchain (GCC, Binutils), because changes imply Buildroot rebuilds
 * runtime is sacred. Faster systems are:
 +
@@ -10041,10 +10083,20 @@ Every directory inside it is a Buildroot package.
 Those packages get automatically added to Buildroot's `BR2_EXTERNAL`, so all you need to do is to turn them on during build, e.g.:
 
 ....
-./build-buildroot --buildroot-config BR2_SAMPLE_PACKAGE=y
+./build-buildroot --buildroot-config 'BR2_SAMPLE_PACKAGE=y'
+....
+
+or force a rebuild after the first one with:
+
+....
+./build-buildroot --buildroot-config 'BR2_SAMPLE_PACKAGE=y' -- sample_package-reconfigure
 ....
 
-If you want to add something to the root filesystem, possibly involving cross-compilation, then packages are the way to go.
+then test it out with:
+
+....
+./run --eval-busybox '/sample_package.out'
+....
 
 In particular, our kernel modules are stored inside a Buildroot package: link:packages/kernel_modules[].
 

build-buildroot

@@ -22,6 +22,7 @@ defaults = {
     'kernel_config_fragment': [],
     'kernel_custom_config_file': None,
     'kernel_modules': False,
+    'no_kernel_modules': False,
     'linux_reconfigure': False,
     'no_all': False,
     'nproc': None,
@@ -33,6 +34,7 @@ defaults = {
 def get_argparse():
     parser = common.get_argparse(argparse_args={'description':'Run Linux on an emulator'})
     common.add_build_arguments(parser)
+    kernel_module_group = parser.add_mutually_exclusive_group()
     parser.add_argument(
         '-B', '--buildroot-config', default=defaults['buildroot_config'], action='append',
         help='''Add a single Buildroot config to the current build.
@@ -79,9 +81,9 @@ Pass multiple times to use multiple fragment files.'''
         help='''Ignore all default kernel configurations and use this file instead.
 Still uses options explicitly passed with `-C` and `-c` on top of it.'''
     )
-    parser.add_argument(
+    kernel_module_group.add_argument(
         '-k', '--kernel-modules', default=defaults['kernel_modules'], action='store_true',
-        help='Reconfigure and rebuild the kernel modules'
+        help='Reconfigure and rebuild the kernel modules package'
     )
     parser.add_argument(
         '-l', '--linux-reconfigure', default=defaults['linux_reconfigure'], action='store_true',
@@ -94,6 +96,10 @@ https://stackoverflow.com/questions/49260466/why-when-i-change-br2-linux-kernel-
         help='''Don't build the all target which normally gets build by default.
 That target builds the root filesystem and all its dependencies.'''
     )
+    kernel_module_group.add_argument(
+        '--no-kernel-modules', default=defaults['kernel_modules'], action='store_true',
+        help="Don't build the kernel modules package"
+    )
     parser.add_argument(
         '--skip-configure', default=defaults['skip_configure'], action='store_true',
         help='''Skip the Buildroot configuration. Saves a few seconds,
@@ -180,6 +186,8 @@ def main(args, extra_args=None):
                     'BR2_ROOTFS_USERS_TABLES="{}"'.format(
                         path_relative_to_buildroot(os.path.join(common.root_dir, 'user_table'))),
                 ])
+                if not args.no_kernel_modules:
+                    buildroot_configs.append('BR2_PACKAGE_KERNEL_MODULES=y')
                 if args.gem5:
                     buildroot_configs.append('BR2_PACKAGE_GEM5=y')
                 if args.initramfs:

buildroot_config/default

@@ -29,37 +29,11 @@ BR2_PACKAGE_HOST_GDB_PYTHON=y
 BR2_PACKAGE_HOST_GDB_SIM=y
 BR2_PACKAGE_HOST_GDB_TUI=y
 
-# Custom packages
-# Keepding those in because we control them fully
-# and know for sure that are small.
-BR2_PACKAGE_KERNEL_MODULES=y
-BR2_SAMPLE_PACKAGE=y
+# DTC.
+BR2_PACKAGE_HOST_DTC=y
 
 # We were tempted to do this to disable S40network neatly,
 # but that package also creates extra configuration files
 # such as /etc/network/interfaces which we need. So we just
 # remove the init.d file for now.
 #BR2_PACKAGE_IFUPDOWN_SCRIPTS=n
-
-# misc packages
-BR2_PACKAGE_DHRYSTONE=y
-BR2_PACKAGE_FILE=y
-BR2_PACKAGE_PCIUTILS=y
-BR2_PACKAGE_STRACE=y
-
-# lscpu: TODO not installing?
-BR2_PACKAGE_UTIL_LINUX=y
-BR2_PACKAGE_UTIL_LINUX_BINARIES=y
-# taskset
-BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y
-
-# gdbserver
-BR2_PACKAGE_GDB=y
-
-# ftrace
-BR2_PACKAGE_TRACE_CMD=y
-
-# DTC
-BR2_PACKAGE_DTC=y
-BR2_PACKAGE_DTC_PROGRAMS=y
-BR2_PACKAGE_HOST_DTC=y

release

@@ -6,6 +6,7 @@ import subprocess
 import common
 zip_img = imp.load_source('zip_img', os.path.join(common.root_dir, 'zip-img'))
 
+subprocess.check_call([os.path.join(common.root_dir, 'test')])
 subprocess.check_call([os.path.join(common.root_dir, 'build-all')])
 zip_img.main()
 tag = 'sha-{}'.format(common.sha)