prebuilt: attempt gem5 a bit further, but stop at the vmlinux step

Author: cirosantilli

README.adoc

@@ -41,7 +41,7 @@ The design goals are to provide setups that are:
 * highly automated: "just works"
 * thoroughly documented: you know what "just works" means
 * can be fully built from source: to give visibility and allow modifications
-* can also use prebuilt binaries as much as possible: in case you are lazy or unable to build from source
+* can also use <<prebuilt, prebuilt binaries>> as much as possible: in case you are lazy or unable to build from source
 
 == Getting started
 
@@ -522,17 +522,24 @@ Our prebuilts currently include:
 * Linux kernel
 * root filesystem
 
-Advantages: saves time and disk space.
+Advantage: saves time and disk space on the initial install.
 
-Limitations:
+The limitations are severe however:
 
 * can't <<gdb,GDB step debug the kernel>>, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: <<prebuilt-toolchain>>.
 +
 Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.
 * can't create new modules or modify the existing ones, since no cross toolchain
-* can't use things that rely on our QEMU fork, e.g. in-fork <<device-models>> or <<tracing>>
 * you won't get the latest version of this repository. Our <<travis>> attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us.
-* <<gem5>> is not currently supported, although it should not be too hard to do. One annoyance is that there is no Debian package for it, so you have to compile your own, so you might as well just build the image itself.
+* <<gem5>> is not currently supported, although it should not be too hard to do.
++
+Annoyances:
++
+** there is no Debian package for it, so you have to compile your own, so you might as well just build the image itself
+** it does not handle <<gem5-qcow2,qcow2>>, and we haven't gotten <<squashfs>> to work yet, therefore we would have to either distribute large ext2 images, or constantly fight with <<br2_target_rootfs_ext2_size>>
+** QEMU uses  bzImage and gdm5 the raw `vmlinux`, and we don't want to distribute the same thing twice... https://github.com/torvalds/linux/blob/master/scripts/extract-vmlinux We could use `extract-vmlinux`, but that would imply cloning the Linux kernel, which is at least half of the kernel build time :-)
+
+This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the <<qemu-buildroot-setup,full featured setup>> while you take a coffee or a nap? :-)
 
 ==== Prebuilt setup getting started
 
@@ -562,18 +569,32 @@ git submodule update --init "$(./getvar qemu_src_dir)"
 
 and you are done.
 
-Alternatively, you can also try to use the host QEMU:
+Alternatively, you can also try to use the host QEMU directly with:
 
 ....
-sudo apt-get install qemu-system-x86
+sudo apt-get install qemu-system-x86 qemu-utils
 ./run --prebuilt
 ....
 
 but QEMU builds are pretty quick, and this further increases the probability of incompatibilities, are you really that lazy?
 
+////
+For gem5, do:
+
+....
+sudo apt-get install qemu-utils
+./build-gem5
+./run --gem5 --prebuilt
+....
+
+`qemu-utils` is required because we currently distribute `.qcow2` files which <<gem5-qcow2,gem5 can't handle>>, so we need `qemu-img` to extract them first.
+////
+
 [[host]]
 === Host kernel module setup
 
+**THIS IS DANGEROUS, YOU HAVE BEEN WARNED**
+
 This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.
 
 It has however severe limitations, and you will soon see that the compilation time and disk usage are well worth it:
@@ -6488,8 +6509,12 @@ index 17498c42b..76b8b351d 100644
 
 The directory of interest is `src/dev/storage`.
 
+=== gem5 qcow2
+
 qcow2 does not appear supported, there are not hits in the source tree, and there is a mention on Nate's 2009 wishlist: http://gem5.org/Nate%27s_Wish_List
 
+This would be good to allow storing smaller sparse ext2 images locally on disk.
+
 === Snapshot
 
 QEMU allows us to take snapshots at any time through the monitor.

common.py

@@ -219,16 +219,32 @@ def print_time(ellapsed_seconds):
     minutes, seconds = divmod(rem, 60)
     print("time {:02}:{:02}:{:02}".format(int(hours), int(minutes), int(seconds)))
 
-def raw_to_qcow2():
+def raw_to_qcow2(prebuilt=False, reverse=False):
     global this
+    if prebuilt:
+        qemu_img_executable = this.qemu_img_basename
+    else:
+        qemu_img_executable = this.qemu_img_executable
+    infmt = 'raw'
+    outfmt = 'qcow2'
+    infile = this.rootfs_raw_file
+    outfile = this.qcow2_file
+    if reverse:
+        tmp = infmt
+        infmt = outfmt
+        outfmt = tmp
+        tmp = infile
+        infile = outfile
+        outfile = tmp
     assert this.run_cmd([
-        this.qemu_img_executable,
+        qemu_img_executable,
+        # Prevent qemu-img from generating trace files like QEMU. Disgusting.
         '-T', 'pr_manager_run,file=/dev/null',
         'convert',
-        '-f', 'raw',
-        '-O', 'qcow2',
-        this.rootfs_raw_file,
-        this.qcow2_file,
+        '-f', infmt,
+        '-O', outfmt,
+        infile,
+        outfile,
     ]) == 0
 
 def resolve_args(defaults, args, extra_args):
@@ -337,13 +353,15 @@ def setup(parser, **extra_args):
     this.linux_variant_dir = '{}.{}'.format(this.linux_build_dir, args.linux_build_id)
     this.vmlinux = os.path.join(this.linux_variant_dir, "vmlinux")
     this.qemu_build_dir = os.path.join(this.out_dir, 'qemu', args.qemu_build_id)
-    this.qemu_executable = os.path.join(this.qemu_build_dir, '{}-softmmu'.format(args.arch), 'qemu-system-{}'.format(args.arch))
-    this.qemu_img_executable = os.path.join(this.qemu_build_dir, 'qemu-img')
+    this.qemu_executable_basename = 'qemu-system-{}'.format(args.arch)
+    this.qemu_executable = os.path.join(this.qemu_build_dir, '{}-softmmu'.format(args.arch), this.qemu_executable_basename)
+    this.qemu_img_basename = 'qemu-img'
+    this.qemu_img_executable = os.path.join(this.qemu_build_dir, this.qemu_img_basename)
     this.qemu_guest_build_dir = os.path.join(this.build_dir, 'qemu-custom')
     this.host_dir = os.path.join(this.buildroot_build_dir, 'host')
     this.host_bin_dir = os.path.join(this.host_dir, 'usr', 'bin')
     this.images_dir = os.path.join(this.buildroot_build_dir, 'images')
-    this.rootfs_raw_file = os.path.join(this.images_dir, 'rootfs.squashfs')
+    this.rootfs_raw_file = os.path.join(this.images_dir, 'rootfs.ext2')
     this.qcow2_file = this.rootfs_raw_file + '.qcow2'
     this.staging_dir = os.path.join(this.buildroot_build_dir, 'staging')
     this.target_dir = os.path.join(this.buildroot_build_dir, 'target')

release

@@ -7,6 +7,11 @@ import common
 zip_img = imp.load_source('zip_img', os.path.join(common.root_dir, 'zip-img'))
 
 subprocess.check_call([os.path.join(common.root_dir, 'test')])
+# A clean release requires a full rebuild unless we hack it :-(
+# We can't just use our curent build as it contains packages we've
+# installed in random experiments. And with EXT2: we can't easily
+# know what the smallest root filesystem size is and use it either...
+# https://stackoverflow.com/questions/47320800/how-to-clean-only-target-in-buildroot
 subprocess.check_call([os.path.join(common.root_dir, 'build-all')])
 zip_img.main()
 tag = 'sha-{}'.format(common.sha)

run

@@ -3,6 +3,7 @@
 import os
 import re
 import shlex
+import shutil
 import subprocess
 import sys
 import time
@@ -95,7 +96,14 @@ def main(args, extra_args=None):
         do_trace = True
         trace_type = args.trace
 
+    def raise_rootfs_not_found():
+        raise Exception('Root filesystem not found. Did you build it?\n' \
+                        'Tried to use: ' + common.rootfs_raw_file)
     if args.gem5:
+        if not os.path.exists(common.rootfs_raw_file):
+            if not os.path.exists(common.qcow2_file):
+                raise_rootfs_not_found()
+            common.raw_to_qcow2(prebuilt=args.prebuilt, reverse=True)
         os.makedirs(os.path.dirname(common.gem5_readfile), exist_ok=True)
         with open(common.gem5_readfile, 'w') as readfile:
             readfile.write(args.gem5_readfile)
@@ -161,17 +169,18 @@ def main(args, extra_args=None):
         os.makedirs(common.run_dir, exist_ok=True)
         if args.prebuilt:
             common.mkdir()
-            qemu_executable = "qemu-system-{}".format(args.arch)
+            qemu_executable = common.qemu_executable_basename
+            qemu_found = shutil.which(qemu_executable) is not None
         else:
             qemu_executable = common.qemu_executable
-        if not os.path.exists(qemu_executable):
-            raise Exception('QEMU executable does not exist, did you forget to build or install it?\n' \
+            qemu_found = os.path.exists(qemu_executable)
+        if not qemu_found:
+            raise Exception('Could not find the QEMU executable, did you forget to build or install it?\n' \
                             'Tried to use: ' + qemu_executable)
         if not os.path.exists(common.qcow2_file):
             if not os.path.exists(common.rootfs_raw_file):
-                raise Exception('Root filesystem not found. Did you build it?\n' \
-                                'Tried to use: ' + common.rootfs_raw_file)
-            common.raw_to_qcow2()
+                raise_rootfs_not_found()
+            common.raw_to_qcow2(prebuilt=args.prebuilt)
         if args.debug_vm:
             serial_monitor = []
         else:
@@ -401,7 +410,7 @@ Default: %(default)s
     )
     parser.add_argument(
         '-P', '--prebuilt', default=defaults['prebuilt'], action='store_true',
-        help='Run the downloaded prebuilt images.'
+        help='Run the downloaded prebuilt images with pre-packaged host tools.'
     )
     group = parser.add_mutually_exclusive_group()
     group.add_argument(