Testing & Fuzzing Microconference Notes

Welcome to Linux Plumbers Conference 2017

The structure will be short introductions to an issue or topic followed
by a discussion with the audience. A limit of 3 slides per presentation
is enforced to ensure focus and allocate enough time for discussions.

IF SOMEONE PRESENTS MORE THAN 3 SLIDES, PLEASE HECKLE AND THROW THINGS AT THEM!

Please use this Etherpad to take notes. Microconf leaders will be giving
a TWO MINUTE summary of their microconference during the Friday
afternoon closing session.

Please remember there is no video this year, so your notes are the only
record of your microconference.

Miniconf leaders: Please remember to take note of the approximate number
of attendees in your session(s).

Please help us take notes at: https://etherpad.openstack.org/p/LPC2017_Testing_Fuzzing

WIKI: http://wiki.linuxplumbersconf.org/2017:testing_fuzzing

SCHEDULE

- Fuzzers panel: Panel + open discussion regarding progress since last year and future goals
- Upstream testing
- The 0day testing project
- ktest
- KMSAN
- Updates on KASAN/KTSAN
- drm/i915 testing
- Status update on formal verification tools
- (BoF?) Stable/distro testing

Introduction, BOF Recap

Sasha started with discussing about the BoF on Testing last night.
Talking adding testing to CI.

kernelci is getting new test added for driver, device testing
other than kernelci, not many options for stable maintainers to do testing
Recommendation to backport kselftest patches to the -stable trees

Sasha: it's important for distros to send their distro commits to stable

Currently being done by scraping changelogs from Ubuntu
Would be better if distros could package this up and send to stable periodically

Are reverts OK in the stable tree?

Sometimes whitespace,etc changes need to be reverted to apply patches to stable
Distro concern: we have a kernel that we have which works for users, and we want to keep that patch until upstream has the "right" fix...

Fuzzers panel: Panel + open discussion regarding progress since last year and future goals

Have bugs fonud by fuzzers to go as regression tests in kselftest
Trinity will include better testing for eBPF programs
Some suggest that we should push out patches to Linux next even in rc6 or later. But there's no real point to push those patches to
- linux-next becaues it's simply not tested for those changes.
https://github.com/kernelslacker/trinity
https://github.com/google/syzkaller

Ktest:
    Automatic testing on a series of patches, maintaining bisectability
    Performs complete testing for each patch: build, install, boot, etc
    Able to bisect kernel configs
    TODO: email support

Intel GFX CI:
    Tests actual hardware
    Running with a lot of debug configs, KASAN etc
    Pre-merge testing of patch series sent by devs
    They run a fork of patchwork for patch series, full test results are available on the Web, sent back to MLs
    Track persistent test failures to avoid reporting them every time
    People get very attached to patches that are merged, so it's critical to perform the testing before the patch lands
    There're a lot of regressions on linux-next in other subsystems
    Nobody is testing their code before pushing it to linux-next, assuming someone does the testing - this makes it impossible to test
    Suspend/Resume is an area which should be better tested -- very popular and not good coverage now

KTF: Kernel testing framework
    A kernel module that uses gTest as a userspace frontend and communicates to the kernel to execute the actual tests
    Let us make the devs hooked on testing, so they enjoy writing the tests
    Need a shift towards TDD; it's important that the tests must be committed to the kernel tree
    "Developers are bad at testing, needs to be a dedicated person" vs. "Tests written by the dev are better than no tests"

"The further out from the developer that a bug is found, the more expensive it is to fix that bug"
KTF link: https://github.com/knuto/ktf

KMSAN KernelMemorySanitizer

fine-grained detection of uninitialized variables
shadow memory: bit to bit shadow mapping
future work ideas:
- taint analysis - use shadow mapping to mark data coming from untrusted source
- use origin to mark place where value was obtained
- fuzzing assistance - determine which arguments to a function are more useful to mutute
- or... mark origin for arguments to syscalls and trace them through the kernel execution flow
- CVE-2017-1000380: is there a way to prevent this/do someting to kill uninit bugs?
- (probably not a good idea to s/kmalloc/kzalloc/g?)
https://github.com/google/kmsan
(requires clang patches)

kernelABI Specification Program

Follow up from 2016: human & machine readable descriptors for kernel ABI
problem: someone has to write the code
- easy part: skeleton for definitions
- hard part: defining specific constraints for each syscall
- no agreement of how to describe such constraints
can we automate this process, vs requiring a maintainer to keep this up to date. Would libclang work?
- reference: android project trouble (standardizing abi across versions)
- other tools: abidiff (works on exported symbols from kernel modules, extend to syscalls?)
alternative, have a way to generate the syscalls instead of mining them

kernel unit test discussion, continued

Q: what do you do for projects where you have to do 90% of the work before you can run the first test?
A: Need to decompose into smaller testable components during the 90% time.
Proposal for in-tree unit testing for the Linux kernel (submit tests alongside patches)
- in a parallel tree or in the same tree
- Concern: if tests are hard to run, they end up as bitrot -- need to be maintained