Tracing and Visualizing File System Internals with eBPF Superpowers

Linux kernel storage stack consists of several interconnected layers including Virtual File System (VFS), block layer and device driver. VFS provides the main interface to userspace applications and it is where the files and directories are being handled. As we go deep, much of the accesses are translated to actual IO operations in the block layer in the kernel. Investigating storage performance issues requires a full insight into all these layers. In this talk, we begin by discussing the journey of a simple filesystem call from userspace all the way into the kernel. We explain how tools like Ftrace can be used to understand control flow inside the kernel. Once we understand the “points of interest” in the control flow of how the kernel handles the request from userspace, we then move on to discuss eBPF based approaches to compute meaningful storage performance/security metrics. We will showcase this with our small and nifty framework that includes a visualization system with different graphical views that represent the collected information about disk accesses in a convenient way. The goal of our talk is not just to show “yet another iotop like tool”, but to highlight the versatility of eBPF VM in the linux kernel that now allows developing targeted, plug and play tools to gather precise data about a system’s activity for security and performance debugging. To this end, we will explain in-depth what actually happens when such targeted eBPF based probing is used to extract meaningful data from the kernel. We explain the plumbing behind simple observability tools such as biolatency, vfsstat etc. [1] that have been built using eBPF and how to build a custom tool yourself. [https://github.com/iovisor/bcc#tools]

Learning Objectives

Understand the different FS layers in the kernel with a live and visual example,Discovering “points of interest” in the different storage layers in Linux kernel and how they can be used to calculate metrics such as block IO request latency (for example, by calculating delta between points of interest like “blk_start_request” and “bld_account_io_done”),Understand what happens "under the hood" when you probe these points of interest with eBPF based tools.,Answer deeper questions like, “Is a VM running in linux kernel safe?”, “Can I access any kernel data structure?”, “What are other alternatives” etc,How to take the data collected from your custom eBPF tools and present it in modern visualization systems such as Grafana