Kernel debugging essentials workshop: Difference between revisions

From base48
imported>B42
imported>Klubko
mNo edit summary
 
(6 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Event|2014-02-19 20:00}}
== Topics ==
== Topics ==
[[File:angry-tux.png|right]]


We'll be holding a workshop on basics of finding and debugging (and eventually fixing) common issues with Linux kernel.
We'll be holding a workshop on basics of finding and debugging (and eventually fixing) common issues with Linux kernel.
Line 10: Line 14:


* Some elementary knowledge of C
* Some elementary knowledge of C
* A laptop with Linux (Fedora 20 preferred, we'll try to cover older versions and other distribution too) and root access, 2G + size of RAM free space
* A laptop with Linux (Fedora 20 preferred, Debian 7 fine, we'll try to cover older versions and other distribution too) and root access, 2G + size of RAM free space
** If you don't have any available, please let me know in advance
** If you don't have any available, please let me know in advance
== Language ==
I'm fine with holding the workshop in English language if there's anyone who doesn't speak Czech/Slovak.


== Setup instructions ==
== Setup instructions ==


TBD: Describe how should people prepare their laptops in advance so that we don't waste too much time at the workshop.
Please prepare your laptop in advance so that we don't waste too much time at the workshop.


Extra hardware will be available for ones who won't be able to bring their own laptops.
Extra hardware will be available for ones who won't be able to bring their own laptops.


== Language ==
=== Install required packages ===
 
{|
! Debian
! Fedora
|- style="vertical-align:top;"
|
# <code>apt-get -y install kdump-tools crash git linux-headers linux-image-$(uname -r)-dbg linux-source ncurses-dbg</code>
|
# <code>pkcon install kernel{-debug,}{-devel,} kexec-tools crash git gcc ncurses-devel
debuginfo-install -y kernel{-debug,}</code>
|}
 
=== Get the kernel source tree ===
 
<code>git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git</code>
 
== Cheatsheet ==
 
Here are some distro-dependent details. I'll be using Fedora, follow these to translate to Debian equivalents if you're running Debian.
 
These are here just for reference, no need to follow these prior to the workshop.
 
=== Enable kdump ===
 
{|
! Debian
! Fedora
|- style="vertical-align:top;"
|
# Edit <code>/etc/default/kdump-tools</code>
# Add <code>crashkernel=256M</code> to <code>/etc/default/grub</code>
# Run <code>update-grub2</code>
# Reboot
# Check <code>kdump-config status</code>
|
# <code>system-config-kdump</code>
 
or
 
# Add <code>crashkernel=256M</code> to <code>/etc/default/grub</code>
# Run <code>grub2-mkconfig >/etc/grub.conf</code>
# <code>systemctl enable kdump.service</code>
# Reboot
# <code>Check systemctl status kdump.service</code>
|}
 
=== Enable netconsole ===


I'm fine with holding the workshop in English language if there's anyone who doesn't speak Czech/Slovak.
{|
! Debian
! Fedora
|- style="vertical-align:top;"
|
# <code>modprobe netconsole \</code><br><code>    netconsole=@/,514@$(getent hosts odvarok.local |</code><br><code>    awk '{print $1}')/</code>
|
# Edit <code>/etc/sysconfig/netconsole</code>
# <code>systemctl enable netconsole.service</code>
# <code>systemctl start netconsole.service</code>
|}


== Time & Attendance ==
=== Running crash ===


Estimated time is 2 hours; we may have a follow-up if needed.
The location of the raw <code>vmlinux</code> binary from the debugging packages differs in distros.
If you're interested in attending, please your time preference (after February 13th), +1 or -1 below if any of the existing suits you or not:


* Saturday, February 15th; 16:00: lubko,b42 +2
{|
* Sunday, February 16th: 16:00: lubko,trendy +2
! Debian
* Monday, February 17th: 16:00: trendy +1
! Fedora
* Tuesday, February 18th: 20:00: lubko,trendy,b42 +3
|- style="vertical-align:top;"
* Wednesday, February 19th: 20:00: lubko,trendy,b42 +3
|
* Thursday, February 20th: 20:00: lubko,trendy,b42 +3
# <code>crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/201402090936/dump.201402090936</code>
* Friday, February 21th: 20:00: lubko,b42 +2
|
(feel free to propose more)
# <code>crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/127.0.0.1-2013.10.24-02:48:40/vmcore</code>
|}


== Materials ==
== Materials ==


Work-in-progress slides. Content will be extended and polished (as I learn beamer...), but outline will stay roughly the same: [[File:kernel-workshop-draft.pdf.gz]]
Work-in-progress slides. Content will be extended and polished (as I learn beamer...), but outline will stay roughly the same: [[File:kernel-workshop.pdf.gz]]
 
* https://github.com/lkundrak/sad-ugly-and-broken This is a repository for variously broken kernel code, for demonstration
* https://github.com/lkundrak/kexec-tools Patched kexec-tools, for >= 3.11 kernels
* http://v3.sk/~lkundrak/firescope/ Firewire debugging tool

Latest revision as of 10:31, 19 February 2014

Topics

File:Angry-tux.png

We'll be holding a workshop on basics of finding and debugging (and eventually fixing) common issues with Linux kernel.

Topics I'd initially cover would include leveraging in-kernel debugging facilities, techniques for capturing debug output from various crashes, understanding the captured data, and modifying, building and loading the altered kernel code with more debugging code or an actual fix.

Original mailing list reference: [1]

Requirements

  • Some elementary knowledge of C
  • A laptop with Linux (Fedora 20 preferred, Debian 7 fine, we'll try to cover older versions and other distribution too) and root access, 2G + size of RAM free space
    • If you don't have any available, please let me know in advance

Language

I'm fine with holding the workshop in English language if there's anyone who doesn't speak Czech/Slovak.

Setup instructions

Please prepare your laptop in advance so that we don't waste too much time at the workshop.

Extra hardware will be available for ones who won't be able to bring their own laptops.

Install required packages

Debian Fedora
  1. apt-get -y install kdump-tools crash git linux-headers linux-image-$(uname -r)-dbg linux-source ncurses-dbg
  1. pkcon install kernel{-debug,}{-devel,} kexec-tools crash git gcc ncurses-devel

debuginfo-install -y kernel{-debug,}

Get the kernel source tree

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Cheatsheet

Here are some distro-dependent details. I'll be using Fedora, follow these to translate to Debian equivalents if you're running Debian.

These are here just for reference, no need to follow these prior to the workshop.

Enable kdump

Debian Fedora
  1. Edit /etc/default/kdump-tools
  2. Add crashkernel=256M to /etc/default/grub
  3. Run update-grub2
  4. Reboot
  5. Check kdump-config status
  1. system-config-kdump

or

  1. Add crashkernel=256M to /etc/default/grub
  2. Run grub2-mkconfig >/etc/grub.conf
  3. systemctl enable kdump.service
  4. Reboot
  5. Check systemctl status kdump.service

Enable netconsole

Debian Fedora
  1. modprobe netconsole \
    netconsole=@/,514@$(getent hosts odvarok.local |
    awk '{print $1}')/
  1. Edit /etc/sysconfig/netconsole
  2. systemctl enable netconsole.service
  3. systemctl start netconsole.service

Running crash

The location of the raw vmlinux binary from the debugging packages differs in distros.

Debian Fedora
  1. crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/201402090936/dump.201402090936
  1. crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/127.0.0.1-2013.10.24-02:48:40/vmcore

Materials

Work-in-progress slides. Content will be extended and polished (as I learn beamer...), but outline will stay roughly the same: File:Kernel-workshop.pdf.gz