How to Hack Your Own Linux System

Passwords are the sole criteria of system Security for most of the System. And when it comes toLinux, if you know the root password you owns the machine. Passwords are as a Securitymeasure for BIOS, Login, Disk, Application, etc.

Linux is considered to be the most Secure Operating System to be hacked or cracked and in reality it is, still we will be discussing some of the loop-holes and exploits of a Linux System. We will be using CentOS Linux throughout the article as an article to crack our own machine’s security.

Press any key to interrupt the boot, as soon as Linux machine boots and you will get a GRUBmenu.

Linux Boot Screen

Press ‘e‘ to edit and go to the line starting with kernel (Generally 2nd Line).

Switch to Single User Mode

Now press ‘e‘ to edit the kernel and add ‘1‘ at the end of line (after one blank space) forcing it to start in single user mode and thus prohibiting it to enter default run-level. Press ‘Enter’ to close the kernel editing and then boot to the altered option. For booting You need to press ‘b‘

Logged into Single User Mode

Now you are logged in to single-user mode.

Set root Password

Yeah! Now using ‘passwd‘ command we can change the root password. And once you have root password you owns the Linux Machine – Don’t you Remember? You can now switch to graphical screen to edit anything and everything.

Add new root Password

Note: In case the above ‘passwd‘ command doesn’t work for you and you didn’t get any output, it simply means that your SELinux is in enforcing mode and you need to disable it first, before proceeding further. Run following command at your prompt.

# setenforce 0

An then run the ‘passwd‘ command, to change root password. Moreover command.

Switch to X Windows

Use command “init 5” (Fedora Based) systems and “gdm3” (Debian Based) systems.

Switch to X Window

So was this not a cake-walk to hack a Linux box? Think about the scenario if somebody did this to your server, Panic! Now we will be learning how to safeguard our Linux Machine from being modified using single user mode.

How we breaked into the system? Using Single-user mode. OK, so the loophole here was – logging into single user mode without the need of entering any password.

Fixing this loophole i.e., password protecting the single user mode.

open file “/etc/rc1.d/S99single” in your favourite editor and search for line.

exec init -t1 s

Just add the following line above it. save it an exit.

exec sbin/sulogin

Before

Before Preview

After

After Preview

Now before entering single user mode you will need to provide root password to proceed. Check again trying to enter single user mode after these changing above said file.

Enter Root Password for Single User Mode

Why don’t you check it, Yourself.

Hack Your Linux System Without Using Single User Mode

OK, so now you will be feeling better that your system is secure. However this is partially true. It is true that your Linux Box can’t be cracked using single user mode but still it can be hacked the other way.

In the above step we modified the kernel to enter single user mode. This time also we will be editing the kernel but with a different parameter, let us see how ?

As a kernel parameter we added ‘1‘ in the above process however now we will be adding ‘init=/bin/bash’ and boot using ‘b‘.

Add ‘init=/bin/bash’

And OOPS you again hacked into your system and the prompt is enough to justify this.

Hacked into Your System

Now Trying to change the root password using the same process as stated in the first method using ‘passwd‘ command, we got something like.

Changing Root Password

Reason and Solution?

1. Reason: The root (/) partition is mounted Read only. (Hence password was not written).
2. Solution: Mount the root (/) partition with read-write permission.

To mount the root partition with read-write permission. Type the following command exactly.

# mount -o remount,rw /

Mount / Partition in Read Write

Now again try to change the password of root using ‘passwd‘ command.

Change Password of root

Hurrah! You hacked into your Linux System once again. Ohhh man is the system so easy to exploit. No! the answer is no. All you need is to configure your system.

All the above two process involved tweaking and passing parameters to kernel. So if we do something to stop kernel tweaking obviously our Linux box would be Secure and not that easy to break. And in order to stop kernel editing at boot we must provide password to boot loader, i.e.,password protect the grub (Lilo is another bootloader for Linux but we won’t be discussing it here) boot loader.

Provide encrypted password to bootloader using ‘grub-md5-crypt‘ followed with your password. First encrypt the password

Password Protect Boot Loader

Copy the above encrypted password, exactly as it is and keep it safe we will be using it in our next step. Now open your ‘grub.conf‘ file using your favourite editor (location might be: /etc/grub.conf) and add the line.

password --md5 $1$t8JvC1$8buXiBsfANd79/X3elp9G1

Change “$1$t8JvC1$8buXiBsfANd79/X3elp9G1” with your encrypted password which you generated above and copied it safely to some other location.

The “grub.conf” file after inserting the above line, save and exit.

grub.conf Preview

Now Cross Checking, editing the kernel at boot, we got.

Cross Cheking Boot Loader

Now you would be breathing that you system is fully secure now and not prone to hack, however still the game is not over.

You better know that you can enforce rescue mode to remove and modify the password using a bootable image.

Just put your installation CD/DVD in your drive and select Rescue Installed System or use any other rescue image, you could even use a Live Linux Distro, mount the HDD and edit the ‘grub.conf‘ file to remove password line, reboot and again you are logged in.

Note: In rescue mode Your HDD is mounted under ‘/mnt/sysimage‘.

# chroot /mnt/sysimage
# vi grub.conf (remove the password line)
# reboot

I know you would be asking- so where is the end. Well i would say is to.

1. Password protect your BIOS.
2. Change you Boot order to HDD first, followed by rest (cd/dvd, network, usb).
3. Use Password sufficiently Long, Easy to remember, Hard to guess.
4. Never write Your Password to anywhere.
5. Obviously use Uppercase, Lowercase, Numbers and Special Character in your password thus making it hard to break.

This guide was just to make you aware of facts and tell you how to secure your System.Tecmint.com and the writer of this article strongly discourage this guide as a base of exploiting other’s system. It is the sole responsibility of the reader if they engage in any such activity and for such kind of act neither the write nor Tecmint.com will be responsible.

Your positive comments makes us feel good and encourages us and that is always sought from you. Enjoy and Stay Tuned.

Linux Kernel Hacking For Advanced Administration

Kernel Hacking Lesson #0: Check for Materials

Your very first assignment is to say, "I can hack the kernel. Kernel hacking is not magic," out loud. :) Good, that's the most important part. Now, read on...

What you need:

1. A Linux box
2. Root on this Linux box (or a sympathetic admin)
3. The ability to reboot this box several times a day
4. A compilation environment installed
5. Some way to get the kernel source

In more detail:

1. Your Linux box needs to have at least 300 MB of free disk space. Find out how much disk space you have free with the command df -h. If you need tips for clearing out some disk space, ask. Note that you can compile on one Linux box and run the kernel you compile on another Linux box. You should compile on the fastest Linux box possible. A 500 Mhz Pentium III with 256 MB RAM will take around 40 minutes to compile a Linux kernel, a 1200 MHz Pentium III with 640 MB RAM will take around 10 minutes, whereas a Duron 800 Mhz with 256 MB RAM will take around 5 minutes.
   I've personally worked with x86's, Alpha's, and PowerPC's, but if you have something else I'm willing to help you figure it out. When asking questions, be sure to let people know if you are compiling for something other than an x86.
2. You don't _need_ root, but you at least need someone with root to set you up. What you need is the ability to boot your new kernel, which may include the ability to run LILO, to copy kernels to places outside your home directory, and the ability to reboot the machine. I do 99% of my kernel work as a non-root user. I compile in a directory owned by user val, and I made the directories I need to write to owned by user val. You can also give the user reboot a password, so you can reboot the machine without su-ing to root. Talk to your sysadmin to find out what you need to do.
3. Ideally, your Linux box is something only you need. If other people need it during the day but don't at night, you can probably still use it. If you are concerned about making the box unusable, there are many ways to test a new kernel and then reboot back into your old kernel. Disk corruption is not as common as you think, but if you do not want to risk the data on your hard disk, you can use a ramdisk with your new kernel. (Ramdisks are explained later on.) You can also create a new partition on your hard disk and use that as your root filesystem when you are experimenting with new kernels.
4. Hopefully, most people installed their Linux box with gcc and other compilation tools included. You can test to see if you have a compilation environment installed by copying the following into a file named hello.c:
   

      #include <stdio.h>
   
      int
      main (void)
      {
    printf ("Hello, world!\n");
    return 0;
      }
   

   And then compiling and running it with:
   

      $ gcc hello.c -o hello
      $ ./hello
   

   If this doesn't work, you will need to install a compilation environment. Usually the easiest way to do this is to just re-install your machine.
5. There are many different ways to get the kernel source, but my preferred one involves having a reasonably high bandwidth net connection. Otherwise, you can get the kernel source from your distribution CD's.
Remember, if you have any questions, please ask them on the grrls-only list, instead of emailing someone privately. The list members will answer your questions politely and helpfully, and other list members will learn from the answers to your questions. Our goal is to never answer a question with "FAQ, read #14 on XYZ," or "Do a web search," or "RTFM." Communicating with other kernel developers is just as important as finding the answer to your question.

Kernel Hacking Lesson #1: Get the Source

Now, you need to get the Linux source. There are many different ways of getting the Linux source, but I'll list them in order of preference. You only need to choose one of these options. While it's easiest to just install the source on your distribution CD, you'll need to stay up to date with the latest version of the Linux source if you want to participate in kernel development.
Linus recommends not having a /usr/src/linux symbolic link to your kernel source tree. However, the /usr/src/linux link will sometimes be used as a default by some modules that compile outside the kernel. If you need it, make sure it points to the correct kernel tree. If it doesn't, modules that compile outside the kernel may use the wrong header files.

• Use BitKeeper to clone a Linux source repository. This is what I use, and it's what many other kernel hackers use, including the entire PowerPC team, Ted T'so, Rik van Riel, and Linus himself. Be warned, this requires downloading many megabytes of data, so don't use it if you have a slow link. This option also requires at least 1 GB of free disk space, which is significantly more than the other options.
  First, get BitKeeper:
  http://www.bitmover.com/cgi-bin/download.cgi
  Follow the instructions and it will tell you how to download and install BitKeeper.
  Then, clone the main Linux tree using BitKeeper:
  

     $ cd /usr/src

  (or wherever you would like to work)

     $ bk clone bk://linux.bkbits.net/linux-2.6 linux-2.6
     $ cd linux-2.6
     $ bk -r co

  And now you have a kernel source tree! Learn more about how to use BitKeeper here:
  http://www.bitkeeper.com/Test.html
• FTP the kernel source from ftp.kernel.org. This is another bandwidth intensive operation, so don't use it if you have a slow link.
  FTP to your local kernel mirror, which is named:
  

     ftp.<country code>.kernel.org

  For example, I live in the US, so I do:

     $ ftp ftp.us.kernel.org

  Login (if necessary) with username ftp or anonymous. Change directory to pub/linux/kernel/v2.6 and download the latest kernel. For example, if the latest version is 2.6.9, download the file named:

     linux-2.6.9.tar.gz

  Usually there is a file named LATEST-IS-<version> to tell you what the latest version is. I recommend the gzipped format (filename ending in .gz) instead of bzip2 format (filename ending in .bz2) since bzip2 takes a long time to decompress.
  Untar and uncompress the file in the directory where you are planning to work.
  You now have your Linux kernel source.
• Install the kernel source from your distribution CD. If you already have a directory named /usr/src/linux and it contains more than one directory, you already have the source installed. If you don't, read on.
  Mount your installation CDROM. On a RedHat based system, the source RPM is usually in <DistributionName>/RPMS/ and is named:
  

     kernel-source-<version>.<arch>.rpm

  One way to find the kernel source package is to run this command, assuming your CD is mounted at /mnt/cdrom:
  

     $ find /mnt/cdrom -name \*kernel-\*

  Install the RPM using:

     $ rpm -iv <pathname>/kernel-source-<version>.<arch>.rpm

  The v switch will tell you if it fails or not.
  If your system is not RPM-based, please let us know how to install the kernel source from your distribution CD.
• There are various other ways to get the kernel source, involving CVS or rsync. If you would like to write instructions for one of these other methods of getting the kernel source, go ahead and we'll include them here.
Why do I recommend BitKeeper over ftp, and ftp over vendor source?
BitKeeper handles creating patches for you. (A patch contains the differences between one source tree and another source tree.) BitKeeper also applies the latest changes for you. When I want to update my tree, I just type:


   $ bk pull

And most of the time, it just works. The only time I have to do work is if I have written code that conflicts with the new code downloaded from the parent tree.

When I want to make a patch to send to somebody, I just type:


   $ bk citool
   $ bk -hr<latest revision number> | bk gnupatch > newpatch

When I want to undo my latest changes, I type:


   $ bk undo -r<latest revision number>

BitKeeper has all kinds of pretty GUI tools to make debugging and merging code easier.

I like BitKeeper so much I wrote a paper about it. You can find the paper and some slides on my web page:
http://www.nmt.edu/~val
I prefer the vanilla kernel source from ftp.kernel.org over the vendor source because you never know what changes the vendor has made to the source. Most of the time, the vendor has improved the tree, but often they make changes of dubious value to a kernel hacker. For example, RedHat 6.2 shipped a kernel that compiled differently depending on whether you were running an SMP kernel at the time of compilation. This makes sense if you are just recompiling a kernel for that machine, but it was useless if you were trying to compile a kernel for a different machine.
The other reason to use vanilla source instead of vendor source is if you want to create and send patches to other kernel developers. If you create a patch and send it to the Linux kernel mailing list for inclusion in the main kernel tree, it had better be against the latest vanilla source tree. If you create a patch on a vendor source tree, it's unlikely to apply to a vanilla source tree.
The one place where vendor source is crucial is for non-x86 architectures. The vanilla source almost never builds and boots on an architecture other than x86. Often, the best place to get a working kernel for a non-x86 is the vendor source. Each architecture usually has some quirky way of getting the latest source in addition to the vendor-supplied source.

Kernel Hacking Lesson #2: Configure Your Kernel

The Linux kernel comes with several configuration tools. Each one is run by typing make <something>config in the top-level kernel source directory. (All make commands need to be run from the top-level kernel directory.)

make config

• This is the barebones configuration tool. It will ask each and every configuration question in order. The Linux kernel has a LOT of configuration questions. Don't use it unless you are a masochist.

make oldconfig

• This will take the config file named .config and only ask the configuration questions which are not already answered in that file. This is most often used when upgrading to a more recent version of the kernel.

make menuconfig

• This is the kernel hacker mainstay. This command pops up a text-based menu-style configurator using the ncurses library. You can descend into the menus that interest you and change only the configuration options you care about.

make xconfig, make gconfig

• If you're running Xwindows, these are prettier, clickable versions of menuconfig. They tends to work less often than menuconfig since they are more sensitive to buggy configuration input files than menuconfig (and because more people use menuconfig). xconfig uses the qt libraries, and gconfig uses the gtk libraries.
Each of the configuration programs produces these end products:

• A file named .config in the top-level directory containing all your configuration choices.
• A file named autoconf.h in the include/linux/ directory defining (or not defining) the CONFIG_* symbols so that the C preprocessor knows whether or not they are turned on.
If you have a working .config file for your machine, just copy it into your kernel source directory and run make oldconfig. You should double check the configuration with make menuconfig. If you don't already have a .config file, you can create one by visiting each submenu and turning on or off the options you need. menuconfig and xconfig have a "Help" option that shows you the Configure.help entry for that option, which may help you decide whether or not it should be turned on. RedHat and other distributions often include sample .config files with their distribution specific kernel source in a subdirectory named configs in the top-level source directory. If you are compiling for PowerPC, you have a variety of default configs to choose from inarch/ppc/configs. make defconfig will copy the default ppc config file to .config. If you know another source of default configuration files, let us know.
Tips for configuring a kernel:

• Always turn on "Prompt for development... drivers".
• From kernel 2.6.8, you can add your own string (such as your initials) at "Local version - append to kernel release" to personalize your kernel version string (for older kernel versions, you have to edit the EXTRAVERSION line in the Makefile).
• Always turn off module versioning, but always turn on kernel module loader, kernel modules and module unloading.
• In the kernel hacking section, turn on all options except for "Use 4Kb for kernel stacks instead of 8Kb". If you have a slow machine, don't turn on "Debug memory allocations" either.
• Turn off features you don't need.
• Only use modules if you have a good reason to use a module. For example, if you are working on a driver, you'll want to load a new version of it without rebooting.
• Be extra sure you selected the right processor type. Find out what processor you have now with cat /proc/cpuinfo.
• Find out what PCI devices you have installed with lspci -v.
• Find out what your current kernel has compiled in with dmesg | less. Most device drivers print out messages when they detect a device.
If all else fails, copy a .config file from someone else. The grrls-only list is a good place to ask.

Kernel Hacking Lesson #3: Compile Your Kernel

At this point, you have your kernel source, and you've run one of the configuration programs and (very important) saved your new configuration file. Don't laugh - I've done this many times.
First, build the kernel:
On an x86:


   # make -j<number of jobs> bzImage

On a PPC:


   # make -j<number of jobs> zImage

Where <number of jobs> is two times the number of cpus in your system. So if you have a single cpu Pentium III, you'd do:


   # make -j2 bzImage

What the -j argument tells the make program is to run that many jobs (commands in the Makefile) at once. make knows which jobs can be run at the same time and which jobs need to wait for other jobs to finish before they can run. Kernel compilation jobs spend enough time waiting for I/O (for example, reading the source file from disk) that running two of the jobs per processor results in the shortest compilation time. NOTE: If you get a compile error, run make with only one job so that it's easy to see where the error occurred. Otherwise, the error message will be hidden in the output from the other make jobs.
If you have enabled modules, you'll need to compile them with the command:


   # make modules

If you are planning on loading these modules on the same machine they are compiled on, then run the automatic module install command. But FIRST - save your old modules!


   # mv /lib/modules/`uname -r` /lib/modules/`uname r`.bak
   # make modules_install

This will put all the modules in subdirectories of the /lib/modules/<version> directory. You can find out what <version> is by looking in include/linux/version.h.

Recompiling the kernel

So, now you've compiled the kernel once. Now you want to change part of the kernel and recompile it. What do you need to do?
In most cases, simply running make -j2 bzImage (or whatever your kernel compile command is) again will do the trick. If you've altered a module's source file, then just do make modules and make modules_install (if appropriate).
Sometimes, you'll change things so much that make can't figure out how to recompile the files correctly. make clean will remove all the object and kernel object files (ending in .o and .ko) and a few other things. make mrproper will do everything make clean does, plus remove your config file, the dependency files, and everything else that make config creates. Be sure to save your config file in another file before running make mrproper. Afterwards, copy the config file back to .config and start over, beginning at make menuconfig. A make mrproper will often fix strange kernel crashes that make no sense and strange compilation errors that make no sense.
Here are some tips for recompilation when you are working on just one or two files. Say you are changing the filedrivers/char/foo.c and you are having trouble getting it to compile at all. You can just type (from the drivers/char/ directory)make -C path/to/kernel/src SUBDIRS=$PWD modules and make will immediately attempt to compile the modules in that directory, instead of descending through all the subdirectories in order and looking for files that need recompilation. If drivers/char/foo.c is a module, you can then insmod the drivers/char/foo.ko file when it actually does compile, without going through the full-blownmake modules command. If drivers/char/foo.c is not a module, you'll then have to run make -j2 bzImage to link it with all the other parts of the kernel.
The && construct in bash is very useful for kernel compilation. The bash command line:


   # thing1 && thing2

Says, "Do thing1, and if it doesn't return an error, do thing2." This is useful for doing something only if a compile command succeeds.
When I'm working on a module, I often use the following command:


   # rmmod foo.ko && make -C path/to/kernel/src SUBDIRS=$PWD modules \
     && insmod foo.ko

This says, "Remove the module foo.ko, and if that succeeds, compile the modules in drivers/char/, and if that succeeds, load the kernel module object file." That way, if I get a compile error, it doesn't load the old module file and it's obvious that I need to fix my code.
Don't let yourself get too frustrated! If you just can't figure out what's going on, try the make mrproper command. If you still can't figure it out, post your question on grrls-only. It's good to try to figure things out by yourself, but it's not good if it makes you give up trying at all.

Kernel Hacking Lesson #4: Boot Your New Kernel

Now that you've successfully compiled your kernel, you need to boot your new kernel. We're going to cover booting a new kernel on three different architectures: x86, PowerPC, and Alpha. Some similarities exist between all three, so we'll start with those.
Rule number one is never ever delete your current working kernel or bootloader configuration files. Never copy your new kernel over your original kernel. You don't need to keep every kernel you ever compile, but do pick one or more "safe" kernels and keep them and their configuration files intact.

Copy your new kernel to a safe location

First, copy the new kernel you just compiled to a safe location. I suggest using /boot/mynewkernel as the destination. If you're on x86, you want to copy the bzImage:


   # cp arch/i386/boot/bzImage /boot/mynewkernel

If you're on a PowerPC or Alpha, copy the vmlinux:


   # cp vmlinux /boot/mynewkernel

Booting your new kernel with LILO on an x86

LILO is the most common boot loader for x86 machines. You are running LILO if you see "LILO:" when you boot. We'll just go over the bare minimum of information you need to boot a new kernel in LILO. For more information on LILO, type man lilo andman lilo.conf. You will need to be root to edit /etc/lilo.conf and run LILO.
The LILO configuration file is in /etc/lilo.conf. The basic idea here is to copy the chunk of your configuration file for your current kernel and then change the name of the kernel file to boot. Remember, you are copying the information for your current kernel and changing the copy, NOT editing the original. Say you have something like this in your /etc/lilo.conf:


   image=/boot/vmlinuz-2.2.14-5.0
           label=linux
           initrd=/boot/initrd-2.2.14-5.0.img
           read-only
           root=/dev/sda1

The image field tells LILO where to find the file containing your new kernel. The label field is what you type at the LILO prompt to boot that kernel. It's a good idea to make the label something short and easy to type. The initrd field is optional and specifies an initial ramdisk to load before mounting the root filesystem. The read-only field says to initially mount the root filesystem read-only, rather than read-write. The root field tells LILO which device contains the root filesystem.
You'll copy this information and change the following fields:


   image=<filename of your new kernel>
            label=<choose whatever name you'd like here>

Remove the initrd field if it exists:


           initrd=/boot/initrd-2.2.14-5.0.img

The reason you should remove the initrd field is because the initrd file does not contain anything that would be useful to your new kernel. In a normal Linux distribution, the initrd contains a lot of kernel modules which can only be loaded by the distribution kernel. We'll talk more about initrd's when we cover ramdisks later.
The new lines you just added to your configuration file should look something like this when you're done:


   image=/boot/mynewkernel
           label=new
           read-only
           root=/dev/sda1

LILO does not know that you changed anything in /etc/lilo.conf until you run the lilo command. You will probably do this more than once: compile a new kernel, copy it to the new place, forget to run LILO, and reboot only to find that LILO doesn't know about the new kernel. So, after you change the /etc/lilo.conf file, always run LILO:


   # lilo -v

The -v says to be verbose about what it's doing. LILO will complain if it can't find the files you told it to use.
Now, reboot, and you'll see the LILO prompt:


   LILO:

Type the label you chose (what you entered in the label field) for your new kernel. LILO will then try to boot that new kernel. To avoid typing a command line into LILO, run this just before you reboot:


   # lilo -v -R "<LILO command line>"

This tells LILO to use that command line the next time you reboot. It then erases that command line and goes back to normal behavior the next time you reboot. This is really useful for testing new kernels, since you can boot your new kernel, see it crash, reset the machine, and automatically boot your working kernel without ever typing anything.
If you are having trouble with LILO, try the LILO mini-HOWTO.

Booting your new kernel with GRUB on an x86

GRUB is becoming a common boot loader for x86 machines. You are running GRUB if you see a pretty graphical menu which lets you select which image to boot with the arrow keys when you boot. We'll just go over the bare minimum of information you need to boot a new kernel in GRUB. For more information on GRUB, type info grub. You will need to be root to edit /etc/grub.conf.
The GRUB configuration file is in /etc/grub.conf. The basic idea here is to copy the chunk of your configuration file for your current kernel and then change the name of the kernel file to boot. Remember, you are copying the information for your current kernel and changing the copy, NOT editing the original. Say you have something like this in your /etc/grub.conf:


title Linux (2.4.9-31)
        root (hd0,0)
        kernel /vmlinuz-2.4.9-31 ro root=/dev/hda3
        initrd /initrd-2.4.9-31.img

The title field is the title in the GRUB menu that corresponds to your new kernel. The root field tells GRUB where your root filesystem is (your root filesystem is what is mounted at /). The kernel field tells GRUB where to find the file containing your kernel and, after the kernel filename, it gives some options to pass the kernel when it boots. NOTE: The filename of the kernel is given relative to the boot partition, which usually means that you have to remove the /boot part from the filename. The initrd field is optional and specifies an initial ramdisk to load before mounting the root filesystem.
You'll copy this information and change the following fields:


title <choose whatever name you'd like here>
        kernel <filename of your new kernel> ro root=/dev/hda3

Remember, you should almost certainly remove the /boot part from the filename of the kernel, unless you know what you're doing.
Remove the initrd field if it exists:


        initrd /initrd-2.4.9-31.img

The reason you should remove the initrd field is because the initrd file does not contain anything that would be useful to your new kernel. In a normal Linux distribution, the initrd contains a lot of kernel modules which can only be loaded by the distribution kernel. We'll talk more about initrd's when we cover ramdisks later.
The new lines you just added to your configuration file should look something like this when you're done:


title new
        root (hd0,0)
        kernel /mynewkernel ro root=/dev/hda3

Note again, that in this case my new kernel is in /boot/mynewkernel, but since I have a separate partition mounted on /boot, and since GRUB will only look inside that partition when I boot, I need to take off the mountpoint part of the name, which is /boot. If I had everything, including /boot, on one big partition, then I would tell GRUB that my kernel was named /boot/mynewkernel.
Now, reboot, and you'll see the GRUB menu. Select the title you chose earlier with the arrow keys and hit enter. GRUB will now try to boot your new kernel.
For more information on GRUB, try the GRUB manual:
http://www.gnu.org/manual/grub/index.html

Booting your new kernel with yaboot and ybin on a PowerPC

Once again, we'll only cover the bare minimum to boot a new kernel under yaboot. For more information on yaboot and ybin, tryman yaboot, man yaboot.conf, and man ybin. You will need to be root to edit /etc/yaboot.conf and run yaboot and ybin.
The yaboot configuration file is in /etc/yaboot.conf and is very similar to the LILO configuration file. The basic idea here is to copy the chunk of your configuration file for your current kernel and then change the name of the kernel file to boot. Remember, you are copying the information for your current kernel and changing the copy, NOT editing the original. Say you have something like this in your /etc/yaboot.conf:


   image=/boot/vmlinux-2.2.15-2.7.0
           label=linux
           root=/dev/hda10
           novideo

The image field tells yaboot where to find the file containing your new kernel. The label field is what you type at the yaboot prompt to boot that kernel. It's a good idea to make the label something short and easy to type. The root field tells LILO which device contains the root filesystem. The novideo field works around a bug on some PowerMacs, and may or may not be necessary on your system. Just copy a yaboot entry that works for your particular machine.
After you copy this information, change the following fields:


   image=<filename of your new kernel>
           label=<choose whatever name you'd like here>

The new lines you just added to your configuration file should look something like this when you're done:


   image=/boot/mynewkernel
           label=new
           root=/dev/hda10
           novideo

Next, run ybin if you have it. Ybin is a helper program that takes care of copying all the necessary files to the bootable partition, running yaboot, and all the little details that Apple's schizophrenic Open Firmware requires for booting a kernel. If you don't have ybin, consult the yaboot man page, man yaboot, for information about what you need to do.

Booting your new kernel with SRM on an Alpha

Note: These instruction were written without the benefit of an Alpha to test them on.
SRM is the firmware installed on most Alpha's. SRM can boot a kernel from an ext2 filesystem as long as the disk has a BSD disklabel. All you really have to do is put your new kernel in the same directory as your current kernel (for simplicity), reboot, and boot substituting your new kernel name for the old kernel name. For more information on SRM, see:


   http://en.tldp.org/HOWTO/SRM-HOWTO/

You should already know how to boot your old kernel from the SRM prompt. All you need to do is change the name of the kernel file to the name of your new kernel file. The easiest way to do this is:


   $ show boot_file
   $ set boot_file <your new kernel filename>

If your old kernel is in /boot/vmlinux, and your new kernel is in /boot/mynewkernel, and show boot_file says this:


   2/boot/vmlinux

Then you would type:


   set boot_file 2/boot/vmlinux

(The "2" at the beginning of the filename tells SRM to look in the second partition of the boot device.)

Troubleshooting booting

At this point, if your kernel doesn't boot, reboot into your old kernel (this is why you never overwrite your old kernel or configuration) and try to figure out what's gone wrong.
A good way to troubleshoot is to start out by copying the lines for your current kernel from your bootloader's configuration file and only changing the label or title of the entry. See if you can boot that kernel (and initrd, if it has one). Once you're sure that's working, then try substituting the name of your new kernel (and removing the initrd entry, if there is one). At this point, you can be fairly sure that it's your kernel that doesn't boot, rather than a problem in your bootloader configuration.

Kernel Hacking Lesson #5: Your First printk

Here's where you prove that kernel hacking is NOT magic. Remember, at this point you have compiled and booted your new Linux kernel. Now we're going to "hack the kernel." We're going to add a printk, a function that prints out a message, during bootup.
First, bring up the file init/main.c in your favorite editor. If you are using source control, be sure to check out the file first. In BitKeeper, use the command bk edit init/main.c. In CVS, use cvs co init/main.c. Find the calibrate_delay(void) function in the file.
The first lines are declarations of a few variables, and will look something like this:


   unsigned long ticks, loopbit;
   int lps_precision = LPS_PREC;

Now, type in the following line just below them:


   printk("*** I am a kernel hacker! ***\n");

And now, recompile using make bzImage or make zImage, run LILO or ybin or whatever you need, and boot your new kernel. You don't need to run make clean. Watch the screen when you boot up - you should see your new message. If it scrolls off the screen too quickly for you to read it, login and type:


   $ dmesg | less

You should see your message near the beginning of the output.
Congratulations! You have just hacked the kernel. printk's are one of the main tools of the kernel hacker. You will need to use them constantly.

More printk Tricks

printk is in many ways the kernel equivalent of the C standard function printf. The formats are mostly the same. You will use certain formats far more often in kernel code than you do in user code. The "%x" or "%p" formats are especially useful. "%x" says to print out the value in hexadecimal, which is base 16, and is usually far more useful information than the value in decimal, which is base 10. This is because the logical unit of information in a computer is a byte (8 bits), and a byte fits into two digits in a hexadecimal representation. So the hexadecimal value:


   0x12345678

Represents these 4 bytes (ignoring endian-ness):


   0x12
   0x34
   0x56
   0x78

As you can see, it's easy to separate out a hexadecimal value into individual byte values - just read it two digits at a time. The "0x" part just says that the following numbers are in base 16, not base 10. (Note: only obsolete and irrelevant machines have bytes which are not 8 bits. We don't care about bytes which are not 8 bits long.)
The "%p" format says to print out the value as a pointer, or an address in memory. This format will depend on the machine, but is always in hexadecimal. It prints leading 0's, too.
Your next assignment is to print out the address of a variable. Below your first printk, add this printk:


   printk("The address of loops_per_jiffy is %p\n", &loops_per_jiffy);

Recompile and boot. Now you know what virtual address in memory the variable loops_per_jiffy is stored at. You can also find this out from the file System.map in your top-level Linux source directory:


   $ grep loops_per_jiffy System.map

Next, we're going to learn about loglevels. You can specify each printk to have a certain level of importance. Depending on the current loglevel, some messages will be printed to the console (your video monitor, usually) and some won't. Add this below your other printk's:


   printk(KERN_DEBUG "*** This is a debug message only. ***\n");

If your loglevel is configured normally, you won't see this message printed out during bootup. Once you've finished booting, login and look at the kernel messages again:


   $ dmesg | less

Below your other messages, you should see:


   *** This is a debug message only. ***

This is convenient since you only have to look at the output when you want to, instead of having it printed on the screen (possibly while you're typing). The definition of all the different KERN_* loglevels in include/linux/kernel.h. Usually, you will only needKERN_INFO and KERN_DEBUG.
Whenever you want to find out what's going on inside the kernel, a good way to start is by putting printk's in strategic places. Be careful that you don't print out too much information - it can slow down your kernel to the point of unusability. In certain places, adding a printk can cause a crash, or cause a bug, or fix a bug. To find out exactly what printk does, start reading in the filekernel/printk.c.
Have fun with your printk's!

Kernel Hacking Lesson #6: Overview of the Kernel Source

In this lesson, you'll get a general idea of where the various parts of the kernel are located in the source tree, what order they execute in, and how to go looking for a particular piece of code.

Where is all the code?

Let's start with the top-level directory of the Linux source tree, which is usually but not always in /usr/src/linux-<version>. We won't get too detailed, because the Linux source changes constantly, but we'll try to give you enough information to figure out where a certain driver or function is.

Makefile

This file is the top-level Makefile for the whole source tree. It defines a lot of useful variables and rules, such as the default gcc compilation flags.

Documentation/

This directory contains a lot of useful (but often out of date) information about configuring the kernel, running with a ramdisk, and similar things. The help entries corresponding to different configuration options are not found here, though - they're found inKconfig files in each source directory.

arch/

All the architecture specific code is in this directory and in the include/asm-<arch> directories. Each architecture has its own directory underneath this directory. For example, the code for a PowerPC based computer would be found under arch/ppc. You will find low-level memory management, interrupt handling, early initialization, assembly routines, and much more in these directories.

crypto/

This is a cryptographic API for use by the kernel itself.

drivers/

As a general rule, code to run peripheral devices is found in subdirectories of this directory. This includes video drivers, network card drivers, low-level SCSI drivers, and other similar things. For example, most network card drivers are found in drivers/net. Some higher level code to glue all the drivers of one type together may or may not be included in the same directory as the low-level drivers themselves.

fs/

Both the generic filesystem code (known as the VFS, or Virtual File System) and the code for each different filesystem are found in this directory. Your root filesystem is probably an ext2 filesystem; the code to read the ext2 format is found in fs/ext2. Not all of the filesystems compile or run, and the more obscure filesystems are always a good candidate for someone looking for a kernel project.

include/

Most of the header files included at the beginning of a .c file are found in this directory. Architecture specific include files are inasm-<arch>. Part of the kernel build process creates the symbolic link from asm to asm-<arch>, so that #include <asm/file.h> will get the proper file for that architecture without having to hard code it into the .c file. The other directories contain non-architecture specific header files. If a structure, constant, or variable is used in more than one .c file, it should be probably be in one of these header files.

init/

This directory contains the files main.c, version.c, and code for creating "early userspace". version.c defines the Linux version string. main.c can be thought of as the kernel "glue". We'll talk more about main.c in the next section. Early userspace provides functionality that needs to be available while a Linux kernel is coming up, but that doesn't need to be run inside the kernel itself.

ipc/

"IPC" stands for "Inter-Process Communication". It contains the code for shared memory, semaphores, and other forms of IPC.

kernel/

Generic kernel level code that doesn't fit anywhere else goes in here. The upper level system call code is here, along with theprintk() code, the scheduler, signal handling code, and much more. The files have informative names, so you can type ls kernel/and guess fairly accurately at what each file does.

lib/

Routines of generic usefulness to all kernel code are put in here. Common string operations, debugging routines, and command line parsing code are all in here.

mm/

High level memory management code is in this directory. Virtual memory (VM) is implemented through these routines, in conjunction with the low-level architecture specific routines usually found in arch/<arch>/mm/. Early boot memory management (needed before the memory subsystem is fully set up) is done here, as well as memory mapping of files, management of page caches, memory allocation, and swap out of pages in RAM (along with many other things).

net/

The high-level networking code is here. The low-level network drivers pass received packets up to and get packets to send from this level, which may pass the data to a user-level application, discard the data, or use it in-kernel, depending on the packet. Thenet/core directory contains code useful to most of the different network protocols, as do some of the files in the net/ directory itself. Specific network protocols are implemented in subdirectories of net/. For example, IP (version 4) code is found in the directory net/ipv4.

scripts/

This directory contains scripts that are useful in building the kernel, but does not include any code that is incorporated into the kernel itself. The various configuration tools keep their files in here, for example.

security/

Code for different Linux security models can be found here, such as NSA Security-Enhanced Linux and socket and network security hooks.

sound/

Drivers for sound cards and other sound related code is placed here.

usr/

This directory contains code that builds a cpio-format archive containing a root filesystem image, which will be used for early userspace.

Where does it all come together?

The central connecting point of the whole Linux kernel is the file init/main.c. Each architecture executes some low-level setup functions and then executes the function called start_kernel, which is found in init/main.c.
The order of execution of code looks something like this:


   Architecture-specific setup code (in arch/<arch>/*)
    |
    v
   The function start_kernel() (in init/main.c)
    |
    v
   The function init() (in init/main.c)
    |
    v
   The user level "init" program

In more detail, this is what happens:

• Architecture-specific set up code that does:
  • Unzip and move the kernel code itself, if necessary
  • Initialize the hardware
    • This may include setting up low-level memory management
  • Transfer control to the function start_kernel()
• start_kernel() does, among other things:
  • Print out the kernel version and command line
  • Start output to the console
  • Enable interrupts
  • Calibrate the delay loop
  • Calls rest_init(), which does:
    • Start a kernel thread to run the init() function
    • Enter the idle loop
• init() does:
  • Start the other processors (on SMP machines)
  • Start the device subsystems
  • Mount the root filesystem
  • Free up unused kernel memory
  • Run /sbin/init (or /etc/init, or...)
At this point, the userlevel init program is running, which will do things like start networking services and run getty (the login program) on your console(s).
You can figure out when a subsystem is initialized from start_kernel() or init() by putting in your own printk's and seeing when the printk's from that subsystem appear with regard to your own printk's. For example, if you wanted to find out when the ALSA sound system was initialized, put printk's at the beginning of start_kernel() and init() and look for where "Advanced Linux Sound Architecture [...]" is printed out relative to your printk's. (See Kernel Hacking Lesson #5 for help with using the printk()function.)

Finding things in the kernel source tree

So, you want to start working on, say, the USB driver. Where do you start looking for the USB code?
First, you can try a find command from the top-level kernel directory:


   $ find . -name \*usb\*

This command will print out every filename that has the string "usb" in the middle of it.
Another thing you might try is looking for a unique string. This unique string can be the output of a printk(), the name of a file in/proc, or any other unique string that might be found in the source code for that driver. For example, USB prints out the message:


   usb-ohci.c: USB OHCI at membase 0xcd030000, IRQ 27

So you might try using a recursive grep to find the part of that printk that is not a conversion character like %d:


   $ grep -r "USB OHCI at" .

Another way you might try to find the USB source code is by looking in /proc. If you type find /proc -name usb, you might find that there is a directory named /proc/bus/usb. You might be able to find a unique string to grep for by reading the entries in that directory.
If all else fails, try descending into individual directories and listing the files, or looking at the output of ls -lR. You may see a filename that looks related. But this should really be a last resort, and something to be tried only after you have run many different find and grep commands.
Once you've found the source code you are interested in, you can start reading it. Reading and understanding Linux kernel code is another lesson in itself. Just remember that the more you read kernel code, the easier it gets. Have fun exploring the kernel!

Kernel Hacking Lesson #7: Understanding System Calls

By now, you're probably looking around at device driver code and wondering, "How does the function foo_read() get called?" Or perhaps you're wondering, "When I type cat /proc/cpuinfo, how does the cpuinfo() function get called?"
Once the kernel has finished booting, the control flow changes from a comparatively straightforward "Which function is called next?" to being dependent on system calls, exceptions, and interrupts. Today, we'll talk about system calls.

What is a system call?

In the most literal sense, a system call (also called a "syscall") is an instruction, similar to the "add" instruction or the "jump" instruction. At a higher level, a system call is the way a user level program asks the operating system to do something for it. If you're writing a program, and you need to read from a file, you use a system call to ask the operating system to read the file for you.

System calls in detail

Here's how a system call works. First, the user program sets up the arguments for the system call. One of the arguments is the system call number (more on that later). Note that all this is done automatically by library functions unless you are writing in assembly. After the arguments are all set up, the program executes the "system call" instruction. This instruction causes an exception: an event that causes the processor to jump to a new address and start executing the code there.
The instructions at the new address save your user program's state, figure out what system call you want, call the function in the kernel that implements that system call, restores your user program state, and returns control back to the user program. A system call is one way that the functions defined in a device driver end up being called.
That was the whirlwind tour of how a system call works. Next, we'll go into minute detail for those who are curious about exactly how the kernel does all this. Don't worry if you don't quite understand all of the details - just remember that this is one way that a function in the kernel can end up being called, and that no magic is involved. You can trace the control flow all the way through the kernel - with difficulty sometimes, but you can do it.

A system call example

This is a good place to start showing code to go along with the theory. We'll follow the progress of a read() system call, starting from the moment the system call instruction is executed. The PowerPC architecture will be used as an example for the architecture specific part of the code. On the PowerPC, when you execute a system call, the processor jumps to the address 0xc00. The code at that location is defined in the file:
arch/ppc/kernel/head.S
It looks something like this:


/* System call */
        . = 0xc00
SystemCall:
        EXCEPTION_PROLOG
        EXC_XFER_EE_LITE(0xc00, DoSyscall)

/* Single step - not used on 601 */
        EXCEPTION(0xd00, SingleStep, SingleStepException, EXC_XFER_STD)
        EXCEPTION(0xe00, Trap_0e, UnknownException, EXC_XFER_EE)

What this code does is save some state and call another function called DoSyscall. Here's a more detailed explanation (feel free to skip this part):
EXCEPTION_PROLOG is a macro that handles the switch from user to kernel space, which requires things like saving the register state of the user process. EXC_XFER_EE_LITE is called with the address of this routine, and the address of the function DoSyscall. Eventually, some state will be saved and DoSyscall will be called. The next two lines save two exception vectors on the addresses 0xd00 and0xe00.
EXC_XFER_EE_LITE looks like this:


#define EXC_XFER_EE_LITE(n, hdlr)       \
        EXC_XFER_TEMPLATE(n, hdlr, n+1, COPY_EE, transfer_to_handler, \
                          ret_from_except)

EXC_XFER_TEMPLATE is another macro, and the code looks like this:


#define EXC_XFER_TEMPLATE(n, hdlr, trap, copyee, tfer, ret)     \
        li      r10,trap;                                       \
        stw     r10,TRAP(r11);                                  \
        li      r10,MSR_KERNEL;                                 \
        copyee(r10, r9);                                        \
        bl      tfer;                                           \
i##n:                                                           \
        .long   hdlr;                                           \
        .long   ret

li stands for "load immediate", which means that a constant value known at compile time is stored in a register. First, trap is loaded into the register r10. On the next line, that value is stored on the address given by TRAP(r11). TRAP(r11) and the next two lines do some hardware specific bit manipulation. After that we call the tfer function (i.e. the transfer_to_handler function), which does yet more housekeeping, and then transfers control to hdlr (i.e. DoSyscall). Note that transfer_to_handler loads the address of the handler from the link register, which is why you see .long DoSyscall instead of bl DoSyscall.
Now, let's look at DoSyscall. It's in the file:
arch/ppc/kernel/entry.S
Eventually, this function loads up the address of the syscall table and indexes into it using the system call number. The syscall table is what the OS uses to translate from a system call number to a particular system call. The system call table is namedsys_call_table and defined in:
arch/ppc/kernel/misc.S
The syscall table contains the addresses of the functions that implement each system call. For example, the read() system call function is named sys_read. The read() system call number is 3, so the address of sys_read() is in the 4th entry of the system call table (since we start numbering the system calls with 0). We read the data from the address sys_call_table + (3 * word_size) and we get the address of sys_read().
After DoSyscall has looked up the correct system call address, it transfers control to that system call. Let's look at where sys_read()is defined, in the file:
fs/read_write.c
This function finds the file struct associated with the fd number you passed to the read() function. That structure contains a pointer to the function that should be used to read data from that particular kind of file. After doing some checks, it calls that file-specific read function in order to actually read the data from the file, and then returns. This file-specific function is defined somewhere else - the socket code, filesystem code, or device driver code, for example. This is one of the points at which a specific kernel subsystem finally interfaces with the rest of the kernel. After our read function finishes, we return from the sys_read(), back to DoSyscall(), which switches control to ret_from_except, which is in defined in:
arch/ppc/kernel/entry.S
This checks for tasks that might need to be done before switching back to user mode. If nothing else needs to be done, we fall through to the restore function, which restores the user process's state and returns control back to the user program. There! Yourread() call is done! If you're lucky, you even got your data back.
You can explore syscalls further by putting printks at strategic places. Be sure to limit the amount of output from these printks. For example, if you add a printk to sys_read() syscall, you should do something like this:


  static int mycount = 0;

  if (mycount < 10) {
           printk ("sys_read called\n");
    mycount++;
  }

Have fun!

Lesson 8: Your First Kernel Module

In this lesson, we'll write and load a simple kernel module. Writing your own module lets you write some standalone kernel code, learn how to use modules, and discover a few rules about how the kernel links together. Note: These instructions were written for the 2.6.x kernels and may not work with different kernel versions.

Does your kernel support modules?

For this lesson, your kernel must have been compiled with these options:


Loadable module support  --->
  [*] Enable loadable module support
  [*]   Module unloading
  [ ]   Module versioning support (EXPERIMENTAL)
  [*]   Automatic kernel module loading    

If you compiled your kernel according to the instructions in the first few kernel lessons, you should already have these options properly set. Otherwise, change these options, recompile the kernel, and boot into your new kernel.

A simple module skeleton

First, find the source that your current Linux kernel was compiled from. Change directory to drivers/misc/ in your Linux source code directory. Now, copy and paste the following code into a file named mymodule.c:


#include <linux/module.h>
#include <linux/config.h>
#include <linux/init.h>

static int __init mymodule_init(void)
{
 printk ("My module worked!\n");
        return 0;
}

static void __exit mymodule_exit(void)
{
 printk ("Unloading my module.\n");
        return;
}

module_init(mymodule_init);
module_exit(mymodule_exit);

MODULE_LICENSE("GPL");

Save the file and edit the Makefile in the same directory. Add this line:


obj-m += mymodule.o

Compile your module:


  # make -C [top directory of your kernel source] SUBDIRS=$PWD modules

Load the module. Depending on your kernel version, do that with either:


  # insmod ./mymodule.o

Or:


  # insmod ./mymodule.ko

And check to see if your message printed out:


  # dmesg | tail

You should see this at the end of the output:


  My module worked!

Now remove the kernel module:


  # rmmod mymodule

Check the output of dmesg again, you should see:


  Unloading my module.

You just wrote and ran a new kernel module! Congratulations!

The module/kernel interface

Now, let's do some more interesting things with your module. One of the key things to realize is that modules can only "see" functions and variables that the kernel deliberately makes visible to the modules. First, let's try to do things the wrong way.
Edit the file kernel/printk.c and add this line after all the included files and near the other global variable declarations (but outside all functions):


  int my_variable = 0;

Now recompile your kernel and reboot into your new kernel. Next, add this to the beginning of your module's mymodule_initfunction, before the other code:


  extern int my_variable;
  printk ("my_variable is %d\n", my_variable);
  my_variable++;

Save your changes and recompile your module:


  # make -C path/to/kernel/src SUBDIRS=$PWD modules

And load the module (this will fail):


  # insmod ./mymodule.ko

Loading your module should fail with the message:


  insmod: error inserting './mymodule.ko': -1 Unknown symbol in module

What this is saying is that the kernel is not allowing modules to see that variable. When the module loads, it has to resolve all it's external references, like function names or variable names. If it can't find all of it's unresolved names in the list of symbols that the kernel exports, then the module can't write to that variable or call that function. The variable my_variable has space allocated for it somewhere in the kernel, but the module can't figure out where.
To fix this, we're going to add my_variable to the list of symbols that the kernel exports. Many kernel directories have a file specifically for exporting symbols defined in that directory. Bring up the file kernel/printk.c again and add this line after the declaration of your variable:


  EXPORT_SYMBOL_NOVERS(my_variable);

Recompile and reboot into your new kernel. Now try to load your module again:


  # insmod ./mymodule.ko

This time, when you check dmesg, you should see:


  my_variable is 0
  My module worked!

Reload your module:


  # rmmod mymodule && insmod ./mymodule.ko

Now you should see:


  Unloading my module.
  my_variable is 1    
  My module worked!

Each time you reload the module, my_variable should increase by one. You are reading and writing to a variable which is defined in the main kernel. Your module can access any variable or function in the main kernel, as long as it is explicitly exported via theEXPORT_SYMBOL() declaration. For example, the function printk() is defined in the kernel and exported in the file kernel/printk.c.
A simple loadable kernel module is a fun way to explore the kernel. For example, you can use a module to turn a printk on or off, by defining a variable do_print in the kernel which is initially set to 0. Then make all your printk's dependent on "do_print":


  if (do_print)
    printk ("Big long obnoxious message\n");

And turn on do_print only when your module is loaded. You can add a function defined in your module to the list of functions that are called when the kernel receives a certain interrupt (use cat /proc/interrupts to find out what interrupts are in use). The function request_irq() adds your function to the list of handlers for a selected irq line, which you can use to print out a message each time you receive an interrupt on that line. You can investigate the current value of any exported variable by loading a module that reads that value and immediately exits (returns a non-zero value from the module_init() function). The variable jiffies, which increments every 1/100th of a second (on most platforms), is a good candidate for this kind of module.
Play with your new kernel module - modules are fun!

Kernel Hacking Lesson #9: Creating, Applying, and Submitting Patches

As a kernel developer, you'll spend a lot of time creating, applying, and submitting patches. Creating and applying patches can be tricky - a lot of conventions must be learned and a lot of common mistakes avoided. Submitting a patch also takes some work.
At first, submitting patches might seem like the easiest part of kernel development. After all, it can't be as hard as fixing an ethernet driver bug, right? Well, often it's easier to fix a kernel bug than to get a kernel patch accepted into the mainline kernel. Part of the reason is the sheer limitations of one person - the kernel maintainer can only read and accept so many patches per release. But other reasons why patches are hard to get accepted are controversial changes, territoriality, personality conflicts, and apathy. And finally, whenever you submit a patch, you are putting your reputation and ego on the line and that's more than a little scary.
That being said, submitting a patch can be a lot of fun and very encouraging. Some kernel developers had parties to celebrate the first patch they wrote that was accepted into the mainline kernel. Knowing that you wrote some code that other people thought was good enough to include in the Linux kernel is a great feeling! So, let's learn how to apply, create, and submit patches.

How patches work

A "patch" is a file that describes the differences between two versions of a file. The program diff compares the original file and the new file line-by-line and prints the differences to standard out in a specific format. The program patch can read the output of diffand apply those changes to another copy of the original file. (Note that the word "patch" refers both to the output of the diffcommand and to the command that applies the patch.) For example:


  val@evilcat <~>$ cat old/file.txt
  This
  is
  a
  simple
  file.
  val@evilcat <~>$ cat new/file.txt
  This
  is
  a
  slightly more complex
  file.
  val@evilcat <~>$ diff -uNr old new
  diff -uNr old/file.txt new/file.txt
  --- old/file.txt        Tue May 28 23:00:21 2002
  +++ new/file.txt        Tue May 28 23:01:01 2002
  @@ -1,5 +1,5 @@
   This
   is
   a
  -simple
  +slightly more complex
   file.

As you can see, the two files differ in only one line. The line from the first file listed on the command line is shown with a "-" in front of it, followed by the line from the second file on the command line is shown with a "+" in front of it. Intuitively, you are "subtracting" the line from the old file and "adding" the line from the new file. Remember, the old files always come first and the newer files come second.
Now, lets apply the patch we just created. A patch updates the older version of the file to the newer version of the file, so we want to apply the patch to the older version of the file.


  val@evilcat <~>$ diff -uNr old new > patchfile
  val@evilcat <~>$ cd old
  val@evilcat <~/old>$ patch -p1 < ../patchfile
  patching file file.txt
  val@evilcat <~/old>$ cat file.txt
  This
  is
  a
  slightly more complex
  file.

After applying the output of the diff command using patch, the "old" file is now the same as the "new" file.

Applying patches

Next, we'll learn how to apply patches. One of the common reasons you'll need to apply a patch is in order to get a particular kernel version which isn't available as one big tarball downloadable from ftp.kernel.org, or else to get an incremental patch so you don't have to download an entire new kernel when most of the kernel files are still the same.
The kernel patch naming and creation standards are not particularly simple. Say that you want to get the kernel 2.6.9-rc4 for some reason, and you currently have the full kernel source for version 2.6.7. You'll need to download the following patches to get from2.6.7 to 2.6.10-rc1:
2.6.7 to 2.6.8
2.6.8 to 2.6.9-rc4
Each prepatch (the patches that come between the major releases and are named patch-2.6.x-rcN, usually found in a directory on the ftp site called testing) is created by diffing against the previous major release. A common mistake is to download kernel version 2.6.9 and then attempt to apply the 2.6.9-rc4 prepatch. If you want kernel version 2.6.9-rc4, you should download kernel2.6.8 and then apply the 2.6.9-rc4 prepatch. This is because 2.6.9-rc4 is a predecessor of 2.6.9, not the other way around. NOTE: The naming convention and location of kernel prepatches tends to change frequently. You may have to read the linux-kernel mailing list to find out where the very latest patches are being kept and what they are being named.
The official kernel patches are all made so that you can simply do:


cd <your linux source tree>
patch -p1 < ../patchfile

What the -p1 option to the patch command says is "Strip the part of the pathname up through the first forward slash and then try to apply the patch to the file with the stripped down pathname."
If all this seems incredibly complex and annoying, you might want to try using BitKeeper. See the end of this lesson for a brief introduction to BitKeeper.

Creating a patch

The first thing to remember is to always keep an untouched, pristine version of the kernel source somewhere. Don't compile in it, don't edit any files in it, don't do anything to it - just copy it to make your working copy of the source tree. The original kernel source should be in a directory named linux.vanilla or linux.orig, and your working directory should be in the same directory as the original source. For example, if your pristine source is in /usr/src/linux.vanilla, your working source should be in /usr/src/also.
After you make your changes to your working copy, you'll create a patch using diff. Assuming that your working tree is namedlinux.new, you would run this command:


  val@evilcat <~>$ diff -uNr linux.vanilla linux.new > patchfile

All the differences between the original kernel source and your new kernel source are now in patchfile. NOTE: Do not ever create a patch with uneven directories, for example (DON'T do this):


  val@evilcat <~>$ diff -uNr linux.vanilla working/usb/thing1/linux > patchfile

This will not create a patch in the standard patch format and no one will bother trying out your patch since it's hard to apply.
Now that you've created a patch - read it! It's almost guaranteed that your patch includes files that you don't want as part of your patch, such as old editor backup files, object files, or random cruft you created during development. To get rid of these files, you can tell diff to ignore certain files, you can delete the files, or you can hand-edit the diff. Be sure you understand the patch format before you hand-edit a patch, or you can easily create a patch that won't apply. One useful command for getting rid of most of the extra files created during a kernel build is:


  make mrproper

But remember, this deletes your .config file and forces you to do a complete recompile of your kernel.
Also, make sure that your patch goes in the correct direction. Are your new lines the ones with "+"'s in front of them? And, make sure those are the changes you wanted to send. It's surprisingly easy to make a diff against the wrong source tree entirely.
After you think you've got a final version of the patch, apply it to a a copy of your pristine source tree (don't ruin your only copy of the pristine source tree). If it doesn't apply without any errors, redo the patch.
Once again, if this seems awfully complex, you may want to try BitKeeper, described at the end of this lesson.

Submitting a patch

After you've created a patch, you'll hopefully want to share it with other people. Ideally, you'll test the patch yourself, get other people to test it too, and have other people read the patch itself. In summary, you want your patch to be bug-free, well-written, and easy to apply.

Testing

Always compile and test your patches yourself. You'll see people posting "totally untested" patches to linux-kernel, but don't fall for it - a totally untested patch is likely to be a useless patch. Kernel maintainers have more than once released a kernel which doesn't compile at all. No one is perfect - always test your patches.

Coding style

Be sure that your code fits in with the code around it and follows the kernel coding style conventions. See the fileDocumentation/CodingStyle for specific directions, although looking at other source files is often the best way to figure out what the current conventions are.

Easy to apply

If it's difficult to apply your patch, it almost certainly won't be accepted. In addition to creating the patch with the proper level of directories, you need to create it against the kernel that is identical (or nearly so) to the kernel that other people will be applying your patch to. So, if you want person XYZ to apply your patch, find out what version of the kernel person XYZ is using and try to get something as close to that as possible. Usually this is the latest vanilla kernel released by the kernel maintainer. For example, if you have a patch against 2.6.9-rc2, and 2.6.9-rc4 is the latest version released, then you should recreate your patch against 2.6.9-rc4. The easiest way to do this is to apply your patch from 2.6.9-rc2 to 2.6.9-rc4 and fix up any changes that occurred between the two versions, then rediff against 2.6.9-rc4.

Who to submit your patch to

The answer to "Who should I submit this patch to?" is "It depends." Subscribe to linux-kernel and any list which is more specific to the area you are working on, and you will begin to get an idea of who the appropriate person is. Some general rules of thumb:
Try to find the person most specifically involved in maintaining the part of the kernel you are changing. If you make a change to the foo driver in the bar subsystem, and the foo driver has a maintainer, you should probably submit your patch to the foo maintainer, and only to the bar subsystem maintainer if the foo maintainer is ignoring you.
The file in the toplevel kernel source directory, MAINTAINERS, is frequently out of date, but often helpful anyway. No one will fault you if you send your patch to the person listed in the MAINTAINERS file and CC linux-kernel. When in doubt, this is always the safest route to take.
And finally, ask a friend! Send mail to the linuxchix lists asking for advice on who to submit a patch to. We can help.

Distributing your patch

Most patches are small enough to be included in an email. While some maintainers refuse to accept patches in attachments, and some refuse MIME encoded attachments, all maintainers will accept a patch that is included in the body of a text-only email. Make sure your mail client isn't mangling your patch - if you aren't sure, email your patch to yourself and apply it to make sure other people will be able to apply it to. Most Linux mailing lists like patches with a meaningful English-language subject, prefixed with the string [PATCH] so that it's easy to find and read patches.
If your patch is too big to send by email (around 20K or larger), put it on a web page or ftp site where other people can download it, and put the URL in your email.

Political considerations

If all that mattered is that your patch was well-formed, correct, and fixed a bug, submitting a patch would be a lot simpler. Instead, your patch needs to be tasteful, timely, interesting, and considerate of the maintainer's ego. Most of the time, a simple bugfix will be immediately accepted. Occasionally though, you'll run into bigger problems. The important thing to remember is that you can't work around the Linux maintainer system, you have to work through it. Read a few threads on linux-kernel in which people tried to wheedle their patch into the kernel (I've tried - and failed). If your patch isn't accepted, listen to what other people are saying about it and try to fix the problems with it. The most often rejected patch is the feature patch - adding a new feature that is considered tasteless by the other maintainers. Don't waste your time trying to get that patch accepted, just maintain it separately. If enough people find the patch useful, you'll gain a reputation as being a useful kernel hacker among all the people who download and use your patch.
Sometimes, a maintainer just can't accept a patch because of his or her ego. When this happens, the only option is to maintain a better version of the code independently of the main kernel. Often, externally maintained code that proves to be better will replace the in-kernel code after a while, which is one way to become a maintainer.

The alternative to diff and patch - BitKeeper

BitKeeper is currently being used by many kernel developers as a replacement for diff and patch. It simplifies a lot of kernel development tasks, such as updating to the latest version, creating patches, and applying patches. For more information, see the BitKeeper website:
http://www.bitkeeper.com
Or "BitKeeper for Kernel Developers"

http://www.nmt.edu/~val/ols/bk.ps