Documentation/backports/hacking =============================== .. note:: the backports project's process is being reworked and redefined. Some of the information in this page may be outdated and will be updated when the changes in the process are finalized. The project has a :doc:`To-Do List ` and a :doc:`License <../../license>`. Philosophy ---------- The project aims to optimize the backporting process by not only providing backported drivers but by also aiming to `automatically backport the Linux kernel `__ by `optimizing backporting collateral evolutions `__. Backports development flow -------------------------- #. The `backports.git `__ "master" development branch always tracks "linux-next", allowing it to track all the development trees. - This ensures that, at the close of each merge window, the state of the backports will be very close to the state of the first release candidate. #. At the close of each merge window, the Backports Project creates a new branch. - This new branch tracks the progress of the impending release throughout the release candidate evaluation period, to the major release, and on to its lifetime as a stable kernel. The backports project thus makes three kinds of backports releases possible: - those derived from linux-next - those derived from the most recent release candidate (if any) - those derived from recent stable kernels. Backports git tree tags and branches ------------------------------------ The project uses two types of git tags: - daily tags following linux-next - stable tags following either Linus' tree for rc releases or Greg's stable tree for stable extra version releases The project uses a master branch and a series of stable branches, a stable branch to track each stable release of the kernel. The master branch always tags linux-next, for example the backports tag backports-20141114 supports using linux-next tag next-20141114. Once Linus makes a release candidate of the kernel a respective *stable backports branch* is created based on the version of Linux Linus released, this backports branch will be based on a recent backports tag which was tracking linux-next. Future release candidates of the same version of Linux as well as stable release of Linux under the same version will be backported using the respective *backports stable branch* created. If any of this is fuzzy please refer to the `Increasing Automation in the Backporting of Linux Drivers Using Coccinelle `__ paper, in particular the sections: - 1A - A. The Linux kernel development model - 1B - A brief history of the Linux kernel backports project Git trees you will need ======================= The project backports a few subsystems down to older kernels. To be able to synchronize backporting the latest and greatest the `linux-next.git `__ tree is used as its main source for kernel updates. backports brings a general kernel compatibility part containing some backported c functions and header files in addition to patches for code in linux-next to make it work on older kernel versions and with the general kernel compatibility part. You will need first the `backports.git `__ tree. Once cloned you can run:: ./devel/backports-update-manager That will get you all other trees and sources required for development, which are: - linux-next - Stephen Rothwell's tree which tracks all development trees daily - linux - Linus' tree - linux-stable - Greg's stable tree for stable extra version releases This will use git clone --reference to minimize disk usage. You can generate backports based on different trees depending on your needs. We are supporting build based on `linux-next.git `__, `linux.git `__ and `linux-stable.git `__. You can also generate builds based on your own tree, but then you probably have to patch backports yourself to add support for that. Tool prerequisites ------------------ git, python patch and coccinelle are needed to generate a backports release. The tools will complain if you do not have any of the required dependencies. Generate new release -------------------- To generate a own backport release based on linux-next you need the following git trees:: git://git.kernel.org/pub/scm/linux/kernel/git/backports/backports.git git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git As linux-next changes often you should start with a working version by using a tagged version from linux-next.git and backports.git with the same date in it like backports-20130607 and next-20130607. This is a combination where someone already tested that the patches are still applying. After you have checked out some version of these git trees, you can start generating a tar by running ./gentree.py in the backports tree. This will take the path to the already checked out linux-next kernel as the second last argument and the path to a directory where the newly created release should be written to as the last parameter. The releases are not created in place any more, but in a new path specified in the last parameter:: ./gentree.py --clean --git-revision v3.x /path/to/your/linux-next /path/to/your/newly/created/release Adding new driver ================= This is a list of steps to be done when adding a new driver. Not all steps are needed for every new driver it depends on what type of driver is added. You can take the patch adding support for the ieee802154 subsystem as a reference: https://git.kernel.org/cgit/linux/kernel/git/backports/backports.git/commit/?id=a42aa4d474e5859164a00bf451970fb58b676fcf You should target the master branch of the backports repository. #. Add directories and files that should be copied from the mainline kernel to ``copy-list`` #. Add patches to ``patches/collateral-evolutions/`` if needed, it is better to add or modify a header file in ``backport/backport-include/`` than adding a patch. #. Add reference to the newly added driver directory in ``backport/Makefile.kernel`` if needed (just needed for new substems) #. Add reference to the newly added driver directory in ``backport/Kconfig.sources`` if needed (just needed for new subsystems) #. Add a defconfig file for this driver in ``backport/defconfigs/`` and modify other defconfig files if appropriate, e.g. if you add a wifi driver add it to ``backport/defconfigs/wifi`` #. If the driver does not build on all kernel versions specify a minimum kernel version in ``dependencies`` Backports currently supports the latest minor version of every mainline kernel starting from 2.6.24. You should check that backports still builds after making your changes. To do so you should install all supported kernel with the script in ``devel/get-compat-kernels`` and then run ``devel/ckmake`` to build backports against every kernel version. Backporting a kernel feature ---------------------------- Features being backported should either be backported into a respective backports-3.x.c file under backports/compat/ and/or their respective header file under backport/backport-include/. Backports Kconfig operation --------------------------- Go read the `backports kconfig operation `__ documentation. Backports Makefile operation ---------------------------- Go read the `backports Makefile operation `__ documentation. Sending patches --------------- backports contributions follow the contribution model implemented by the Linux kernel. Patches or pull requests for backports must have be signed-offed. If you don't sign off on them **they will not accepted**. This means adding a line that says "Signed-off-by: Name " at the end of each commit, indicating that you wrote the code and have the right to pass it on as an open source patch. For exact definition of what the Signed-off-by tag is you can read the definition of the "**Developer's Certificate of Origin 1.1**", which you can read here: http://gerrit.googlecode.com/svn/documentation/2.0/user-signedoffby.html Remember there are three trees. linux-next itself is a conglomeration of kernel git trees itself, so patches for linux-next.git should be sent to each respective subsystem for which the patches are targeted for. So for example for 802.11 you will want to send them to John Linville and cc linux-wireless, for further guidelines on this see the `Submitting Patches guidelines for 802.11 `__. As another example, for bluetooth you will want to send them to Gustavo Padovan and cc the linux-bluetooth mailing list. If your patch touches on others areas of the kernel refer to the MAINTAINERS file on the kernel. For backports.git please send patches against to:: To: hauke@hauke-m.de, mcgrof@kernel.org CC: backports@vger.kernel.org Subject: [PATCH] backports: fix foo Patches are preferred sent with a clear commit log entry, if unfamiliar with how to send patches please refer to `a git guide `__. Requirements for contributions ------------------------------ The code quality, and coding style is precisely the same as followed by the Linux kernel, for that refer to `the Linux kernel SubmittingPatches `__ guide. Other than this, since we a supporting backporting you are expected to refer to the respective upstream commit which you are providing a backport for. This should describe both the git commit sha1sum on Linux, and the commit description. The description should also include the Linux kernel version which merged the commit upstream first, this can be used to help understand what target kernels the backport work was for. An example of good commit log follows:: commit 54584122b3279700f5ef2ba4f260a84273215947 Author: Stefan Assmann Date: Fri May 16 13:21:19 2014 +0200 backports: handle new get_module_info and get_module_eeprom pointers in struct ethtool_ops In kernel 3.5 struct ethtool_ops received 2 new function pointers get_module_info and get_module_eeprom. Address this by putting ifdef around the code. commit 41c3cb6d20f0252308e9796fa4f3dacb4960de91 Author: Stuart Hodgson Date: Thu Apr 19 09:44:42 2012 +0100 ethtool: Extend the ethtool API to obtain plugin module eeprom data git describe --contains 41c3cb6d20f0252308e9796fa4f3dacb4960de91 v3.5-rc1~109^2~124^2~2 Signed-off-by: Stefan Assmann Signed-off-by: Hauke Mehrtens Backporting preference for EXPORT_SYMBOL_GPL() and proprietary drivers ---------------------------------------------------------------------- This framework was designed by intent since its inception **only** for *upstream* Linux kernel drivers to avoid excuses about the difficulty to support **upstream** Linux kernel development while also supporting **users on older kernels**. To ensure the intent is respected currently all symbols that we do work on to backport are exported via **EXPORT_SYMBOL_GPL()** as we do work to backport them for the supported kernels. By using EXPORT_SYMBOL_GPL() we make it **clear** through the only current technical means we have that if you use the backported symbols your software **is considered derivative works of the Linux kernel**. Backporting more subsystems --------------------------- Feel free to any backport subsystems. The requirement is you at least list its kernel dependency and also that you have tested compilation success with ckmake. If you are adding a new subsystem we are in hopes you will also help maintain it. Generating backports stats -------------------------- To help backports developers evaluate the gains of use of Coccinelle SmPL a tool has been written as part of the work done towards the `Increasing Automation in the Backporting of Linux Drivers Using Coccinelle paper `__, the tool has a temporary git tree on github, integration of the tool into the upstream backports tree is desirable but porting of the tool to Python must be done first. Below are instructions on how to use the tool to get stats on the latest tree. The statistics give a slew of metrics over the gains of using of Coccinelle towards automatically backporting the Linux kernel on each release. First git clone upstream backports git tree, and have a linux-next tree available (using Linus' tree is possible as well, you just need to use the respective stable backports branch). Then generate a backports release using --gitdebug so that each step of the backports process creates a commit. This lets the tool evaluate the impact of using Coccinelle at each step. Assuming you have backports cloned on ~/backports/ and the latest linux-next tree with the respective linux-next tag associated with the latest backports tree tag you would do:: cd linux-next git reset --hard next-20151218 cd ~/backports/ mkdir -p ~/build/ rm -rf ~/build/backports-20151218 ./gentree.py --gitdebug ~/linux-next ~/build/backports-20151218 Now go get the temporary backports stats tool and copy over the required binary and script:: cd ~/ git clone https://github.com/mcgrof/backports-cocci-stats.git cd backports-cocci-stats make cp gen-cocci-stats.sh ~/build/backports-20151218/ cp clean ~/build/backports-20151218/ Now generate the stats, you should see something like this:: mcgrof@ergon ~/build/backports-20151218 (git::master)$ ./gen-cocci-stats.sh Development and Maintenance efficiency metrics:    dev-efficiency    maint-efficiency   diff-wc      diffstat       clean       SmPL-Patch          0.555556             2.33333        21             5           9  0065-ndisc_send_na-argument.cocci              2.75                   8        64            22           8  0062-iff-no-queue.cocci          0.666667             1.88889        17             6           9  skb_no_xmit_more.cocci           1.16667             2.79167        67            28          24  ptp_getsettime64.cocci          0.142857                   1        14             2          14  features_check.cocci           1.18182             5.45455        60            13          11  0055-netdev-tstats.cocci           1.55906             3.55118       451           198         127  0054-struct-proto_ops-sig.cocci          0.666667             1.88889        17             6           9  no-pfmemalloc.cocci          0.634146              1.4878        61            26          41  set_vf_rate.cocci              3.75              10.625        85            30           8  igb_pci_error_handlers.cocci           1.07692             4.23077        55            14          13  ethtool_cmd_mdix.cocci          0.588235             1.23529        21            10          17      rxnfc.cocci          0.285714             1.53571        43             8          28  get_module.cocci          0.285714             1.53571        43             8          28  ethtool_eee.cocci          0.714286             2.28571        16             5           7  skb_no_fcs.cocci              0.25             1.59375        51             8          32  set_vf_spoofchk.cocci          0.428571             2.85714        40             6          14  sriov_configure.cocci           0.87037              2.7037       146            47          54  0031-sk_data_ready.cocci               4.4                17.9       179            44          10  genl-const.cocci           7.11111             49.6667       447            64           9  0019-usb_driver_lpm.cocci          0.571429             4.14286        58             8          14  get_ts_info.cocci           11.5333                45.4       681           173          15  0001-netlink-portid.cocci           2.96667               16.35       981           178          60  0002-no_dmabuf.cocci          0.444444             1.77778        32             8          18  0002-gpio-parent.cocci          0.512821             1.89744        74            20          39  0002-group_attr_bus.cocci          0.769231             2.79487       109            30          39  0001-group_attr_class.cocci           1.47184              5.6484      3711           967         657   all-SmPL.cocci Patch total diff wc -l: 3075 SmPL total diff wc -l: 3711 Total total diff wc -l: 6786 Patch diff % contribution: 45.3139 SmPL diff % contribution: 54.6861 Now say you wanted to generate stats for a newer release, say for stats based on linux-next tag next-20160122, you would do:: cd ~/linux-next/ git reset --hard next-20160122 cd ~/backports/ ./gentree.py --gitdebug ~/linux-next/ ~/build/backports-20160122 cd ~/build/backports-20160122 cp ~/backports-cocci-stats/gen-cocci-stats.sh . cp ~/backports-cocci-stats/clean . ./gen-cocci-stats.sh Development and Maintenance efficiency metrics:    dev-efficiency    maint-efficiency   diff-wc      diffstat       clean       SmPL-Patch               0.4                 2.6        13             2           5  0067-mdio-addr.cocci          0.555556             2.33333        21             5           9  0065-ndisc_send_na-argument.cocci              2.75                   8        64            22           8  0062-iff-no-queue.cocci          0.666667             1.88889        17             6           9  skb_no_xmit_more.cocci           1.16667             2.79167        67            28          24  ptp_getsettime64.cocci          0.142857                   1        14             2          14  features_check.cocci           1.18182             5.45455        60            13          11  0055-netdev-tstats.cocci           1.55906             3.55118       451           198         127  0054-struct-proto_ops-sig.cocci          0.666667             1.88889        17             6           9  no-pfmemalloc.cocci          0.634146              1.4878        61            26          41  set_vf_rate.cocci              3.75              10.625        85            30           8  igb_pci_error_handlers.cocci           1.07692             4.23077        55            14          13  ethtool_cmd_mdix.cocci          0.588235             1.23529        21            10          17      rxnfc.cocci          0.285714             1.53571        43             8          28  get_module.cocci          0.285714             1.53571        43             8          28  ethtool_eee.cocci          0.714286             2.28571        16             5           7  skb_no_fcs.cocci              0.25             1.59375        51             8          32  set_vf_spoofchk.cocci          0.428571             2.85714        40             6          14  sriov_configure.cocci           0.87037              2.7037       146            47          54  0031-sk_data_ready.cocci               4.4                17.9       179            44          10  genl-const.cocci           7.11111             49.6667       447            64           9  0019-usb_driver_lpm.cocci          0.571429             4.14286        58             8          14  get_ts_info.cocci           11.5333                45.4       681           173          15  0001-netlink-portid.cocci           2.96667               16.35       981           178          60  0002-no_dmabuf.cocci          0.444444             1.77778        32             8          18  0002-gpio-parent.cocci          0.512821             1.89744        74            20          39  0002-group_attr_bus.cocci          0.769231             2.79487       109            30          39  0001-group_attr_class.cocci           1.46375             5.62538      3724           969         662   all-SmPL.cocci Patch total diff wc -l: 3087 SmPL total diff wc -l: 3724 Total total diff wc -l: 6811 Patch diff % contribution: 45.3237 SmPL diff % contribution: 54.6763 See also -------- - :doc:`To-Do List ` - :doc:`License <../../license>`