XFS status update for 2010

XFS status update for December 2010

The release process of the Linux 2.6.37 kernel with it's large XFS updates was in it's final days in December, which explains why we only saw a single one-liner regression fix for XFS in Linus' tree. The XFS development tree finally saw some updates when the writeback updates and some small cleanups to the allocator and log recovery code were merged, but the large metadata scalability updates that have been posted to the list multiple times are still missing. In addition to this on-going work the list also saw patches that fix smaller issues, which are also still waiting to be merged.

On the userspace side xfsprogs and xfsdump development has been quit, with no commits to either repository in December, although a large series of updates to the metadump command has been reposted near the end of the month. The xfstests repository saw a new regression test for a btrfs problem, and various updates to existing tests.

XFS status update for November 2010

From looking at the kernel git commits November looked like a pretty slow month with just two hand full fixes going into the release candidates for Linux 2.6.37, and none at all going into the development tree. But in this case git statistics didn't tell the whole story - there was a lot of activity on patches for the next merge window on the list. The focus in November was still at metadata scalability, with various patchsets that improves parallel creates and unlinks again, and also improves 8-way dbench throughput by 30%. In addition to that there were patches to improve preallocation for NFS servers, to simplify the writeback code, and to remove the XFS-internal percpu counters for free space for the generic kernel percpu counters, which just needed a small improvement.

On the user space side we saw the release of xfsprogs 3.1.4, which contains various accumulated bug fixes and Debian packaging updates. The xfsdump tree saw a large update to speed up restore by using mmap for an internal database and remove the limitation of ~ 214 million directory entries per dump file. The xfstests test suite saw three new testcases and various fixes, including support for the hfsplus filesystem.

XFS status update for October 2010

Near the end of the month we finally saw the release of Linux 2.6.36. Just a single fix made it into mainline in this month, showing that the stabilization period before has worked very well.

Linux 2.6.36 has been another impressive release for XFS, seeing various performance improvements in the new delayed logging code, for direct I/O and the sync system call, a few bug fixes, and lots of cleanups, resulting in a net removal of over 2000 lines of code:

       89 files changed, 1998 insertions(+), 4279 deletions(-)

The merge window for Linux 2.6.37 opened just a few days after the release of Linux 2.6.36 and already contains another large XFS update at the end of October. Highlights of the XFS tree merged into 2.6.37-rc1 are another large set of metadata scalability patches, support for 32-bit wide project IDs, and support for the new XFS_IOC_ZERO_RANGE ioctl, which allows to punch a whole and convert it to an unwritten extent in a single atomic operation.

The metadata scalability changes improve 8-way fs_mark of 50 million files by over 15% and removal of those files by over 100%, with further improvements expected by the next round of XFS metadata scalability and VFS scalability improvements targeted at Linux 2.6.38.

On the user space side October was a rather quit month for xfsprogs, which only saw the addition of 32-bit project ID handling, and a fix for parsing the mount table in fsr when used together with disk encryption tools. A few patches for xfsdump were posted on the list, but none was applied, leaving the majority of the user space activity to xfstests, which saw very active development. Various patches went into xfstests to improve portability to filesystems with a limited feature set, and to move more filters to generic code. In addition various cleanups to test cases in test programs were applied.

XFS status update for September 2010

Mainline activity has been rather low in September while with only two more fixes going into the 2.6.36 release candidates after the large merge activity in August. Development for the next merge window has been more active. The largest item was the inclusion of the metadata scalability patch series, which provides very large speedups for parallel metadata operations. In addition a new ioctl to punch holes and convert the whole to an unwritten extent was added and a small number of cleanups also made it into the tree.

Patches to add support for 32bit wide project ID identifiers and using group and project quotas concurrently were posted to the list and discussed but not yet included.

Userspace development has been rather quiet again, with a single fix committed to xfsprogs and xfsdump each. The xfstests test suite grew a new test case and received a few additional fixes. Last but not least the XFS Users Guide was updated with various factual corrections and spelling fixes.

XFS status update for August 2010

At the first of August we finally saw the release of Linux 2.6.35, which includes a large XFS update. The most prominent feature in Linux 2.6.35 is the new delayed logging code which provides massive speedups for metadata-intensive workloads, but there has been a large amount of other fixes and cleanups, leading to the following diffstat:

        67 files changed, 4426 insertions(+), 3835 deletions(-)

Given the early release of Linux 2.6.35 the merge window for the next release fully fell into the month of August. The XFS updates for Linux 2.6.36 include various additional performance improvements in the delayed logging code, for direct I/O writes and for avoiding synchronous transactions, as well as various fixed and large amount of cleanups, including the removal of the remaining dead DMAPI code.

On the userspace side we saw the 3.1.3 release of xfsprogs, which includes various smaller fixes, support for the new XFS_IOC_ZERO_RANGE ioctl and Debian packaging updates. The xfstests package saw one new test case and a couple of smaller patches, and xfsdump has not seen any updates at all.

The XMLified versions of the XFS users guide, training labs and filesystem structure documentation are now available as on the fly generated html on the xfs.org website and can be found at Papers & Documentation.

XFS status update for July 2010

July saw three more release candidates for the Linux 2.6.35 kernel, which included a relatively large number of XFS updates. There were two security fixes, a small one to prevent swapext to operate on write-only file descriptors, and a much larger one to properly validate inode numbers coming from NFS clients or userspace applications using the bulkstat or the open-by-handle interfaces. In addition to that another relatively large patch fixes the way inodes get reclaimed in the background, and avoids inode caches growing out of bounds.

In the meantime the code for the Linux 2.6.36 got the last touches before the expected opening of the merge window, by merging a few more last minute fixes and cleanups. The most notable one is a patch series that fixes in-memory corruption when concurrently accessing unwritten extents using the in-kernel AIO code.

The userspace side was still quite slow, but some a bit more activity than June. In xfsprogs the xfs_db code grew two bug fixes, as did the xfs_io script. The xfstests package saw one new test cases and various fixes to existing code. Last but not least a few patches affecting the build system for all userspace tools were committed.

XFS status update for June 2010

The month of June saw a few important bug fixes for the Linux 2.6.35 release candidates. That includes ensuring that files used for the swapext ioctl are writable to the user, and doing proper validation of inodes coming from untrusted sources, such as NFS exporting and the open by handle system calls. The main work however has been focused on development for the Linux 2.6.36 merge window, including merging various patches that have been out on the mainline list for a long time. Highlights include further performance improvements for sync heavy metadata workloads, stack space reduction in the writeback path and improvements of the XFS tracing infrastructure. Also after some discussion the remaining hooks for DMAPI are going to be dropped in mainline. As a replacement a tree containing full DMAPI support with a slightly cleaner XFS interaction will be hosted by SGI.

On the userspace side June was a rather slow month, with no updates to xfsprogs and xfsdump at all, and just one new test case and a cleanup applied to xfstests.

XFS status update for May 2010

In May 2010 we saw the long awaited release of Linux 2.6.34, which includes a large XFS update. The most important features appearing in 2.6.34 was the new inode and quota flushing code, which leads to much better I/O patterns for metadata-intensive workloads. Additionally support for synchronous NFS exports has been improved to give much better performance, and performance for the fsync, fdatasync and sync system calls has been improved slightly. A bug when resizing extremely busy filesystems has been fixed, which required extensive modification to the data structure used for looking up the per-allocation group data. Last but not least there was a steady flow of minor bug fixes and cleanups, leading to the following diffstat from 2.6.33 to 2.6.34:

 86 files changed, 3209 insertions(+), 3178 deletions(-)

Meanwhile active development aimed at 2.6.35 merge progressed. The major feature for this window is the merge of the delayed logging code, which adds a new logging mode that dramatically reduces the bandwidth required for log I/O. See the documentation for details. Testers for this new code are welcome.

In userland xfsprogs saw the long awaited 3.1.2 release, which can be considered a bug fix release for xfs_repair, xfs_fsr and mkfs.xfs. After the release a few more fixes were merged into the development tree. The xfstests package saw various new tests, including many tests to exercise the quota code, and a few fixes to existing tests.

XFS status update for April 2010

In April 2.6.34 still was in the release candidate phase, with a hand full of XFS fixes making it into mainline. Development for the 2.6.35 merge window went ahead full steam at the same time.

While a fair amount of patches hit the development tree these were largely cleanups, with the real development activity happening on the mailing list. There was another round of patches and following discussion on the scalable busy extent tracking and delayed logging features mentioned last month. They are expected to be merged in May and queue up for the Linux 2.6.35 window. Last but not least April saw a large number of XFS fixes backported to the 2.6.32 and 2.6.33 -stable series.

In user land xfsprogs has seen few but important updates, preparing for a new release next month. The xfs_repair tool saw a fix to correctly enable the lazy superblock counters on an existing filesystem, and xfs_fsr saw updates to better deal with dynamic attribute forks. Last but not a least a port to Debian GNU/kFreeBSD got merged. The xfstests test suite saw two new test cases and various smaller fixes.

XFS status update for March 2010

The merge window for Linux 2.6.34 closed in the first week of March, with the important XFS features already landing in February. Not surprisingly the XFS merge activity in March has been rather slow, with only about a dozen bug fixes patches making it towards Linus' tree in that time.

On the other hand active development for the 2.6.35 merge window has been very active. Most importantly there was a lot of work on the transaction and log subsystems. Starting with a large patchset to clean up and refactor the transaction subsystem and introducing more flexible I/O containers in the low-level logging code work is progressing to a new, more efficient logging implementation. While this preparatory work has already been merged in the development tree, the actual delayed logging implementation still needs more work after the initial public posting. The delayed logging implementation which is very briefly modeled after the journaling mode in the ext3/4 and reiserfs filesystems allows to accumulated multiple asynchronous transactions in memory instead of possibly writing them out many times. Using the new delayed logging mechanism I/O bandwidth used for the log decreases by orders of magnitude and performance on metadata intensive workloads increases massively.

In addition to that a new version of the discard (aka TRIM) support has been posted, this time entirely contained in kernel space and without the need of a userspace utility to drive it. Last but not least the usual steady stream of cleanups and bug fixes has not ceased this month either.

Besides the usual flow of fixes and new test cases in the xfstests test suite development on the userspace side has been rather slow. Xfsprogs has only seen a single fix for SMP locking in xfs_repair and support for building on Debian GNU/kFreeBSD, and xfsdump has seen no commit at all.

XFS status update for February 2010

February saw the release of the Linux 2.6.33 kernel, which includes a large XFS update. The biggest user-visible change in Linux 2.6.33 is that XFS now support the generic Linux trace event infrastructure, which allows tracing lots of XFS behavior with a normal production built kernel. Except for this Linux 2.6.33 has been mostly a bug-fix release, fixing various user reported bugs in previous releases. The total diffstat for XFS in Linux 2.6.33 looks like:

 84 files changed, 3023 insertions(+), 3550 deletions(-)

In addition to that the merge window for Linux 2.6.34 opened and the first merge of the XFS tree made it into Linus tree. Unlike Linux 2.6.33 this merge window includes major feature work. The most important change for users is a new algorithm for inode and quota writeback that leads to better I/O locality and improved metadata performance. The second big change is a rewrite of the per-allocation group data lookup which fixes a long-standing problem in the code to grow a life filesystem and will also ease future filesystem shrinking support. Not merged through the XFS tree, but of great importance for embedded users is a new API that allows XFS to properly flush cache lines on it's log and large directory buffers, making XFS work properly on architectures with virtually indexed caches, such as parisc and various arm and mips variants. Last but not least there is an above average amount of cleanups that went into Linus tree in this cycle.

There have been more patches on the mailing list that haven't made it to Linus tree yet, including an optimized implementation of fdatasync(2) and massive speedups for metadata workloads on NFS exported XFS filesystems.

On the userspace side February has been a relatively quiet month. Lead by xfstests only a moderate amount of fixes made it into the respective trees.

XFS status update for January 2010

January saw additional release candidates of the Linux 2.6.33 kernel, including a couple of bug fixes for XFS. In the meantime the XFS tree has been growing a large number of patches destined for the Linux 2.6.34 merge window: a large rework of the handling of per-AG data, support for the quota netlink interface, and better power saving behavior of the XFS kernel threads, and of course various cleanups.

A large patch series to replace the current asynchronous inode writeback with a new scheme that uses the delayed write buffers was posted to the list. The new scheme, which allows archive better I/O locality by dispatching meta-data I/O from a single place has been discussed extensively and is expected to be merged in February.

On the userspace side January saw the 3.1.0 and 3.1.1 releases of xfsprogs, as well as the 3.0.4 release of xfsdump. The biggest changes in xfsprogs 3.1.0 were optimizations in xfs_repair that lead to a much lower memory usage, and optional use of the blkid library for filesystem detection and retrieving storage topology information. The 3.1.1 release contained various important bug fixes for these changes and a various improvements to the build system. The major feature of xfsdump 3.0.4 were fixes for time stamp handling on 64-bit systems.

The xfstests package also lots of activity including various new testcases and an improved build system.