Pibroch: New page: == XFS status update for December 2010 == The release process of the Linux 2.6.37 kernel with it's large XFS updates was in it's final days in December, which explains why we only saw a s...

2012-03-19T06:13:02Z

New page: == XFS status update for December 2010 == The release process of the Linux 2.6.37 kernel with it's large XFS updates was in it's final days in December, which explains why we only saw a s...

New page

== XFS status update for December 2010 ==

The release process of the Linux 2.6.37 kernel with it's large XFS updates
was in it's final days in December, which explains why we only saw a single
one-liner regression fix for XFS in Linus' tree. The XFS development tree
finally saw some updates when the writeback updates and some small cleanups
to the allocator and log recovery code were merged, but the large metadata
scalability updates that have been posted to the list multiple times are
still missing. In addition to this on-going work the list also saw patches
that fix smaller issues, which are also still waiting to be merged.

On the userspace side xfsprogs and xfsdump development has been quit, with
no commits to either repository in December, although a large series of
updates to the metadump command has been reposted near the end of the month.
The xfstests repository saw a new regression test for a btrfs problem,
and various updates to existing tests.

== XFS status update for November 2010 ==

From looking at the kernel git commits November looked like a pretty
slow month with just two hand full fixes going into the release candidates
for Linux 2.6.37, and none at all going into the development tree.
But in this case git statistics didn't tell the whole story - there
was a lot of activity on patches for the next merge window on the list.
The focus in November was still at metadata scalability, with various
patchsets that improves parallel creates and unlinks again, and also
improves 8-way dbench throughput by 30%. In addition to that there
were patches to improve preallocation for NFS servers, to simplify
the writeback code, and to remove the XFS-internal percpu counters
for free space for the generic kernel percpu counters, which just needed
a small improvement.

On the user space side we saw the release of xfsprogs 3.1.4, which
contains various accumulated bug fixes and Debian packaging updates.
The xfsdump tree saw a large update to speed up restore by using
mmap for an internal database and remove the limitation of ~ 214
million directory entries per dump file. The xfstests test suite
saw three new testcases and various fixes, including support for the
hfsplus filesystem.

== XFS status update for October 2010 ==

Near the end of the month we finally saw the release of Linux 2.6.36.
Just a single fix made it into mainline in this month, showing that the
stabilization period before has worked very well.

Linux 2.6.36 has been another impressive release for XFS, seeing
various performance improvements in the new delayed logging code,
for direct I/O and the sync system call, a few bug fixes, and lots
of cleanups, resulting in a net removal of over 2000 lines of code:

89 files changed, 1998 insertions(+), 4279 deletions(-)

The merge window for Linux 2.6.37 opened just a few days after the
release of Linux 2.6.36 and already contains another large XFS update
at the end of October. Highlights of the XFS tree merged into 2.6.37-rc1
are another large set of metadata scalability patches, support for 32-bit
wide project IDs, and support for the new XFS_IOC_ZERO_RANGE ioctl,
which allows to punch a whole and convert it to an unwritten extent
in a single atomic operation.

The metadata scalability changes improve 8-way fs_mark of 50 million files
by over 15% and removal of those files by over 100%, with further
improvements expected by the next round of XFS metadata scalability
and VFS scalability improvements targeted at Linux 2.6.38.

On the user space side October was a rather quit month for xfsprogs, which
only saw the addition of 32-bit project ID handling, and a fix for
parsing the mount table in fsr when used together with disk encryption
tools. A few patches for xfsdump were posted on the list, but none
was applied, leaving the majority of the user space activity to
xfstests, which saw very active development. Various patches went
into xfstests to improve portability to filesystems with a limited
feature set, and to move more filters to generic code. In addition
various cleanups to test cases in test programs were applied.

== XFS status update for September 2010 ==

Mainline activity has been rather low in September while with only
two more fixes going into the 2.6.36 release candidates after the
large merge activity in August. Development for the next merge
window has been more active. The largest item was the inclusion
of the metadata scalability patch series, which provides very large
speedups for parallel metadata operations. In addition a new
ioctl to punch holes and convert the whole to an unwritten extent
was added and a small number of cleanups also made it into the tree.

Patches to add support for 32bit wide project ID identifiers and
using group and project quotas concurrently were posted to the list
and discussed but not yet included.

Userspace development has been rather quiet again, with a single fix
committed to xfsprogs and xfsdump each. The xfstests test suite grew
a new test case and received a few additional fixes. Last but not least
the [http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/index.html XFS Users Guide]
was updated with various factual corrections and spelling fixes.

== XFS status update for August 2010 ==

At the first of August we finally saw the release of Linux 2.6.35,
which includes a large XFS update. The most prominent feature in
Linux 2.6.35 is the new delayed logging code which provides massive
speedups for metadata-intensive workloads, but there has been
a large amount of other fixes and cleanups, leading to the following
diffstat:

67 files changed, 4426 insertions(+), 3835 deletions(-)

Given the early release of Linux 2.6.35 the merge window for the
next release fully fell into the month of August. The XFS updates
for Linux 2.6.36 include various additional performance improvements
in the delayed logging code, for direct I/O writes and for avoiding
synchronous transactions, as well as various fixed and large amount
of cleanups, including the removal of the remaining dead DMAPI
code.

On the userspace side we saw the 3.1.3 release of xfsprogs, which includes
various smaller fixes, support for the new XFS_IOC_ZERO_RANGE ioctl and
Debian packaging updates. The xfstests package saw one new test case
and a couple of smaller patches, and xfsdump has not seen any updates at
all.

The XMLified versions of the XFS users guide, training labs and filesystem
structure documentation are now available as on the fly generated html on
the xfs.org website and can be found at [[XFS_Papers_and_Documentation|Papers & Documentation]].

== XFS status update for July 2010 ==

July saw three more release candidates for the Linux 2.6.35 kernel, which
included a relatively large number of XFS updates. There were two security
fixes, a small one to prevent swapext to operate on write-only file
descriptors, and a much larger one to properly validate inode numbers
coming from NFS clients or userspace applications using the bulkstat or
the open-by-handle interfaces. In addition to that another relatively
large patch fixes the way inodes get reclaimed in the background, and
avoids inode caches growing out of bounds.

In the meantime the code for the Linux 2.6.36 got the last touches before
the expected opening of the merge window, by merging a few more last
minute fixes and cleanups. The most notable one is a patch series
that fixes in-memory corruption when concurrently accessing unwritten
extents using the in-kernel AIO code.

The userspace side was still quite slow, but some a bit more activity
than June. In xfsprogs the xfs_db code grew two bug fixes, as did
the xfs_io script. The xfstests package saw one new test cases and
various fixes to existing code. Last but not least a few patches
affecting the build system for all userspace tools were committed.

== XFS status update for June 2010 ==

The month of June saw a few important bug fixes for the Linux 2.6.35
release candidates. That includes ensuring that files used for the
swapext ioctl are writable to the user, and doing proper validation
of inodes coming from untrusted sources, such as NFS exporting and
the open by handle system calls. The main work however has been
focused on development for the Linux 2.6.36 merge window, including
merging various patches that have been out on the mainline list
for a long time. Highlights include further performance improvements
for sync heavy metadata workloads, stack space reduction in the
writeback path and improvements of the XFS tracing infrastructure.
Also after some discussion the remaining hooks for DMAPI are going
to be dropped in mainline. As a replacement a tree containing
full DMAPI support with a slightly cleaner XFS interaction will be
hosted by SGI.

On the userspace side June was a rather slow month, with no updates
to xfsprogs and xfsdump at all, and just one new test case and a cleanup
applied to xfstests.

== XFS status update for May 2010 ==

In May 2010 we saw the long awaited release of Linux 2.6.34, which includes
a large XFS update. The most important features appearing in 2.6.34 was the
new inode and quota flushing code, which leads to much better I/O patterns
for metadata-intensive workloads. Additionally support for synchronous NFS
exports has been improved to give much better performance, and performance
for the fsync, fdatasync and sync system calls has been improved slightly.
A bug when resizing extremely busy filesystems has been fixed, which required
extensive modification to the data structure used for looking up the
per-allocation group data. Last but not least there was a steady flow of
minor bug fixes and cleanups, leading to the following diffstat from
2.6.33 to 2.6.34:

86 files changed, 3209 insertions(+), 3178 deletions(-)

Meanwhile active development aimed at 2.6.35 merge progressed. The
major feature for this window is the merge of the delayed logging code,
which adds a new logging mode that dramatically reduces the bandwidth
required for log I/O. See the
[http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs-delayed-logging-design.txt;h=96d0df28bed323d5596fc051b0ffb96ed8e3c8df;hb=HEAD documentation] for details. Testers
for this new code are welcome.

In userland xfsprogs saw the long awaited 3.1.2 release, which can be
considered a bug fix release for xfs_repair, xfs_fsr and mkfs.xfs. After
the release a few more fixes were merged into the development tree.
The xfstests package saw various new tests, including many tests to
exercise the quota code, and a few fixes to existing tests.

== XFS status update for April 2010 ==

In April 2.6.34 still was in the release candidate phase, with
a hand full of XFS fixes making it into mainline. Development for
the 2.6.35 merge window went ahead full steam at the same time.

While a fair amount of patches hit the development tree these were
largely cleanups, with the real development activity happening on
the mailing list. There was another round of patches and following
discussion on the scalable busy extent tracking and delayed logging
features mentioned last month. They are expected to be merged in
May and queue up for the Linux 2.6.35 window. Last but not least
April saw a large number of XFS fixes backported to the 2.6.32 and
2.6.33 -stable series.

In user land xfsprogs has seen few but important updates, preparing
for a new release next month. The xfs_repair tool saw a fix to
correctly enable the lazy superblock counters on an existing
filesystem, and xfs_fsr saw updates to better deal with dynamic
attribute forks. Last but not a least a port to Debian GNU/kFreeBSD
got merged. The xfstests test suite saw two new test cases and various
smaller fixes.

== XFS status update for March 2010 ==

The merge window for Linux 2.6.34 closed in the first week of March,
with the important XFS features already landing in February. Not
surprisingly the XFS merge activity in March has been rather slow,
with only about a dozen bug fixes patches making it towards Linus'
tree in that time.

On the other hand active development for the 2.6.35 merge window has
been very active. Most importantly there was a lot of work on the
transaction and log subsystems. Starting with a large patchset to
clean up and refactor the transaction subsystem and introducing more
flexible I/O containers in the low-level logging code work is
progressing to a new, more efficient logging implementation. While
this preparatory work has already been merged in the development tree,
the actual delayed logging implementation still needs more work after
the initial public posting. The delayed logging implementation which
is very briefly modeled after the journaling mode in the ext3/4
and reiserfs filesystems allows to accumulated multiple asynchronous
transactions in memory instead of possibly writing them out
many times. Using the new delayed logging mechanism I/O bandwidth
used for the log decreases by orders of magnitude and performance
on metadata intensive workloads increases massively.

In addition to that a new version of the discard (aka TRIM) support
has been posted, this time entirely contained in kernel space
and without the need of a userspace utility to drive it. Last but
not least the usual steady stream of cleanups and bug fixes has not
ceased this month either.

Besides the usual flow of fixes and new test cases in the xfstests
test suite development on the userspace side has been rather slow.
Xfsprogs has only seen a single fix for SMP locking in xfs_repair
and support for building on Debian GNU/kFreeBSD, and xfsdump
has seen no commit at all.

== XFS status update for February 2010 ==

February saw the release of the Linux 2.6.33 kernel, which includes
a large XFS update. The biggest user-visible change in Linux 2.6.33
is that XFS now support the generic Linux trace event infrastructure,
which allows tracing lots of XFS behavior with a normal production
built kernel. Except for this Linux 2.6.33 has been mostly a bug-fix
release, fixing various user reported bugs in previous releases.
The total diffstat for XFS in Linux 2.6.33 looks like:

84 files changed, 3023 insertions(+), 3550 deletions(-)

In addition to that the merge window for Linux 2.6.34 opened and the
first merge of the XFS tree made it into Linus tree. Unlike Linux
2.6.33 this merge window includes major feature work. The most
important change for users is a new algorithm for inode and quota
writeback that leads to better I/O locality and improved metadata
performance. The second big change is a rewrite of the per-allocation
group data lookup which fixes a long-standing problem in the code
to grow a life filesystem and will also ease future filesystem
shrinking support. Not merged through the XFS tree, but of great
importance for embedded users is a new API that allows XFS to properly
flush cache lines on it's log and large directory buffers, making
XFS work properly on architectures with virtually indexed caches,
such as parisc and various arm and mips variants. Last but not
least there is an above average amount of cleanups that went into
Linus tree in this cycle.

There have been more patches on the mailing list that haven't made
it to Linus tree yet, including an optimized implementation of
fdatasync(2) and massive speedups for metadata workloads on
NFS exported XFS filesystems.

On the userspace side February has been a relatively quiet month.
Lead by xfstests only a moderate amount of fixes made it into
the respective trees.

== XFS status update for January 2010 ==

January saw additional release candidates of the Linux 2.6.33 kernel,
including a couple of bug fixes for XFS. In the meantime the XFS tree
has been growing a large number of patches destined for the Linux 2.6.34
merge window: a large rework of the handling of per-AG data, support for
the quota netlink interface, and better power saving behavior of the
XFS kernel threads, and of course various cleanups.

A large patch series to replace the current asynchronous inode writeback
with a new scheme that uses the delayed write buffers was posted to
the list. The new scheme, which allows archive better I/O locality by
dispatching meta-data I/O from a single place has been discussed
extensively and is expected to be merged in February.

On the userspace side January saw the 3.1.0 and 3.1.1 releases of xfsprogs,
as well as the 3.0.4 release of xfsdump. The biggest changes in xfsprogs
3.1.0 were optimizations in xfs_repair that lead to a much lower memory
usage, and optional use of the blkid library for filesystem detection
and retrieving storage topology information. The 3.1.1 release contained
various important bug fixes for these changes and a various improvements to
the build system. The major feature of xfsdump 3.0.4 were fixes for
time stamp handling on 64-bit systems.

The xfstests package also lots of activity including various new testcases
and an improved build system.

XFS status update for 2010 - Revision history

Pibroch: New page: == XFS status update for December 2010 == The release process of the Linux 2.6.37 kernel with it's large XFS updates was in it's final days in December, which explains why we only saw a s...