xfs.org - User contributions [en]

XFS FAQ

2018-07-30T22:32:12Z

Sandeen: /* Q: Where can I find information about the internals of XFS? */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Darrick has updated Barry Naujok's documentation of the [https://www.kernel.org/pub/linux/utils/fs/xfs/docs/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAIDs have a persistent write cache which is preserved across power failure, interface resets, system crashes, etc. The same may be true of some SSD devices. This sort of hardware should report to the operating system that no flushes are required, and in that case barriers will not be issued, even without the "nobarrier" option. Quoting Christoph Hellwig [http://oss.sgi.com/archives/xfs/2015-12/msg00281.html on the xfs list],
If the device does not need cache flushes it should not report requiring
flushes, in which case nobarrier will be a noop. Or to phrase it
differently: If nobarrier makes a difference skipping it is not safe.
On modern kernels with hardware which properly reports write cache behavior, there is no need to change barrier options at mount time.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/archives/xfs/2009-01/msg01023.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. This was an issue with the older "inode32" inode allocation mode, where inode allocation is restricted to lower filesysetm blocks. To fix this, [http://oss.sgi.com/archives/xfs/2009-01/msg01031.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Alternately, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 No space left on device on xfs filesystem with 7.7TB free]

However, 'inode64' has been the default behavior since kernel v3.7...

Unfortunately, v3.7 also added a bug present from kernel v3.7 to v3.17 which caused new allocation groups added by growfs to be unavailable for inode allocation. This was fixed by commit <tt>[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9de67c3ba9ea961ba420573d56479d09d33a7587 9de67c3b xfs: allow inode allocations in post-growfs disk space.]</tt> in kernel v3.17.
Without that commit, the problem can be worked around by doing a "mount -o remount,inode64" after the growfs operation.

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

== Q: Why do files on XFS use more data blocks than expected? ==

The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimize file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistance in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.

This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.

See the FAQ entry on speculative preallocation for details.

== Q: What is speculative preallocation? ==

XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.

Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).

In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor.

== Q: How can I speed up or avoid delayed removal of speculative preallocation? ==

Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):

/proc/sys/fs/xfs/speculative_prealloc_lifetime

== Q: Is speculative preallocation permanent? ==

Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate) or following a crash. In the event of a crash, the in-memory state used to track and reclaim the speculative preallocation is lost.

== Q: My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size? ==

Speculative preallocation can not be disabled but XFS can be tuned to a fixed allocation size with the 'allocsize=' mount option. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. Use 'allocsize=64k' to revert to the default XFS behavior prior to support for dynamic speculative preallocation.

== Q: mount (or umount) takes minutes or even hours - what could be the reason ? ==

In some cases xfs log (journal) can become quite big. For example if it accumulates many entries and doesn't get chance to apply these to disk (due to lockup, crash, hard reset etc). xfs will try to reapply these at mount (in dmesg: "Starting recovery (logdev: internal)").

That process with big log to be reapplied can take very long time (minutes or even hours). Similar problem can happen with unmount taking hours when there are hundreds of thousands of dirty inode in memory that need to be flushed to disk.

(http://oss.sgi.com/pipermail/xfs/2015-October/044457.html)

== Q: Which I/O scheduler for XFS? ==

=== On rotational disks without hardware raid ===

''CFQ'': not great for XFS parallelism:

< dchinner> it doesn't allow other threads to get IO issued immediately after the first one
< dchinner> it waits, instead, for a timeslice to expire before moving to the IO of a different process.
< dchinner> so instead of interleaving the IO of multiple jobs in a single sweep across the disk,
it enforces single threaded access to the disk

''deadline'': good option, doesn't have such problem

Note that some kernels have block multiqueue enabled which (currently - 08/2016) doesn't support I/O schedulers at all thus there is no optimisation and reordering IO for best seek order, so disable blk-mq for rotational disks (see CONFIG_SCSI_MQ_DEFAULT, CONFIG_DM_MQ_DEFAULT options and use_blk_mq parameter for scsi-mod/dm-mod kernel modules).

Also hardware raid can be smart enough to cache and reorder I/O requests thus additional layer of reordering
(like Linux I/O scheduler) can potentially conflict and make performance worse. If you have such raid card
then try method described below.

=== SSD disks or rotational disks but with hardware raid card that has cache enabled ===

''Block multiqueue'' enabled (and thus no I/O scheduler at all) or block multiqueue disabled and ''noop'' or ''deadline'' I/O scheduler activated is good solution. SSD disks don't really need I/O schedulers while smart raid cards do I/O ordering on their own.

Note that if your raid is very dumb and/or has no cache enabled then it likely cannot reorder I/O requests and thus it could benefit from I/O scheduler.

== Q: Why does userspace say "filesystem uses v1 dirs, limited functionality provided?" ==

Either you have a very old or a rather new filesystem coupled with too-old userspace.

Very old filesystems used a format called "directory version 1" and in this case the error is correct and hopefully self explanatory.

However, if you have a newer filesystem with version 5 superblocks and the metadata CRC feature enabled, older releases of xfsprogs may incorrectly issue the "v1 dir" message. In this case, get newer xfsprogs; at least v3.2.0, but preferably the latest release.

Getting the latest source code

2018-07-30T22:20:51Z

Sandeen: /* Current XFS kernel source */

== XFS Released/Stable source ==

Note: as of September 2016, the XFS project is moving away from oss.sgi.com infrastructure. As we move to other infrastructure the links below will be updated to point to the new locations.

* '''Mainline kernels'''
:XFS has been maintained in the official Linux kernel [http://www.kernel.org/ kernel trees] starting with [http://lkml.org/lkml/2003/12/8/35 Linux 2.4] and is frequently updated with the latest stable fixes and features from the XFS development team.

* '''Vendor kernels'''
:All modern Linux distributions include support for XFS.

* '''XFS userspace'''
:[https://kernel.org/pub/linux/utils/fs/xfs source code tarballs] of the xfs userspace tools. These tarballs form the basis of the xfsprogs packages found in Linux distributions.

== Development and bleeding edge Development ==

* [[XFS git howto]]

=== Current XFS kernel source ===

* [https://git.kernel.org/pub/scm/fs/xfs/xfs-linux.git xfs]

$ git clone git.kernel.org/pub/scm/fs/xfs/xfs-linux.git

=== XFS user space tools ===
* [https://git.kernel.org/cgit/fs/xfs/xfsprogs-dev.git/ xfsprogs ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git

A few packages are needed to compile <tt>xfsprogs</tt>, depending on your package manager:

apt-get install libtool automake gettext libblkid-dev uuid-dev pkg-config
yum install libtool automake gettext libblkid-devel libuuid-devel

=== XFS dump ===
* [https://git.kernel.org/cgit/fs/xfs/xfsdump-dev.git/ xfsdump ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git

=== XFS tests ===
* [https://git.kernel.org/cgit/fs/xfs/xfstests-dev.git/ xfstests ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

=== DMAPI user space tools ===
* [https://git.kernel.org/cgit/fs/xfs/dmapi-dev.git/ dmapi ]

git clone git://git.kernel.org/pub/scm/fs/xfs/dmapi-dev.git

=== git-cvsimport generated trees ===

The Git trees are automated mirrored copies of the CVS trees using [http://www.kernel.org/pub/software/scm/git/docs/git-cvsimport.html git-cvsimport].
Since git-cvsimport utilized the tool [http://www.cobite.com/cvsps/ cvsps] to recreate the atomic commits of ptools or "mod" it is easier to see the entire change that was committed using git.

* [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=archive/xfs-import.git;a=summary linux-2.6-xfs-from-cvs]
* [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=archive/xfs-cmds.git;a=summary xfs-cmds]

Before building in the <tt>xfsdump</tt> or <tt>dmapi</tt> directories (after building <tt>xfsprogs</tt>), you will need to run:
# cd xfsprogs
# make install-dev
to create <tt>/usr/include/xfs</tt> and install appropriate files there.

Before building in the xfstests directory, you will need to run:
# cd xfsprogs
# make install-qa
to install a somewhat larger set of files in <tt>/usr/include/xfs</tt>.

== XFS cvs trees ==

The cvs trees were created using a script that converted sgi's internal
ptools repository to a cvs repository, so the cvs trees were considered read only.

At this point all new development is being managed by the git trees thus the cvs trees
are no longer active in terms of current development and should only be used
for reference.

* [[XFS CVS howto]]

Getting the latest source code

2018-07-24T06:32:03Z

Sandeen: /* XFS user space tools */

== XFS Released/Stable source ==

Note: as of September 2016, the XFS project is moving away from oss.sgi.com infrastructure. As we move to other infrastructure the links below will be updated to point to the new locations.

* '''Mainline kernels'''
:XFS has been maintained in the official Linux kernel [http://www.kernel.org/ kernel trees] starting with [http://lkml.org/lkml/2003/12/8/35 Linux 2.4] and is frequently updated with the latest stable fixes and features from the XFS development team.

* '''Vendor kernels'''
:All modern Linux distributions include support for XFS.

* '''XFS userspace'''
:[https://kernel.org/pub/linux/utils/fs/xfs source code tarballs] of the xfs userspace tools. These tarballs form the basis of the xfsprogs packages found in Linux distributions.

== Development and bleeding edge Development ==

* [[XFS git howto]]

=== Current XFS kernel source ===

* [https://git.kernel.org/cgit/linux/kernel/git/dgc/linux-xfs.git/ xfs]

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git

Note: the old kernel tree on [http://oss.sgi.com/cgi-bin/gitweb.cgi oss.sgi.com] is no longer kept up to date with the master tree on kernel.org.

=== XFS user space tools ===
* [https://git.kernel.org/cgit/fs/xfs/xfsprogs-dev.git/ xfsprogs ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git

A few packages are needed to compile <tt>xfsprogs</tt>, depending on your package manager:

apt-get install libtool automake gettext libblkid-dev uuid-dev pkg-config
yum install libtool automake gettext libblkid-devel libuuid-devel

=== XFS dump ===
* [https://git.kernel.org/cgit/fs/xfs/xfsdump-dev.git/ xfsdump ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfsdump-dev.git

=== XFS tests ===
* [https://git.kernel.org/cgit/fs/xfs/xfstests-dev.git/ xfstests ]

git clone git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git

=== DMAPI user space tools ===
* [https://git.kernel.org/cgit/fs/xfs/dmapi-dev.git/ dmapi ]

git clone git://git.kernel.org/pub/scm/fs/xfs/dmapi-dev.git

=== git-cvsimport generated trees ===

The Git trees are automated mirrored copies of the CVS trees using [http://www.kernel.org/pub/software/scm/git/docs/git-cvsimport.html git-cvsimport].
Since git-cvsimport utilized the tool [http://www.cobite.com/cvsps/ cvsps] to recreate the atomic commits of ptools or "mod" it is easier to see the entire change that was committed using git.

* [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=archive/xfs-import.git;a=summary linux-2.6-xfs-from-cvs]
* [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=archive/xfs-cmds.git;a=summary xfs-cmds]

Before building in the <tt>xfsdump</tt> or <tt>dmapi</tt> directories (after building <tt>xfsprogs</tt>), you will need to run:
# cd xfsprogs
# make install-dev
to create <tt>/usr/include/xfs</tt> and install appropriate files there.

Before building in the xfstests directory, you will need to run:
# cd xfsprogs
# make install-qa
to install a somewhat larger set of files in <tt>/usr/include/xfs</tt>.

== XFS cvs trees ==

The cvs trees were created using a script that converted sgi's internal
ptools repository to a cvs repository, so the cvs trees were considered read only.

At this point all new development is being managed by the git trees thus the cvs trees
are no longer active in terms of current development and should only be used
for reference.

* [[XFS CVS howto]]

XFS FAQ

2016-12-05T21:04:01Z

Sandeen: /* Q: Why does userspace say "filesystem uses v1 dirs, limited functionality provided?" */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAIDs have a persistent write cache which is preserved across power failure, interface resets, system crashes, etc. The same may be true of some SSD devices. This sort of hardware should report to the operating system that no flushes are required, and in that case barriers will not be issued, even without the "nobarrier" option. Quoting Christoph Hellwig [http://oss.sgi.com/archives/xfs/2015-12/msg00281.html on the xfs list],
If the device does not need cache flushes it should not report requiring
flushes, in which case nobarrier will be a noop. Or to phrase it
differently: If nobarrier makes a difference skipping it is not safe.
On modern kernels with hardware which properly reports write cache behavior, there is no need to change barrier options at mount time.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/archives/xfs/2009-01/msg01023.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. This was an issue with the older "inode32" inode allocation mode, where inode allocation is restricted to lower filesysetm blocks. To fix this, [http://oss.sgi.com/archives/xfs/2009-01/msg01031.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Alternately, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 No space left on device on xfs filesystem with 7.7TB free]

However, 'inode64' has been the default behavior since kernel v3.7...

Unfortunately, v3.7 also added a bug present from kernel v3.7 to v3.17 which caused new allocation groups added by growfs to be unavailable for inode allocation. This was fixed by commit <tt>[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9de67c3ba9ea961ba420573d56479d09d33a7587 9de67c3b xfs: allow inode allocations in post-growfs disk space.]</tt> in kernel v3.17.
Without that commit, the problem can be worked around by doing a "mount -o remount,inode64" after the growfs operation.

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

== Q: Why do files on XFS use more data blocks than expected? ==

The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimize file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistance in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.

This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.

See the FAQ entry on speculative preallocation for details.

== Q: What is speculative preallocation? ==

XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.

Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).

In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor.

== Q: How can I speed up or avoid delayed removal of speculative preallocation? ==

Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):

/proc/sys/fs/xfs/speculative_prealloc_lifetime

== Q: Is speculative preallocation permanent? ==

Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate) or following a crash. In the event of a crash, the in-memory state used to track and reclaim the speculative preallocation is lost.

== Q: My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size? ==

Speculative preallocation can not be disabled but XFS can be tuned to a fixed allocation size with the 'allocsize=' mount option. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. Use 'allocsize=64k' to revert to the default XFS behavior prior to support for dynamic speculative preallocation.

== Q: mount (or umount) takes minutes or even hours - what could be the reason ? ==

In some cases xfs log (journal) can become quite big. For example if it accumulates many entries and doesn't get chance to apply these to disk (due to lockup, crash, hard reset etc). xfs will try to reapply these at mount (in dmesg: "Starting recovery (logdev: internal)").

That process with big log to be reapplied can take very long time (minutes or even hours). Similar problem can happen with unmount taking hours when there are hundreds of thousands of dirty inode in memory that need to be flushed to disk.

(http://oss.sgi.com/pipermail/xfs/2015-October/044457.html)

== Q: Which I/O scheduler for XFS? ==

=== On rotational disks without hardware raid ===

''CFQ'': not great for XFS parallelism:

< dchinner> it doesn't allow other threads to get IO issued immediately after the first one
< dchinner> it waits, instead, for a timeslice to expire before moving to the IO of a different process.
< dchinner> so instead of interleaving the IO of multiple jobs in a single sweep across the disk,
it enforces single threaded access to the disk

''deadline'': good option, doesn't have such problem

Note that some kernels have block multiqueue enabled which (currently - 08/2016) doesn't support I/O schedulers at all thus there is no optimisation and reordering IO for best seek order, so disable blk-mq for rotational disks (see CONFIG_SCSI_MQ_DEFAULT, CONFIG_DM_MQ_DEFAULT options and use_blk_mq parameter for scsi-mod/dm-mod kernel modules).

Also hardware raid can be smart enough to cache and reorder I/O requests thus additional layer of reordering
(like Linux I/O scheduler) can potentially conflict and make performance worse. If you have such raid card
then try method described below.

=== SSD disks or rotational disks but with hardware raid card that has cache enabled ===

''Block multiqueue'' enabled (and thus no I/O scheduler at all) or block multiqueue disabled and ''noop'' or ''deadline'' I/O scheduler activated is good solution. SSD disks don't really need I/O schedulers while smart raid cards do I/O ordering on their own.

Note that if your raid is very dumb and/or has no cache enabled then it likely cannot reorder I/O requests and thus it could benefit from I/O scheduler.

== Q: Why does userspace say "filesystem uses v1 dirs, limited functionality provided?" ==

Either you have a very old or a rather new filesystem coupled with too-old userspace.

Very old filesystems used a format called "directory version 1" and in this case the error is correct and hopefully self explanatory.

However, if you have a newer filesystem with version 5 superblocks and the metadata CRC feature enabled, older releases of xfsprogs may incorrectly issue the "v1 dir" message. In this case, get newer xfsprogs; at least v3.2.0, but preferably the latest release.

XFS FAQ

2016-12-05T21:02:57Z

Sandeen:

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAIDs have a persistent write cache which is preserved across power failure, interface resets, system crashes, etc. The same may be true of some SSD devices. This sort of hardware should report to the operating system that no flushes are required, and in that case barriers will not be issued, even without the "nobarrier" option. Quoting Christoph Hellwig [http://oss.sgi.com/archives/xfs/2015-12/msg00281.html on the xfs list],
If the device does not need cache flushes it should not report requiring
flushes, in which case nobarrier will be a noop. Or to phrase it
differently: If nobarrier makes a difference skipping it is not safe.
On modern kernels with hardware which properly reports write cache behavior, there is no need to change barrier options at mount time.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/archives/xfs/2009-01/msg01023.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. This was an issue with the older "inode32" inode allocation mode, where inode allocation is restricted to lower filesysetm blocks. To fix this, [http://oss.sgi.com/archives/xfs/2009-01/msg01031.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Alternately, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 No space left on device on xfs filesystem with 7.7TB free]

However, 'inode64' has been the default behavior since kernel v3.7...

Unfortunately, v3.7 also added a bug present from kernel v3.7 to v3.17 which caused new allocation groups added by growfs to be unavailable for inode allocation. This was fixed by commit <tt>[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9de67c3ba9ea961ba420573d56479d09d33a7587 9de67c3b xfs: allow inode allocations in post-growfs disk space.]</tt> in kernel v3.17.
Without that commit, the problem can be worked around by doing a "mount -o remount,inode64" after the growfs operation.

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

== Q: Why do files on XFS use more data blocks than expected? ==

The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimize file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistance in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.

This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.

See the FAQ entry on speculative preallocation for details.

== Q: What is speculative preallocation? ==

XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.

Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).

In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor.

== Q: How can I speed up or avoid delayed removal of speculative preallocation? ==

Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):

/proc/sys/fs/xfs/speculative_prealloc_lifetime

== Q: Is speculative preallocation permanent? ==

Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate) or following a crash. In the event of a crash, the in-memory state used to track and reclaim the speculative preallocation is lost.

== Q: My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size? ==

Speculative preallocation can not be disabled but XFS can be tuned to a fixed allocation size with the 'allocsize=' mount option. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. Use 'allocsize=64k' to revert to the default XFS behavior prior to support for dynamic speculative preallocation.

== Q: mount (or umount) takes minutes or even hours - what could be the reason ? ==

In some cases xfs log (journal) can become quite big. For example if it accumulates many entries and doesn't get chance to apply these to disk (due to lockup, crash, hard reset etc). xfs will try to reapply these at mount (in dmesg: "Starting recovery (logdev: internal)").

That process with big log to be reapplied can take very long time (minutes or even hours). Similar problem can happen with unmount taking hours when there are hundreds of thousands of dirty inode in memory that need to be flushed to disk.

(http://oss.sgi.com/pipermail/xfs/2015-October/044457.html)

== Q: Which I/O scheduler for XFS? ==

=== On rotational disks without hardware raid ===

''CFQ'': not great for XFS parallelism:

< dchinner> it doesn't allow other threads to get IO issued immediately after the first one
< dchinner> it waits, instead, for a timeslice to expire before moving to the IO of a different process.
< dchinner> so instead of interleaving the IO of multiple jobs in a single sweep across the disk,
it enforces single threaded access to the disk

''deadline'': good option, doesn't have such problem

Note that some kernels have block multiqueue enabled which (currently - 08/2016) doesn't support I/O schedulers at all thus there is no optimisation and reordering IO for best seek order, so disable blk-mq for rotational disks (see CONFIG_SCSI_MQ_DEFAULT, CONFIG_DM_MQ_DEFAULT options and use_blk_mq parameter for scsi-mod/dm-mod kernel modules).

Also hardware raid can be smart enough to cache and reorder I/O requests thus additional layer of reordering
(like Linux I/O scheduler) can potentially conflict and make performance worse. If you have such raid card
then try method described below.

=== SSD disks or rotational disks but with hardware raid card that has cache enabled ===

''Block multiqueue'' enabled (and thus no I/O scheduler at all) or block multiqueue disabled and ''noop'' or ''deadline'' I/O scheduler activated is good solution. SSD disks don't really need I/O schedulers while smart raid cards do I/O ordering on their own.

Note that if your raid is very dumb and/or has no cache enabled then it likely cannot reorder I/O requests and thus it could benefit from I/O scheduler.

== Q: Why does userspace say "filesystem uses v1 dirs, limited functionality provided?" ==

Either you have a very old or a very new filesystem. Very old filesystems used a format called "directory version 1" and in this case the error is self explanatory.

However, if you have a new filesystem with version 5 superblocks and the metadata CRC feature enabled, older releases of xfsprogs may incorrectly issue the "v1 dir" message. In this case, get newer xfsprogs; at least v3.2.0, but preferably the latest release.

XFS email list and archives

2016-09-15T14:33:28Z

Sandeen:

= XFS mailing list =
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

As of October 1 2016, the XFS mailing list will move away from the long standing address of xfs@oss.sgi.com due to the prospective shutdown of the oss.sgi.com infrastructure.

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Archives ==

Archives of both the old and new lists can be found at

* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)

== Other / Older XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)
* [http://marc.info/?l=linux-xfs Marc] (old oss.sgi.com list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber] (old oss.sgi.com list)

XFS email list and archives

2016-09-15T14:31:08Z

Sandeen:

__FORCETOC__

== XFS mailing list ==
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

As of October 1 2016, the XFS mailing list will move away from the long standing address of xfs@oss.sgi.com due to the prospective shutdown of the oss.sgi.com infrastructure.

Archives of both the old and new lists can be found at

* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Other / Older XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)
* [http://marc.info/?l=linux-xfs Marc] (old oss.sgi.com list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber] (old oss.sgi.com list)

XFS email list and archives

2016-09-15T14:23:54Z

Sandeen: /* XFS email list */

== XFS email list ==
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

Current archives of the linux-xfs@vger.kernel.org list can be found at

* [http://www.spinics.net/lists/linux-xfs/ Spinics]

(As of October 1 2016, the XFS mailing list will move away from the long standing address of xfs@oss.sgi.com because of the prospective shutdown of the oss.sgi.com infrastructure. See below for links to the old archives.)

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Old XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)
* [http://marc.info/?l=linux-xfs Marc] (old oss.sgi.com list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber]

XFS email list and archives

2016-08-30T15:50:19Z

Sandeen: /* Old XFS archives */

== XFS email list ==
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

As of September 2016, the XFS will move from the long standing address of xfs@oss.sgi.com because of the propsective shutdown of the oss.sgi.com infrastructure. See below for links to the old archives.

Current crchives of the linux-xfs@vger.kernel.org list can be found at

* [http://www.spinics.net/lists/linux-xfs/ Spinics]

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Old XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)
* [http://marc.info/?l=linux-xfs Marc] (old oss.sgi.com list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber]

XFS email list and archives

2016-08-30T14:54:17Z

Sandeen: /* Old XFS archives */

== XFS email list ==
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

As of September 2016, the XFS will move from the long standing address of xfs@oss.sgi.com because of the propsective shutdown of the oss.sgi.com infrastructure. See below for links to the old archives.

Current crchives of the linux-xfs@vger.kernel.org list can be found at

* [http://www.spinics.net/lists/linux-xfs/ Spinics]

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Old XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgi.com list)
* [http://marc.info/?l=linux-xfs] Marc] (old oss.sgi.com list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber]

XFS email list and archives

2016-08-30T14:48:08Z

Sandeen: /* Old XFS archives */

== XFS email list ==
Patches, comments, requests and questions should go to [mailto:linux-xfs@vger.kernel.org linux-xfs@vger.kernel.org]

As of September 2016, the XFS will move from the long standing address of xfs@oss.sgi.com because of the propsective shutdown of the oss.sgi.com infrastructure. See below for links to the old archives.

Current crchives of the linux-xfs@vger.kernel.org list can be found at

* [http://www.spinics.net/lists/linux-xfs/ Spinics]

== Subscribing to the list ==

Details for subscribing to the list can be found at the [http://vger.kernel.org/vger-lists.html#linux-xfs vger list info page].

Subscribing is *only* possible by sending an email with the body:

<pre>subscribe linux-xfs</pre>

to [mailto:majordomo@vger.kernel.org majordomo@vger.kernel.org]

== Old XFS archives ==

The list archives on oss.sgi.com are available [http://oss.sgi.com/archives/xfs here] (MHonArc) and [http://oss.sgi.com/pipermail/xfs here] (mailman). Note that as we transition away from oss.sgi.com infrastructure, these archives will go stale or disappear.

Other archives include:
* [https://www.spinics.net/lists/linux-xfs/ Spinics] (new vger.kernel.org list)
* [https://www.spinics.net/lists/xfs/ Spinics] (old oss.sgicom list)
* [http://www.opensubscriber.com/messages/xfs@oss.sgi.com/topic.html OpenSubscriber]

Improving Metadata Performance By Reducing Journal Overhead

2016-04-12T16:07:24Z

Sandeen:

= Future Directions - Journalling =

From http://oss.sgi.com/archives/xfs/2008-09/msg00800.html

== Improving Metadata Performance By Reducing Journal Overhead ==

XFS currently uses asynchronous write-ahead logging to ensure that changes to
the filesystem structure are preserved on crash. It does this by logging
detailed records of the changes being made to each object on disk during a
transaction. Every byte that is modified needs to be recorded in the journal.

There are two issues with this approach. The first is that transactions can
modify a *lot* of metadata to complete a single operation. Worse is the fact
that the average size of a transactions grows as structures get larger and
deeper, so performance on larger, fuller filesystem drops off as log bandwidth
is consumed by fewer, larger transactions.

The second is that we re-log previous changes that are active in the journal
if the object is modified again. hence if an object is modified repeatedly, the
dirty parts of the object get rewritten over and over again. in the worst case,
frequently logged buffers will be entirely dirty and so even if we only change
a single byte in the buffer we'll log the entire buffer.

The problem can be approached along two different axes:

- reduce the amount we log in a given transaction
- change the way we re-log objects.

Both of these things give the same end result - we require less bandwidth to
the journal to log changes that are happening in the filesystem.

== Asynchronous Transaction Aggregation ==

Status: Done, known as delayed logging.

Experimental in 2.6.35, stable for production in 2.6.37, planned for default
in 2.6.39.

Design documentation can be found here:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs-delayed-logging-design.txt

== Atomic Multi-Transaction Operations ==

A feature asynchronous transaction aggregation makes possible is atomic
multi-transaction operations. On the first transaction we hold the queue in
memory, preventing it from being committed. We can then do further transactions
that will end up in the same commit record, and on the final transaction we
unlock the async transaction queue. This will allow all those transaction to be
applied atomically. This is far simpler than any other method I've been looking
at to do this.

After a bit of reflection, I think this feature may be necessary for correct
implementation of existing logging techniques. The way we currently implement
rolling transactions (with permanent log reservations and rolling
dup/commit/re-reserve sequences) would seem to require all the commits in a
rolling transaction to be including in a single commit record. If I understand
history and the original design correctly, these rolling transactions were
implemented so that large, complex transactions would not pin the tail of the
log as they progressed. IOWs, they implicitly use re-logging to keep the tail
of the log moving forward as they progress and continue to modify items in the
transaction.

Given we are using asynchronous transaction aggregation as a method of reducing
re-logging, it would make sense to prevent these sorts of transactions from
pinning the tail of the log at all. Further, because we are effectively
disturbing the concept of unique transactions, I don't think that allowing a
rolling transaction to span aggregated commits is valid as we are going to be
ignoring the transaction IDs that are used to identify individual transactions.

Hence I think it is a good idea to simply replace rolling transactions with
atomic multi-transaction operations. This may also allow us to split some of
the large compound transactions into smaller, more self contained transactions.
This would reduce reservation pressure on log space in the common case where
all the corner cases in the transactions are not taken. In terms of
implementation, I think we can initially augment the permanent transaction
reservation/release interface to acheive this. With a working implementation,
we can then look to changing to a more explicit interface and slowly work to
remove the 'permanent log transaction' concept entirely. This shold simplify
the log code somewhat....

Note: This asynchronous transaction aggregation is originally based on a
concept floated by Nathan Scott called 'Delayed Logging' after observing how
ext3 implemented journalling. This never passed more than a concept
description phase....

== Operation Based Logging ==

The second approach to reducing log traffic is to change exactly what we
log in the transactions. At the moment, what we log is the exact change to
the item that is being made. For things like inodes and dquots, this isn't
particularly expensive because it is already a very compact form. The issue
comes with changes that are logged in buffers.

The prime example of this is a btree modification that involves either removing
or inserting a record into a buffer. The records are kept in compact form, so an
insert or remove will also move other records around in the buffer. In the worst
case, a single insert or remove of a 16 byte record can dirty an entire block
(4k generally, but could be up to 64k). In this case, if we were to log the
btree operation (e.g. insert {record, index}) rather than the resultant change
on the buffer the overhead of a btree operation is fixed. Such logging also
allows us to avoid needing to log the changes due to splits and merges - we just
replay the operation and subsequent splits/merges get done as part of replay.

The result of this is that complex transactions no longer need as much log space
as all possible change they can cause - we only log the basic operations that
are occurring and their result. Hence transaction end up being much smaller,
vary less in size between empty and full filesystems, etc. An example set of
operations describing all the changes made by an extent allocation on an inode
would be:

- inode X intent to allocate extent {off, len}
- AGCNT btree update record in AG X {old rec} {new rec values}
- AGBNO btree delete record in AG X {block, len}
- inode X BMBT btree insert record {off, block, len}
- inode X delta

This comes down to a relatively small, bound amount of space which is close the
minimun and existing allocation transaction would consume. However, with this
method of logging the transaction size does not increase with the size of
structures or the amount of updates necessary to complete the operations.

A major difference to the existing transaction system is that re-logging
of items doesn't fit very neatly with operation based logging.

There are three main disadvantages to this approach:

- recovery becomes more complex - it will need to change substantially
to accomodate operation replay rather than just reading from disk
and applying deltas.
- we have to create a whole new set of item types and add the necessary
hooks into the code to log all the operations correctly.
- re-logging is probably not possible, and that introduces
differences to the way we'll need to track objects for flushing. It
may, in fact, require transaction IDs in all objects to allow us
to determine what the last transaction that modified the item
on disk was during recovery.

Changing the logging strategy as described is a much more fundamental change to
XFS than asynchronous transaction aggregation. It will be difficult to change
to such a model in an evolutionary manner; it is more of a 'flag day' style
change where then entire functionality needs to be added in one hit. Given that
we will also still have to support the old log format, it doesn't enable us to
remove any code, either.

Given that we are likely to see major benefits in the problem workloads as a
result of asynchronous transaction aggregation, it may not be necessary to
completely rework the transaction subsystem. Combining aggregation with an
ongoing process of targeted reduction of transaction size will provide benefits
out to at least the medium term. It is unclear whether this direction will be
sufficient in the long run until we can measure the benefit that aggregation
will provide.

== Reducing Transaction Overhead ==

Per iclog callback list locks: Done

AIL tail pushing in it's own thread: Done

Bulk AIL insert and delete operations: Done

Log grant lock split-up: Done

Lock free transaction reserve path: Done

Moving all of the log interfacing out of the direct transaction commit path may provide similar benefits to moving the AIL pushing into it's own thread. This will mean that there will typically only be a single thread formatting and writing to iclog buffers. This will remove much of the parallelism that puts excessive pressure on many of these locks.

== Reducing Recovery Time ==

With 2GB logs, recovery can take an awfully long time due to the need
to read each object synchronously as we process the journal. An obvious
way to avoid this is to add another pass to the processing to do asynchronous
readahead of all the objects in the log before doing the processing passes.
This will populate the cache as quickly as possible and hide any read latency
that could occur as we process commit records.

A logical extension to this is to sort the objects in ascending offset order
before issuing I/O on them. That will further optimise the readahead I/O
to reduce seeking and hence should speed up the read phase of recovery
further.

== ToDo ==
Further investigation of recovery for future optimisation.

Improving inode Caching

2016-04-12T16:06:37Z

Sandeen:

= Future Directions for XFS - Inode Subsystems =

From http://oss.sgi.com/archives/xfs/2008-09/msg00799.html

== Improving Inode Caching and Operation in XFS ==
--------------------------------------------

Thousand foot view:

We want to drive inode lookup in a manner that is as parallel, scalable and low
overhead as possible. This means efficient indexing, lowering memory
consumption, simplifying the caching heirachy, removing duplication and
reducing/removing lock traffic.

In addition, we want to provide a good foundation for simplifying inode I/O,
improving writeback clustering, preventing RMW of inode buffers under memory
pressure, reducing creation and deletion overhead and removing writeback of
unlogged changes completely.

There are a variety of features in disconnected trees and patch sets that need
to be combined to acheive this - the basic structure needed to implement this is
already in mainline and that is the radix tree inode indexing. Further
improvements are going to be based around this structure and using it
effectively to avoid needing other indexing mechanisms.

Discussion:

== Combining XFS and VFS inodes ==

Status: Done (October 2008)

== Compressed Inode Cache ==

The XFs inode cache uses a lot of memory. We can avoid this problem by making
use of the compressed inode cache - only the active inodes are held in a
non-compressed form, hence most inodes will end up being cached in compressed
form rather than in the XFS/linux inode form. The compressed form can reduce
the cached inode footprint to 200-300 bytes per inode instead of 1-1.1k that
they currently take on a 64bit system. Hence by moving to a compressed cache we
can greatly increase the number of inodes cached in a given amount of memory
which more that offsets any comparitive increase we will see from inodes in
reclaim. the compressed cache should really have a LRU and a shrinker as well
so that memory pressure will slowly trim it as memory demands occur. [Note:
this compressed cache is discussed further later on in the reclaim context.]

== Fixed Inode Cache Size ==

It is worth noting that for embedded systems and appliances it may be worth while allowing
the size of the caches to be fixed. Also, to prevent memory fragmentation
problems, we could simply allocate that memory to the compressed cache slab.
In effect, this would become a 'static slab' in that it has a bound maximum
size and never frees and memory. When the cache is full, we reclaim an
object out of it for reuse - this could be done by triggering the shrinker
to reclaim from the LRU. This would prevent the compressed inode cache from
consuming excessive amounts of memory in tightly constrained evironments.
Such an extension to the slab caches does not look difficult to implement,
and would allow such customisation with minimal deviation from mainline code.

== Bypassing the Linux Inode Cache ==

Lookups: Done (October 2008)

Tracking dirty inodes: Done

Writeback of dirty inodes: Done

Writeback of dirty pages: still executed the by VFS

Now that we can track dirty inodes ourselves, we can pretty much isolate
writeback of both data and inodes from the generic pdflush code. If we add a
hook high up in the pdflush path that simply passes us a writeback control
structure with the current writeback guidelines, we can do writeback within
those guidelines in the most optimal fashion for XFS.

== Avoiding the Generic pdflush Code ==

Writeback of inodes via AIL: Done

For pdflush driven writeback, we only want to write back data; all other inode
writeback should be driven from the AIL (our time ordered dirty metadata list)
or xfssyncd in a manner that is most optimal for XFS.

Furthermore, if we implement our own pdflush method, we can parallelise it in
several ways. We can ensure that each filesystem has it's own flush thread or
thread pool, we can have a thread pool shared by all filesystems (like pdflush
currently operates), we can have a flush thread per inode radix tree, and so
one. The method of paralleisation is open for interpretation, but enabling
multiple flush threads to operate on a single filesystem is one of the necessary
requirements to avoid data writeback (and hence delayed allocation) being
limited to the throughput of a single CPU per filesystem.

== Improving Inode Writeback ==

To optimise inode writeback, we really need to reduce the impact of inode
buffer read-modify-write cycles. XFS is capable of caching far more inodes in
memory than it has buffer space available for, so RMW cycles during inode
writeback under memory pressure are quite common. Firstly, we want to avoid
blocking pdflush at all costs. Secondly, we want to issue as much localised
readahead as possible in ascending offset order to allow both elevator merging
of readahead and as little seeking as possible. Finally, we want to issue all
the write cycles as close together as possible to allow the same elevator and
I/O optimisations to take place.

To do this, firstly we need the non-blocking inode flush semantics to issue
readahead on buffers that are not up-to-date rather than reading them
synchronously. Inode writeback already has the interface to handle inodes that
weren't flushed - we return EAGAIN from xfs_iflush() and the higher inode
writeback layers handle this appropriately. It would be easy to add another
flag to pass down to the buffer layer to say 'issue but don't wait for any
read'. If we use a radix tree traversal to issue readahead in such a manner,
we'll get ascending offset readahead being issued.

One problem with this is that we can issue too much readahead and thrash the
cache. A possible solution to this is to make the readahead a 'delayed read'
and on I/o completion add it to a queue that holds a reference on the buffer.
If a followup read occurs soon after, we remove it from the queue and drop that
reference. This prevents the buffer from being reclaimed in betwen the
readahead completing and the real read being issued. We should also issue this
delayed read on buffers that are in the cache so that they don't get reclaimed
to make room for the readahead.

To prevent buildup of delayed read buffers, we can periodically purge them -
those that are older than a given age (say 5 seconds) can be removed from the
list and their reference dropped. This will free the buffer and allow it's
pages to be reclaimed.

Once we have done the readahead pass, we can then do a modify and writeback
pass over all the inodes, knowing that there will be no read cycles to delay
this step. Once again, a radix tree traversal gives us ascending order
writeback and hence the modified buffers we send to the device will be in
optimal order for merging and minimal seek overhead.

== Contiguous Inode Allocation ==

To make optimal use of the radix tree cache and enable wide-scale clustering of
inode writeback across multiple clusters, we really need to ensure that inode
allocation occurs in large contiguous chunks on disk. Right now we only
allocate chunks of 64 inodes at a time; ideally we want to allocate a stripe
unit (or multiple of) full of inodes at a time. This would allow inode
writeback clustering to do full stripe writes to the underlying RAID if there
are dirty inodes spanning the entire stripe unit.

The problem with doing this is that we don't want to introduce the latency of
creating megabytes of inodes when only one is needed for the current operation.
Hence we need to push the inode creation into a background thread and use that
to create contiguous inode chunks asynchronously. This moves the actual on-disk
allocation of inodes out of the normal create path; it should always be able to
find a free inode without doing on disk allocation. This will simplify the
create path by removing the allocate-on-disk-then-retry-the-create double
transaction that currently occurs.

As an aside, we could preallocate a small amount of inodes in each AG (10-20MB
of inodes per AG?) without impacting mkfs time too greatly. This would allow
the filesystem to be used immediately on the first mount without triggering
lots of background allocation. This could alsobe done after the first mount
occurs, but that could interfere with typical benchmarking situations. Another
good reason for this preallocation is that it will help reduce xfs_repair
runtime for most common filesystem usages.

One of the issues that the background create will cause is a substantial amount
of log traffic - every inode buffer initialised will be logged in whole. Hence
if we create a megabyte of inodes, we'll be causing a megabyte of log traffic
just for the inode buffers we've initialised. This is relatively simple to fix
- we don't log the buffer, we just log the fact that we need to initialise
inodes in a given range. In recovery, when we see this transaction, then we
build the buffers, initialise them and write them out. Hence, we don't need to
log the buffers used to initialise the inodes.

Also, we can use the background allocations to keep track of recently allocated
inode regions in the per-ag. Using that information to select the next inode to
be used rather than requiring btree searches on every create will greatly reduce
the CPU overhead of workloads that create lots of new inodes. It is not clear
whether a single background thread will be able to allocate enough inodes
to keep up with demand from the rest of the system - we may need multiple
threads for large configurations.

== Single Block Inode Allocation ==

One of the big problems we have withe filesystems that are approaching
full is that it can be hard to find a large enough extent to hold 64 inodes.
We've had ENOSPC errors on inode allocation reported on filesystems that
are only 85% full. This is a sign of free space fragmentation, and it
prevents inode allocation from succeeding. We could (and should) write
a free space defragmenter, but that does not solve the problem - it's
reactive, not preventative.

The main problem we have is that XFS uses inode chunk size and alignment
to optimise inode number to disk location conversion. That is, the conversion
becomes a single set of shifts and masks instead of an AGI btree lookup.
This optimisation substantially reduces the CPU and I/O overhead of
inode lookups, but it does limit our flexibility. If we break the
alignment restriction, every lookup has to go back to a btree search.
Hence we really want to avoid breaking chunk alignment and size
rules.

An approach to avoiding violation of this rule is to be able to determine which
index to look up when parsing the inode number. For example, we could use the
high bit of the inode number to indicate that it is located in a non-aligned
inode chunk and hence needs to be looked up in the btree. This would avoid
the lookup penalty for correctly aligned inode chunks.

If we then redefine the meaning of the contents of the AGI btree record for
such inode chunks, we do not need a new index to keep these in. Effectively,
we need to add a bitmask to the record to indicate which blocks inside
the chunk can actually contain inodes. We still use aligned/sized records,
but mask out the sections that we are not allowed to allocate inodes in.
Effectively, this would allow sparse inode chunks. There may be limitations
on the resolution of sparseness depending on inode size and block size,
but for the common cases of 4k block size and 256 or 512 byte inodes I
think we can run a fully sparse mapping for each inode chunk.

This would allow us to allocate inode extents of any alignment and size
that fits *inside* the existing alignment/size limitations. That is,
a single extent allocation could not span two btree records, but can
lie anywhere inside a single record. It also means that we can do
multiple extent allocations within one btree record to make optimal
use of the fragmented free space.

It should be noted that this will probably have impact on some of the
inode cluster buffer mapping and clustering algorithms. It is not clear
exactly what impact yet, but certainly write clustering will be affected.
Fortunately we'll be able to detect the inodes that will have this problem
by the high bit in the inode number.

== Inode Unlink ==

If we turn to look at unlink and reclaim interactions, there are a few
optimisations that can be made. Firstly, we don't need to do inode inactivation
in reclaim threads - these transactions can easily be pushed to a background
thread. This means that xfs_inactive would be little more than a vmtruncate()
call and queuing to a workqueue. This will substantially speed up the processing
of prune_icache() - we'll get inodes moved into reclaim much faster than we do
right now.

This will have a noticable effect, though. When inodes are unlinked the space
consumed by those inodes may not be immediately freed - it will be returned as
the inodes are processed through the reclaim threads. This means that userspace
monitoring tools such as 'df' may not immediately reflect the result of a
completed unlink operation. This will be a user visible change in behaviour,
though in most cases should not affect anyone and for those that it does affect
a 'sync' should be sufficient to wait for the space to be returned.

Now that inodes to be unlinked are out of general circulation, we can make the
unlinked path more complex. It is desirable to move the unlinked list from the
inode buffer to the inode core, but that has locking implications for incore
unlinked. Hence we really need background thread processing to enable this to
work (i.e. being able to requeue inodes for later processing). To ensure that
to overhead of this work is not a limiting factor, we will probably need
multiple workqueue processing threads for this.

Moving the logging to the inode core enables two things - it allows us to keep
an in-memory copy of the unlinked list off the perag and that allows us to remove
xfs_inotobp(). The in-memory unlinked list means we don't have to read and
traverse the buffers every time we need to find the previous buffer to remove an
inode from the list, but it does mean we have to take the inode lock. If the
previous inode is locked, then we can't remove the inode from the unlinked list
so we must requeue it for this to occur at a later time.

Combined with the changes to inode create, we effectively will only use the
inode buffer in the transaction subsystem for marking the region stale when
freeing an inode chunk from disk (i.e. the default noikeep configuration). If
we are using large inode allocation, we don't want to be freeing random inode
chunks - this will just leave us with fragmented inode regions and undo all the
good work that was done originally.

To avoid this, we should not be freeing inode chunks as soon as they no longer
have any empty inodes in them. We should periodically scan the AGI btree
looking for contiguous chunks that have no inodes allocated in them, and then
freeing the large contiguous regions we find in one go. It is likely this can
be done in a single transaction; it's one extent to be freed, along with a
contiguous set of records to be removed from the AGI btree so should not
require logging much at all. Also, the background scanning could be triggered
by a number of different events - low space in an AG, a large number of free
inodes in an AG, etc - as it doesn't need to be done frequently. As a result
of the lack of frequency that this needs to be done, it can probably be
handled by a single thread or delayed workqueue.

Further optimisations are possible here - if we rule that the AGI btree is the
sole place that inodes are marked free or in-use (with the exception of
unlinked inodes attached to the AGI lists), then we can avoid the need to
write back unlinked inodes or read newly created inodes from disk. This would
require all inodes to effectively use a random generation number assigned at
create time as we would not be reading it from disk - writing/reading the current
generation number appears to be the only real reason for doing this I/O. This
would require extra checks to determine if an inode is unlinked - we
need to do an imap lookup rather than reading it and then checking it is
valid if it is not already in memory. Avoiding the I/O, however, will greatly speed
up create and remove workloads. Note: the impact of this on the bulkstat algorithm
has not been determined yet.

One of the issues we need to consider with this background inactivation is that
we will be able to defer a large quantity of inactivation transactions so we are
going to need to be careful about how much we allow to be queued. Simple queue
depth throttling should be all that is needed to keep this under control.

== Reclaim Optimizations ==

Tracking inodes for reclaim in radix tree: Done

Using RCU for radix tree reclaim walks: Done

Non-blocking background reclaim: Done

Parallelised shrinker based reclaim: Done

Now that we have efficient unlink, we've got to handle the reclaim of all the
inodes that are now dead or simply not referenced. For inodes that are dirty,
we need to write them out to clean them. For inodes that are clean and not
unlinked, we need to compress them down for more compact storage. This involves
some CPU overhead, but it is worth noting that reclaiming of clean inodes
typically only occurs when we are under memory pressure.

By compressing the XFS inode in this case, we are effectively reducing the
memory usage of the inode rather than freeing it directly. If we then get
another operation on that inode (e.g. the working set is slightly larger than
can be held in linux+XFS inode pairs, we avoid having to read the inode off
disk again - it simply gets uncompressed out of the cache. In essence we use
the compressed inode cache as an exclusive second level cache - it has higher
density than the primary cache and higher load latency and CPU overhead,
but it still avoids I/O in exactly the same manner as the primary cache.

We cannot allow unrestricted build-up of reclaimable inodes - the memory they
consume will be large, so we should be aiming to compress reclaimable inodes as
soon as they are clean. This will prevent buildup of memory consuming
uncompressed inodes that are not likely to be referenced again immediately.

This clean inode reclaimation process can be accelerated by triggering reclaim
on inode I/O completion. If the inode is clean and reclaimable we should
trigger immediate reclaim processing of that inode. This will mean that
reclaim of newly cleaned inodes will not get held up behind reclaim of dirty
inodes.

For inodes that are unlinked, we can simply free them in reclaim as theƦ
are no longer in use. We don't want to poison the compressed cache with
unlinked inodes, nor do we need to because we can allocate new inodes
without incurring I/O.

Still, we may end up with lots of inodes queued for reclaim. We may need
to implement a throttle mechanism to slow down the rate at which inodes
are queued for reclaimation in the situation where the reclaim process
is not able to keep up. It should be noted that if we parallelise inode
writeback we should also be able to parallelise inode reclaim via
the same mechanism, so the need for throttling may relatively low
if we can have multiple inodes under reclaim at once.

It should be noted that complexity is exposed by interactions with concurrent
lookups, especially if we move to RCU locking on the radix tree. Firstly, we
need to be able to do an atomic swap of the compressed inode for the
uncompressed inode in the radix tree (and vice versa), to be able to tell them
apart (magic #), and to have atomic reference counts to ensure we can avoid use
after free situations when lookups race with compression or freeing.

Secondly, with the complex unlink/reclaim interactions we will need to be
careful to detect inodes in the process of reclaim - the lookupp process
will need to do different things depending on the state of reclaim. Indeed,
we will need to be able to cancel reclaim of an unlinked inode if we try
to allocate it before it has been fully unlinked or reclaimed. The same
can be said for an inode in the process of being compressed - if we get
a lookup during the compression process, we want to return the existing
inode, not have to wait, re-allocate and uncompress it again. These
are all solvable issues - they just add complexity.

== Accelerated Reclaim of buftarg Page Cache for Inodes ==
----------------------------------------------------

Per-buftarg buffer LRU reclaim: Done

Per-buftarg shrinker: Done

Per-buffer type reclaim prioritisation: Done

For single use inodes or even read-only inodes, we read them in, use them, then
reclaim them. With the compressed cache, they'll get compressed and live a lot
longer in memory. However, we also will have the inode cluster buffer pages
sitting in memory for some length of time after the inode was read in. This can
consume a large amount of memory that will never be used again, and does not
get reclaimed until they are purged from the LRU by the VM. It would be
advantageous to accelerate the reclaim of these pages so that they do not build
up unneccessarily.

A better method would appear to be to leverage the delayed read queue
mechanism. This delayed read queue pins read buffers for a short period of
time, and then if they have not been referenced they get torn down. If, as
part of this delayed read buffer teardown procedure we all free the backing
pages completely, we acheive the exact same result as having our own LRUs to
manage the page cache. This seems much simpler and a much more holistic
approach to solving the problem than implementing page LRUs.

== Killing Bufferheads (a.k.a "Die, buggerheads, Die!") ==

[This is not strictly about inode caching, but doesn't fit into
other areas of development as closely as it does to inode caching
optimisations.]

XFS is extent based. The Linux page cache is block based. Hence for
every cached page in memory, we have to attach a structure for mapping
the blocks on that page back to to the on-disk location. In XFs, we also
use this to hold state for delayed allocation and unwritten extent blocks
so the generic code can do the right thing when necessary. We also
use it to avoid extent lookups at various times within the XFS I/O
path.

However, this has a massive cost. While XFS might represent the
disk mapping of a 1GB extent in 24 bytes of memory, the page cache
requires 262,144 bufferheads (assuming 4k block size) to represent the
same mapping. That's roughly 14MB of memory neededtoo represent that.

Chris Mason wrote an extent map representation for page cache state
and mappings for BTRFS; that code is mostly generic and could be
adapted to XFS. This would allow us to hold all the page cache state
in extent format and greatly reduce the memory overhead that it currently
has. The tradeoff is increased CPU overhead due to tree lookups where
structure lookups currently are used. Still, this has much lower
overhead than xfs_bmapi() based lookups, so the penalty is going to
be lower than if we did these lookups right now.

If we make this change, we would then have three levels of extent
caching:

- the BMBT buffers
- the XFS incore inode extent tree (iext*)
- the page cache extent map tree

Effectively, the XFS incore inode extent tree becomes redundant - all
the extent state it holds can be moved to the generic page cache tree
and we can do all our incore operations there. Our logging of changes
is based on the BMBT buffers, so getting rid of the iext layer would
not impact the transaction subsystem at all.

Such integration with the generic code will also allow development
of generic writeback routines for delayed allocation, unwritten
extents, etc that are not specific to a given filesystem.

== Demand Paging of Large Inode Extent Maps ==

Currently the inode extent map is pinned in memory until the inode is
reclaimed. Hence an inode with millions of extents will pin a large
amount of memory and this can cause serious issues in low memory
situations. Ideally we would like to be able to page the extent
map in and out once they get to a certain size to avoid this
problem. This feature requires more investigation before an overall
approach can be detailed here.

It should be noted that if we move to an extent-based page cache mapping
tree, the associated extent state tree can be used to track sparse
regions. That is, regions of the extent map that are not in memory
can be easily represented and acceesses to an unread region can then
be used to trigger demand loading.

== Food For Thought (Crazy Ideas) ==

If we are not using inode buffers for logging changes to inodes, we should
consider whether we need them at all. What benefit do the buffers bring us when
all we will use them for is read or write I/O? Would it be better to go
straight to the buftarg page cache and do page based I/O via submit_bio()?

Reliable Detection and Repair of Metadata Corruption

2016-04-12T16:05:00Z

Sandeen:

= Future Directions - Reliability =

From http://oss.sgi.com/archives/xfs/2008-09/msg00802.html

== Reliable Detection and Repair of Metadata Corruption ==

This can be broken down into specific phases. Firstly, we cannot repair a
corruption we have not detected. Hence the first thing we need to do is
reliable detection of errors and corruption. Once we can reliably detect errors
in structures and verified that we are propagating all the errors reported from
lower layers into XFS correctly, we can look at ways of handling them more
robustly. In many cases, the same type of error needs to be handled differently
due to the context in which the error occurs. This introduces extra complexity
into this problem.

Rather than continually refering to specific types of problems (such as
corruption or error handling) I'll refer to them as 'exceptions'. This avoids
thinking about specific error conditions through specific paths and so helps us
to look at the issues from a more general or abstract point of view.

== Exception Detection ==

Our current approach to exception detection is entirely reactive and rather
slapdash - we read a metadata block from disk and check certain aspects of it
(e.g. the magic number) to determine if it is the block we wanted. We have no
way of verifying that it is the correct block of metadata of the type
we were trying to read; just that it is one of that specific type. We
do bounds checking on critical fields, but this can't detect bit errors
in those fields. There's many fields we don't even bother to check because
the range of valid values are not limited.

Effectively, this can be broken down into three separate areas:

- ensuring what we've read is exactly what we wrote
- ensuring what we've read is the block we were supposed to read
- robust contents checking

Firstly, if we introduce a mechanism that we can use to ensure what we read is
something that the filesystem wrote, we can detect a whole range of exceptions
that are caused in layers below the filesystem (software and hardware). The
best method for this is to use a guard value that travels with the metadata it
is guarding. The guard value needs to be derived from the contents of the
block being guarded. Any event that changes the guard or the contents it is
guarding will immediately trigger an exception handling process when the
metadata is read in. Some examples of what this will detect are:

- bit errors in media/busses/memory after guard is calculated
- uninitialised blocks being returned from lower layers (dmcrypt
had a readahead cancelling bug that could do this)
- zeroed sectors as a result of double sector failures
in RAID5 systems
- overwrite by data blocks
- partial overwrites (e.g. due to power failure)

The simplest method for doing this is introducing a checksum or CRC into each
block. We can calculate this for each different type of metadata being written
just before they are written to disk, hence we are able to provide a guard that
travels all the way to and from disk with the metadata itself. Given that
metadata blocks can be a maximum of 64k in size, we don't need a hugely complex
CRC or number of bits to protect blocks of this size. A 32 bit CRC will allow
us to reliably detect 15 bit errors on a 64k block, so this would catch almost
all types of bit error exceptions that occur. It will also detect almost all
other types of major content change that might occur due to an exception.
It has been noted that we should select the guard algorithm to be one that
has (or is targetted for) widespread hardware acceleration support.

The other advantage this provides us with is a very fast method of determining
if a corrupted btree is a result of a lower layer problem or indeed an XFS
problem. That is, instead of always getting a WANT_CORRUPTED_GOTO btree
exception and shutdown, we'll get a'bad CRC' exception before we even start
processing the contents. This will save us much time when triaging corrupt
btrees - we won't spend time chasing problems that result from (potentially
silent or unhandled) lower layer exceptions.

While a metadata block guard will protect us against content change, it won't
protect us against blocks that are written to the wrong location on disk. This,
unfortunately, happens more often that anyone would like and can be very
difficult to track down when it does occur. To protect against this problem,
metadata needs to be self-describing on disk. That is, if we read a block
on disk, there needs to be enough information in that block to determine
that it is the correct block for that location.

Currently we have a very simplistic method of determining that we really have
read the correct block - the magic numbers in each metadata structure. This
only enables us to identify type - we still need location and filesystem to
really determine if the block we've read is the correct one. We need the
filesystem identifier because misdirected writes can cross filesystem
boundaries. This is easily done by including the UUID of the filesystem in
every individually referencable metadata structure on disk.

For block based metadata structures such as btrees, AG headers, etc, we
can add the block number directly to the header structures hence enabling
easy checking. e.g. for btree blocks, we already have sibling pointers in the
header, so adding a long 'self' pointer makes a great deal of sense.
For inodes, adding the inode number into the inode core will provide exactly
the same protection - we'll now know that the inode we are reading is the
one we are supposed to have read. We can make similar modifications to dquots
to make them self identifying as well.

So now we are able to verify the metadata we read from disk is what we wrote
and it's the correct metadata block, the only thing that remains is more
robust checking of the content. In many cases we already do this in DEBUG
code but not in runtime code. For example, when we read an inode cluster
in we only check the first inode for a matching magic number, whereas in
debug code we check every inode in the cluster.

In some cases, there is not much point in doing this sort of detailed checking;
it's pretty hard to check the validity of the contents of a btree block without
doing a full walk of the tree and that is prohibitive overhead for production
systems. The added block guards and self identifiers should be sufficient to
catch all non-filesystem based exceptions in this case, whilst the existing
exception detection should catch all others. With the btree factoring that
is being done on for this work, all of the btrees should end up protected by
WANT_CORRUPTED_GOTO runtime exception checking.

We also need to verify that metadata is sane before we use it. For example, if
we pull a block number out of a btree record in a block that has passed all
other validity it still may be invalid due to corruption prior to writing
it to disk. In these cases we need to ensure the block number lands
within the filesystem and/or within the bounds of the specific AG.

Similar checking is needed for pretty much any forward or backwards reference
we are going to follow or using in an algorithm somewhere. This will help
prevent kernel panics by out of bound references (e.g. using an unchecked ag
number to index the per-AG array) by turning them into a handled exception
(which will initially be a shutdown). That is, we will turn a total system
failure into a (potentially recoverable) filesystem failure.

Another failures that we often have reported is that XFS has 'hung' and
traige indicates that the filesystem appears to be waiting for a metadata
I/O completion to occur. We have seen in the past I/O errors not being
propagated from the lower layers back into the filesystem causing these
sort of problems. We have also seen cases where there have been silent
I/O errors and the first thing to go wrong is 'XFS has hung'.

To catch situations like this, we need to track all I/O we have in flight and
have some method of timing them out. That is, if we haven't completed the I/O
in N seconds, issue a warning and enter an exception handling process that
attempts to deal with the problem.

My initial thoughts on this is that it could be implemented via the MRU cache
without much extra code being needed. The complexity with this is that we
can't catch data read I/O because we use the generic I/O path for read. We do
our own data write and metadata read/write, so we can easily add hooks to track
all these types of I/O. Hence we will initially target just metadata I/O as
this would only need to hook into the xfs_buf I/O submission layer.

To further improve exception detection, once guards and self-describing
structures are on disk, we can add filesystem scrubbing daemons that can verify
the structure of the filesystem pro-actively. That is, we can use background
processes to discovery degradation in the filesystem before it is found by a
user intiated operation. This gives us the ability to do exception handling in
a context that enables further checking and potential repair of the exception.
This sort of exception handling may not be possible if we are in a
user-initiated I/O context, and certainly not if we are in a transaction
context.

This will also allow us to detect errors in rarely referenced parts of
the filesystem, thereby giving us advance warning of degradation in filesystems
that we might not otherwise get (e.g. in systems without media scrubbing).
Ideally, data scrubbing woul dneed to be done as well, but without data guards
it is rather hard to detect that there's been a change in the data....

== Exception Handling ==

Once we can detect exceptions, we need to handle them in a sane manner.
The method of exception handling is two-fold:

- retry (write) or cancel (read) asynchronous I/O
- shut down the filesystem (fatal).

Effectively, we either defer non-critical failures to a later point in
time or we come to a complete halt and prevent the filesystem from being
accessed further. We have no other methods of handling exceptions.

If we look at the different types of exceptions we can have, they
broadly fall into:

- media read errors
- media write errors
- successful media read, corrupted contents

The context in which the errors occur also influences the exception processing
that is required. For example, an unrecoverable metadata read error within a
dirty transaction is a fatal error, whilst the same error during a read-only
operation will simply log the error to syslog and return an error to userspace.

Furthermore, the storage subsystem plays a part indeciding how to handle
errors. The reason is that in many storage configurations I/O errors can be
transient. For example, in a SAN a broken fibre can cause a failover to a
redundant path, however the inflight I/O on the failed is usually timed out and
an error returned. We don't want to shut down the filesystem on such an error -
we want to wait for failover to a redundant path and then retry the I/O. If the
failover succeeds, then the I/O will succeed. Hence any robust method of
exception handling needs to consider that I/O exceptions may be transient.

In the abscence of redundant metadata, there is little we can do right now
on a permanent media read error. There are a number of approaches we
can take for handling the exception:

- try reading the block again. Normally we don't get an error
returned until the device has given up on trying to recover it.
If it's a transient failure, then we should eventually get a
good block back. If a retry fails, then:

- inform the lower layer that it needs to perform recovery on that
block before trying to read it again. For path failover situations,
this should block until a redundant path is brought online. If no
redundant path exists or recovery from parity/error coding blocks
fails, then we cannot recover the block and we have a fatal error
situation.

Ultimately, however, we reach a point where we have to give up - the metadata
no longer exists on disk and we have to enter a repair process to fix the
problem. That is, shut down the filesystem and get a human to intervene
and fix the problem.

At this point, the only way we can prevent a shutdown situation from occurring
is to have redundant metadata on disk. That is, whenever we get an error
reported, we can immediately retry by reading from an alternate metadata block.
If we can read from the alternate block, we can continue onwards without
the user even knowing there is a block in the filesystem. Of course, we'd
need to log the event for the administrator to take action on at some point
in the future.

Even better, we can mostly avoid this intervention if we have alternate
metadata blocks. That is, we can repair blocks that are returning read errors
during the exception processing. In the case of media errors, they can
generally be corrected simply by re-writing the block that was returning the
error. This will force drives to remap the bad blocks internally so the next
read from that location will return valid data. This, if my understanding is
correct, is the same process that ZFS and BTRFS use to recover from and correct
such errors.

NOTE: Adding redundant metadata can be done in several different ways. I'm not
going to address that here as it is a topic all to itself. The focus of this
document is to outline how the redundant metadata could be used to enhance
exception processing and prevent a large number of cases where we currently
shut down the filesystem.

TODO:
Transient write error
Permanent write error
Corrupted data on read
Corrupted data on write (detected during guard calculation)
I/O timeouts
Memory corruption

== Reverse Mapping ==

It is worth noting that even redundant metadata doesn't solve all our
problems. Realistically, all that redundant metadata gives us is the ability
to recover from top-down traversal exceptions. It does not help exception
handling of occurences such as double sector failures (i.e. loss of redundancy
and a metadata block). Double sector failures are the most common cause
of RAID5 data loss - loss of a disk followed by a sector read error during
rebuild on one of the remaining disks.

In this case, we've got a block on disk that is corrupt. We know what block it
is, but we have no idea who the owner of the block is. If it is a metadata
block, then we can recover it if we have redundant metadata. Even if this is
user data, we still want to be able to tell them what file got corrupted by the
failure event. However, without doing a top-down traverse of the filesystem we
cannot find the owner of the block that was corrupted.

This is where we need a reverse block map. Every time we do an allocation of
an extent we know who the owner of the block is. If we record this information
in a separate tree then we can do a simple lookup to find the owner of any
block and start an exception handling process to repair the damage. Ideally
we also need to include information about the type of block as well. For
example, and inode can own:

- data blocks
- data fork BMBT blocks
- attribute blocks
- attribute fork BMBT blocks

So keeping track of owner + type would help indicate what sort of exception
handling needs to take place. For example, a missing data fork BMBT block means there
will be unreferenced extents across the filesystem. These 'lost extents'
could be recovered by reverse map traversal to find all the BMBT and data
blocks owned by that inode and finding the ones that are not referenced.
If the reverse map held suffient extra metadata - such as the offset within the
file for the extent - the exception handling process could rebuild the BMBT
tree completely without needing ænd external help.

It would seem to me that the reverse map needs to be a long-pointer format
btree and held per-AG. it needs long pointers because the owner of an extent
can be anywhere in the filesystem, and it needs to be per-AG to avoid adverse
effect on allocation parallelism.

The format of the reverse map record will be dependent on the amount of
metdata we need to store. We need:

- owner (64 bit, primary record)
- {block, len} extent descriptor
- type
- per-type specific metadata (e.g. offset for data types).

Looking at worst case here, say we have 32 bytes per record, the worst case
space usage of the reverse map btree woul dbe roughly 62 records per 4k
block. With a 1TB allocation group, we have 228 4k blocks in the AG
that could require unique reverse mappings. That gives us roughly 222
4k blocks to for the reverse map, or 234 bytes - roughly 16GB per 1TB
of space.

It may be a good idea to allocate this space at mkfs time (tagged as unwritten
so it doesn't need zeroing) to avoid allocation overhead and potential free
space fragmentation as the reverse map index grows and shrinks. If we do
this we could even treat this as a array/skip list where a given block in the
AG has a fixed location in the map. This will require more study to determine
the advantages and disadvantages of such approaches.

== Recovering From Errors During Transactions ==

One of the big problems we face with exception recovery is what to do
when we take an exception inside a dirty transaction. At present, any
error is treated as a fatal error, the transaction is cancelled and
the filesystem is shut down. Even though we may have a context which
can return an error, we are unable to revert the changes we have
made during the transaction and so cannot back out.

Effectively, a cancelled dirty transaction looks exactly like in-memory
structure corruption. That is, what is in memory is different to that
on disk, in the log or in asynchronous transactions yet to be written
to the log. Hence we cannot simply return an error and continue.

To be able to do this, we need to be able to undo changes made in a given
transaction. The method XFS uses for journalling - write-ahead logging -
makes this diffcult to do. A transaction proceeds in the following
order:

- allocate transaction
- reserve space in the journal for transaction
- repeat until change is complete:
- lock item
- join item to transaction
- modify item
- record region of change to item
- transaction commit

Effectively, we modify structures in memory then record where we
changed them for the transaction commit to write to disk. Unfortunately,
this means we overwrite the original state of the items in memory,
leaving us with no way to back out those changes from memory if
something goes wrong.

However, based on the observation that we are supposed to join an item to the
transaction *before* we start modifying it, it is possible to record the state
of the item before we start changing it. That is, we have a hoook that can
allow us take a copy of the unmodified item when we join it to the
transaction.

If we have an unmodified copy of the item in memory, then if the transaction
is cancelled when dirty, we have the information necessary to undo, or roll
back, the changes made in the transaction. This would allow us to return
the in-memory state to that prior to the transaction starting, thereby
ensuring that the in-memory state matches the rest of the filesystem and
allowing us to return an error to the calling context.

This is not without overhead. we would have to copy every metadata item
entirely in every transaction. This will increase the CPU overhead
of each transaction as well as the memory required. It is the memory
requirement more than the CPU overhead that concerns me - we may need
to ensure we have a memory pool associated with transaction reservation
that guarantees us enough memory is available to complete the transaction.
However, given that we could roll back transactions, we could now *fail
transactions* with ENOMEM and not have to shut down the filesystem, so this
may be an acceptible trade-off.

In terms of implementation, it is worth noting that there is debug code in
the xfs_buf_log_item for checking that all the modified regions of a buffer
were logged. Importantly, this is implemented by copying the original buffer
in the item initialisation when it is first attached to a transaction. In
other words, this debug code implements the mechanism we need to be able
to rollback changes made in a transaction. Other item types would require
similar changes to be made.

Overall, this doesn't look like a particularly complex change to make; the
only real question is how much overhead is it going to introduce. With CPUs
growing more cores all the time, and XFS being aimed at extremely
multi-threaded workloads, this overhead may not be a concern for long.

== Failure Domains ==

If we plan to have redundant metadata, or even try to provide fault isolation
between different parts of the filesystem namespace, we need to know about
independent regions of the filesystem. 'Independent Regions' (IR) are ranges
of the filesystem block address space that don't share resources with
any other range.

A classic example of a filesystem made up of multiple IRs is a linear
concatenation of multiple drives into a larger address space. The address
space associated with each drive can operate independently from the other
drives, and a failure of one drive will not affect the operation of the address
spaces associated with other drives in the linear concatenation.

A Failure Domain (FD) is made up of one or more IRs. IRs cannot be shared
between FDs - IRs are not independent if they are shared! Effectively, an
ID is an encoding of the address space within the filesystem that lower level
failures (from below the filesystem) will not propagate outside. The geometry
and redundancy in the underlying storage will determine the nature of the
IRs available to the filesystem.

To use redundant metadata effectively for recovering from fatal lower layer
loss or corruption, we really need to be able to place said redundant
metadata in a different FDs. That way a loss in one domain can be recovered
from a domain that is still intact. It also means that it is extremely
difficult to lose or corrupt all copies of a given piece of metadata;
that would require multiple independent faults to occur in a localised
temporaral window. Concurrent multiple component failure in multiple
IRs is considered to be quite unlikely - if such an event were to
occur, it is likely that there is more to worry about than filesystem
consistency (like putting out the fire in the data center).

Another use of FDs is to try to minimise the number of domain boundaries
each object in the filesystem crosses. If an object is wholly contained
within a FD, and that object is corrupted, then the repair problem is
isolated to that FD, not the entire filesystem. That is, by making
allocation strategies and placement decisions aware of failure domain
boundaries we can constraint the location of related data and metadata.
Once locality is constrained, the scope of repairing an object if
it becomes corrupted is reduced to that of ensuring the FD is consistent.

There are many ways of limiting cross-domain dependencies; I will
not try to detail them here. Likewise, there are many ways of introducing
such information into XFS - mkfs, dynamically via allocation policies,
etc - so I won't try to detail them, either. The main point to be
made is that to make full use of redundant metadata and to reduce
the scope of common reapir problems we need to pay attention to
how the system can fail to ensure that we can recover from failures
as quickly as possible.

XFS FAQ

2015-12-14T16:39:32Z

Sandeen: /* Q. Should barriers be enabled with storage which has a persistent write cache? */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAIDs have a persistent write cache which is preserved across power failure, interface resets, system crashes, etc. The same may be true of some SSD devices. This sort of hardware should report to the operating system that no flushes are required, and in that case barriers will not be issued, even without the "nobarrier" option. Quoting Christoph Hellwig [http://oss.sgi.com/archives/xfs/2015-12/msg00281.html on the xfs list],
If the device does not need cache flushes it should not report requiring
flushes, in which case nobarrier will be a noop. Or to phrase it
differently: If nobarrier makes a difference skipping it is not safe.
On modern kernels with hardware which properly reports write cache behavior, there is no need to change barrier options at mount time.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/archives/xfs/2009-01/msg01023.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. This was an issue with the older "inode32" inode allocation mode, where inode allocation is restricted to lower filesysetm blocks. To fix this, [http://oss.sgi.com/archives/xfs/2009-01/msg01031.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Alternately, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 No space left on device on xfs filesystem with 7.7TB free]

However, 'inode64' has been the default behavior since kernel v3.7...

Unfortunately, v3.7 also added a bug present from kernel v3.7 to v3.17 which caused new allocation groups added by growfs to be unavailable for inode allocation. This was fixed by commit <tt>[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9de67c3ba9ea961ba420573d56479d09d33a7587 9de67c3b xfs: allow inode allocations in post-growfs disk space.]</tt> in kernel v3.17.
Without that commit, the problem can be worked around by doing a "mount -o remount,inode64" after the growfs operation.

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

== Q: Why do files on XFS use more data blocks than expected? ==

The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimize file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistance in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.

This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.

See the FAQ entry on speculative preallocation for details.

== Q: What is speculative preallocation? ==

XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.

Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).

In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor.

== Q: How can I speed up or avoid delayed removal of speculative preallocation? ==

Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):

/proc/sys/fs/xfs/speculative_prealloc_lifetime

== Q: Is speculative preallocation permanent? ==

Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate) or following a crash. In the event of a crash, the in-memory state used to track and reclaim the speculative preallocation is lost.

== Q: My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size? ==

Speculative preallocation can not be disabled but XFS can be tuned to a fixed allocation size with the 'allocsize=' mount option. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. Use 'allocsize=64k' to revert to the default XFS behavior prior to support for dynamic speculative preallocation.

== Q: mount (or umount) takes minutes or even hours - what could be the reason ? ==

In some cases xfs log (journal) can become quite big. For example if it accumulates many entries and doesn't get chance to apply these to disk (due to lockup, crash, hard reset etc). xfs will try to reapply these at mount (in dmesg: "Starting recovery (logdev: internal)").

That process with big log to be reapplied can take very long time (minutes or even hours). Similar problem can happen with unmount taking hours when there are hundreds of thousands of dirty inode in memory that need to be flushed to disk.

(http://oss.sgi.com/pipermail/xfs/2015-October/044457.html)

XFS FAQ

2014-10-07T22:54:54Z

Sandeen: /* Q: Why do I receive No space left on device after xfs_growfs? */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. This was an issue with the older "inode32" inode allocation mode, where inode allocation is restricted to lower filesysetm blocks. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Alternately, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 No space left on device on xfs filesystem with 7.7TB free]

However, 'inode64' has been the default behavior since kernel v3.7...

Unfortunately, v3.7 also added a bug present from kernel v3.7 to v3.17 which caused new allocation groups added by growfs to be unavailable for inode allocation. This was fixed by commit <tt>[http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=9de67c3ba9ea961ba420573d56479d09d33a7587 9de67c3b xfs: allow inode allocations in post-growfs disk space.]</tt> in kernel v3.17.
Without that commit, the problem can be worked around by doing a "mount -o remount,inode64" after the growfs operation.

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

== Q: Why do files on XFS use more data blocks than expected? ==

The XFS speculative preallocation algorithm allocates extra blocks beyond end of file (EOF) to minimize file fragmentation during buffered write workloads. Workloads that benefit from this behaviour include slowly growing files, concurrent writers and mixed reader/writer workloads. It also provides fragmentation resistance in situations where memory pressure prevents adequate buffering of dirty data to allow formation of large contiguous regions of data in memory.

This post-EOF block allocation is accounted identically to blocks within EOF. It is visible in 'st_blocks' counts via stat() system calls, accounted as globally allocated space and against quotas that apply to the associated file. The space is reported by various userspace utilities (stat, du, df, ls) and thus provides a common source of confusion for administrators. Post-EOF blocks are temporary in most situations and are usually reclaimed via several possible mechanisms in XFS.

See the FAQ entry on speculative preallocation for details.

== Q: What is speculative preallocation? ==

XFS speculatively preallocates post-EOF blocks on file extending writes in anticipation of future extending writes. The size of a preallocation is dynamic and depends on the runtime state of the file and fs. Generally speaking, preallocation is disabled for very small files and preallocation sizes grow as files grow larger.

Preallocations are capped to the maximum extent size supported by the filesystem. Preallocation size is throttled automatically as the filesystem approaches low free space conditions or other allocation limits on a file (such as a quota).

In most cases, speculative preallocation is automatically reclaimed when a file is closed. Applications that repeatedly trigger preallocation and reclaim cycles (e.g., this is common in file server or log file workloads) can cause fragmentation. Therefore, this pattern is detected and causes the preallocation to persist beyond the lifecycle of the file descriptor.

== Q: How can I speed up or avoid delayed removal of speculative preallocation? ==

Linux 3.8 (and later) includes a scanner to perform background trimming of files with lingering post-EOF preallocations. The scanner bypasses dirty files to avoid interference with ongoing writes. A 5 minute scan interval is used by default and can be adjusted via the following file (value in seconds):

/proc/sys/fs/xfs/speculative_prealloc_lifetime

== Q: Is speculative preallocation permanent? ==

Preallocated blocks are normally reclaimed on file close, inode reclaim, unmount or in the background once file write activity subsides. They can be explicitly made permanent via fallocate or a similar interface. They can be implicitly made permanent in situations where file size is extended beyond a range of post-EOF blocks (i.e., via an extending truncate) or following a crash. In the event of a crash, the in-memory state used to track and reclaim the speculative preallocation is lost.

== Q: My workload has known characteristics - can I disable speculative preallocation or tune it to an optimal fixed size? ==

Speculative preallocation can not be disabled but XFS can be tuned to a fixed allocation size with the 'allocsize=' mount option. Speculative preallocation is not dynamically resized when the allocsize mount option is set and thus the potential for fragmentation is increased. Use 'allocsize=64k' to revert to the default XFS behavior prior to support for dynamic speculative preallocation.

XFS FAQ

2014-01-21T01:44:56Z

Sandeen: /* Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

XFS FAQ

2014-01-21T01:44:21Z

Sandeen: /* Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.9 through v3.12 */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.10 through v3.12 ==

This may happen when running xfs_growfs under a v3.10-v3.12 kernel,
if the filesystem was previously grown under a kernel prior to v3.8.

Old kernel versions prior to v3.8 did not zero the empty part of
new secondary superblocks when growing the filesystem with xfs_growfs.

Kernels v3.10 and later began detecting this non-zero part of the
superblock as corruption, and emit the

Internal error xfs_sb_read_verify

error message.

Kernels v3.13 and later are more forgiving about this - if the non-zero
data is found on a Version 4 superblock, it will not be flagged as
corruption.

The problematic secondary superblocks may be repaired by using an xfs_repair
version 3.2.0-alpha1 or above.

The relevant kernelspace commits are as follows:

v3.8 1375cb6 xfs: growfs: don't read garbage for new secondary superblocks <- fixed underlying problem
v3.10 04a1e6c xfs: add CRC checks to the superblock <- detected old underlying problem
v3.13 59e5a0e xfs: don't break from growfs ag update loop on error <- less consequential
v3.13 10e6e65 xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields <- is more forgiving of old underlying problem

This commit allows xfs_repair to detect and correct the problem:

v3.2.0-alpha1 cbd7508 xfs_repair: zero out unused parts of superblocks

XFS FAQ

2014-01-20T22:10:58Z

Sandeen: /* Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.9 through v3.12 */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.9 through v3.12 ==

Old kernel versions didn't zero the empty part of the secondary
superblocks when growing the filesystem. However, kernels v3.10 and
newer have started flagging this as corruption. (Kernels v3.13 and later
are more forgiving about this, if the garbage is found on a V4 (not V5)
superblock).

This commit in 3.8 fixed
the kernel growfs code not to put garbage in the new secondary
superblocks:

commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Oct 9 14:50:52 2012 +1100

xfs: growfs: don't read garbage for new secondary superblocks

When updating new secondary superblocks in a growfs operation, the
superblock buffer is read from the newly grown region of the
underlying device. This is not guaranteed to be zero, so violates
the underlying assumption that the unused parts of superblocks are
zero filled. Get a new buffer for these secondary superblocks to
ensure that the unused regions are zero filled correctly.

The only time the kernel reads secondary superblocks is during a
growfs operation, so that's the only time the kernel will detect
such an error. More extensive validity tests were added during 3.9
and 3.10, and these now throw corruption errors over secondary
superblocks that have not been correctly zeroed.

To fix this, you need to grab xfsprogs from the git repo
(3.2.0-alpha1 or newer will do) as this commit to xfs_repair detects and fixes
the corrupted superblocks:

commit cbd7508db4c9597889ad98d5f027542002e0e57c
Author: Eric Sandeen <sandeen@redhat.com>
Date: Thu Aug 15 02:26:40 2013 +0000

xfs_repair: zero out unused parts of superblocks

Prior to:
1375cb65 xfs: growfs: don't read garbage for new secondary superblocks

we ran the risk of allowing garbage in secondary superblocks
beyond the in-use sb fields. With kernels 3.10 and beyond, the
verifiers will kick these out as invalid, but xfs_repair does
not detect or repair this condition.

There is superblock stale-data zeroing code, but it is under a
narrow conditional - the bug addressed in the above commit did not
meet that conditional. So change this to check unconditionally.

Further, the checking code was looking at the in-memory
superblock buffer, which was zeroed prior to population, and
would therefore never possibly show any stale data beyond the
last up-rev superblock field.

So instead, check the disk buffer for this garbage condition.

If we detect garbage, we must zero out both the in-memory sb
and the disk buffer; the former may contain unused data
in up-rev sb fields which will be written back out; the latter
may contain garbage beyond all fields, which won't be updated
when we translate the in-memory sb back to disk.

The V4 superblock case was zeroing out the sb_bad_features2
field; we also fix that to leave that field alone.

In kernels v3.13 and newer, this commit:

commit 10e6e65dfcedff63275c3d649d329c044caa8e26
Author: Eric Sandeen <sandeen@sandeen.net>
Date: Mon Sep 9 15:33:29 2013 -0500

xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields

will cause the kernel to be more forgiving of this situation.

XFS FAQ

2014-01-20T21:51:32Z

Sandeen:

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

== Q: I'm getting "Internal error xfs_sb_read_verify" errors when I try to run xfs_growfs under kernels v3.9 through v3.12 ==

Old kernel versions didn't zero the empty part of the secondary
superblocks when growing the filesystem. However, kernels v3.10 and
newer have started flagging this as corruption. (Kernels v3.13 and later
are more forgiving about this, if the garbage is found on a V4 (not V5)
superblock).

This commit in 3.8 fixed
the kernel growfs code not to put garbage in the new secondary
superblocks:

commit 1375cb65e87b327a8dd4f920c3e3d837fb40e9c2
Author: Dave Chinner <dchinner@redhat.com>
Date: Tue Oct 9 14:50:52 2012 +1100

xfs: growfs: don't read garbage for new secondary superblocks

When updating new secondary superblocks in a growfs operation, the
superblock buffer is read from the newly grown region of the
underlying device. This is not guaranteed to be zero, so violates
the underlying assumption that the unused parts of superblocks are
zero filled. Get a new buffer for these secondary superblocks to
ensure that the unused regions are zero filled correctly.

The only time the kernel reads secondary superblocks is during a
growfs operation, so that's the only time the kernel will detect
such an error. More extensive validity tests were added during 3.9
and 3.10, and these now throw corruption errors over secondary
superblocks that have not been correctly zeroed.

To fix this, you need to grab xfsprogs from the git repo
(3.2.0-alpha1 or newer will do) as this commit to xfs_repair detects and fixes
the corrupted superblocks:

commit cbd7508db4c9597889ad98d5f027542002e0e57c
Author: Eric Sandeen <sandeen@redhat.com>
Date: Thu Aug 15 02:26:40 2013 +0000

xfs_repair: zero out unused parts of superblocks

Prior to:
1375cb65 xfs: growfs: don't read garbage for new secondary superblocks

we ran the risk of allowing garbage in secondary superblocks
beyond the in-use sb fields. With kernels 3.10 and beyond, the
verifiers will kick these out as invalid, but xfs_repair does
not detect or repair this condition.

There is superblock stale-data zeroing code, but it is under a
narrow conditional - the bug addressed in the above commit did not
meet that conditional. So change this to check unconditionally.

Further, the checking code was looking at the in-memory
superblock buffer, which was zeroed prior to population, and
would therefore never possibly show any stale data beyond the
last up-rev superblock field.

So instead, check the disk buffer for this garbage condition.

If we detect garbage, we must zero out both the in-memory sb
and the disk buffer; the former may contain unused data
in up-rev sb fields which will be written back out; the latter
may contain garbage beyond all fields, which won't be updated
when we translate the in-memory sb back to disk.

The V4 superblock case was zeroing out the sb_bad_features2
field; we also fix that to leave that field alone.

In kernels v3.13 and newer, this commit:

commit 10e6e65dfcedff63275c3d649d329c044caa8e26
Author: Eric Sandeen <sandeen@sandeen.net>
Date: Mon Sep 9 15:33:29 2013 -0500

xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields

will cause the kernel to be more forgiving of this situation.

XFS FAQ

2012-11-02T17:36:16Z

Sandeen: /* Q: Does the filesystem have an undelete capability? */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to keep good backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

XFS FAQ

2012-11-02T17:35:39Z

Sandeen: /* Q: Does the filesystem have an undelete capability? */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS.

However, if an inode is unlinked but neither it nor its associated data blocks get immediately re-used and overwritten, there is some small chance to recover the file from the disk.

''photorec'', ''xfs_irecover'' or ''xfsr'' are some tools which attempt to do this, with varying success.

There are also commercial data recovery services and closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS] which claims to recover data, although this has not been tested by the XFS developers.

As always, the best advice is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

XFS FAQ

2012-07-19T15:51:03Z

Sandeen: /* Q: I want to tune my XFS filesystems for */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS (so far).

However at least some XFS driver implementations do not wipe file information nodes completely so there are chance to recover files with specialized commercial closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS].

In this kind of XFS driver implementation it does not re-use directory entries immediately so there are chance to get back recently deleted files even with their real names.

''xfs_irecover'' or ''xfsr'' may help too, [http://www.who.is.free.fr/wiki/doku.php?id=recover this site] has a few links.

This applies to most recent Linux distributions (versions?), as well as to most popular NAS boxes that use embedded linux and XFS file system.

Anyway, the best is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

''Premature optimization is the root of all evil.'' - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

XFS FAQ

2012-07-19T15:50:34Z

Sandeen: /* Q: I want to tune my XFS filesystems for */

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS (so far).

However at least some XFS driver implementations do not wipe file information nodes completely so there are chance to recover files with specialized commercial closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS].

In this kind of XFS driver implementation it does not re-use directory entries immediately so there are chance to get back recently deleted files even with their real names.

''xfs_irecover'' or ''xfsr'' may help too, [http://www.who.is.free.fr/wiki/doku.php?id=recover this site] has a few links.

This applies to most recent Linux distributions (versions?), as well as to most popular NAS boxes that use embedded linux and XFS file system.

Anyway, the best is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

"Premature optimization is the root of all evil." - Donald Knuth

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

XFS FAQ

2012-06-21T00:44:00Z

Sandeen: DIE xfs_check DIE!

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

What you need to report depend on the problem you are seeing. Firstly, your machine hardware and storage configuration needs to be described. That includes:

* kernel version (uname -a)
* xfsprogs version (xfs_repair -V)
* number of CPUs
* contents of /proc/meminfo
* contents of /proc/mounts
* contents of /proc/partitions
* RAID layout (hardware and/or software)
* LVM configuration
* type of disks you are using
* write cache status of drives
* size of BBWC and mode it is running in
* xfs_info output on the filesystem in question
* dmesg output showing all error messages and stack traces

Then you need to describe your workload that is causing the problem, and a demonstration of the bad behaviour that is occurring. If it is a performance problem, then 30s - 1 minute samples of:

# iostat -x -d -m 5
# vmstat 5

can give us insight into the IO and memory utilisation of your machine at the time of the problem.

If the filesystem is hanging, then capture the output of the dmesg command after running:

# echo w > /proc/sysrq-trigger
# dmesg

will tell us all the hung processes in the machine, often pointing us directly to the cause of the hang.

And for advanced users, capturing an event trace using '''trace-cmd''' (git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git) will be very helpful. In many cases the XFS developers will ask for this information anyway, so it's a good idea to be ready with it in advance. Start the trace with this command, either from a directory not on an XFS filesystem or with an output file destination on a non-XFS filesystem:

# trace-cmd record -e xfs\*

before the problem occurs, and once it has occurred, kill the trace-cmd with ctrl-C, and then run:

# trace-cmd report > trace_report.txt

Compress the trace_report.txt file and include that with the bug report. The reason for trying to host the output of the record command on a different filesystem is so that the writing of the output file does not pollute the trace of the problem we are trying to diagnose.

If you have a problem with '''xfs_repair(8)''', make sure that you save the entire output of the problematic run so that the developers can see exactly where it encountered the problem. You might be asked to capture the metadata in the filesystem using '''xfs_metadump(8)''' (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS (so far).

However at least some XFS driver implementations do not wipe file information nodes completely so there are chance to recover files with specialized commercial closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS].

In this kind of XFS driver implementation it does not re-use directory entries immediately so there are chance to get back recently deleted files even with their real names.

''xfs_irecover'' or ''xfsr'' may help too, [http://www.who.is.free.fr/wiki/doku.php?id=recover this site] has a few links.

This applies to most recent Linux distributions (versions?), as well as to most popular NAS boxes that use embedded linux and XFS file system.

Anyway, the best is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE) (for SATA this requires at least kernel 2.6.15 because [http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b095518ef51c37658c58367bd19240b8a113f25c ATA command passthrough support]): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

'''xfs_repair -n''' should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

XFS FAQ

2012-04-12T04:06:37Z

Sandeen:

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

Things to include are what version of XFS you are using, if this is a CVS version of what date and version of the kernel. If you have problems with userland packages please report the version of the package you are using.

If the problem relates to a particular filesystem, the output from the '''xfs_info(8)''' command and any '''mount(8)''' options in use will also be useful to the developers.

If you experience an oops, please run it through '''ksymoops''' so that it can be interpreted.

If you have a filesystem that cannot be repaired, make sure you have xfsprogs 2.9.0 or later and run '''xfs_metadump(8)''' to capture the metadata (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS (so far).

However at least some XFS driver implementations do not wipe file information nodes completely so there are chance to recover files with specialized commercial closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS].

In this kind of XFS driver implementation it does not re-use directory entries immediately so there are chance to get back recently deleted files even with their real names.

''xfs_irecover'' or ''xfsr'' may help too, [http://www.who.is.free.fr/wiki/doku.php?id=recover this site] has a few links.

This applies to most recent Linux distributions (versions?), as well as to most popular NAS boxes that use embedded linux and XFS file system.

Anyway, the best is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_check and xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (although for SATA this only works on a recent kernel with ATA command passthrough):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE): (although for SATA this only works on a recent kernel with ATA command passthrough): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery or flash backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. This battery backed cache should ensure that if power fails or a PSU dies, the contents of the cache will be written to disk on next boot. However, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents.

If you do not have a battery or flash backed cache you should seriously consider disabling write cache if you value your data.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, this will disable the controller and disk cache (see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf, page 86);

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

The '''xfs_check''' tool, or '''xfs_repair -n''', should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

As of kernel 3.2.12, the default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

Note that above a few average extents per file, the fragmentation factor rapidly approaches 100%:
[[Image:Frag_factor.png|500px]]

File:Frag factor.png

2012-04-12T04:02:26Z

Sandeen: Graph of fragmentation factor vs. avg extents per file

Graph of fragmentation factor vs. avg extents per file

XFS FAQ

2012-02-16T23:14:44Z

Sandeen:

Info from: [http://oss.sgi.com/projects/xfs/faq.html main XFS faq at SGI]

Many thanks to earlier maintainers of this document - Thomas Graichen and Seth Mos.

== Q: Where can I find documentation about XFS? ==

The SGI XFS project page http://oss.sgi.com/projects/xfs/ is the definitive reference. It contains pointers to whitepapers, books, articles, etc.

You could also join the [[XFS_email_list_and_archives|XFS mailing list]] or the '''<nowiki>#xfs</nowiki>''' IRC channel on ''irc.freenode.net''.

== Q: Where can I find documentation about ACLs? ==

Andreas Gruenbacher maintains the Extended Attribute and POSIX ACL documentation for Linux at http://acl.bestbits.at/

The '''acl(5)''' manual page is also quite extensive.

== Q: Where can I find information about the internals of XFS? ==

An [http://oss.sgi.com/projects/xfs/training/ SGI XFS Training course] aimed at developers, triage and support staff, and serious users has been in development. Parts of the course are clearly still incomplete, but there is enough content to be useful to a broad range of users.

Barry Naujok has documented the [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS ondisk format] which is a very useful reference.

== Q: What partition type should I use for XFS on Linux? ==

Linux native filesystem (83).

== Q: What mount options does XFS have? ==

There are a number of mount options influencing XFS filesystems - refer to the '''mount(8)''' manual page or the documentation in the kernel source tree itself ([http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/filesystems/xfs.txt;hb=HEAD Documentation/filesystems/xfs.txt])

== Q: Is there any relation between the XFS utilities and the kernel version? ==

No, there is no relation. Newer utilities tend to mainly have fixes and checks the previous versions might not have. New features are also added in a backward compatible way - if they are enabled via mkfs, an incapable (old) kernel will recognize that it does not understand the new feature, and refuse to mount the filesystem.

== Q: Does it run on platforms other than i386? ==

XFS runs on all of the platforms that Linux supports. It is more tested on the more common platforms, especially the i386 family. Its also well tested on the IA64 platform since thats the platform SGI Linux products use.

== Q: Quota: Do quotas work on XFS? ==

Yes.

To use quotas with XFS, you need to enable XFS quota support when you configure your kernel. You also need to specify quota support when mounting. You can get the Linux quota utilities at their sourceforge website [http://sourceforge.net/projects/linuxquota/ http://sourceforge.net/projects/linuxquota/] or use '''xfs_quota(8)'''.

== Q: Quota: What's project quota? ==

The project quota is a quota mechanism in XFS can be used to implement a form of directory tree quota, where a specified directory and all of the files and subdirectories below it (i.e. a tree) can be restricted to using a subset of the available space in the filesystem.

== Q: Quota: Can group quota and project quota be used at the same time? ==

No, project quota cannot be used with group quota at the same time. On the other hand user quota and project quota can be used simultaneously.

== Q: Quota: Is umounting prjquota (project quota) enabled fs and mounting it again with grpquota (group quota) removing prjquota limits previously set from fs (and vice versa) ? ==

To be answered.

== Q: Are there any dump/restore tools for XFS? ==

'''xfsdump(8)''' and '''xfsrestore(8)''' are fully supported. The tape format is the same as on IRIX, so tapes are interchangeable between operating systems.

== Q: Does LILO work with XFS? ==

This depends on where you install LILO.

Yes, for MBR (Master Boot Record) installations.

No, for root partition installations because the XFS superblock is written at block zero, where LILO would be installed. This is to maintain compatibility with the IRIX on-disk format, and will not be changed.

== Q: Does GRUB work with XFS? ==

There is native XFS filesystem support for GRUB starting with version 0.91 and onward. Unfortunately, GRUB used to make incorrect assumptions about being able to read a block device image while a filesystem is mounted and actively being written to, which could cause intermittent problems when using XFS. This has reportedly since been fixed, and the 0.97 version (at least) of GRUB is apparently stable.

== Q: Can XFS be used for a root filesystem? ==

Yes, with one caveat: Linux does not support an external XFS journal for the root filesystem via the "rootflags=" kernel parameter. To use an external journal for the root filesystem in Linux, an init ramdisk must mount the root filesystem with explicit "logdev=" specified. [http://mindplusplus.wordpress.com/2008/07/27/scratching-an-i.html More information here.]

== Q: Will I be able to use my IRIX XFS filesystems on Linux? ==

Yes. The on-disk format of XFS is the same on IRIX and Linux. Obviously, you should back-up your data before trying to move it between systems. Filesystems must be "clean" when moved (i.e. unmounted). If you plan to use IRIX filesystems on Linux keep the following points in mind: the kernel needs to have SGI partition support enabled; there is no XLV support in Linux, so you are unable to read IRIX filesystems which use the XLV volume manager; also not all blocksizes available on IRIX are available on Linux (only blocksizes less than or equal to the pagesize of the architecture: 4k for i386, ppc, ... 8k for alpha, sparc, ... is possible for now). Make sure that the directory format is version 2 on the IRIX filesystems (this is the default since IRIX 6.5.5). Linux can only read v2 directories.

== Q: Is there a way to make a XFS filesystem larger or smaller? ==

You can ''NOT'' make a XFS partition smaller online. The only way to shrink is to do a complete dump, mkfs and restore.

An XFS filesystem may be enlarged by using '''xfs_growfs(8)'''.

If using partitions, you need to have free space after this partition to do so. Remove partition, recreate it larger with the ''exact same'' starting point. Run '''xfs_growfs''' to make the partition larger. Note - editing partition tables is a dangerous pastime, so back up your filesystem before doing so.

Using XFS filesystems on top of a volume manager makes this a lot easier.

== Q: What information should I include when reporting a problem? ==

Things to include are what version of XFS you are using, if this is a CVS version of what date and version of the kernel. If you have problems with userland packages please report the version of the package you are using.

If the problem relates to a particular filesystem, the output from the '''xfs_info(8)''' command and any '''mount(8)''' options in use will also be useful to the developers.

If you experience an oops, please run it through '''ksymoops''' so that it can be interpreted.

If you have a filesystem that cannot be repaired, make sure you have xfsprogs 2.9.0 or later and run '''xfs_metadump(8)''' to capture the metadata (which obfuscates filenames and attributes to protect your privacy) and make the dump available for someone to analyse.

== Q: Mounting an XFS filesystem does not work - what is wrong? ==

If mount prints an error message something like:

mount: /dev/hda5 has wrong major or minor number

you either do not have XFS compiled into the kernel (or you forgot to load the modules) or you did not use the "-t xfs" option on mount or the "xfs" option in <tt>/etc/fstab</tt>.

If you get something like:

mount: wrong fs type, bad option, bad superblock on /dev/sda1,
or too many mounted file systems

Refer to your system log file (<tt>/var/log/messages</tt>) for a detailed diagnostic message from the kernel.

== Q: Does the filesystem have an undelete capability? ==

There is no undelete in XFS (so far).

However at least some XFS driver implementations do not wipe file information nodes completely so there are chance to recover files with specialized commercial closed source software like [http://www.ufsexplorer.com/rdr_xfs.php Raise Data Recovery for XFS].

In this kind of XFS driver implementation it does not re-use directory entries immediately so there are chance to get back recently deleted files even with their real names.

''xfs_irecover'' or ''xfsr'' may help too, [http://www.who.is.free.fr/wiki/doku.php?id=recover this site] has a few links.

This applies to most recent Linux distributions (versions?), as well as to most popular NAS boxes that use embedded linux and XFS file system.

Anyway, the best is to always keep backups.

== Q: How can I backup a XFS filesystem and ACLs? ==

You can backup a XFS filesystem with utilities like '''xfsdump(8)''' and standard '''tar(1)''' for standard files. If you want to backup ACLs you will need to use '''xfsdump''' or [http://www.bacula.org/en/dev-manual/Current_State_Bacula.html Bacula] (> version 3.1.4) or [http://rsync.samba.org/ rsync] (>= version 3.0.0) to backup ACLs and EAs. '''xfsdump''' can also be integrated with [http://www.amanda.org/ amanda(8)].

== Q: I see applications returning error 990 or "Structure needs cleaning", what is wrong? ==

The error 990 stands for [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=blob;f=fs/xfs/linux-2.6/xfs_linux.h#l145 EFSCORRUPTED] which usually means XFS has detected a filesystem metadata problem and has shut the filesystem down to prevent further damage. Also, since about June 2006, we [http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/xfs.git;a=commit;h=da2f4d679c8070ba5b6a920281e495917b293aa0 converted from EFSCORRUPTED/990 over to using EUCLEAN], "Structure needs cleaning."

The cause can be pretty much anything, unfortunately - filesystem, virtual memory manager, volume manager, device driver, or hardware.

There should be a detailed console message when this initially happens. The messages have important information giving hints to developers as to the earliest point that a problem was detected. It is there to protect your data.

You can use xfs_check and xfs_repair to remedy the problem (with the file system unmounted).

== Q: Why do I see binary NULLS in some files after recovery when I unplugged the power? ==

Update: This issue has been addressed with a CVS fix on the 29th March 2007 and merged into mainline on 8th May 2007 for 2.6.22-rc1.

XFS journals metadata updates, not data updates. After a crash you are supposed to get a consistent filesystem which looks like the state sometime shortly before the crash, NOT what the in memory image looked like the instant before the crash.

Since XFS does not write data out immediately unless you tell it to with fsync, an O_SYNC or O_DIRECT open (the same is true of other filesystems), you are looking at an inode which was flushed out, but whose data was not. Typically you'll find that the inode is not taking any space since all it has is a size but no extents allocated (try examining the file with the '''xfs_bmap(8)''' command).

== Q: What is the problem with the write cache on journaled filesystems? ==

Many drives use a write back cache in order to speed up the performance of writes. However, there are conditions such as power failure when the write cache memory is never flushed to the actual disk. Further, the drive can de-stage data from the write cache to the platters in any order that it chooses. This causes problems for XFS and journaled filesystems in general because they rely on knowing when a write has completed to the disk. They need to know that the log information has made it to disk before allowing metadata to go to disk. When the metadata makes it to disk then the transaction can effectively be deleted from the log resulting in movement of the tail of the log and thus freeing up some log space. So if the writes never make it to the physical disk, then the ordering is violated and the log and metadata can be lost, resulting in filesystem corruption.

With hard disk cache sizes of currently (Jan 2009) up to 32MB that can be a lot of valuable information. In a RAID with 8 such disks these adds to 256MB, and the chance of having filesystem metadata in the cache is so high that you have a very high chance of big data losses on a power outage.

With a single hard disk and barriers turned on (on=default), the drive write cache is flushed before and after a barrier is issued. A powerfail "only" loses data in the cache but no essential ordering is violated, and corruption will not occur.

With a RAID controller with battery backed controller cache and cache in write back mode, you should turn off barriers - they are unnecessary in this case, and if the controller honors the cache flushes, it will be harmful to performance. But then you *must* disable the individual hard disk write cache in order to ensure to keep the filesystem intact after a power failure. The method for doing this is different for each RAID controller. See the section about RAID controllers below.

== Q: How can I tell if I have the disk write cache enabled? ==

For SCSI/SATA:

* Look in dmesg(8) output for a driver line, such as: "SCSI device sda: drive cache: write back"
* <nowiki># sginfo -c /dev/sda | grep -i 'write cache' </nowiki>

For PATA/SATA (although for SATA this only works on a recent kernel with ATA command passthrough):

* <nowiki># hdparm -I /dev/sda</nowiki> and look under "Enabled Supported" for "Write cache"

For RAID controllers:

* See the section about RAID controllers below

== Q: How can I address the problem with the disk write cache? ==

=== Disabling the disk write back cache. ===

For SATA/PATA(IDE): (although for SATA this only works on a recent kernel with ATA command passthrough): 

* <nowiki># hdparm -W0 /dev/sda</nowiki> # hdparm -W0 /dev/hda
* <nowiki># blktool /dev/sda wcache off</nowiki> # blktool /dev/hda wcache off

For SCSI:

* Using sginfo(8) which is a little tedious It takes 3 steps. For example:
*# <nowiki>#sginfo -c /dev/sda</nowiki> which gives a list of attribute names and values
*# <nowiki>#sginfo -cX /dev/sda</nowiki> which gives an array of cache values which you must match up with from step 1, e.g. 0 0 0 1 0 1 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0
*# <nowiki>#sginfo -cXR /dev/sda 0 0 0 1 0 0 0 0 0 0 65535 0 65535 65535 1 0 0 0 3 0 0</nowiki> allows you to reset the value of the cache attributes.

For RAID controllers:

* See the section about RAID controllers below

This disabling is kept persistent for a SCSI disk. However, for a SATA/PATA disk this needs to be done after every reset as it will reset back to the default of the write cache enabled. And a reset can happen after reboot or on error recovery of the drive. This makes it rather difficult to guarantee that the write cache is maintained as disabled. 

=== Using an external log. ===

Some people have considered the idea of using an external log on a separate drive with the write cache disabled and the rest of the file system on another disk with the write cache enabled. However, that will '''not''' solve the problem. For example, the tail of the log is moved when we are notified that a metadata write is completed to disk and we won't be able to guarantee that if the metadata is on a drive with the write cache enabled.

In fact using an external log will disable XFS' write barrier support.

=== Write barrier support. ===

Write barrier support is enabled by default in XFS since kernel version 2.6.17. It is disabled by mounting the filesystem with "nobarrier". Barrier support will flush the write back cache at the appropriate times (such as on XFS log writes). This is generally the recommended solution, however, you should check the system logs to ensure it was successful. Barriers will be disabled and reported in the log if any of the 3 scenarios occurs:

* "Disabling barriers, not supported with external log device"
* "Disabling barriers, not supported by the underlying device"
* "Disabling barriers, trial barrier write failed"

If the filesystem is mounted with an external log device then we currently don't support flushing to the data and log devices (this may change in the future). If the driver tells the block layer that the device does not support write cache flushing with the write cache enabled then it will report that the device doesn't support it. And finally we will actually test out a barrier write on the superblock and test its error state afterwards, reporting if it fails.

== Q. Should barriers be enabled with storage which has a persistent write cache? ==

Many hardware RAID have a persistent write cache which preserves it across power failure, interface resets, system crashes, etc. Using write barriers in this instance is not recommended and will in fact lower performance. Therefore, it is recommended to turn off the barrier support and mount the filesystem with "nobarrier", assuming your RAID controller is infallible and not resetting randomly like some common ones do. But take care about the hard disk write cache, which should be off.

== Q. Which settings does my RAID controller need ? ==

It's hard to tell because there are so many controllers. Please consult your RAID controller documentation to determine how to change these settings, but we try to give an overview here:

Real RAID controllers (not those found onboard of mainboards) normally have a battery backed cache (or an [http://en.wikipedia.org/wiki/Electric_double-layer_capacitor ultracapacitor] + flash memory "[http://www.tweaktown.com/articles/2800/adaptec_zero_maintenance_cache_protection_explained/ zero maintenance cache]") which is used for buffering writes to improve speed. Even if it's battery backed, the individual hard disk write caches need to be turned off, as they are not protected from a powerfail and will just lose all contents in that case.

* onboard RAID controllers: there are so many different types it's hard to tell. Generally, those controllers have no cache, but let the hard disk write cache on. That can lead to the bad situation that after a powerfail with RAID-1 when only parts of the disk cache have been written, the controller doesn't even see that the disks are out of sync, as the disks can resort cached blocks and might have saved the superblock info, but then lost different data contents. So, turn off disk write caches before using the RAID function.

* 3ware: /cX/uX set cache=off, see http://www.3ware.com/support/UserDocs/CLIGuide-9.5.1.1.pdf , page 86

* Adaptec: allows setting individual drives cache
arcconf setcache <disk> wb|wt
wb=write back, which means write cache on, wt=write through, which means write cache off. So "wt" should be chosen.

* Areca: In archttp under "System Controls" -> "System Configuration" there's the option "Disk Write Cache Mode" (defaults "Auto")

"Off": disk write cache is turned off

"On": disk write cache is enabled, this is not safe for your data but fast

"Auto": If you use a BBM (battery backup module, which you really should use if you care about your data), the controller automatically turns disk writes off, to protect your data. In case no BBM is attached, the controller switches to "On", because neither controller cache nor disk cache is safe so you don't seem to care about your data and just want high speed (which you get then).

That's a very sensible default so you can let it "Auto" or enforce "Off" to be sure.

* LSI MegaRAID: allows setting individual disks cache:
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL # flushes the controller cache
MegaCli -LDGetProp -Cache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the controller cache settings
MegaCli -LDGetProp -DskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # shows the disk cache settings (for all phys. disks in logical disk)
MegaCli -LDSetProp -EnDskCache|DisDskCache -LN|-L0,1,2|-LAll -aN|-a0,1,2|-aALL # set disk cache setting

* Xyratex: from the docs: "Write cache includes the disk drive cache and controller cache.". So that means you can only set the drive caches and the unit caches together. To protect your data, turn it off, but write performance will suffer badly as also the controller write cache is disabled.

== Q: Which settings are best with virtualization like VMware, XEN, qemu? ==

The biggest problem is that those products seem to also virtualize disk
writes in a way that even barriers don't work any more, which means even
a fsync is not reliable. Tests confirm that unplugging the power from
such a system even with RAID controller with battery backed cache and
hard disk cache turned off (which is safe on a normal host) you can
destroy a database within the virtual machine (client, domU whatever you
call it).

In qemu you can specify cache=off on the line specifying the virtual
disk. For others information is missing.

== Q: What is the issue with directory corruption in Linux 2.6.17? ==

In the Linux kernel 2.6.17 release a subtle bug was accidentally introduced into the XFS directory code by some "sparse" endian annotations. This bug was sufficiently uncommon (it only affects a certain type of format change, in Node or B-Tree format directories, and only in certain situations) that it was not detected during our regular regression testing, but it has been observed in the wild by a number of people now.

''Update: the fix is included in 2.6.17.7 and later kernels.''

To add insult to injury, '''xfs_repair(8)''' is currently not correcting these directories on detection of this corrupt state either. This '''xfs_repair''' issue is actively being worked on, and a fixed version will be available shortly.

''Update: a fixed '''xfs_repair''' is now available; version 2.8.10 or later of the xfsprogs package contains the fixed version.''

No other kernel versions are affected. However, using a corrupt filesystem on other kernels can still result in the filesystem being shutdown if the problem has not been rectified (on disk), making it seem like other kernels are affected.

The '''xfs_check''' tool, or '''xfs_repair -n''', should be able to detect any directory corruption.

Until a fixed '''xfs_repair''' binary is available, one can make use of the '''xfs_db(8)''' command to mark the problem directory for removal (see the example below). A subsequent '''xfs_repair''' invocation will remove the directory and move all contents into "lost+found", named by inode number (see second example on how to map inode number to directory entry name, which needs to be done _before_ removing the directory itself). The inode number of the corrupt directory is included in the shutdown report issued by the kernel on detection of directory corruption. Using that inode number, this is how one would ensure it is removed:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
core.mode = 040755
core.version = 2
core.format = 3 (btree)
...
xfs_db> write core.mode 0
xfs_db> quit
</nowiki>

A subsequent '''xfs_repair''' will clear the directory, and add new entries (named by inode number) in lost+found.

The easiest way to map inode numbers to full paths is via '''xfs_ncheck(8)'''<nowiki>: </nowiki>

<nowiki>
# xfs_ncheck -i 14101 -i 14102 /dev/sdXXX
14101 full/path/mumble_fratz_foo_bar_1495
14102 full/path/mumble_fratz_foo_bar_1494
</nowiki>

Should this not work, we can manually map inode numbers in B-Tree format directory by taking the following steps:

<nowiki>
# xfs_db -x /dev/sdXXX
xfs_db> inode NNN
xfs_db> print
core.magic = 0x494e
...
next_unlinked = null
u.bmbt.level = 1
u.bmbt.numrecs = 1
u.bmbt.keys[1] = [startoff] 1:[0]
u.bmbt.ptrs[1] = 1:3628
xfs_db> fsblock 3628
xfs_db> type bmapbtd
xfs_db> print
magic = 0x424d4150
level = 0
numrecs = 19
leftsib = null
rightsib = null
recs[1-19] = [startoff,startblock,blockcount,extentflag]
1:[0,3088,4,0] 2:[4,3128,8,0] 3:[12,3308,4,0] 4:[16,3360,4,0]
5:[20,3496,8,0] 6:[28,3552,8,0] 7:[36,3624,4,0] 8:[40,3633,4,0]
9:[44,3688,8,0] 10:[52,3744,4,0] 11:[56,3784,8,0]
12:[64,3840,8,0] 13:[72,3896,4,0] 14:[33554432,3092,4,0]
15:[33554436,3488,8,0] 16:[33554444,3629,4,0]
17:[33554448,3748,4,0] 18:[33554452,3900,4,0]
19:[67108864,3364,4,0]
</nowiki>

At this point we are looking at the extents that hold all of the directory information. There are three types of extent here, we have the data blocks (extents 1 through 13 above), then the leaf blocks (extents 14 through 18), then the freelist blocks (extent 19 above). The jumps in the first field (start offset) indicate our progression through each of the three types. For recovering file names, we are only interested in the data blocks, so we can now feed those offset numbers into the '''xfs_db''' dblock command. So, for the fifth extent - 5:[20,3496,8,0] - listed above:

...
xfs_db> dblock 20
xfs_db> print
dhdr.magic = 0x58443244
dhdr.bestfree[0].offset = 0
dhdr.bestfree[0].length = 0
dhdr.bestfree[1].offset = 0
dhdr.bestfree[1].length = 0
dhdr.bestfree[2].offset = 0
dhdr.bestfree[2].length = 0
du[0].inumber = 13937
du[0].namelen = 25
du[0].name = "mumble_fratz_foo_bar_1595"
du[0].tag = 0x10
du[1].inumber = 13938
du[1].namelen = 25
du[1].name = "mumble_fratz_foo_bar_1594"
du[1].tag = 0x38
...

So, here we can see that inode number 13938 matches up with name "mumble_fratz_foo_bar_1594". Iterate through all the extents, and extract all the name-to-inode-number mappings you can, as these will be useful when looking at "lost+found" (once '''xfs_repair''' has removed the corrupt directory).

== Q: Why does my > 2TB XFS partition disappear when I reboot ? ==

Strictly speaking this is not an XFS problem.

To support > 2TB partitions you need two things: a kernel that supports large block devices (<tt>CONFIG_LBD=y</tt>) and a partition table format that can hold large partitions. The default DOS partition tables don't. The best partition format for
> 2TB partitions is the EFI GPT format (<tt>CONFIG_EFI_PARTITION=y</tt>).

Without CONFIG_LBD=y you can't even create the filesystem, but without <tt>CONFIG_EFI_PARTITION=y</tt> it works fine until you reboot at which point the partition will disappear. Note that you need to enable the <tt>CONFIG_PARTITION_ADVANCED</tt> option before you can set <tt>CONFIG_EFI_PARTITION=y</tt>.

== Q: Why do I receive <tt>No space left on device</tt> after <tt>xfs_growfs</tt>? ==

After [http://oss.sgi.com/pipermail/xfs/2009-January/039828.html growing a XFS filesystem], df(1) would show enough free space but attempts to write to the filesystem result in -ENOSPC. To fix this, [http://oss.sgi.com/pipermail/xfs/2009-January/039835.html Dave Chinner advised]:

The only way to fix this is to move data around to free up space
below 1TB. Find your oldest data (i.e. that was around before even
the first grow) and move it off the filesystem (move, not copy).
Then if you copy it back on, the data blocks will end up above 1TB
and that should leave you with plenty of space for inodes below 1TB.

A complete dump and restore will also fix the problem ;)

Also, you can add 'inode64' to your mount options to allow inodes to live above 1TB.

example:[https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30703&forum=38 | No space left on device on xfs filesystem with 7.7TB free]

== Q: Is using noatime or/and nodiratime at mount time giving any performance benefits in xfs (or not using them performance decrease)? ==

The default atime behaviour is relatime, which has almost no overhead compared to noatime but still maintains sane atime values. All Linux filesystems use this as the default now (since around 2.6.30), but XFS has used relatime-like behaviour since 2006, so no-one should really need to ever use noatime on XFS for performance reasons.

Also, noatime implies nodiratime, so there is never a need to specify nodiratime when noatime is also specified.

== Q: How to get around a bad inode repair is unable to clean up ==

The trick is go in with xfs_db and mark the inode as a deleted, which will cause repair to clean it up and finish the remove process.

xfs_db -x -c 'inode XXX' -c 'write core.nextents 0' -c 'write core.size 0' /dev/hdXX

== Q: How to calculate the correct sunit,swidth values for optimal performance ==

XFS allows to optimize for a given RAID stripe unit (stripe size) and stripe width (number of data disks) via mount options.

These options can be sometimes autodetected (for example with md raid and recent enough kernel (>= 2.6.32) and xfsprogs (>= 3.1.1) built with libblkid support) but manual calculation is needed for most of hardware raids.

The calculation of these values is quite simple:

su = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>

So if your RAID controller has a stripe size of 64KB, and you have a RAID-6 with 8 disks, use

su = 64k
sw = 6 (RAID-6 of 8 disks has 6 data disks)

A RAID stripe size of 256KB with a RAID-10 over 16 disks should use

su = 256k
sw = 8 (RAID-10 of 16 disks has 8 data disks)

Alternatively, you can use "sunit" instead of "su" and "swidth" instead of "sw" but then sunit/swidth values need to be specified in "number of 512B sectors"!

Note that <tt>xfs_info</tt> and <tt>mkfs.xfs</tt> interpret sunit and swidth as being specified in units of 512B sectors; that's unfortunately not the unit they're reported in, however.
<tt>xfs_info</tt> and <tt>mkfs.xfs</tt> report them in multiples of your basic block size (bsize) and not in 512B sectors.

Assume for example: swidth 1024 (specified at mkfs.xfs command line; so 1024 of 512B sectors) and block size of 4096 (bsize reported by mkfs.xfs at output). You should see swidth 128 (reported by mkfs.xfs at output). 128 * 4096 == 1024 * 512.

When creating XFS filesystem on top of LVM on top of hardware raid please use sunit/swith values as when creating XFS filesystem directly on top of hardware raid.

== Q: Why doesn't NFS-exporting subdirectories of inode64-mounted filesystem work? ==

The default <tt>fsid</tt> type encodes only 32-bit of the inode number for subdirectory exports. However, exporting the root of the filesystem works, or using one of the non-default <tt>fsid</tt> types (<tt>fsid=uuid</tt> in <tt>/etc/exports</tt> with recent <tt>nfs-utils</tt>) should work as well. (Thanks, Christoph!)

== Q: What is the inode64 mount option for? ==

By default, with 32bit inodes, XFS places inodes only in the first 1TB of a disk. If you have a disk with 100TB, all inodes will be stuck in the first TB. This can lead to strange things like "disk full" when you still have plenty space free, but there's no more place in the first TB to create a new inode. Also, performance sucks.

To come around this, use the inode64 mount options for filesystems >1TB. Inodes will then be placed in the location where their data is, minimizing disk seeks.

Beware that some old programs might have problems reading 64bit inodes, especially over NFS. Your editor used inode64 for over a year with recent (openSUSE 11.1 and higher) distributions using NFS and Samba without any corruptions, so that might be a recent enough distro.

== Q: Can I just try the inode64 option to see if it helps me? ==

Starting from kernel 2.6.35, you can try and then switch back. Older kernels have a bug leading to strange problems if you mount without inode64 again. For example, you can't access files & dirs that have been created with an inode >32bit anymore.

== Q: Performance: mkfs.xfs -n size=64k option ==

Asking the implications of that mkfs option on the XFS mailing list, Dave Chinner explained it this way:

Inodes are not stored in the directory structure, only the directory entry name and the inode number. Hence the amount of space used by a
directory entry is determined by the length of the name.

There is extra overhead to allocate large directory blocks (16 pages instead of one, to begin with, then there's the vmap overhead, etc), so for small directories smaller block sizes are faster for create and unlink operations.

For empty directories, operations on 4k block sized directories consume roughly 50% less CPU that 64k block size directories. The 4k block size directories consume less CPU out to roughly 1.5 million entries where the two are roughly equal. At directory sizes of 10 million entries, 64k directory block operations are consuming about 15% of the CPU that 4k directory block operations consume.

In terms of lookups, the 64k block directory will take less IO but consume more CPU for a given lookup. Hence it depends on your IO latency and whether directory readahead can hide that latency as to which will be faster. e.g. For SSDs, CPU usage might be the limiting factor, not the IO. Right now I don't have any numbers on what the difference might be - I'm getting 1 billion inode population issues worked out first before I start on measuring cold cache lookup times on 1 billion files....

== Q: I want to tune my XFS filesystems for <something> ==

The standard answer you will get to this question is this: use the defaults.

There are few workloads where using non-default mkfs.xfs or mount options make much sense. In general, the default values already used are optimised for best performance in the first place. mkfs.xfs will detect the difference between single disk and MD/DM RAID setups and change the default values it uses to configure the filesystem appropriately.

There are a lot of "XFS tuning guides" that Google will find for you - most are old, out of date and full of misleading or just plain incorrect information. Don't expect that tuning your filesystem for optimal bonnie++ numbers will mean your workload will go faster. You should only consider changing the defaults if either: a) you know from experience that your workload causes XFS a specific problem that can be worked around via a configuration change, or b) your workload is demonstrating bad performance when using the default configurations. In this case, you need to understand why your application is causing bad performance before you start tweaking XFS configurations.

In most cases, the only thing you need to to consider for <tt>mkfs.xfs</tt> is specifying the stripe unit and width for hardware RAID devices. For mount options, the only thing that will change metadata performance considerably are the <tt>logbsize</tt> and <tt>delaylog</tt> mount options. Increasing <tt>logbsize</tt> reduces the number of journal IOs for a given workload, and <tt>delaylog</tt> will reduce them even further. The trade off for this increase in metadata performance is that more operations may be "missing" after recovery if the system crashes while actively making modifications.

== Q: Which factors influence the memory usage of xfs_repair? ==

This is best explained with an example. The example filesystem is 16Tb, but basically empty (look at icount).

# xfs_repair -n -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 64, imem = 0, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2096.
#

xfs_repair is saying it needs at least 2096MB of RAM to repair the filesystem,
of which 2,097,152KB is needed for tracking free space.
(The -m 1 argument was telling xfs_repair to use ony 1 MB of memory.)

Now if we add some inodes (50 million) to the filesystem (look at icount again), and the result is:

# xfs_repair -vv -m 1 /dev/vda
Phase 1 - find and verify superblock...
- max_mem = 1024, icount = 50401792, imem = 196882, dblock = 4294967296, dmem = 2097152
Required memory for repair is greater that the maximum specified
with the -m option. Please increase it to at least 2289.

That is now needs at least another 200MB of RAM to run.

The numbers reported by xfs_repair are the absolute minimum required and approximate at that;
more RAM than this may be required to complete successfully.
Also, if you only give xfs_repair the minimum required RAM, it will be slow;
for best repair performance, the more RAM you can give it the better.

== Q: Why some files of my filesystem shows as "?????????? ? ? ? ? ? filename" ? ==

If ls -l shows you a listing as

# ?????????? ? ? ? ? ? file1
?????????? ? ? ? ? ? file2
?????????? ? ? ? ? ? file3
?????????? ? ? ? ? ? file4

and errors like:
# ls /pathtodir/
ls: cannot access /pathtodir/file1: Invalid argument
ls: cannot access /pathtodir/file2: Invalid argument
ls: cannot access /pathtodir/file3: Invalid argument
ls: cannot access /pathtodir/file4: Invalid argument

or even:
# failed to stat /pathtodir/file1

It is very probable your filesystem must be mounted with inode64
# mount -oremount,inode64 /dev/diskpart /mnt/xfs

should make it work ok again.
If it works, add the option to fstab.

== Q: The xfs_db "frag" command says I'm over 50%. Is that bad? ==

It depends. It's important to know how the value is calculated. xfs_db looks at the extents in all files, and returns:

(actual extents - ideal extents) / actual extents

This means that if, for example, you have an average of 2 extents per file, you'll get an answer of 50%. 4 extents per file would give you 75%. This may or may not be a problem, especially depending on the size of the files in question. (i.e. 400GB files in four 100GB extents would hardly be considered badly fragmented). The xfs_bmap command can be useful for displaying the actual fragmentation/layout of individual files.

XFS Companies

2012-02-03T19:00:32Z

Sandeen:

= These are companies that either use XFS or have a product that utilizes XFS . =

Info gathered from: [http://oss.sgi.com/projects/xfs/users.html XFS Users] on [http://oss.sgi.com/ oss.sgi.com]

== [http://www.dell.com/ Dell's HPC NFS Storage Solution] ==

"The Dell NFS Storage Solution (NSS) is a unique new storage solution providing cost-effective NFS storage as an appliance. Designed to scale from 20 TB installations up to 80 TB of usable space, the NSS is delivered as a fully configured, ready-to-go storage solution and is available with full hardware and software support from Dell. ... XFS was chosen for the NSS because XFS is capable of scaling beyond 16 TB and provides good performance for a broad range of applications."

More information is available in [http://i.dell.com/sites/content/business/solutions/hpcc/en/Documents/Dell-NSS-NFS-Storage-solution-final.pdf this solution guide].

== [http://www.kernel.org/ The Linux Kernel Archives] ==

"A bit more than a year ago (as of October 2008) kernel.org, in an ever increasing need to squeeze more performance out of it's machines, made the leap of migrating the primary mirror machines (mirrors.kernel.org) to XFS. We site a number of reasons including fscking 5.5T of disk is long and painful, we were hitting various cache issues, and we were seeking better performance out of our file system."

"After initial tests looked positive we made the jump, and have been quite happy with the results. With an instant increase in performance and throughput, as well as the worst xfs_check we've ever seen taking 10 minutes, we were quite happy. Subsequently we've moved all primary mirroring file-systems to XFS, including www.kernel.org , and mirrors.kernel.org. With an average constant movement of about 400mbps around the world, and with peaks into the 3.1gbps range serving thousands of users simultaneously it's been a file system that has taken the brunt we can throw at it and held up spectacularly."

== [http://www.sdss.org/ The Sloan Digital Sky Survey] ==

"The Sloan Digital Sky Survey is an ambitious effort to map one-quarter of the sky at optical and very-near infrared wavelengths and take spectra of 1 million extra-galactic objects. The estimated amount of data that will be acquired over the 5 year lifespan of the project is 15TB, however, the total amount of storage space required for object informational databases, corrected frames, and reduced spectra will be several factors more than this. The goal is to have all the data online and available to the collaborators at all times. To accomplish this goal we are using commodity, off the shelf (COTS) Intel servers with EIDE disks configured as RAID50 arrays using XFS. Currently, 14 machines are in production accounting for over 18TB. By the scheduled end of the survey in 2005, 50TB of XFS disks will be online serving SDSS data to collaborators and the public."

"For complete details and status of the project please see [http://www.sdss.org/ http://www.sdss.org]. For details of the storage systems, see the [http://home.fnal.gov/~yocum/storageServerTechnicalNote.html SDSS Storage Server Technical Note]."

== [http://www-d0.fnal.gov/ The DØ Experiment at Fermilab] ==

"At the DØ experiment at the Fermi National Accelerator Laboratory we have a ~150 node cluster of desktop machines all using the SGI-patched kernel. Every large disk (>40Gb) or disk array in the cluster uses XFS including 4x640Gb disk servers and several 60-120Gb disks/arrays. Originally we chose reiserfs as our journaling filesystem, however, this was a disaster. We need to export these disks via NFS and this seemed perpetually broken in 2.4 series kernels. We switched to XFS and have been very happy. The only inconvenience is that it is not included in the standard kernel. The SGI guys are very prompt in their support of new kernels, but it is still an extra step which should not be necessary."

== [http://www.ciprico.com/pDiMeda.shtml Ciprico DiMeda NAS Solutions] ==

"The Ciprico DiMeda line of Network Attached Storage solutions combine the ease of connectivity of NAS with the SAN like performance levels required for digital media applications. The DiMeda 3600 provides high availability and high performance through dual NAS servers and redundant, scalable Fibre Channel RAID storage. The DiMeda 1700 provides high performance files services at a low price by using the latest Serial ATA RAID technology. All DiMeda systems are based on Linux and use XFS as the filesystem. We tested a number of filesystem alternatives and XFS was chosen because it provided the highest performance in digital media applications and the journaling feature ensures rapid failover in our dual node fault tolerant configurations."

== [http://www.quantum.com/Products/NAS+Servers/Guardian+14000/Default.htm The Quantum Guardian™ 14000] ==

"The Quantum Guardian™ 14000, the latest Network Attached Storage (NAS) solution from Quantum, delivers 1.4TB of enterprise-class storage for less than $25,000. The Guardian 14000 is a Linux-based device which utilizes XFS to provide a highly reliable journaling filesystem with simultaneous support for Windows, UNIX, Linux and Macintosh environments. As dedicated appliance optimized for fast, reliable file sharing, the Guardian 14000 combines the simplicity of NAS with a robust feature set designed for the most demanding enterprise environments. Support for tools such as Active Directory Service (ADS), UNIX Network Information Service (NIS) and Simple Network Management Protocol (SNMP) provides ease of management and seamless integration. Hardware redundancy, Snapshots and StorageCare™ on-site service ensure security for business-critical data."

== [http://www.bigstorage.com/products_approach_overview.html BigStorage K2~NAS] ==

"At BigStorage we pride ourselves on tailoring our NAS systems to meet our customer's needs, with the help of XFS we are able to provide them with the most reliable Journaling Filesystem technology available. Our open systems approach, which allows for cross-platform integration, gives our customers the flexibility to grow with their data requirements. In addition, BigStorage offers a variety of other features including total hardware redundancy, snapshotting, replication and backups directly from the unit. All of our products include BigStorageï¿½s 24/7 LiveResponse™ support. With LiveResponse™, we keep our team of experienced technical experts on call 24 hours a day, every day, to ensure that your storage investment remains online, all the time."

== [http://www.echostar.com Echostar DishPVR 721] ==

"Echostar uses the XFS filesystem for its latest generation of satellite receivers, the DP721. Echostar chose XFS for its performance, stability and unique set of features."

"XFS allowed us to meet a demanding requirement of recording two mpeg2 streams to the internal hard drive while simultaneously viewing a third pre-recorded stream. In addition, XFS allowed us to withstand unexpected power loss without filesystem corruption or user interaction."

"We tested several other filesystems, but XFS emerged as the clear winner."

== [http://www.sun.com/hardware/serverappliances/raq550/ Sun Cobalt RaQ™ 550] ==

From the [http://www.sun.com/hardware/serverappliances/raq550/features.html features] page:

"XFS is a journaling file system capable of quick fail over recovery after unexpected interruptions. XFS is an important feature for mission-critical applications as it ensures data integrity and dramatically reduces startup time by avoiding FSCK delay."

== [http://pingu.salk.edu/ Center for Cytometry and Molecular Imaging at the Salk Institute] ==

"I run the Center for Cytometry and Molecular Imaging at the Salk Institute in La Jolla, CA. We're a core facility for the Institute, offering flow cytometry, basic and deconvolution microscopy, phosphorimaging (radioactivity imaging) and fluorescent imaging."

"I'm currently in the process of migrating our data server to Linux/XFS. Our web server currently uses Linux/XFS. We have about 60 Gb on the data server which has a 100Gb SCSI RAID 5 array. This is a bit restrictive for our microscopists so in order that they can put more data online, I'm adding another machine, also running Linux/XFS, with about 420 Gb of IDE-RAID5, based on Adaptec controllers...."

"Servers are configured with quota and run Samba, NFS, and Netatalk for connectivity to the mixed bag of computers we have around here. I use the CVS XFS tree most of the time. I have not seen any problems in the several months I have been testing."

== [http://coltex.nl/ Coltex Retail Group BV] ==

"Coltex Retail group BV in the Netherlands uses Red Hat Linux with XFS for their main database server which collects the data from over 240 clothing retail stores throughout the Netherlands. Coltex depends on the availability of the server for over 100 hundred employees in the main office for retrieval of logistical and sales figures. The database size is roughly 10GB large containing both historical and current data."

"The entire production and logistical system depends on the availability of the system and downtime would mean a significant financial penalty. The speed and reliability of the XFS filesystem which has a proven track record and mature tools to go with it is fundamental to the availability of the system."

"XFS has saved us a lot of time during testing and implementation. A long filesystems check is no longer needed when bad things happen when they do. The increased speed of our database system which is based on Progress 9.1C is also a nice benefit to this filesystem."

== [http://www.dkp.com/ DKP Effects] ==

"We're a 3D computer graphics/post-production house. We've currently got four fileservers using XFS under Linux online - three 350GB servers and one 800GB server. The servers are under fairly heavy load - network load to and from the dual NICs on the box is basically maxed out 18 hours a day - and we do have occasional lockups and drive failures. Thanks to Linux SW RAID5 and XFS, though, we haven't had any data loss, or significant down time."

== [http://www.epigenomics.com/ Epigenomics] ==

"We currently have several IDE-to-SCSI-RAID systems with XFS in production. The largest has a capacity of 1.5TB, the other 2 have 430GB each."

"Data stored on these filesystems is on the one hand "normal" home directories and corporate documents and on the other hand scientific data for our laboratory and IT department."

== [http://www.incyte.com/ Incyte Genomics] ==

"I'm currently in the process of slowly converting 21 clusters totaling 2300+ processors over to XFS."

"These machines are running a fairly stock RH7.1+XFS. The application is our own custom scheduler for doing genomic research. We have one of the worlds largest sequencing labs which generates a tremendous amount of raw data. Vast amounts of CPU cycles must be applied to it to turn it into useful data we can then sell access to."

"Currently, a minority of these machines are running XFS, but as I can get downtime on the clusters I am upgrading them to 7.1+XFS. When I'm done, it'll be about 10TB of XFS goodness... across 9G disks mostly."

== [http://www.monmouth.edu/ Monmouth University] ==

"We've replaced our NetApp filer (80GB, $40,000). NetApp ONTAP software [runs on NetApp filers] is basically an NFS and CIFS server with their own proprietary filesystem. We were quickly running out of space and our annual budget almost depleted. What were we to do?"

"With an off-the-shelf Dell 4400 series server and 300GB of disks ($8,000 total). We were able to run Linux and Samba to emulate a NetApp filer."

"XFS allowed us to manage 300GB of data with absolutely no downtime (now going on 79 days) since implementation. Gone are the days of fearing the fsck of 300GB."

== [http://www.astro.wisc.edu The University of Wisconsin Astronomy Department] ==

"At the University of Wisconsin Astronomy Department we have been using Linux XFS since the first release. We currently have 31 Linux boxes running XFS on all filesystems with about 2.6 TB of disk space on these machines. We use XFS primarily on our data reduction systems, but we also use it on our web server and on one of the remote observing machines at the WIYN 3.5m Telescope at Kitt Peak (http://www.noao.edu/wiyn/wiyn.html)."

"We will likely be using Linux XFS at least in part on the GLIMPSE program (http://www.astro.wisc.edu/sirtf/) which will likely require several TB of disk space to process the data."

== [http://www.amoa.org/ The Austin Museum of Art] ==

"The Austin Museum of Art has two file servers running RedHat 7.2_XFS upgraded from RedHat 7.1_XFS. Our webserver runs Domino on top of RedHat 7.3_XFS and we're getting about 70% better performance than the Domino server running on Windows 2000 Server. We're moving our workstations away from Windows and Microsoft Office to an LTSP server running on RedHat 7.3_XFS.

"We've become solely dependent on XFS for all of our data systems."

== [http://www.tecmath.com/ tecmath AG] ==

"We use a production server with a 270 GB RAID 5 (hardware) disk array. It is based on a Suse 7.2 distribution, but with a standard 2.4.12 kernel with XFS and LVM patches. The server provides NFS to 8 Unix clients as well as Samba to about 80 PCs. The machine also runs Bind 9, Apache, Exim, DHCP, POP3, MySQL. I have tried out different configurations with ReiserFS, but I didn't manage to find a stable configuration with respect to NFS. Since I converted all disks to XFS some 3 months ago, we never had any filesystem-related problems."

== [http://www.theiqgroup.com/ The IQ Group] ==

"Here at the IQ Group, Inc. we use XFS for all our production and development servers."

"Our OS of choice is Slackware Linux 8.0. Our hardware of choice is Dell and VALinux servers."

"As for applications, we run the standard Unix/Linux apps like Sendmail, Apache, BIND, DHCP, iptables, etc.; as well as Oracle 9i and Arkeia."

"We've been running XFS across the board for about 3 months now without a hitch (so far)."

"Size-wise, our biggest server is about 40 GB, but that will be increasing substantially in the near future."

"Our production servers are collocated so a journaled FS was a must. Reboots are quick and no human interaction is required like with a bad fsck on ext2. Additionally, our database servers gain additional integrity and robustness."

"We originally chose XFS over ReiserFS and ext3 because of it's age (it's been in production on SGI boxes for probably longer than all the other journaling FS's combined) and it's speed appeared comparable as well."

== [http://www.artsit.usyd.edu.au Arts IT Unit, Sydney University] ==

"I've got XFS on a 'production' file server. The machine could have up to 500 people logged in, but typically less than 200. Most are Mac users, connected via NetAtalk for 'personal files', although there are shared areas for admin units. Probably about 30-40 windows users. (Samba) It's the file server for an Academic faculty at a University."

"Hardware RAID, via Mylex dual channel controller with 4 drives, Intel Tupelo MB, Intel 'SC5000' server chassis with redundant power and hot-swap scsi bays. The system boots off a non RAID single 9gb UW-scsi drive."

"Only system 'crash' was caused by some one accidentally unplugging it, just before we put it into production. It was back in full operation within 5 minutes. Without journaling, the fsck would have taken well over an hour. In day to day use it has run well."

== [http://structbio.vanderbilt.edu/comp/ Vanderbilt University Center for Structural Biology] ==

"I run a high-performance computing center for Structural Biology research at Vanderbilt University. We use XFS extensively, and have been since the late prerelease versions. I've had nothing but good experiences with it."

"We began using XFS in our search for a good solution for our RAID fileservers. We had such good experiences with it on these systems that we've begun putting it on the root/usr/var partitions of every Linux system we run here. I even have it on my laptop these days. XFS in combination with the 2.4 NFS3 implementation performs very well for us, and we have great uptimes on these systems (Our 750GB ArenaII setup is at 143 days right now)."

"All told, we've got about 1.2TB of XFS filesystems spinning right now. It's spread out across maybe a dozen or so filesystems and will continue to increase as we are growing fast and that's all we use now. Next up is putting it on our 17-node Linux cluster, which will bring that up to 1.5TB spread across 30 filesystems."

"I, for one, would LOVE to see XFS make it into the kernel tree. From my perspectives, it's one of the best things to happen to Linux in the 7 years I've been using/administering it."

==== 2008 Update ====

"We've since moved our main home directories to a proprietary NAS, but continue to use XFS on 10TB of LVM storage for doing backup-to-disk from the same NAS"

== [http://www-cdf.fnal.gov/ CDF Experiment at Fermi National Lab] ==

CDF, an elementary particle physics experiment at Fermi National Lab, is using XFS for all our cache disks.

The usage model is that we have a PB tape archive (2 STK silos) as permanent storage. In front of this archive we are deploying a roughly 100TB disk cache system. The cache is made up of 50 2TB file server based on cheap commodity hardware (3ware based hardware raid using IDE drives). The data is then processed by a cluster of 300 Dual CPU Linux PC's. The cache software is dCache, a DESY/FNAL product.

The whole system is used by more than 300 active users from all over the world for batch processing for their physics data analysis.

== [http://www.get2chip.com Get2Chip, Inc.] ==

"We are using XFS on 3 production file servers with approximately 1.5T of data. Quite impressive especially when we had a power outage and all three servers shutdown. All servers came back up in minutes with no problems! We are looking at creating two more servers that would manage 2+ TB of data store."

== [http://www.lando.co.za Lando International Group Technologies] ==

Lando International Group Technologies is the home of:

* [www.lando.co.za Lanndo Technologies Africa (Pty) Ltd] - Internet Service Provider
* [www.lbsd.net Linux Based Systems Design] (Article 21). Not-For-Profit company established to provide free Linux distributions and programs.
* Cell Park South Africa (Pty) Ltd. RSA Pat Appln 2001/10406. Collecting parking fees by means of cell phone SMS or voice.
* Read Plus Education (Pty) Ltd. Software based reading skills training and testing for ages 4 to 100.
* Mobivan. Mobile office including Internet access, fax, copying, printing, telephone, collection and delivery services, legal services, pre-paid phone and electricity services, bill payment email, secretarial services, training facilities and management services.
* Lando International Marketing Agency. Direct marketing services, design and supply of promotional material, consulting, sourcing of capital and other funding.
* Illico. Software development and systems analysis on most platforms.

"Throughout these companies, we use the XFS filesystem with [http://idms.lbsd.net IDMS Linux] on high-end Intel servers, with an average of 100 GB storage each. XFS stores our customer and user data, including credit card details, mail, routing tables, etc.. We have not had one problem since the release of the first XFS patch."

== [http://www.fcb-wilkens.com Foote, Cone, & Belding] ==

"We are an advertisement company in Germany, and the use of the XFS filesystem is a story of success for us. In our Hamburg office, we have two file servers having a 420 Gig RAID in XFS format serving (almost) all our data to about 180 Macs and about 30 PCs using Samba and Netatalk. Some of the data is used in our offices in Frankfurt and Berlin, and in fact the Berlin office is just getting it's own 250 Gig fileserver (using XFS) right now."

"The general success with XFS has led us to switch over all our Linux servers to run on XFS as well (with the exception of two systems that are tied to tight specifications configuration wise). XFS, even the old 1.0 version, has happily taken on various abuse - broken SCSI controllers, broken RAID systems."

== [http://www.moving-picture.co.uk/ Moving Picture Company] ==

"We here at MPC use XFS/RedHat 7.2 on all of our graphics-workstations and file-servers. More info can be found in an [http://www.linuxuser.co.uk/articles/issue20/lu20-Linux_at_work-In_the_picture.pdf article] LinuxUser magazine did on us recently."

== [http://www.coremetrics.com/ Coremetrics, Inc.] ==

"We are currently using XFS for 25+ production web-servers, ~900GB Oracle db servers, with potentially 15+ more servers by mid 2003, with ~900GB+ databases. All XFS installed."

"Also, our dev environment, except for the Sun boxes which all are being migrated to X86 in the aforementioned server additions, plus the dev Sun boxes as well, are all x86 dual proc servers running Oracle, application servers, or web services as needed. All servers run XFS from images we've got on our SystemImager servers."

"All production back-end servers are connected via FC1 or FC2 to a SAN containing ~13TB of raw storage, which, will soon be converted from VxFS to XFS with the migration of Oracle to our x86 platforms."

== [http://evolt.org Evolt.org] ==

"evolt.org, a world community for web developers promoting the mutual free exchange of ideas, skills and experiences, has had a great deal of success using XFS. Our primary webserver which serves 100K hosts/month, primary Oracle database with ~25Gb of data, and free member hosting for 1000 users haven't had a minute of downtime since XFS has been installed. Performance has been spectacular and maintenance a breeze."

 ''All testimonials on this page represent the views of the submitters, and references to other products and companies should not be construed as an endorsement by either the organizations profiled, or by SGI. All trademarks (r) their respective owners.''

XFS Companies

2012-02-03T18:59:02Z

Sandeen:

== These are companies that either use XFS or have a product that utilizes XFS . ==

Info gathered from: [http://oss.sgi.com/projects/xfs/users.html XFS Users] on [http://oss.sgi.com/ oss.sgi.com]

== [http://www.dell.com/ Dell's HPC NFS Storage Solution] ==

"The Dell NFS Storage Solution (NSS) is a unique new storage solution providing cost-effective NFS storage as an appliance. Designed to scale from 20 TB installations up to 80 TB of usable space, the NSS is delivered as a fully configured, ready-to-go storage solution and is available with full hardware and software support from Dell."

More information is available in [http://i.dell.com/sites/content/business/solutions/hpcc/en/Documents/Dell-NSS-NFS-Storage-solution-final.pdf this solution guide].

== [http://www.kernel.org/ The Linux Kernel Archives] ==

"A bit more than a year ago (as of October 2008) kernel.org, in an ever increasing need to squeeze more performance out of it's machines, made the leap of migrating the primary mirror machines (mirrors.kernel.org) to XFS. We site a number of reasons including fscking 5.5T of disk is long and painful, we were hitting various cache issues, and we were seeking better performance out of our file system."

"After initial tests looked positive we made the jump, and have been quite happy with the results. With an instant increase in performance and throughput, as well as the worst xfs_check we've ever seen taking 10 minutes, we were quite happy. Subsequently we've moved all primary mirroring file-systems to XFS, including www.kernel.org , and mirrors.kernel.org. With an average constant movement of about 400mbps around the world, and with peaks into the 3.1gbps range serving thousands of users simultaneously it's been a file system that has taken the brunt we can throw at it and held up spectacularly."

== [http://www.sdss.org/ The Sloan Digital Sky Survey] ==

"The Sloan Digital Sky Survey is an ambitious effort to map one-quarter of the sky at optical and very-near infrared wavelengths and take spectra of 1 million extra-galactic objects. The estimated amount of data that will be acquired over the 5 year lifespan of the project is 15TB, however, the total amount of storage space required for object informational databases, corrected frames, and reduced spectra will be several factors more than this. The goal is to have all the data online and available to the collaborators at all times. To accomplish this goal we are using commodity, off the shelf (COTS) Intel servers with EIDE disks configured as RAID50 arrays using XFS. Currently, 14 machines are in production accounting for over 18TB. By the scheduled end of the survey in 2005, 50TB of XFS disks will be online serving SDSS data to collaborators and the public."

"For complete details and status of the project please see [http://www.sdss.org/ http://www.sdss.org]. For details of the storage systems, see the [http://home.fnal.gov/~yocum/storageServerTechnicalNote.html SDSS Storage Server Technical Note]."

== [http://www-d0.fnal.gov/ The DØ Experiment at Fermilab] ==

"At the DØ experiment at the Fermi National Accelerator Laboratory we have a ~150 node cluster of desktop machines all using the SGI-patched kernel. Every large disk (>40Gb) or disk array in the cluster uses XFS including 4x640Gb disk servers and several 60-120Gb disks/arrays. Originally we chose reiserfs as our journaling filesystem, however, this was a disaster. We need to export these disks via NFS and this seemed perpetually broken in 2.4 series kernels. We switched to XFS and have been very happy. The only inconvenience is that it is not included in the standard kernel. The SGI guys are very prompt in their support of new kernels, but it is still an extra step which should not be necessary."

== [http://www.ciprico.com/pDiMeda.shtml Ciprico DiMeda NAS Solutions] ==

"The Ciprico DiMeda line of Network Attached Storage solutions combine the ease of connectivity of NAS with the SAN like performance levels required for digital media applications. The DiMeda 3600 provides high availability and high performance through dual NAS servers and redundant, scalable Fibre Channel RAID storage. The DiMeda 1700 provides high performance files services at a low price by using the latest Serial ATA RAID technology. All DiMeda systems are based on Linux and use XFS as the filesystem. We tested a number of filesystem alternatives and XFS was chosen because it provided the highest performance in digital media applications and the journaling feature ensures rapid failover in our dual node fault tolerant configurations."

== [http://www.quantum.com/Products/NAS+Servers/Guardian+14000/Default.htm The Quantum Guardian™ 14000] ==

"The Quantum Guardian™ 14000, the latest Network Attached Storage (NAS) solution from Quantum, delivers 1.4TB of enterprise-class storage for less than $25,000. The Guardian 14000 is a Linux-based device which utilizes XFS to provide a highly reliable journaling filesystem with simultaneous support for Windows, UNIX, Linux and Macintosh environments. As dedicated appliance optimized for fast, reliable file sharing, the Guardian 14000 combines the simplicity of NAS with a robust feature set designed for the most demanding enterprise environments. Support for tools such as Active Directory Service (ADS), UNIX Network Information Service (NIS) and Simple Network Management Protocol (SNMP) provides ease of management and seamless integration. Hardware redundancy, Snapshots and StorageCare™ on-site service ensure security for business-critical data."

== [http://www.bigstorage.com/products_approach_overview.html BigStorage K2~NAS] ==

"At BigStorage we pride ourselves on tailoring our NAS systems to meet our customer's needs, with the help of XFS we are able to provide them with the most reliable Journaling Filesystem technology available. Our open systems approach, which allows for cross-platform integration, gives our customers the flexibility to grow with their data requirements. In addition, BigStorage offers a variety of other features including total hardware redundancy, snapshotting, replication and backups directly from the unit. All of our products include BigStorageï¿½s 24/7 LiveResponse™ support. With LiveResponse™, we keep our team of experienced technical experts on call 24 hours a day, every day, to ensure that your storage investment remains online, all the time."

== [http://www.echostar.com Echostar DishPVR 721] ==

"Echostar uses the XFS filesystem for its latest generation of satellite receivers, the DP721. Echostar chose XFS for its performance, stability and unique set of features."

"XFS allowed us to meet a demanding requirement of recording two mpeg2 streams to the internal hard drive while simultaneously viewing a third pre-recorded stream. In addition, XFS allowed us to withstand unexpected power loss without filesystem corruption or user interaction."

"We tested several other filesystems, but XFS emerged as the clear winner."

== [http://www.sun.com/hardware/serverappliances/raq550/ Sun Cobalt RaQ™ 550] ==

From the [http://www.sun.com/hardware/serverappliances/raq550/features.html features] page:

"XFS is a journaling file system capable of quick fail over recovery after unexpected interruptions. XFS is an important feature for mission-critical applications as it ensures data integrity and dramatically reduces startup time by avoiding FSCK delay."

== [http://pingu.salk.edu/ Center for Cytometry and Molecular Imaging at the Salk Institute] ==

"I run the Center for Cytometry and Molecular Imaging at the Salk Institute in La Jolla, CA. We're a core facility for the Institute, offering flow cytometry, basic and deconvolution microscopy, phosphorimaging (radioactivity imaging) and fluorescent imaging."

"I'm currently in the process of migrating our data server to Linux/XFS. Our web server currently uses Linux/XFS. We have about 60 Gb on the data server which has a 100Gb SCSI RAID 5 array. This is a bit restrictive for our microscopists so in order that they can put more data online, I'm adding another machine, also running Linux/XFS, with about 420 Gb of IDE-RAID5, based on Adaptec controllers...."

"Servers are configured with quota and run Samba, NFS, and Netatalk for connectivity to the mixed bag of computers we have around here. I use the CVS XFS tree most of the time. I have not seen any problems in the several months I have been testing."

== [http://coltex.nl/ Coltex Retail Group BV] ==

"Coltex Retail group BV in the Netherlands uses Red Hat Linux with XFS for their main database server which collects the data from over 240 clothing retail stores throughout the Netherlands. Coltex depends on the availability of the server for over 100 hundred employees in the main office for retrieval of logistical and sales figures. The database size is roughly 10GB large containing both historical and current data."

"The entire production and logistical system depends on the availability of the system and downtime would mean a significant financial penalty. The speed and reliability of the XFS filesystem which has a proven track record and mature tools to go with it is fundamental to the availability of the system."

"XFS has saved us a lot of time during testing and implementation. A long filesystems check is no longer needed when bad things happen when they do. The increased speed of our database system which is based on Progress 9.1C is also a nice benefit to this filesystem."

== [http://www.dkp.com/ DKP Effects] ==

"We're a 3D computer graphics/post-production house. We've currently got four fileservers using XFS under Linux online - three 350GB servers and one 800GB server. The servers are under fairly heavy load - network load to and from the dual NICs on the box is basically maxed out 18 hours a day - and we do have occasional lockups and drive failures. Thanks to Linux SW RAID5 and XFS, though, we haven't had any data loss, or significant down time."

== [http://www.epigenomics.com/ Epigenomics] ==

"We currently have several IDE-to-SCSI-RAID systems with XFS in production. The largest has a capacity of 1.5TB, the other 2 have 430GB each."

"Data stored on these filesystems is on the one hand "normal" home directories and corporate documents and on the other hand scientific data for our laboratory and IT department."

== [http://www.incyte.com/ Incyte Genomics] ==

"I'm currently in the process of slowly converting 21 clusters totaling 2300+ processors over to XFS."

"These machines are running a fairly stock RH7.1+XFS. The application is our own custom scheduler for doing genomic research. We have one of the worlds largest sequencing labs which generates a tremendous amount of raw data. Vast amounts of CPU cycles must be applied to it to turn it into useful data we can then sell access to."

"Currently, a minority of these machines are running XFS, but as I can get downtime on the clusters I am upgrading them to 7.1+XFS. When I'm done, it'll be about 10TB of XFS goodness... across 9G disks mostly."

== [http://www.monmouth.edu/ Monmouth University] ==

"We've replaced our NetApp filer (80GB, $40,000). NetApp ONTAP software [runs on NetApp filers] is basically an NFS and CIFS server with their own proprietary filesystem. We were quickly running out of space and our annual budget almost depleted. What were we to do?"

"With an off-the-shelf Dell 4400 series server and 300GB of disks ($8,000 total). We were able to run Linux and Samba to emulate a NetApp filer."

"XFS allowed us to manage 300GB of data with absolutely no downtime (now going on 79 days) since implementation. Gone are the days of fearing the fsck of 300GB."

== [http://www.astro.wisc.edu The University of Wisconsin Astronomy Department] ==

"At the University of Wisconsin Astronomy Department we have been using Linux XFS since the first release. We currently have 31 Linux boxes running XFS on all filesystems with about 2.6 TB of disk space on these machines. We use XFS primarily on our data reduction systems, but we also use it on our web server and on one of the remote observing machines at the WIYN 3.5m Telescope at Kitt Peak (http://www.noao.edu/wiyn/wiyn.html)."

"We will likely be using Linux XFS at least in part on the GLIMPSE program (http://www.astro.wisc.edu/sirtf/) which will likely require several TB of disk space to process the data."

== [http://www.amoa.org/ The Austin Museum of Art] ==

"The Austin Museum of Art has two file servers running RedHat 7.2_XFS upgraded from RedHat 7.1_XFS. Our webserver runs Domino on top of RedHat 7.3_XFS and we're getting about 70% better performance than the Domino server running on Windows 2000 Server. We're moving our workstations away from Windows and Microsoft Office to an LTSP server running on RedHat 7.3_XFS.

"We've become solely dependent on XFS for all of our data systems."

== [http://www.tecmath.com/ tecmath AG] ==

"We use a production server with a 270 GB RAID 5 (hardware) disk array. It is based on a Suse 7.2 distribution, but with a standard 2.4.12 kernel with XFS and LVM patches. The server provides NFS to 8 Unix clients as well as Samba to about 80 PCs. The machine also runs Bind 9, Apache, Exim, DHCP, POP3, MySQL. I have tried out different configurations with ReiserFS, but I didn't manage to find a stable configuration with respect to NFS. Since I converted all disks to XFS some 3 months ago, we never had any filesystem-related problems."

== [http://www.theiqgroup.com/ The IQ Group] ==

"Here at the IQ Group, Inc. we use XFS for all our production and development servers."

"Our OS of choice is Slackware Linux 8.0. Our hardware of choice is Dell and VALinux servers."

"As for applications, we run the standard Unix/Linux apps like Sendmail, Apache, BIND, DHCP, iptables, etc.; as well as Oracle 9i and Arkeia."

"We've been running XFS across the board for about 3 months now without a hitch (so far)."

"Size-wise, our biggest server is about 40 GB, but that will be increasing substantially in the near future."

"Our production servers are collocated so a journaled FS was a must. Reboots are quick and no human interaction is required like with a bad fsck on ext2. Additionally, our database servers gain additional integrity and robustness."

"We originally chose XFS over ReiserFS and ext3 because of it's age (it's been in production on SGI boxes for probably longer than all the other journaling FS's combined) and it's speed appeared comparable as well."

== [http://www.artsit.usyd.edu.au Arts IT Unit, Sydney University] ==

"I've got XFS on a 'production' file server. The machine could have up to 500 people logged in, but typically less than 200. Most are Mac users, connected via NetAtalk for 'personal files', although there are shared areas for admin units. Probably about 30-40 windows users. (Samba) It's the file server for an Academic faculty at a University."

"Hardware RAID, via Mylex dual channel controller with 4 drives, Intel Tupelo MB, Intel 'SC5000' server chassis with redundant power and hot-swap scsi bays. The system boots off a non RAID single 9gb UW-scsi drive."

"Only system 'crash' was caused by some one accidentally unplugging it, just before we put it into production. It was back in full operation within 5 minutes. Without journaling, the fsck would have taken well over an hour. In day to day use it has run well."

== [http://structbio.vanderbilt.edu/comp/ Vanderbilt University Center for Structural Biology] ==

"I run a high-performance computing center for Structural Biology research at Vanderbilt University. We use XFS extensively, and have been since the late prerelease versions. I've had nothing but good experiences with it."

"We began using XFS in our search for a good solution for our RAID fileservers. We had such good experiences with it on these systems that we've begun putting it on the root/usr/var partitions of every Linux system we run here. I even have it on my laptop these days. XFS in combination with the 2.4 NFS3 implementation performs very well for us, and we have great uptimes on these systems (Our 750GB ArenaII setup is at 143 days right now)."

"All told, we've got about 1.2TB of XFS filesystems spinning right now. It's spread out across maybe a dozen or so filesystems and will continue to increase as we are growing fast and that's all we use now. Next up is putting it on our 17-node Linux cluster, which will bring that up to 1.5TB spread across 30 filesystems."

"I, for one, would LOVE to see XFS make it into the kernel tree. From my perspectives, it's one of the best things to happen to Linux in the 7 years I've been using/administering it."

==== 2008 Update ====

"We've since moved our main home directories to a proprietary NAS, but continue to use XFS on 10TB of LVM storage for doing backup-to-disk from the same NAS"

== [http://www-cdf.fnal.gov/ CDF Experiment at Fermi National Lab] ==

CDF, an elementary particle physics experiment at Fermi National Lab, is using XFS for all our cache disks.

The usage model is that we have a PB tape archive (2 STK silos) as permanent storage. In front of this archive we are deploying a roughly 100TB disk cache system. The cache is made up of 50 2TB file server based on cheap commodity hardware (3ware based hardware raid using IDE drives). The data is then processed by a cluster of 300 Dual CPU Linux PC's. The cache software is dCache, a DESY/FNAL product.

The whole system is used by more than 300 active users from all over the world for batch processing for their physics data analysis.

== [http://www.get2chip.com Get2Chip, Inc.] ==

"We are using XFS on 3 production file servers with approximately 1.5T of data. Quite impressive especially when we had a power outage and all three servers shutdown. All servers came back up in minutes with no problems! We are looking at creating two more servers that would manage 2+ TB of data store."

== [http://www.lando.co.za Lando International Group Technologies] ==

Lando International Group Technologies is the home of:

* [www.lando.co.za Lanndo Technologies Africa (Pty) Ltd] - Internet Service Provider
* [www.lbsd.net Linux Based Systems Design] (Article 21). Not-For-Profit company established to provide free Linux distributions and programs.
* Cell Park South Africa (Pty) Ltd. RSA Pat Appln 2001/10406. Collecting parking fees by means of cell phone SMS or voice.
* Read Plus Education (Pty) Ltd. Software based reading skills training and testing for ages 4 to 100.
* Mobivan. Mobile office including Internet access, fax, copying, printing, telephone, collection and delivery services, legal services, pre-paid phone and electricity services, bill payment email, secretarial services, training facilities and management services.
* Lando International Marketing Agency. Direct marketing services, design and supply of promotional material, consulting, sourcing of capital and other funding.
* Illico. Software development and systems analysis on most platforms.

"Throughout these companies, we use the XFS filesystem with [http://idms.lbsd.net IDMS Linux] on high-end Intel servers, with an average of 100 GB storage each. XFS stores our customer and user data, including credit card details, mail, routing tables, etc.. We have not had one problem since the release of the first XFS patch."

== [http://www.fcb-wilkens.com Foote, Cone, & Belding] ==

"We are an advertisement company in Germany, and the use of the XFS filesystem is a story of success for us. In our Hamburg office, we have two file servers having a 420 Gig RAID in XFS format serving (almost) all our data to about 180 Macs and about 30 PCs using Samba and Netatalk. Some of the data is used in our offices in Frankfurt and Berlin, and in fact the Berlin office is just getting it's own 250 Gig fileserver (using XFS) right now."

"The general success with XFS has led us to switch over all our Linux servers to run on XFS as well (with the exception of two systems that are tied to tight specifications configuration wise). XFS, even the old 1.0 version, has happily taken on various abuse - broken SCSI controllers, broken RAID systems."

== [http://www.moving-picture.co.uk/ Moving Picture Company] ==

"We here at MPC use XFS/RedHat 7.2 on all of our graphics-workstations and file-servers. More info can be found in an [http://www.linuxuser.co.uk/articles/issue20/lu20-Linux_at_work-In_the_picture.pdf article] LinuxUser magazine did on us recently."

== [http://www.coremetrics.com/ Coremetrics, Inc.] ==

"We are currently using XFS for 25+ production web-servers, ~900GB Oracle db servers, with potentially 15+ more servers by mid 2003, with ~900GB+ databases. All XFS installed."

"Also, our dev environment, except for the Sun boxes which all are being migrated to X86 in the aforementioned server additions, plus the dev Sun boxes as well, are all x86 dual proc servers running Oracle, application servers, or web services as needed. All servers run XFS from images we've got on our SystemImager servers."

"All production back-end servers are connected via FC1 or FC2 to a SAN containing ~13TB of raw storage, which, will soon be converted from VxFS to XFS with the migration of Oracle to our x86 platforms."

== [http://evolt.org Evolt.org] ==

"evolt.org, a world community for web developers promoting the mutual free exchange of ideas, skills and experiences, has had a great deal of success using XFS. Our primary webserver which serves 100K hosts/month, primary Oracle database with ~25Gb of data, and free member hosting for 1000 users haven't had a minute of downtime since XFS has been installed. Performance has been spectacular and maintenance a breeze."

 ''All testimonials on this page represent the views of the submitters, and references to other products and companies should not be construed as an endorsement by either the organizations profiled, or by SGI. All trademarks (r) their respective owners.''

XFS Papers and Documentation

2010-08-23T15:23:04Z

Sandeen:

=== Primary XFS Documentation ===

The XFS documentation started by SGI has been converted to docbook/[https://fedorahosted.org/publican/ Publican] format. The material is suitable for experienced users as well as developers and support staff. The XML source is available in a [http://git.kernel.org/?p=fs/xfs/xfsdocs-xml-dev.git;a=summary git repository] and builds of the documentation are available here:

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/index.html XFS User Guide]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/index.html XFS File System Structure]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Labs/tmp/en-US/html/index.html XFS Training Labs]

* (Original versions of this material are still available at [http://oss.sgi.com/projects/xfs/training/index.html XFS Overview and Internals (html)] and [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS Filesystem Structure (pdf)]

The format of <tt>/proc/fs/xfs/stat</tt> also has been documented:
* [[Runtime_Stats|Runtime_Stats]]

=== Papers, Presentations, Etc ===

The October 2009 issue of the USENIX ;login: magazine published an article about XFS targeted at system administrators:

* ''XFS: The big storage file system for Linux'' [[http://oss.sgi.com/projects/xfs/papers/hellwig.pdf pdf]]

At the Ottawa Linux Symposium (July 2006), Dave Chinner presented a paper on filesystem scalability in Linux 2.6 kernels:

* ''High Bandwidth Filesystems on Large Systems'' (July 2006) [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-paper.pdf paper]] [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-presentation.pdf presentation]]

At linux.conf.au 2008 Dave Chinner gave a presentation about xfs_repair that he co-authored with Barry Naujok:

* Fixing XFS Filesystems Faster [[http://mirror.linux.org.au/pub/linux.conf.au/2008/slides/135-fixing_xfs_faster.pdf]]

In July 2006, SGI storage marketing updated the XFS datasheet:

* ''Open Source XFS for Linux'' [[http://oss.sgi.com/projects/xfs/datasheet.pdf pdf]]

At UKUUG 2003, Christoph Hellwig presented a talk on XFS:

* ''XFS for Linux'' (July 2003) [[http://oss.sgi.com/projects/xfs/papers/ukuug2003.pdf pdf]] [[http://verein.lst.de/~hch/talks/ukuug2003/ html]]

Originally published in Proceedings of the FREENIX Track: 2002 Usenix Annual Technical Conference:

* ''Filesystem Performance and Scalability in Linux 2.4.17'' (June 2002) [[http://oss.sgi.com/projects/xfs/papers/filesystem-perf-tm.pdf pdf]]

At the Ottawa Linux Symposium, an updated presentation on porting XFSÂ to Linux was given:

* ''Porting XFS to Linux'' (July 2000) [[http://oss.sgi.com/projects/xfs/papers/ols2000/ols-xfs.htm html]]

At the Atlanta Linux Showcase, SGI presented the following paper on the port of XFS to Linux:

* ''Porting the SGI XFS File System to Linux'' (October 1999) [[http://oss.sgi.com/projects/xfs/papers/als/als.ps ps]] [[http://oss.sgi.com/projects/xfs/papers/als/als.pdf pdf]]

At the 6th Linux Kongress & the Linux Storage Management Workshop (LSMW) in Germany in September, 1999, SGI had a few presentations including the following:

* ''SGI's port of XFS to Linux'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/linux_kongress/index.htm html]]
* ''Overview of DMF'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/DMF-over/index.htm html]]

At the LinuxWorld Conference & Expo in August 1999, SGI published:

* ''An Open Source XFS data sheet'' (August 1999) [[http://oss.sgi.com/projects/xfs/papers/xfs_GPL.pdf pdf]]

From the 1996 USENIX conference:

* ''An XFS white paper'' [[http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html html]]

=== Other historical articles, press-releases, etc ===

* IBM's ''Advanced Filesystem Implementor's Guide'' has a chapter ''Introducing XFS'' [[http://www-106.ibm.com/developerworks/library/l-fs9.html html]]

* An editorial titled ''Tired of fscking? Try a journaling filesystem!'', Freshmeat (February 2001) [[http://freshmeat.net/articles/view/212/ html]]

* ''Who give a fsck about filesystems'' provides an overview of the Linux 2.4 filesystems [[http://www.linuxuser.co.uk/articles/issue6/lu6-All_you_need_to_know_about-Filesystems.pdf html]]

* ''Journal File Systems'' in issue 55 of ''Linux Gazette'' provides a comparison of journaled filesystems.

* The original XFS beta release announcement was published in ''Linux Today'' (September 2000) [[http://linuxtoday.com/news_story.php3?ltsn=2000-09-26-017-04-OS-SW html]]

* ''XFS: It's worth the wait'' was published on ''EarthWeb'' (July 2000) [[http://networking.earthweb.com/netos/oslin/article/0,,12284_623661,00.html html]]

* An ''IRIX-XFS data sheet'' (July 1999) [[http://oss.sgi.com/projects/xfs/papers/IRIX_xfs_data_sheet.pdf pdf]]

* The ''Getting Started with XFS'' book (1994) [[http://oss.sgi.com/projects/xfs/papers/getting_started_with_xfs.pdf pdf]]

* Original ''XFS design documents'' (1993) ([http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_ps/ ps], [http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_pdf/ pdf])

XFS Papers and Documentation

2010-08-23T15:01:46Z

Sandeen:

=== Primary XFS Documentation ===

The XFS documentation started by SGI has been converted to docbook/[https://fedorahosted.org/publican/ Publican] format. The material is suitable for experienced users as well as developers and support staff. The XML source is available in a [http://git.kernel.org/?p=fs/xfs/xfsdocs-dev.git;a=summary git repository] and builds of the documentation are available here:

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/index.html XFS User Guide]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/index.html XFS File System Structure]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Labs/tmp/en-US/html/index.html XFS Training Labs]

* (Original versions of this material are still available at [http://oss.sgi.com/projects/xfs/training/index.html XFS Overview and Internals (html)] and [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS Filesystem Structure (pdf)]

The format of <tt>/proc/fs/xfs/stat</tt> also has been documented:
* [[Runtime_Stats|Runtime_Stats]]

=== Papers, Presentations, Etc ===

The October 2009 issue of the USENIX ;login: magazine published an article about XFS targeted at system administrators:

* ''XFS: The big storage file system for Linux'' [[http://oss.sgi.com/projects/xfs/papers/hellwig.pdf pdf]]

At the Ottawa Linux Symposium (July 2006), Dave Chinner presented a paper on filesystem scalability in Linux 2.6 kernels:

* ''High Bandwidth Filesystems on Large Systems'' (July 2006) [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-paper.pdf paper]] [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-presentation.pdf presentation]]

At linux.conf.au 2008 Dave Chinner gave a presentation about xfs_repair that he co-authored with Barry Naujok:

* Fixing XFS Filesystems Faster [[http://mirror.linux.org.au/pub/linux.conf.au/2008/slides/135-fixing_xfs_faster.pdf]]

In July 2006, SGI storage marketing updated the XFS datasheet:

* ''Open Source XFS for Linux'' [[http://oss.sgi.com/projects/xfs/datasheet.pdf pdf]]

At UKUUG 2003, Christoph Hellwig presented a talk on XFS:

* ''XFS for Linux'' (July 2003) [[http://oss.sgi.com/projects/xfs/papers/ukuug2003.pdf pdf]] [[http://verein.lst.de/~hch/talks/ukuug2003/ html]]

Originally published in Proceedings of the FREENIX Track: 2002 Usenix Annual Technical Conference:

* ''Filesystem Performance and Scalability in Linux 2.4.17'' (June 2002) [[http://oss.sgi.com/projects/xfs/papers/filesystem-perf-tm.pdf pdf]]

At the Ottawa Linux Symposium, an updated presentation on porting XFSÂ to Linux was given:

* ''Porting XFS to Linux'' (July 2000) [[http://oss.sgi.com/projects/xfs/papers/ols2000/ols-xfs.htm html]]

At the Atlanta Linux Showcase, SGI presented the following paper on the port of XFS to Linux:

* ''Porting the SGI XFS File System to Linux'' (October 1999) [[http://oss.sgi.com/projects/xfs/papers/als/als.ps ps]] [[http://oss.sgi.com/projects/xfs/papers/als/als.pdf pdf]]

At the 6th Linux Kongress & the Linux Storage Management Workshop (LSMW) in Germany in September, 1999, SGI had a few presentations including the following:

* ''SGI's port of XFS to Linux'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/linux_kongress/index.htm html]]
* ''Overview of DMF'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/DMF-over/index.htm html]]

At the LinuxWorld Conference & Expo in August 1999, SGI published:

* ''An Open Source XFS data sheet'' (August 1999) [[http://oss.sgi.com/projects/xfs/papers/xfs_GPL.pdf pdf]]

From the 1996 USENIX conference:

* ''An XFS white paper'' [[http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html html]]

=== Other historical articles, press-releases, etc ===

* IBM's ''Advanced Filesystem Implementor's Guide'' has a chapter ''Introducing XFS'' [[http://www-106.ibm.com/developerworks/library/l-fs9.html html]]

* An editorial titled ''Tired of fscking? Try a journaling filesystem!'', Freshmeat (February 2001) [[http://freshmeat.net/articles/view/212/ html]]

* ''Who give a fsck about filesystems'' provides an overview of the Linux 2.4 filesystems [[http://www.linuxuser.co.uk/articles/issue6/lu6-All_you_need_to_know_about-Filesystems.pdf html]]

* ''Journal File Systems'' in issue 55 of ''Linux Gazette'' provides a comparison of journaled filesystems.

* The original XFS beta release announcement was published in ''Linux Today'' (September 2000) [[http://linuxtoday.com/news_story.php3?ltsn=2000-09-26-017-04-OS-SW html]]

* ''XFS: It's worth the wait'' was published on ''EarthWeb'' (July 2000) [[http://networking.earthweb.com/netos/oslin/article/0,,12284_623661,00.html html]]

* An ''IRIX-XFS data sheet'' (July 1999) [[http://oss.sgi.com/projects/xfs/papers/IRIX_xfs_data_sheet.pdf pdf]]

* The ''Getting Started with XFS'' book (1994) [[http://oss.sgi.com/projects/xfs/papers/getting_started_with_xfs.pdf pdf]]

* Original ''XFS design documents'' (1993) ([http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_ps/ ps], [http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_pdf/ pdf])

XFS Papers and Documentation

2010-08-23T15:00:39Z

Sandeen:

The XFS documentation started by SGI has been converted to docbook/[https://fedorahosted.org/publican/ Publican] format. The material is suitable for experienced users as well as developers and support staff. The XML source is available in a [http://git.kernel.org/?p=fs/xfs/xfsdocs-dev.git;a=summary git repository] and builds of the documentation are available here:

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide//tmp/en-US/html/index.html XFS User Guide]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure//tmp/en-US/html/index.html XFS File System Structure]

* [http://xfs.org/docs/xfsdocs-xml-dev/XFS_Labs/tmp/en-US/html/index.html XFS Training Labs]

* (Original versions of this material are still available at [http://oss.sgi.com/projects/xfs/training/index.html XFS Overview and Internals (html)] and [http://oss.sgi.com/projects/xfs/papers/xfs_filesystem_structure.pdf XFS Filesystem Structure (pdf)]

The format of <tt>/proc/fs/xfs/stat</tt> also has been documented:
* [[Runtime_Stats|Runtime_Stats]]

The October 2009 issue of the USENIX ;login: magazine published an article about XFS targeted at system administrators:

* ''XFS: The big storage file system for Linux'' [[http://oss.sgi.com/projects/xfs/papers/hellwig.pdf pdf]]

At the Ottawa Linux Symposium (July 2006), Dave Chinner presented a paper on filesystem scalability in Linux 2.6 kernels:

* ''High Bandwidth Filesystems on Large Systems'' (July 2006) [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-paper.pdf paper]] [[http://oss.sgi.com/projects/xfs/papers/ols2006/ols-2006-presentation.pdf presentation]]

At linux.conf.au 2008 Dave Chinner gave a presentation about xfs_repair that he co-authored with Barry Naujok:

* Fixing XFS Filesystems Faster [[http://mirror.linux.org.au/pub/linux.conf.au/2008/slides/135-fixing_xfs_faster.pdf]]

In July 2006, SGI storage marketing updated the XFS datasheet:

* ''Open Source XFS for Linux'' [[http://oss.sgi.com/projects/xfs/datasheet.pdf pdf]]

At UKUUG 2003, Christoph Hellwig presented a talk on XFS:

* ''XFS for Linux'' (July 2003) [[http://oss.sgi.com/projects/xfs/papers/ukuug2003.pdf pdf]] [[http://verein.lst.de/~hch/talks/ukuug2003/ html]]

Originally published in Proceedings of the FREENIX Track: 2002 Usenix Annual Technical Conference:

* ''Filesystem Performance and Scalability in Linux 2.4.17'' (June 2002) [[http://oss.sgi.com/projects/xfs/papers/filesystem-perf-tm.pdf pdf]]

At the Ottawa Linux Symposium, an updated presentation on porting XFSÂ to Linux was given:

* ''Porting XFS to Linux'' (July 2000) [[http://oss.sgi.com/projects/xfs/papers/ols2000/ols-xfs.htm html]]

At the Atlanta Linux Showcase, SGI presented the following paper on the port of XFS to Linux:

* ''Porting the SGI XFS File System to Linux'' (October 1999) [[http://oss.sgi.com/projects/xfs/papers/als/als.ps ps]] [[http://oss.sgi.com/projects/xfs/papers/als/als.pdf pdf]]

At the 6th Linux Kongress & the Linux Storage Management Workshop (LSMW) in Germany in September, 1999, SGI had a few presentations including the following:

* ''SGI's port of XFS to Linux'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/linux_kongress/index.htm html]]
* ''Overview of DMF'' (September 1999) [[http://oss.sgi.com/projects/xfs/papers/DMF-over/index.htm html]]

At the LinuxWorld Conference & Expo in August 1999, SGI published:

* ''An Open Source XFS data sheet'' (August 1999) [[http://oss.sgi.com/projects/xfs/papers/xfs_GPL.pdf pdf]]

From the 1996 USENIX conference:

* ''An XFS white paper'' [[http://oss.sgi.com/projects/xfs/papers/xfs_usenix/index.html html]]

=== Other historical articles, press-releases, etc ===

* IBM's ''Advanced Filesystem Implementor's Guide'' has a chapter ''Introducing XFS'' [[http://www-106.ibm.com/developerworks/library/l-fs9.html html]]

* An editorial titled ''Tired of fscking? Try a journaling filesystem!'', Freshmeat (February 2001) [[http://freshmeat.net/articles/view/212/ html]]

* ''Who give a fsck about filesystems'' provides an overview of the Linux 2.4 filesystems [[http://www.linuxuser.co.uk/articles/issue6/lu6-All_you_need_to_know_about-Filesystems.pdf html]]

* ''Journal File Systems'' in issue 55 of ''Linux Gazette'' provides a comparison of journaled filesystems.

* The original XFS beta release announcement was published in ''Linux Today'' (September 2000) [[http://linuxtoday.com/news_story.php3?ltsn=2000-09-26-017-04-OS-SW html]]

* ''XFS: It's worth the wait'' was published on ''EarthWeb'' (July 2000) [[http://networking.earthweb.com/netos/oslin/article/0,,12284_623661,00.html html]]

* An ''IRIX-XFS data sheet'' (July 1999) [[http://oss.sgi.com/projects/xfs/papers/IRIX_xfs_data_sheet.pdf pdf]]

* The ''Getting Started with XFS'' book (1994) [[http://oss.sgi.com/projects/xfs/papers/getting_started_with_xfs.pdf pdf]]

* Original ''XFS design documents'' (1993) ([http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_ps/ ps], [http://oss.sgi.com/projects/xfs/design_docs/xfsdocs93_pdf/ pdf])

Shrinking Support

2008-12-21T18:36:17Z

Sandeen: minor cleanup

Currently XFS Filesystems can't be shrunk.

To support shrinking XFS filesystems a few things needs to be implemented, based on a list by Dave Chinner [http://marc.info/?l=linux-xfs&m=118091640624488&w=2]:

* A way to check space is available for shrink

* An ioctl or similar interface to prevent new allocations from a given allocation group.

* A variant of the xfs_reno tool to support moving inodes out of filesystem areas that go away.

* A variant of the xfs_fsr tool to support moving data out of the filesystem areas that go away.

* Some way to move out orphan metadata out of the AGs truncated off

* A transaction to shrink the filesystem.

At that point, we'll have a "working" shrink that will allow
shrinking to only 50% of the original size because the log
(in the middle of the filesystem) will
get in the way. To fix that, we'll need to implement transactions
to move the log...

== Available pieces ==

* A script from Ruben Porras to check if enough free space is available to support shrinking [[http://marc.info/?l=linux-xfs&m=118581682117599&w=2]]

* A patch from Ruben Porras to allow / disallow allocation from an allocation group [http://marc.info/?l=linux-xfs&m=118302806818420&w=2] plus userspace support for setting / clearing it [http://marc.info/?l=linux-xfs&m=118881137031101&w=2]

* The xfs_fsr tool in xfsprogs

* The xfs_reno too, see [[Unfinished_work#The_xfs_reno_tool]]

* An untested patch from Dave Chinner for a xfs_swap_inodes ioctl that allows to not just defragment extents but moving the whole inode [http://marc.info/?l=linux-xfs&m=119552278931942&w=2] and a patch to xfs_reno to use it from Ruben Porras [http://marc.info/?l=linux-xfs&m=119582841808985&w=2]

Shrinking Support

2008-12-21T18:34:34Z

Sandeen: remove dup bullet point

Currently XFS Filesystems can't be shrunk.

To suppor shrinking XFS filesystems a few thinks needs to be implemented, based on a list by Dave Chinner [http://marc.info/?l=linux-xfs&m=118091640624488&w=2]:

* A way to check space is available for shrink

* An ioctl or similar interface to prevent new allocations from a given allocation group.

* A variant of the xfs_reno tool to support moving inodes out of filesystem areas that go away.

* A variant of the xfs_fsr tool to support moving data out of the filesystem areas that go away.

* Some way to move out orphan metadata out of the AGs truncated off

* A transaction to shrink the filesystem.

At that point, we'll got a "working" shrink that will allow
shrinking to only 50% of the original size because the log will
get in the way. To fix that, we'll need to implement transactions
to move the log...

== Available pieces ==

* A script from Ruben Porras to check if enough free space is available to support shrinking [[http://marc.info/?l=linux-xfs&m=118581682117599&w=2]]

* A patch from Ruben Porras to allow / disallow allocation from an allocation group [http://marc.info/?l=linux-xfs&m=118302806818420&w=2] plus userspace support for setting / clearing it [http://marc.info/?l=linux-xfs&m=118881137031101&w=2]

* The xfs_fsr tool in xfsprogs

* The xfs_reno too, see [[Unfinished_work#The_xfs_reno_tool]]

* An untested patch from Dave Chinner for a xfs_swap_inodes ioctl that allows to not just defragment extents but moving the whole inode [http://marc.info/?l=linux-xfs&m=119552278931942&w=2] and a patch to xfs_reno to use it from Ruben Porras [http://marc.info/?l=linux-xfs&m=119582841808985&w=2]

Xfs.org:About

2008-12-21T17:39:06Z

Sandeen: Removing all content from page