Notes about Linux filesystems

File system hints generic (20210419)

Release dependent:

• Linux 5.12: added id-mapping for files.
• Linux 5.10: F2FS 5.10 has a file-corrupting bug for inline data that needs a patch.
• Linux 4.10: F2FS support for zoned (SMR host managed) HDDs.

File system hints for F2FS (20221102)

• Support for variable length inline extended attributes added in 2017.

File system hints for JFS (121226)

• Support from TRIM only from kernel version 3.7 and later.
• No support for barriers; but the flush interval to the journal is very short.
• JFS in kernel version 2.6.8 has a singificant memory leak.
• JFS can handle bad blocks only when a filetree is created, additional ones cannot be handled.
• The journal can be disabled for fast writing, but disabling the journal is not safe and should not be used for anything other than reloading backups.
• Since each growing extent is allocated space from an allocation group, and each allocation group will only allocate space to a single extents, if the number of extents (or files) being grown is greater than that of allocation groups some processes will block.

File system hints for ext3 (120304)

• An ext3 filetree can get very fragmented.
• Creating a large filetree can take a long time to initialize block groups.
• Timestamps (including modification time) have a granularity of 1 second, which means that multiple updates per second are not recorded. This can impact make processing and fsync.
• When fsync is issued all outstanding updates in the journal are written out (in some popular cases), making it a very expensive operation.
• Flushing for memory to disk is every 5 seconds by default, which can be too frequent, and can mask the lack of fsync in applications.
• Older kernels (such as in RHEL5) support only filetrees with 128B inodes, but newer tools create filetrees with 256B inodes by default.
• Older versions of GRUB can only read filetrees with 128B inodes.
• Maximum filetree size is 8TiB.
• Support for only 32k subdirectories in a directory.
• Support for only 32k hard links to a file.
• Support and TRIM and FSTRIM only from kernel version 2.6.36.
• Checking a damaged filesystem can take months.
• Directory indices or ACL blocks can be allocated away from directory data and lead to terrible performance.

File system hints for ext4 (120922)

• In-place conversion from ext3 leaves existing files allocated as they were.
• Timestamps (including modification time) have a granularity of 1 second if the older ext3 compatible inode size of 128 bytes is used or kept. If so, multiple updates per second are not recorded. This can impact make processing and fsync.
• Flushing to disk is much less frequent than for ext3, which increases the chances of data loss unless barriers are enabled and fsync is used by applications.
• It is possible to create ext4 filetrees larger than 8TiB but this requires a recent kernel with a page cache that supports that.
• It is possible to create ext4 filetrees larger than 16TiB but this requires not just a recent kernel but also mke2fs 1of 1.42 or newer.
• Online resize of ext4 filetree from less than 16TiB to more than 16TiB is possible from kernel release 3.3 but only if the filetree was created with 64 bit offsets.
• Since kernel version 3.5 ext4 metadata is optionally checksummed.

File system hints for XFS (140925)

Kernel version dependent hints:

• 3.17 fixes a long standing bug that means that by default when a filetree space is increased any newly created allocation groups are not used for inode allocations. A workaround is to remount with first with option inode32 and then again remount with option inode64.
• 3.4.5: fixes a delayed segment allocation issue.
• 3.2.x: bug that causes needless wakeups in xfsaild.
• 3.2.x: Bug of double unlocking of the ilock.
• 3.2.12 and older: The default i/o scheduler, CFQ, will defeat much of the parallelization in XFS.
• 2.6.39 and newer: Support for TRIM and FSTRIM.
• 2.6.37: There a known bug in the VFS which impacts XFS.
• 2.6.35 and older: Mounting a filesystem that was mounted with inode64 without it can cause problems.
• 2.6.32 and newer: The su and sw parameters are obtained automatically from Linux MD, with xfsprogs 3.1.1 or newer.
• 2.6.27: Case insensitive filenames.
• 2.6.27.3: Fixes for some regressions.
• 2.6.21 and older: On crash a file can have nulls inside.
• 2.6.17 (point versions up to .6): Bug that leads to directory corruption.
• 2.6.17 and newer: Barriers are enabled by default.

Kernel version independent hints:

• Since the superblock is at sector 0 of the filetree volume, one cannot have a partition boot loader or other metadata in the same volume.
• XFS does not handle bad blocks at all.
• Applications that don't issue fsync can get files full of zero blocks if there is a crash. This is not an issue with XFS, but with the applications.
• In 32b mode it is possible to use a filetree that is larger than fsck can repair.
• In 32b mode or without the inode64 option inodes will only be allocated in the first 1TiB (if the sector size is 512B) of space.
• In 32b mode with 4KiB kernel stacks there is a strong possibility of stack overflow, and a near certainty if exported by NFS.
• In 64b mode kernel stacks are 8KiB by default, but there is still some possibility of stack overflow with NFS exports, especially on top od DM/LVM2.
• The inode64 option rotors directories across AGs, and then attempts to allocate space for new files in the AG containing the directory, which is quite different from the alternative because if you create a bunch of files in the same directory, without inode64 XFS will scatter the extents all over the disk rather than trying to allocate them next to each other.
• When a file is opened the entire list of extents is loaded and kept in memory, which can consume a lot of memory for highly fragmented files.
• Larger inode sizes can store more extended attributes and inode extents maps for greater efficiency.
• Up to 64KiB of extended attributes are supported.
• Filetree freezing hangs all applications accessing files in that filetree.
• It is possible to move an internal journal to an external journal, but not viceversa, unless the filetree had an internal journal to start with.
• Since metadata access is serialized by allocation group, if all allocation groups are in use to grown extents writing can stop for all other files, or similarly if the files are in the same allocation group. Having more allocations groups typically improves multithreaded performance.
• If the XFS allocation group size is a multiple of the underlying RAID stripe then the allocation groups and (and their metadata) may end up on the same disks, preventing parallel IO across the stripe. If mkfs.xfs can discover the underlying RAID geometry it will warn about this with the message:

  Warning: AG size is a multiple of stripe width. This can cause performance problems by aligning all AGs on the same disk. To avoid this, run mkfs with an AG size that is one stripe unit smaller, for example %llu.

  The solution indicated is to manually specify an allocation group size that is not congruent with the stripe width, usually a bit smaller.
• Configuring an external journal will disable XFS' write barrier support.

File system hints for Btrfs

Inside the Btrfs notes page.

File system hints for ZFS (201213)

• The arecord must match the largest record size.
• Vdevs cannot be changed or expanded or deleted.
• All IO is done by at least a full recordsize, regardless of how much is actually read.
• Removing snapshots can be very, very slow.

• ZFS 2.0 (2020) has been renamed as OpenZFS and can have a persistent cache, the Zstd algorithm, and a PAM module for file based encryption.
• ZFS 0.6.4 (2015) has redundant_metadata and it is set by default to all. It also has as other notable new features LZ4 compression and bookmarks.
• In Ubuntu LTS 16, hot spares don't work.
• In older FreeBSD versions spares can become accidentally UNAVAIL, to fix detach and reattach them.

File system hints for NFS (210203)

Release dependent:

• NFS protocol versions up to NFSv3 only report timestamps, including modification times, with a granularity of 1 second, even if the exported filesystem has a finer timestamp granularity.
• The Linux NFS client does not do incremental flushing, but it issues COMMIT packets with no range, that flush all pending writes.
• NFS over TCP will not restart a half-open connection.
• The NFSv4 exports can all be under the same directory, and that directory must be exported with fsid=0, and the exported filetree paths do not contain the name of that directory. But this is optional and exports can be done as separate filetrees too.
• Using des-cbc-crc:normal keytab entries with newer versions of Kerberos may require editing /etc/krb5.conf.
• NFSv4 ID mapping must be enabled, and the mapping domain must be explicitly set and exactly the same between client and host. It does not need to be the DNS domain, but conventionally it is.
• NFSv4 ID mapping with Kerberos authentication only works if the undocumented variable Local-Realm is set and must be the same between client and host, and must be the name of the Kerberos realm.
• When mounting with Kerberos authentication the name given for the server must be identical to that of the service principal for the server, and must be a canonical name of the server, that is the address it resolves to must resolve back to the same name, perhaps because of a bug in GNU LIBC which affects the Kerberos library.

Release dependent:

• Fixed from 4.14 and 4.19: in some cases where programs rename an NFS file, under NFS 4.0 but not 4.1, the Linux NFS client does not revalidatea a file handle resulting in some programs reporting Stale file error.
• In kernel 2.6.32 there is a bug in UDP offloading that causes freezes and corruption in NFS.
• In some recent versions of the Linux kernel's NFS client autotuning can result in instability:

  • a change in RHEL 6.3: sunrpc.tcp_max_slot_table_entries dynamically allocating RPC slots up to the maximum (65536).
  • Reverting to previous limit of 128 recovered system stability.

• Kerberos security export syntax must be done with a pseudo client of gss/krb5 or gss/krb5[ip] with kernel versions older than 2.6.23 or nfs-utils versions older than 1.11.
• NFSv4 with Kerberos can only use des-cbc-crc:normal (also in RHEL) keytab entries in kernels older than 2.6.35 thanks to a massive update to the Linux GSS support module.
  With kernel versions that only support des-cbc-crc enctypes if unsupported enctypes are used the GSS server dæmon will print debug message like:

  prepare_krb5_rfc_cfx_buffer: not implemented

Summary of conditions for a working NFSv4 with Kerberos GSSAPI authentication and/or encryption:

• The rpc_pipefs filesystem must be mounted on both fileserver and client, usually at /var/lib/nfs/rpc_pipepfs/ or /var/run/rpc_pipefs/ depending on distribution, and the Pipefs-Directory parameter in idmapd.conf must be set to that path.
• The id mapping parameter Domain in idmapd.conf must be set to the same value on client(s) and server(s), and usually is the lower case version of the relevant DNS domain name.
• The Local-Realm parameter in idmapd.conf must be set to the same value on client(s) and server(s) and must be the Kerberos realm name used for both.
• ID mapping in the kernel must be enabled with a command similar to:

  echo -n N > /sys/module/nfs/parameters/nfs4_disable_idmapping

• The rpcsec_gss_krb5 kernel module must be loaded.
• The file gssapi_mech.conf must list the gssapi_krb5 shared object with mechglue_internal_krb5_init as the initialization function.
• On all involved systems the /etc/krb5.keytab must have the relevant host and service entries:
  • The server must have in its keytab the keys for the nfs/ and host/ service principal for its canonical DNS name, unless the svcgss dæmon is configured otherwise.
  • Each client must have in its keytab the keys for the host/ and nfs/ service principal for its own canonical DNS name, unless the gss dæmon is configured to run with the -n option, or the gss dæmon is a new one that can use just the host/ service principal.
  The canonical DNS names for the machine should be all lowercase, and because of canonical name ambiguities using multiple IP addresses on a host may not work.
  To sort out canonicalization issues the /etc/hosts file or the DNS zone may need to be carefully edited.
• These dæmons should be running:
  • The usual rpcbind (or portmap) plus idmap on both server and client.
  • svcgss on the server, unless this is a recent versions of the NFS Linux utilities in which case it is not needed. Reading carefully the man page can be useful.
  • gss dæmon on the client. Reading carefully the man page can be useful.
• The exports file on the server must list the relevant filetrees as exported with one of the gss/ security types, and the exporting filetree must be mounted on the client using the same security type.
• Each user accessing a NFS filetree mounted with Kerberos security must have their own Kerberos principal, the nfs/ principal for the client is used only to mount the filetree, but does not grant any user on the client access to the files.
• If the mounting succeeds the Kerberos credential cache /tmp/krb5cc_machine_REALM will have a key from the local host's service principal and a key for the server's service principal.
• The type of salt should be normal unless using a kaserver in which case it should be afs3 and v4 only if relying on a Kerberos4 KDC.
• The Kerberos host and service keytab entries must have enctype des-cbc-src for kernel versions older than 2.6.35.

Some useful pages for using NFSv4 with Kerberos:

• http://wiki.linux-nfs.org/wiki/Nfsv4_configuration.
• http://www.citi.umich.edu/projects/nfsv4/linux/using-nfsv4.html.
• http://linuxcostablanca.blogspot.co.uk/2012/02/nfsv4-myths-and-legends.html.
• https://we.riseup.net/stefani/kerberos-and-nfs4.
• http://wiki.debian.org/NFS/Kerberos.
• http://wiki.linux-nfs.org/wiki/Enduser_doc_kerberos.
• http://www.itp.uzh.ch/~dpotter/howto/kerberos.
• http://sadiquepp.blogspot.co.uk/2009/02/how-to-configure-nfsv4-with-kerberos-in.html.
• http://wiki.linux-nfs.org/wiki/General_troubleshooting_recommendations.
• https://docs.fedoraproject.org/en-US/Fedora/17/html/FreeIPA_Guide/kerb-nfs.html.

File system hints for OpenAFS (130913)

Version dependent:

• From version 1.6.2 to 1.6.5 inclusive a bug in the new asynchronous IO code means that core dumps (and probably other kernel initiated IO) cannot be made to OpenAFS volumes.
• From OpenAFS versions 1.4.15, 1.6.5, 1.7.26 it is possible to use Kerberos keys with better enctypes than des-cbc-crc and des-cbc-md5, and they should be used because 56-bit DES encryption is quite easy to break.
• In OpenAFS up to and including 1.6.1 there is a bug with select and having more than 1024 file descriptors open that can cause memory corruption in the fileserver or the salvageserver; this often results in a hung salvageserver process and error messages like:

  Wed Dec 18 08:52:47 2013 SYNC_getCom:  error receiving command
  Wed Dec 18 08:52:47 2013 FSYNC_com:  read failed; dropping connection (cnt=8568)
  Wed Dec 18 08:52:47 2013 SYNC_getCom:  error receiving command
  Wed Dec 18 08:52:47 2013 FSYNC_com:  read failed; dropping connection (cnt=8569)
  mode c1ff
  6: dev 7, inode 25811, length 0, type/mode c1ff
  7: dev 7, inode 9532, length 0, type/mode c1ff
  8: dev 7, inode 21498, length 0, type/mode c1ff
  9: dev 905, inode 95580989, length 65, type/mode 8000
  10: dev 905, inode 94903997, length 2048, type/mode 8000
  11: dev 905, inode 97388566, length 8448, type/mode 8003

  in the FileLog.
  The easiest solution is to put the line

  ulimit -n 1021

  in the script that starts BOS. The best solution is to upgrade to the latest release, as 1.6.1 has this and other known issues.
  This bug affects the Debian 7/Wheezy packages for OpenAFS at least up to version 1.6.1-3+deb7u1. There are version Debian 7/Wheezy 1.6.5.1 packages in the wheezy-backports archive.
• In OpenAFS version 1.6.0 a bug can lead to extreme overpinging of file servers.
• AFS protocol encryption is available in OpenAFS from version 1.4.11 or from version 1.5.60, and probably it has always been in OpenAFS.
• OpenAFS versions 1.6 and newer can put the cache on any filesystem as the relevant code has been rewritten. However it is useful to have the cache on a filetree without a journal, as the cache is ephemeral.
• The client cache and the partitions up to and including OpenAFS version 1.4 must be on an ext2 or ext3 filesystem.
• OpenAFS with Kerberos can only use des-cbc-crc:normal tickets and since version 1.2.11 it can also use des-cbc-md4 and des-cbc-md5 and using other may require editing /etc/krb5.conf.

Version independent:

• OpenAFS terminology is sometimes different from common usage. In particular a volume is actually a subtree of AFS directories and files, and a partition that holds volumes is actually a subtree of some native operating system filesystem, whether the partition is on a fileserver or is the cache on a client.
• OpenAFS file servers will use as a partition anything that is mounted under directories whose name begins with vicep in the system's root directory.
• It is possible to use a file as the block device for a partition holding OpenAFS volumes, as long as it is mounted vi a loop device.
• The partition for AFS volumes on an OpenAFS fileserver does not need to be in its own dedicated block device, and neither does the AFS cache filetree on an OpenAFS client, but out of space conditions caused by space in the filetree being less than that declared for the OpenAFS may be handled badly. The /vicepAB partitions which are not mount points will be however ignored unless they contain a file called AlwaysAttach.
• AFS cell names are case insensitive but they are stored internally in uppercase and printed in lower case. As a rule by convention they should always bwe specified in lower case, as there are default mappings to case sensitive Kerberos realm names in all upper cases and to case insentitive DNS domain names in all lower case.
• The afsio program cannot use dynroot because it relies on libafscp which does not handle synthetic roots.
• OpenAFS uses UDP and implements a window style flow control algorithm similar to TCP, but the maximumwindow size is much smaller, which limits performance links with a large BDP. The protocol allows up to 256 outstanding packets, but versions of OpenAFS limit that for 32 packets, with the exception of the YFS version which allows for the full 256 packets.
• The default network buffers sizes for OpenAFS fileservers are usually very inadequate:

  So, setting a UDP buffer of 8Mbytes from user space is _just_ enough to handle 4096 incoming RX packets on a standard ethernet. However, it doesn't give you enough overhead to handle pings and other management packets. 16Mbytes should be plenty providing that you don't

  a) Dramatically increase the number of threads on your fileserver
  b) Increase the RX window size
  c) Increase the ethernet frame size of your network (what impact this has depends on the internals of your network card implementation)
  d) Have a large number of 1.6.0 clients on your network

  To summarise, and to stress Dan's original point - if you're running with the fileserver default buffer size (64k, 16 packets), or with the standard Linux maximum buffer size (128k, 32 packets), you almost certainly don't have enough buffer space for a loaded fileserver.

• Since a read-only replica will always be preferred to a read-write one, even if all read-only replicas are not available as long as one exists OpenAFS will not use the read-write one.
  Therefore it is always a good idea if a volume has read-only replicas to create an additional read-only replica in the same partition as the read-write one, as that is essentially free as it does not require file copying.
  It also is a good idea because it in case of release (updating read-only volumes to have the same content as a read-write volume) the read-only replica in the same partitions gets updated very quickly, and then other read-only replicas get updated from it, reducing the latency of the release operation.
• Because of a (difficult to fix) bug it is very recommended to avoid having read-only and read-write replicas in different partitions on the same server because at boot it could happen that the partition with the read-only replica is the first to be discovered by OpenAFS and then the read-write replica is never attached.
  Having read-only and read-write replicas of the same volume in different partitions on the same server is a design error, and there are checks against that, but some corner cases can be missed by the checks.
• When reconfiguring an OpenAFS service special care must be taken when changing the database server with the lowest IP address.
• When reconfiguring the addresses of OpenAFS DB servers the client caches must be restarted, or reset with fs newcell.
• When changing the addresses of OpenAFS file servers there are important cautions concerning the use of fs changeaddr which is however rarely needed.
• For various quorum related reasons the number of AFS db servers should be odd (1, 2).
• It is possible to authenticate OpenAFS clients against multiple cells or as multiple users, because the client authentication cache can hold distinct AFS tokens; even if the Kerberos credential cache can only only hold those for one principal.

File system hints for Lustre (120307)

• Not all Lustre releases are equally reliable, and choosing a good one for a specific environment can require some experimentation.
• MGS, MDS and OSS can reside on the same system, and the same system can run those of several instances.
• Mounting an OST on the OSS system that holds it can cause memory resource deadlock and is not recommended.
• The MDS can be rather CPU bound.
• The best RPC size for transfers is 1MiB, and aligning storage and user requests to 1MiB boundaries can give very large performance increases.
• Operations that query or update the inodes, such as changing the size of a file, can be very slow, as all OSSes on which the inode is sliced must be contacted.
• It is much better to use the Lustre own find command than the platform one.
• Re-exporting a Lustre mount via SMB or NFS can give very poor performance.
• Dual-linking Lustre servers on two separate network for Lustre-client and intra-Lustre communications can avoid a lot of problems.
• LNET imposes some restrictions on changing IP addresses while the system is running.