Translation of thoughts by Theodore Tsâo, the creator of the Ext4 file system, on the development of ext4, the BcacheFS file system, the Linux kernel, the code of conduct, and file systems in general:
On the development of ext4.
Each release of the ext4 kernel involves contributions from more than half a dozen people. Currently, most of my time goes into reviewing code, conducting tests, and improving the test application {kvm,gce,qemu,android}-xfstests. I heavily rely on 2-3 other developers from SUSE and IBM who assist me with code reviews.
On BcacheFS
Honestly, bcachefs is not a completely standalone project â for example, Kent authored 72% of the patches between kernel releases 6.11 and 6.12, while out of 103 patches for ext4 during the same period, I authored exactly 0%. This is because I firmly believe that programming is a team sport, and my role as a technical leader is to empower ext4 participants to do their best to improve the file system. We hold weekly conferences, and Derrick Wong, a senior XFS developer and former XFS maintainer, participates in these conferences â and I have helped him with XFS testing issues, while Derrick has assisted me with various ext4 testing matters and even reviewed a couple of ext4 patches. We collaborate, and that's a good thing.
I let others decide whether they want to trust their data to someone who is a lone hotshot programmer who may well be more talented than I am, but I'll give you a hint â you can 'cheat' by bringing a team to solve the problem. You don't necessarily have to do it alone. Of course, for this, you need to know how to bring out the best in others, and you have to work together. And polite behavior towards each other in mailing lists never hurts.
On the kernel, CoC, capabilities, and the future of ext4
Ext4 is indeed gaining some new features, but these are ones that companies are willing to fund, as the return on investment in developing features makes sense from a cost-benefit perspective. For instance, fscrypt and case-insensitive directories were features beneficial for Android and Chrome OS, and were funded at least partially by those developer groups (Steam was also concerned about case folding and supported one of the engineers). We want to add support for untorn writes because it will improve the performance of databases on cloud-emulated block devices, where 16k atomic writes can be guaranteed, allowing us to eliminate double buffering in MySQL and PostgreSQL.
(In fact, Amazon and Google can accomplish this in their own DB products by making assumptions about how Amazon EBS and Google Persistent Disk work, but we want to approach it in a more generalized way that will be more sustainable in the long run). This is less appealing than things like reflinks, but the ROI is much easier to justify, both because the costs are lower (less work for development, testing, and qualification for corporate deployment) and because the benefits are much easier to quantify. Claims like 'I can save the salary cost of XX full-time software engineers over five years' are much easier to make for this kind of performance-enhancing functionality.
In contrast, reflinks are fun, but I couldn't find a client willing to pay for the development costs, or a company that believes their customers would buy more of their product if they added reflinks to ext4. This may sound terribly corporate, but there is a story about how ZFS engineers started a project from scratch without asking for permission from management or getting input from the sales department, and presented Sun with what was essentially a done deal.
That sounds great, but remembering that in the end, Sun started losing money until it had to sell itself to another company, and the engineering organization that supported ZFS no longer exists. Around the time ZFS was announced, I was involved in a company-wide study to determine whether it made sense to invest in filesystem features for AIX and Linux â we concluded that it didn't; the ROI was low, and new filesystem features wouldn't increase the number of customers buying IBM's hardware, software, or systems. IBM may have faced tough times, but it still exists, whereas Sun does not.
Around the same time, representatives from several Linux companies came together to figure out how Linux would compete with ZFS. It was at this meeting that the idea was proposed that btrfs would be the long-term answer, while ext4 would serve as a short-term solution that would provide support for features like online resizing, 64-bit block numbers, and other capabilities that were present in traditional legacy Unix OS but absent in ext3.
At that meeting, I was asked to determine what it would take to create a completely new filesystem. I conducted research, looking at how much effort was required to create filesystems like IBM's GPFS and JFS, Digital's advfs, and I evaluated how much Sun needed to create ZFS and bring it to a fully production-ready state. The answer I received was about 100 person-years, with a low estimate of 50 person-years and a high estimate of 200 person-years (but that was for GPFS, which was a clustered filesystem and therefore much more complex).
I reported this at the meeting, and a senior engineer from Intel said, 'No, don't tell the management about this, as they will never approve the project! Tell them that btrfs will be ready in 18 months.' I let people decide for themselves when btrfs will reach 'enterprise readiness,' especially for those new attractive enhanced features meant to compete with ZFS, but I don't think it's debatable that this did not happen in 18 months.
Even before Sun was dissolved, many companies that sent their representatives to the meeting withdrew their engineers from working on btrfs, which certainly didn't help. However, this was likely because companies are rational organizations making their own investment return decisions, and funding a new file system didn't make as much sense as telling people that Linux would have an answer to ZFS.
Looking back, it can be said that while ZFS had some really cool features, they werenât enough to make most users choose Solaris over purchasing much cheaper x86 platforms and installing Linux. By the time Sun decided to try the OpenSolaris and Solaris x86 strategy, it was too late. The network effects were enormous, and the x86 strategy didn't answer how one company, Sun, could pay the salaries of all the super talented engineers working on Solaris. Buying a $5,000 x86 server doesnât provide much return on investment compared to proxy server the $100,000 SunFire E10k Sparc, which Sun called the âpointâ in the âdot Comâ era.
The point is that engineering in the real world is about trade-offs, and the business realities are part of that trade-off. I donât apologize for wanting to eat food and wanting to earn enough money to retire one day. That, in turn, means I need to understand how I deliver value to my employer that is at least 10 times greater than my salary. If I can do that while continuing to work with open source and helping other companies earn money so theyâre willing to contribute to ext4, well, thatâs part of the challenge and why I love working with open source.
Returning to the Code of Conduct, I will say that almost all maintainers of major file systems support the Code not due to any vague liberal considerations. It is because we need every engineer willing to contribute to our project, and many of us have seen people who refused to work on Linux and switched to other operating systems (I know one person who switched to Windows and was a valuable Linux kernel developer at IBM Linux Technology Center) or worked on internal projects but not on those that required interaction with LKML, due to the toxic environment of a few individuals on the mailing list.
In some cases, concerns were unfounded; for example, Linus yelled at a senior developer who really should have known better and with whom Linus had personal interactions in most cases, and they had already established relationships. The problem is that newcomers didnât know this and would get scared â âWhat if Linus humiliates me in public just like he did with Steve?â not understanding that this wouldnât happen in practice. Thatâs why we have a CoC; itâs not for us senior engineers, but to support the younger engineers in our teams, whom we want to train so that at some point they can replace us when itâs time to retire, or when a bus hits us, or we otherwise leave this mortal world.
Keep in mind about 50-100 person-years of work to create a file system ready for enterprise use. We need all the engineers we can get, and many of us do extra work in our spare time because we care. Building a high-quality file system is teamwork, and we need every talented engineer we can get. Even if one engineer is a super 10x programmer, if they end up scaring away a bunch of other engineers who could work on testing, performance tuning, etc., itâs just not worth letting someone be a jerk.
Source: opennet.ru
