Translation of Theodore Ts'o's reflections, the creator of the Ext4 filesystem, on the development of ext4, the BcacheFS filesystem, the Linux kernel, ZFS, the code of conduct, and filesystems in general: <\/p>\n
On the development of ext4<\/b>. <\/p>\n Each release of the ext4 kernel involves contributions from more than half a dozen people. Currently, most of my time goes into reviewing code, conducting tests, and improving the test application {kvm<\/a>,gce,qemu,android}-xfstests. I heavily rely on 2-3 other developers from SUSE and IBM who assist me with code reviews. <\/p><\/blockquote>\n On BcacheFS<\/b> <\/p>\n Honestly, bcachefs is not a completely standalone project \u2014 for instance, Kent authored 72% of the patches between releases 6.11 and 6.12, while out of 103 patches to ext4 during the same timeframe, I authored exactly 0%. This is because I firmly believe that programming is a team sport, and my role as a technical lead is to enable the ext4 developers to do their best in improving the filesystem. We hold weekly meetings, and Derrick Wong, a senior XFS developer and former XFS maintainer, participates in these meetings \u2014 and I have been known to assist him with XFS testing issues, while Derrick has helped me with various ext4 testing matters and even reviewed a couple of ext4 patches. We collaborate with each other, and it\u2019s beneficial. <\/p>\n I leave it to others to decide whether they want to trust their data to a lone hotshot programmer who may well be more talented than I am, but I'll give you a hint \u2014 you can 'cheat' by bringing a team together to solve the problem. You don't have to do it alone. Of course, for this, you need to know how to bring out the best in others, and you must work together. And polite interactions with each other in mailing lists don\u2019t hurt. <\/p><\/blockquote>\n On the kernel, CoC, capabilities, and the future of ext4<\/b> <\/p>\n Ext4 is indeed gaining some new features, but these are ones that companies are willing to fund, as the return on investment in developing features makes sense from a cost-benefit perspective. For instance, fscrypt and case-insensitive directories were features beneficial for Android and Chrome OS, and were funded at least partially by those developer groups (Steam was also concerned about case folding and supported one of the engineers). We want to add support for untorn writes because it will improve the performance of databases on cloud-emulated block devices, where 16k atomic writes can be guaranteed, allowing us to eliminate double buffering in MySQL and PostgreSQL. <\/p>\n (In fact, Amazon and Google can do this in their own DBMS products, making assumptions about how Amazon EBS and Google Persistent Disk work, but we want to do it in a more general way that will be more maintainable in the long term). It\u2019s less appealing than things like reflinks, but the return on investment is much easier to justify, both because the costs are lower (less work in development, testing, and qualification for enterprise deployment) and because the benefits are much easier to quantify. Things like 'I can save the salary costs of XX full-time developer engineers over five years' are much easier to make for this kind of performance-enhancing features. <\/p>\n Unlike this, reflinks are fun, but I couldn\u2019t find a client willing to pay for the development costs, or a company that believes its customers will buy more of its product if they add reflinks to ext4. It may sound terribly corporate, but there\u2019s a story about how ZFS engineers started their project from scratch without asking permission from management and without getting suggestions from the sales department, presenting Sun with something that was essentially a done deal. <\/p>\n It sounds great, but considering that in the end, Sun started losing money and had to sell itself to another company, and that the engineering organization that supported ZFS no longer exists. Around the time ZFS was announced, I was involved in a company-wide study to see if it made sense to invest in filesystem features for AIX and Linux \u2014 and we concluded that it didn\u2019t, the return on investment was low, and new filesystem features wouldn\u2019t lead to an increase in customers buying IBM hardware, software, or systems. Maybe tough times came for IBM, but it still exists, whereas Sun does not. <\/p>\n About the same time, representatives from several Linux companies gathered to brainstorm how Linux would compete with ZFS. It was during this meeting that the idea was put forth that btrfs would be the long-term answer, while ext4 would be a short-term fix that would provide support for things like real-time resizing, 64-bit block numbers, and other features that were in traditional Legacy Unix OS but were absent from ext3. <\/p>\n At that meeting, I was asked to determine what it would take to create a completely new filesystem. I conducted research, looking at how much effort was required to create filesystems like IBM's GPFS and JFS, Digital's advfs, and I evaluated how much Sun needed to create ZFS and bring it to a fully production-ready state. The answer I received was about 100 person-years, with a low estimate of 50 person-years and a high estimate of 200 person-years (but that was for GPFS, which was a clustered filesystem and therefore much more complex). <\/p>\n I mentioned this at the meeting, and a senior engineer from Intel said, 'No, don\u2019t tell management about this because they will never approve the project! Tell them that btrfs will be ready in 18 months.' I leave it to people to decide when btrfs reaches the status of 'ready for enterprise use', especially for those new attractive extended features that were supposed to compete with ZFS, but I don\u2019t think it\u2019s debatable that this didn\u2019t happen in 18 months. <\/p>\n Even before Sun disbanded, many companies that sent representatives to the meeting opted out of involving engineers in the work on btrfs, which of course didn't help. However, this was likely due to the fact that companies are rational organizations making their own decisions about return on investment, and funding a new file system didn\u2019t make as much sense as telling people that Linux would have an answer to ZFS. <\/p>\n Looking back, it can be said that while ZFS had some really cool features, they weren\u2019t enough to make most users choose Solaris over purchasing much cheaper x86 platforms and installing Linux. By the time Sun decided to try the OpenSolaris and Solaris x86 strategy, it was too late. The network effects were enormous, and the x86 strategy didn't answer how one company, Sun, could pay the salaries of all the super talented engineers working on Solaris. Buying a $5,000 x86 server doesn\u2019t provide much return on investment compared to proxy server<\/a> The SunFire E10k Sparc priced at $100,000, which Sun referred to as the 'dot' in 'dot Com.' <\/p>\n The point is that engineering work in the real world is a compromise, and business realities are part of that compromise. I don\u2019t apologize for preferring to eat food, and for wanting to earn enough money to retire someday. This, in turn, means I need to understand well how I provide value to my employer, at least ten times my salary. If I can do this while continuing to work with open source and helping other companies make money so they\u2019re willing to contribute to ext4, well, that\u2019s part of the challenge and the reason I love working with open source. <\/p>\n Returning to the Code of Conduct, I will say that almost all maintainers of major file systems support the Code not due to any vague liberal considerations. It is because we need every engineer willing to contribute to our project, and many of us have seen people who refused to work on Linux and switched to other operating systems (I know one person who switched to Windows and was a valuable Linux kernel developer at IBM Linux Technology Center) or worked on internal projects but not on those that required interaction with LKML, due to the toxic environment of a few individuals on the mailing list. <\/p>\n In some cases, concerns were unwarranted; for instance, Linus shouted at a senior developer who should have known better and whom Linus met in person most of the time, and they had an established relationship. The problem is that newcomers didn\u2019t know this and were frightened \u2014 'What if Linus humiliates me publicly like he did Steve?' \u2014 not realizing that, in practice, this wouldn\u2019t happen. That\u2019s why we have a CoC; it\u2019s not for us senior engineers, but to support the younger engineers in our teams, whom we want to train to eventually replace us when it\u2019s time to retire, or if we get hit by a bus, or otherwise leave this mortal world. <\/p>\n Don't forget about the 50-100 person-years of work required to create a file system ready for use in a corporate environment. We need all the engineers we can get, and many of us do extra work in our free time because we care. Creating a high-quality file system is a team effort, and we need every talented engineer we can attract. Even if one engineer is a super 10x programmer, if in the end he drives away a lot of other engineers who could work on testing, performance tuning, etc., it's just not worth it to let someone be a jerk. <\/p><\/blockquote>\n\n
\n
\n