🥇Something else: Haiku app bundles?

TL; DR: Can Haiku get proper support for app bundles, like app catalogs (like .app on Mac) and/or application images (Linux AppImage)? It seems to me that this will be a worthy addition, which is easier to implement correctly than in other systems, since most of the infrastructure is already in place.

A week ago I discovered Haiku, an unexpectedly good system. And since I've been interested in catalogs and application images (inspired by the simplicity of the Macintosh) for a long time, it's not surprising that the idea came to my mind ...

To be completely clear: I am the creator and author of AppImage, a Linux application distribution format that aims for Mac simplicity and gives full control to application authors and end users (if you want to know more, see this article). wiki и documentation).

What if we make an AppImage for Haiku?

Let's talk a little, purely theoretically, about what needs to be done in order to get AppImage, or something similar, on Haiku? You don't have to create something right now, because the system that Haiku already has works amazingly, but an imaginary experiment would be nice. It also demonstrates the sophistication of Haiku, compared to Linux production environments, where such things are terribly difficult (I have the right to say so: I have been toiling with debugging for 10 years already).

On the Macintosh System 1, each application was a separate file "managed" by the Finder. Using AppImage I'm trying to recreate the same user experience on Linux.

First, what is an AppImage? It is a system for releasing third party applications (for example, Ultimaker Cure) that allows you to release applications when and how they want: no need to know the specifics of various distributions, build policies or build infrastructure, no need for maintainers, and they do not tell users what (not) can be installed on their computers. AppImage should be understood as something similar to a package for Mac in the format .app inside the disk image .dmg. The main difference is that the applications are not copied, but remain inside the AppImage always, in much the same way as Haiku packages .hpkg mounted, and never installed in the usual sense.

AppImage has gained some appeal and popularity over its more than 10 years of existence: Linus Torvalds himself has publicly endorsed it, and common projects (for example, LibreOffice, Krita, Inkscape, Scribus, ImageMagick) have adopted it as the main way to distribute continuous or nightly builds, not interfering with users' installed or uninstalled applications. However, Linux desktops and distributions most often still cling to the traditional, centralized, maintainer-based distribution model and/or promote their own enterprise business and/or engineering programs based on Flatpak (RedHat, Fedora, GNOME) and Snappy (Canonical, Ubuntu). It comes ridiculously.

How does it work

Each AppImage contains 2 parts: a small double-clickable ELF (aka. runtime.c), followed by a filesystem image SquashFS.

The SquashFS file system contains the payload in the form of an application and everything needed to run it, which in good sense should not be considered part of the default installation for every fairly recent target system (Linux distribution). It also contains metadata, such as the application name, icons, MIME types, etc., etc.

When run by the user, runtime uses FUSE and squashfuse to mount the file system, after which it handles running some entry point (called AppRun) inside the mounted AppImage.
The file system is unmounted after the process ends.

Everything seems simple.

And these things complicate things:

With so many different Linux distributions out there, nothing in their right mind can be called "part of the default installation for every fresh target system". We get around this problem by building excludelist, which allows you to determine what will be packaged in the AppImage and what will need to be taken somewhere else. At the same time, we sometimes miss, despite the fact that, in general, everything works fine. For this reason, we recommend that packagers test AppImages on all target systems (distributions).
Payload applications must be file system roamable. Unfortunately, many applications have hard-coded absolute paths to, for example, resources in /usr/share. This needs to be corrected somehow. In addition, you must either export LD_LIBRARY_PATHor fix rpath so that the loader can find the associated libraries. The first method has its drawbacks (which are worked around in complex ways), and the second is just cumbersome.
The biggest UX trap for users is what they need set executable bit AppImage file after download. Believe it or not, for some this is a real barrier. The need to set the executable bit is cumbersome even for experienced users. As a workaround, we suggested installing a small service that monitors AppImage files and sets their executable bit. In its purest form, not the best solution, because it will not work out of the box. Linux distributions don't provide this service, so out-of-the-box users are having a hard time.
Linux users are waiting for the new application to have an icon in the launch menu. You can’t say to the system: “Look, there’s a new application, let’s work.” Instead, according to the XDG specification, you need to copy the file .desktop to the right place in /usr for a system-wide installation, or in $HOME for individual. Icons of certain sizes, according to the XDG specification, need to be placed in certain places in usr or $HOME, then run commands in the desktop to update the icon cache, or hope that the desktop manager will figure it out and automatically detect everything. Likewise with MIME types. As a workaround, it is proposed to use the same service, which, in addition to setting the feasibility flag, will be, if there are icons and so on. in AppImage, copy them from AppImage to the right places according to XDG. When deleted or moved, the service will supposedly clean up everything. Of course, there are differences in the behavior of each working environment, in the formats of graphic files, their sizes, storage locations and how caches are updated, which creates a problem. In short, this method is a crutch.
If the above is not enough, there is still no AppImage icon in the file manager. The Linux world has not yet decided to implement elficon (despite discussion и implementation), so it's not possible to embed the icon directly into the app. So it turns out that applications in the file manager do not have their own icons (no difference, AppImage or something else), they are only in the launch menu. As a workaround, we use thumbnails, a mechanism that was originally designed to allow desktop managers to display thumbnail previews of graphic files as their icons. Therefore, the service for setting the executable bit also works as a "thinnerizer", creating and writing icon thumbnails in the appropriate places. /usr и $HOME. It also cleans up if the AppImage is deleted or moved. Because each desktop manager behaves slightly differently, such as what formats it accepts icons in, what sizes or locations, it's all really painful.
The application simply crashes at runtime if errors occur (for example, there is a library that is not part of the base system and is not supplied by AppImage), and no one tells the user in the GUI what exactly is going on. We started to get around this by using notice on the desktop, which means we need to catch errors from the command line, convert them into user-friendly messages, which then need to be displayed on the desktop. And of course, every desktop environment handles them a little differently.
At the moment (September 2019 - translator's note), I have not found an easy way to tell the system that the file 1.png must be opened with Krita, and 2.png - using GIMP.

Storage location for cross-desktop specifications used in GNOME, KDE и Xfce is freedesktop.org

Achieving the level of sophistication deeply woven into the Haiku desktop environment is difficult, if not impossible, due to the specs XDG from freedesktop.org for cross-desktop, as well as implementations of desktop managers based on these specifications. As an example, one system-wide Firefox icon can be cited: apparently, the authors of XDG did not even think that a user could have several versions of the same application installed.

Icons for different versions of Firefox

I was wondering what the Linux world could learn from Mac OS X to not mess up with system integration. If you have time and you are doing this, be sure to read what Arno Gurdol, one of the first Mac OS X engineers, said:

We wanted to make installing the application as easy as dragging the application icon from anywhere (server, external drive) onto your computer's drive. To do this, the application package stores all the information, including icons, version, file type processed, URL scheme type, that the system needs to know in order to process the application. This also includes information for 'central storage' in the Icon Services and Launch Services database. To support performance, applications are 'discovered' in several 'well-known' places: in the system and user's Applications directory, and in some others automatically if the user navigates the Finder to the directory containing the application. In practice, this has worked very well.

https://youtu.be/qQsnqWJ8D2c
Apple WWDC 2000 session 144 - Mac OS X: packaging applications and printing documents.

None of this infrastructure exists on Linux production environments, so we're looking for workarounds around structural limitations in the AppImage project.

Is Haiku coming to the rescue?

And one more thing: Linux platforms as the basis of desktop environments tend to be so underspecified that many things that are quite simple in a consistent full-stack system become frustratingly fragmented and complex in Linux. I devoted a whole talk to issues related to the Linux platform for desktop environments (knowledgeable developers have confirmed that this will remain so for a very long time).

My talk on Linux desktop issues in 2018

Even Linus Torvalds admitted that it was because of the fragmentation that the idea of working environments failed.

Good to see Haiku!

With Haiku, everything becomes amazingly simple.

While the naive approach to "porting" an AppImage to Haiku is to simply try to assemble (basically runtime.c and service) its components (which might even be possible!), it won't do much good for Haiku. Because in fact, most of these problems are solved on Haiku and are conceptually sound. Haiku provides exactly the kind of system infrastructure building blocks I've been looking for in Linux desktops for so long and couldn't believe they weren't there. Namely:

Believe it or not, this is something that many Linux users cannot overcome. On Haiku, everything is done automatically!

ELF files that do not have an executable bit get it automatically when double-clicking in the file manager.
Applications can have embedded resources, such as icons that appear in the file manager. You don't have to copy a bunch of images into special icon directories, and therefore don't have to clean them up after deleting or moving an application.
There is a database for linking applications to documents, there is no need to copy any files for this.
In the lib/ directory next to the executable file, libraries are searched by default.
There are no multiple distributions and desktop environments, everything that works works everywhere.
There is no separate module to run that is different from the Applications directory.
Applications do not have built-in absolute paths to their resources, there are special functions for determining the location at runtime.
Introduced the idea of compressed filesystem images: any hpkg package. They are all mounted by the kernel.
Each file is opened by the application that created it, unless otherwise specified. How cool is that!

Two png files. Pay attention to the different icons showing that they will be opened by different applications on double-click. Also note the "Open with:" dropdown menu where the user can select an individual application. How simple!

It looks like many of the crutches and workarounds needed by AppImage on Linux become unnecessary on Haiku, which has a simplicity and sophistication at its core that makes it handle most of our needs.

Do Haiku need app packages after all?

This leads to a big question. If it were an order of magnitude easier to create a system like AppImage on Haiku than on Linux, would it be worth it? Or has Haiku, with its hpkg packaging system, effectively eliminated the need to develop such an idea? Well, the answer is to look at the motivation behind the existence of AppImages.

User perspective

Let's look at our end user:

I want to install an application without asking for an administrator (root) password. There is no concept of an administrator on Haiku, the user has full control because it is a personal system! (In principle, you can imagine this in multiplayer mode, I hope the developers keep it simple)
I want to get the latest and greatest versions of applications, not wait for them to appear in my distribution (most often this means "never", at least if you do not update the entire OS). On Haiku, this is "resolved" with floating releases. This means that it is possible to get the latest and greatest versions of applications, but for this you need to constantly update the rest of the system, in fact, turning it into a “moving target”.
I want multiple versions of the same application side by side, because I can't find out what was broken in the latest version, or, let's say, as a web developer, I need to check my work under different browser versions. Haiku solves the first, but not the second problem. Updates are rolled back, but only for the entire system, it is impossible (as far as I know) to run, for example, several versions of WebPositive or LibreOffice at the same time.

One of the developers writes:

In essence, the rationale is this: a use case is so rare that it makes no sense to optimize for it; handling it as a special case in HaikuPorts seems more than acceptable.

I need to save applications where I like, not on my startup drive. I often run out of space on disks, so I need to connect an external drive or a network directory to store applications (all versions that I downloaded). If I connect such a disk, it is necessary that applications be launched by double-clicking. Haiku keeps old versions of packages, but I don't know how to move them to an external drive, and also how to call applications from there later.

Developer comment:

Technically, this is already possible with the mount command. Of course, we will make a GUI for this as soon as there are enough interested users.

I don't need millions of files scattered around the file system that I can't manually manage on my own. I want one file per app that I can easily download, move, delete. On Haiku this problem is solved with packages .hpkg, which transfer, for example, python, from thousands of files to one. But if there is, for example, Scribus using python, then I have to deal with at least two files. And I have to take care to keep versions of them that work with each other.

Multiple versions of AppImages running side by side on the same Linux

App Developer's Perspective

Let's look from an application developer's point of view:

I want to manage the entire user experience. I don't want to depend on an operating system to tell me when and how I should release apps. In Haiku, developers can work with their own hpkg repositories, but this means that users will have to configure them manually, which makes this idea "less attractive".
I have a download page on my website where I distribute .exe for Windows .dmg for Mac and .AppImage for Linux. Or maybe I want to monetize access to this page, everything can be? What should I place there for Haiku? Enough file .hpkg with dependencies only on HaikuPorts
My software needs specific versions of other software. For example, Krita is known to need a patched version of Qt, or Qt that is fine-tuned to a specific version of Krita, at least until the patches are pushed back into Qt. It is possible to package native Qt for an application in a package .hpkg, but most likely, this is not welcome.

The usual app download page. What to post here for Haiku?

Will bundles (those existing as app directories like AppDir or .app in the style of Apple) and/or images (in the form of heavily modified AppImages or .dmg from Apple) applications a useful addition to the Haiku work environment? Or will it dilute the whole picture and lead to fragmentation, and therefore add complexity? I'm torn: on the one hand, the beauty and sophistication of Haiku is based on the fact that there is usually one way to do something, not many. On the other hand, most of the infrastructure for catalogs and/or app bundles is already in place, so the system is crying out for the remaining few percent to fall into place.

According to the developer mr. waddlesplash

On Linux they are (catalogs and application bundles, - approx. translator) are most likely a technical solution to systemic problems. At Haiku, we prefer to simply solve systemic problems.

What do you think?

Before you answer...

Wait, let's do a quick reality check: in fact application directories - already part of Haiku:

Application directories already exist on Haiku, but are not yet supported in the file manager

They're just not as well supported as, say, the Macintosh Finder. How cool would it be if the QtCreator directory had the name and icon "QtCreator" in the top left corner, launching the application on double-click?

A little earlier I asked:

Are you sure you'll launch your decade-old apps today, when all the app stores and distribution repositories have forgotten about them and their dependencies? Are you sure you will still be able to access your current job in the future?

Is there already an answer from Haiku, or can catalogs and app bundles help here? I think they can.

According to mr. waddlesplash:

Yes, we have an answer to the question: we will just support these applications for as long as necessary until someone can read their file formats in the right way or provide one-to-one functionality. Our commitment to keeping BeOS R5 apps running on Haiku is proof of that...

That's right!

What course of action should Haiku take?

I can imagine the peaceful coexistence of hpkg, directories and application images:

The system software uses .hpkg
For the most frequently used software (especially the one that needs to be scheduled for rolling releases), use .hpkg (approximately 80% of all cases)
Some installed through .hpkg, applications will benefit from moving to an application catalog infrastructure (e.g. QtCreator): they will be distributed as .hpkg, as before.

mr. waddlesplash writes:

If all you need is browsing apps in /system/apps, instead make the directories in Deskbar more manageable for users, because /system/apps not designed for users to open and watch it regularly (unlike macOS). For such situations, Haiku has a different paradigm, but this option is, in theory, acceptable.

Haiku gets the infrastructure to run application images, nightly, continuous and test builds of software, as well as when the user wants to “freeze it in time”, for proprietary and internal software, and other special use cases (about 20% of all). These images contain the files necessary to run the application. .hpkg, mounted by the system tools, and after the end of the application - unmounted. (Perhaps a file manager could put files .hpkg into application images, automatically or at the request of the user - well, as when you drag an application to a network directory or an external drive. It's just a song! Or rather, poetry - haiku.) On the other hand, the user may want to install the contents of the image in the form of files.hpkg, after which they will be updated and processed in the same way as if they were installed through HaikuDepot ... Need to brainstorm).

Quote from mr. waddlesplash:

Launching applications from external drives or network directories can potentially be useful. And adding the ability to configure more "zones" for pkgman would definitely be a nice feature.

Such a system would take advantage of hpkg, catalogs, and application images. They are good individually, but together they will be invincible.

Conclusion

Haiku has a framework that provides a simple and sophisticated user interface for PCs, far beyond what is typically provided for Linux PCs. Package system .hpkg is one such example, but the rest of the system is also imbued with sophistication. However, Haiku would benefit from proper support for application catalogs and images. How best to do this is worth discussing with people who know Haiku, its philosophy and architecture is much better than I do. I ended up using Haiku for just over a week. Nevertheless, I believe that this fresh look will be useful to the designers, developers and architects of Haiku. At the very least, I'll be happy to be their "sparring partner." I have over 10 years of hands-on experience with Linux catalogs and bundles and would love to find a use for them in Haiku, which I think they are the perfect concept for. The potential solutions I have proposed are not at all the only true ones for the problems that I have described, and if the Haiku team decides to find other, more elegant ones, I am all for it. Basically, I'm already thinking about the idea of how to make a system hpkg even more amazing without changing the way it works. It turns out that the Haiku team has been thinking about app bundles for a long time when implementing a package management system, but unfortunately (I think) the idea has gone "obsolete". Maybe it's time to revive it?

Try it yourself! After all, the Haiku project provides images for booting from DVD or USB, generated daily.
Got questions? We invite you to the Russian-speaking telegram channel.

Error overview: How to shoot yourself in the foot in C and C++. Haiku OS Recipe Collection

From Author Translation: This is the eighth and final article in the Haiku series.

Something else: Haiku app bundles?