Hey Habr! I present to your attention the translation of the post by Stephen Wolfram
Prerequisites for the solvency of the Wolfram language
Today we are on the verge of great achievements together with the programming language
The Wolfram Function Repository is something made possible by the unique characteristics of the Wolfram Language not only as a programming language, but also as
For example, the Wolfram Functions Repository already contains
Likewise, more than
To go to the page, copy the above object (function BLOB), paste it into the input line and then run the function - it is already built into the Wolfram language and is supported by default, starting with
It should be noted here that when processing
I hope with the support
Inside the functions from the repository, there can be small or rather voluminous code fragments written in the Wolfram Language. For example, these could be calls
The shell and programming structure of the Wolfram Functions Repository is set up in such a way that everyone can contribute to the common cause in the most simple and convenient way for him - in fact, just
There are many trade-offs and details in this approach, but our goal is to optimize the Wolfram feature repository both for the convenience of users and so that new user features make a meaningful contribution to the evolution of the language. As we grow, I have no doubt that we will have to invent new methods of processing and testing the functions that are built into the repository, and this is not the least to organize a large number of functions and find those that users need. However, it cannot but be encouraging that the path we have chosen is a good start. I'm on my own
Increase efficiency while reducing costs
Even before the Internet, there were ways to share Wolfram Language code (our first major centralized project was
For more than 30 years, our company has worked hard to maintain the integrity of the structure of the Wolfram language, and this is very important for the Wolfram language to become not just a programming language, but also
Various computational processes take place in the implementation structure of each function. It should be noted here that in this case it is necessary that the function has a clear and uniform appearance and visual readability for the user. In this context, the built-in functions of the Wolfram Language are represented by more than 6000 consecutive examples of how functions should be properly programmed (these are our
Creating a repository that works really well is an interesting meta-programming challenge. For example, an excess of restrictions in the program will not be able to achieve the required unification and universality of the algorithm. Just as with an insufficient number of functional restrictions, you will not be able to implement a sufficiently correct sequence of algorithm execution. Several previous examples of the implementation of a compromise of these approaches implemented by our company worked quite stably - these are:
All of the above examples have a fundamental feature - the objects and functions assembled in the project have a very high degree of structuring and distribution of processes. Of course, the detail of the structure of what is a demo or a neural network or something else can vary greatly, but the fundamental structure for any current repository always remains the same. So what is your opinion, dear user, about creating such a repository that adds extensions to the Wolfram language? The Wolfram Language is designed to be extremely flexible, so it can be extended and modified in any way. This circumstance is extremely important for the ability to quickly create various large-scale software projects in the Wolfram Language. It should be noted here that with the growth of the flexibility of the language, the cost of projects implemented in such a language will inevitably increase. This is because the more the user uses such a language, the more dedicated functionality they get, but we should not forget that this approach can also have negative sides in terms of the inability to ensure consistent consistency of program modules.
In traditional programming languages, there is a common problem with libraries - if you use one library, for example, then the code works correctly, but if you try to use several libraries, there is no guarantee that they will interact correctly with each other. Also, in traditional programming languages - unlike a full-fledged computing language - there is no way to guarantee control over the presence of consistent built-in representations for any functions or data types other than their basic structures. But, in fact, the problem is even bigger than it seems at first glance: if someone is building a large-scale vertical of functionality, then without the huge cost of centralized programming of the project that we have invested in the Wolfram language, it is impossible to achieve consistency. Therefore, it is important that all program modules always work together correctly.
Thus, the idea behind the Wolfram Function Repository is to avoid the problem mentioned above—simply adding extensions to the language in the form of relatively small pieces of code through individual functions that are easier to develop as consistent modules. However, there are programming features that cannot be made convenient with individual functions (and our company is going to release an optimized software algorithm in the near future to help implement large-scale software packages). However, based on the functions already built into the Wolfram Language, there are many possibilities for programming that are implemented on the basis of individual functions. The idea here is that with relatively little programming effort, a number of new and very useful features can be created that will provide sufficient design coherence, while being well aligned with each other, and also, in addition, they will be able to easily and widely used in the language in the future.
This approach is definitely a compromise. Given the implementation of a larger package, one can imagine a whole new world of functionality that will become extremely powerful and useful. If there is a need to get new functionality that will fit into everything else, but at the same time you are not ready to spend huge efforts on the development of the project, this, unfortunately, can lead to a decrease in the scale of your project. The idea behind the Wolfram Functions Repository is to provide functionality to a defining part of a project, this approach allows you to add powerful functionality while also making it easier to maintain a good programming project consistency.
Help with adding custom functions to the functions repository
Our team has worked hard to make contributing to Wolfram Repository features easy enough for users. On the desktop (already in
There are two main things you will need to do: first, actually write the code for your function, and second, write documentation that illustrates how your function should work.
Click the "Open Sample" button at the top to see an example of what you need to do:
In fact, you are trying to create something that looks like a built-in function in the Wolfram Language. Except that it can do something much more specific than a built-in function. At the same time, expectations regarding its completeness and reliability will be much lower.
You need to give your function a name that follows the Wolfram Language's function naming conventions. In addition, you will need to develop documentation for your function, similar to the built-in functions of the language. I will talk about this in more detail later. For now, just notice that in the row of buttons at the top of the definition notebook file, there is a button
When you are sure that everything is filled in properly and you are ready, click the "Check" button. It is perfectly normal that you have not figured out all the details yet. So the "Check" function will automatically run and do a lot of style and consistency checks. Often, it will immediately prompt you to confirm and accept the corrections (For example: "This line must end with a colon," and it will prompt you to enter a colon). Sometimes she will ask you to add or change something yourself. We will be constantly adding new features to the automatic functionality of the "Check" button, but its main purpose is to ensure that everything you submit to the features repository already exactly matches as many style guidelines as possible.
So, after running "Check" you can use "Preview" ("Preview"). "Preview" creates a preview of the documentation page you have defined for your function. You can also create a preview for a file created on your computer or for a file located in the cloud storage. If, for some reason, you are not satisfied with what you see in the preview, just go back and make the necessary corrections, and then click the Preview button again.
You are now ready to commit your function to the repository. The Deploy button gives you four options:
The important thing at this step is that you can submit your function to the Wolfram Functions Repository so that it is available to any user. In this case, you can also place your function for a limited number of users. For example, you can create a feature that is hosted locally on your computer so that it is available when you use that particular computer. Or you can post it in your
So, let's say you want to actually submit your function to the Wolfram Function Knowledge Base. To do this, you click the "Submit" button to the repository. So what is happening right now? Your application is immediately queued for review and approval by our dedicated team of curators.
As your application progresses through the approval process (which usually takes a few days), you will receive updates on the status of your application, and possibly suggestions on how to further use it. But once your feature is approved, it will immediately be published to the Wolfram Feature Repository and made available for anyone to use. (And it will show up in
What should be in storage?
It should be noted that our company has very high standards for completeness, reliability and overall quality, of the 6000+ features that we have already built into the Wolfram language over the past 30+ years, all meet the above requirements. The goal of the Wolfram Functions Repository is to take advantage of all the structure and functionality that already exists in the Wolfram Language in order to add as many much lighter (that is, higher performance) functions as possible.
Of course, the functions in the Wolfram Functions Repository must conform to the design principles of the Wolfram Language—so that they can fully interact with other functions and the user's expectations of how the functions should work correctly. However, the functions do not have to have the same completeness or reliability.
In built-in functions of the Wolfram language, we work hard to make program functions as generic as possible. That said, when you're in the Wolfram Functions Repository, there's nothing wrong with having a function that just handles some very specific but useful case. For example, the function
Another point about inline functions is that our company makes every effort to handle all non-standard cases, to properly handle invalid input, and so on. In a function repository, it is perfectly normal to have a special function that handles the main cases of solving a problem and ignores all the rest.
Obviously it's better to have functions that do more and do it better, but optimizing for a function repository - as opposed to the built-in functions of the Wolfram language - should have more functions tied to more functions, rather than delving into the implementation processes of each specific function.
Now let's look at an example of testing features in a repository. Consistency expectations for such functions are naturally much lower than for built-in language functions. This is all the more true in cases where functions depend on external resources, such as APIs, it is important to constantly conduct sequential tests, which automatically happens inside the verification algorithms. In the nb file, you can explicitly define definitions (in the "More Information" section) and specify as many tests, defined either by input and output lines or full character objects like
As a result, there will be a number of implementation complexities in the functions repository. Some will be just one line of code, others may be thousands or tens of thousands of lines long, probably using many helper functions. When is it worth adding a function that requires very little code to define? Basically, if a function has
The main purpose of the feature repository (as its name suggests) is to introduce new features into the functionality of the language. If you want to add new data or
Actually there are two ways. You may want to introduce a new object type to be used in new functions in the functions repository. And in this case, you can always just write down its symbolic representation and use it when inputting or outputting functions in the functions repository.
But what if you want to represent an object and then determine, through existing functions in the Wolfram Language, which should work with it? To do this, the Wolfram Language has always had a lightweight mechanism called
So, what can be in the function code in the function repository?
Should all code for the Wolfram Functions Repository be written in the Wolfram language? Definitely, the code inside the external API should not be written in the Wolfram language, which actually does not even make the language code. In fact, if you find a function in almost any external language or library, you can create a wrapper that allows you to use it in the Wolfram Function Repository. (Usually you should use the built-in functions for this.
So what's the point of doing this? In essence, this allows you to use the entire Wolfram Language integrated system and its entire unified set of programming features. If you get the base implementation from an external library or language, then you can use the rich symbolic structure of the Wolfram Language to create a handy top-level function that allows users to easily use any functionality already implemented. At the very least, this should be possible in an ideal world where all the building blocks of library loading and the like exist, in which case they would be resolved automatically by the Wolfram Language. (It should be noted that in practice there may be problems with
By the way, when you first analyze generic external libraries, they often seem too complex to be covered by just a few functions, but in many cases, most of the complexity comes from creating the infrastructure required for the library and all the functions to support it. . However, when using the Wolfram Language, the framework is usually already built into packages, and so it is not necessary to detail all of these support functions, but only to create functions for a few "top-most" application-oriented functions in the library.
"Ecosystem" of the knowledge base
If you've written functions that you use all the time, submit them to the Wolfram Functions Repository! If this does not end up with something more (language development), then it will be much more convenient for you to use functions for personal use. However, it is logical to assume that if you use the features regularly, other users may also find them useful.
Naturally, you may find yourself in a situation where you are not able - or do not want - to share your functions or in case of access to private information resources. Even in such cases, you can simply deploy the features in your own cloud account,
The functions you submit to the Wolfram Functions Repository don't have to be perfect; they just have to be useful. It's a bit like the "Bugs" section of the classic Unix documentation - the "Definitions Section" has an "Author's Notes" section where you can describe limitations, issues, and the like that you already know about your function. Also, when you submit your feature to the repository, you can add submission notes, which will be read by a dedicated team of curators.
Once a feature is published, its page always has two links at the bottom: "
Sometimes it becomes necessary to simply use functions from the Wolfram Function Repository, such as built-in functions, without looking into their code. However, if you want to "look inside", there is always a Notepad button at the top. Click on it and you'll get your own copy of the original definition notebook that was submitted to the functions repository. Sometimes you can just use it as an example for your needs. However, you can also develop your own modification of this function. You may want to put these functions you found from the repository on your computer or in your account in your aphid cloud storage, you may want to submit them to the functions knowledge base, perhaps as an improved, augmented version of the original function.
In the future, we plan to support Git-style forking for feature repositories, but for now we're trying to keep it simple and we always have only one generally accepted version of each feature built into the language. More often than not (if the developers don't stop maintaining the features they've developed and respond to user messages), the original author of the feature takes control of updates to it and submits new versions, which are then reviewed and, if they pass the review process, published in the language.
Consider the question of how the "versioning" of the developed functions works. Right now, when you use a function from the functions repository, its definition will be permanently stored on your computer (or in your cloud account if you use the cloud). If a new version of a function is available, the next time you use it, you will receive a message notifying you of this. And if you want to update a feature to a newer version, you can do so with the command
One of the great things about the Wolfram Functions Repository is that any Wolfram Language program anywhere can use functions from it. If the program appears in notepad, it is often convenient to format the functions of the repository as functions of easily readable "function binaries" (perhaps with an appropriate set of versions).
You can always refer to any function of the function repository using a text
How does it work?
Inside the functions in the Wolfram repository, this is possible using exactly the same
Consider
Inside you can see some information using the function
How does setting the resource function work? The simplest is the purely local case. Here's an example that takes a function (in this case, just a pure function) and defines it as the resource function for a given session in the program:
Once you've made the definition, you can use the resource function:
Note that this function blob has a black icon . This means that the BLOB refers to the in-memory resource function defined for the current session. A resource feature that is permanently stored on your computer or cloud account has a gray icon . And for the official resource function, there is an orange icon in the Wolfram Function Repository .
So what happens when you use the Deploy menu in the definition notebook? First, it takes all the definitions in the notepad and creates a character from them.
Local deployment of a function from the repository on your computer is performed using the command
If you click the Submit button for the Function Repository, then what happens under it
By default, the submission is made under the name associated with your Wolfram ID. But if you are submitting an application on behalf of a development team or organization, you can
After you have submitted any of your functions to the functions knowledge base, it will be queued for verification. If you receive comments in response, they will usually be in the form of a text file with additional "comment cells" added. You can always check the status of your application by going to
Some subtleties at work
At first glance, it may seem that you can just take the definition pad and put it verbatim in the functions repository, however, in fact, there are quite a few subtleties - and processing them requires quite complex meta-programming, processing symbolically as the code that defines the function , and Notepad definition itself. Much of this happens internally, i.e. "behind the scenes", but it can have some implications that are worth understanding if you're going to be contributing to the feature knowledge base.
First immediate subtlety: When you fill out the "Definitions Notebook", you can simply refer to your function everywhere using a name like MyFunction, which looks like a normal function name in the Wolfram Language, but for function repository documentation this is replaced by
The second subtlety is that when you create a resource function from the Definition Notepad, then all the dependencies involved in the function definition must be fixed and explicitly included. However, to ensure that the definitions remain modular, you need to put everything in a unique
You will usually never see any trace of the code used to set up this namespace. But if, for some reason, you call an unexecuted symbol inside your function, you will see that symbol is in the internal context of the function. However, when processing the Definition Notepad, at least the character corresponding to the function itself
The feature repository is for defining new features. And these functions can have options. Often these options (for example,
Most functions just do what they're supposed to do every time they're called, but some functions need to be initialized before they can run in a particular session - and there's an Initialization section in the Definition section to solve this problem.
Functions from the repository can use other functions that are already in the repository, in order to set up definitions for a function repository that include two (or more) functions that reference each other, you need to deploy them in your program session so that you can reference on them like
Development prospects. What should happen when the repository gets really big?
Today we are just launching the Wolfram Functions Repository, but as time goes on we expect it to (remove) its scope and functionality can increase dramatically, and as it grows in development, there will be various problems that we can already foresee can arise.
The first problem concerns function names and their uniqueness. The function repository is designed in such a way that, as with built-in functions in the Wolfram Language, any given function can be referenced simply by specifying its name. But this inevitably means that function names must be globally unique throughout the repository, so that, for example, there can only be one in a repository
At first glance, this may seem like a big problem, but it's worth understanding that this is basically the same problem as for things like internet domains or social media markers. And the fact is that it is simply necessary to have a registrar in the system - and this is one of the roles that our company will perform for the knowledge base of Wolfram functions. (For private versions of the repository, administrators can be their registrars.) Of course, an Internet domain can be registered without having anything on it, but in a functions repository, a function name can only be registered if there is an actual function definition.
Part of our role in managing the knowledge base of Wolfram functions is to ensure that the name chosen for the function is logical given the function's definition, and that it conforms to Wolfram Language naming conventions. We have over 30 years of experience naming built-in functions in the Wolfram Language, and our curation team (curators?) will bring this experience to the function repository as well. Of course, there are always exceptions. For example, it might seem preferable to have a short name for some function, but it's better to "defend" with a longer, more specific name, because it's less likely to run into someone wanting to make a similar function name in the future.
(It should be noted here that simply adding some member tag to disambiguate functions will not have the desired effect. Because, if you do not insist on always assigning a tag, you will need to define a default tag for any given function, as well as allocate author tags, which will again require global coordination.)
As the knowledge base of Wolfram functions grows, one of the problems that will most likely arise is the discoverability of functions, for this the system provides
For built-in functions in the Wolfram Language, there is a so-called detection level provided by
The Wolfram Function Repository is set up as a persistent function repository, where any function in it will always work. Of course, new versions of functions may also appear, and we expect that some functions will of course become obsolete over time. The functions will work if they are used in programs, but their documentation pages will link to newer, more advanced functions.
The Wolfram Functions Repository is designed to quickly get new features and explore new ways to use the Wolfram language. With a lot of optimism, we hope that some of what has been explored in the feature repository will eventually make sense as built-in parts of the core Wolfram Language. Over the past decade, we've had a similar set
The most important thing here is that a function in the functions repository is something that is available to every user for use right now. Perhaps a built-in language feature could be much better and more performant, but a feature repository would allow users to immediately access all new features. And, most importantly, this concept allows everyone to add any of their new features.
Earlier in the history of the Wolfram language, this idea would not have worked as well as it does now, but at this stage so much effort has already been put into the language, as well as such a deep understanding of the principles of language design, that it now seems very possible that a large community of users will add features that will maintain design consistency to make them useful to a wide range of users.
There is an incredible spirit of talent(?) in the Wolfram Language user community. (Of course, this community includes many leading R&D experts in a wide variety of fields.) I hope that the Wolfram Feature Repository provides an effective platform for uncovering and spreading this spirit of talent. Only together can we create something that greatly expands the scope to which the Wolfram computational paradigm can be applied.
We've come a long way in the Wolfram language in 30+ years. Now together, let's go even further. I urge all respected users of the Wolfram language around the world to use the functional repository as a platform for this, as well as such a new software project as the Free Wolfram Engine for developers.
Source: habr.com