HIROSHI WATANABE/GETTY IMAGES
In the book "The Wealth of Nations," Adam Smith illustrates how the division of labor becomes the main source of productivity increases. An example is the assembly line in a pin factory: "One worker pulls the wire, another straightens it, the third cuts it, the fourth sharpens the end, and the fifth shapes the other end to attach the head." Through specialization focused on specific functions, each worker becomes a highly skilled specialist in their narrow task, leading to enhanced process efficiency. The output per worker multiplies several times, and the factory becomes more effective in producing pins.
This functional division of labor is so ingrained in our minds even today that we quickly organize our teams accordingly. Data Science is no exception. Complex algorithmic business opportunities require multiple labor functions, so companies typically create groups of specialists: researchers, data analysts, machine learning engineers, scientists focused on causal relationships, and so on. The work of specialists is coordinated by a product manager, distributing functions in a manner reminiscent of a pin factory: "one person receives the data, another models it, a third executes it, a fourth measures it," and so on.
Unfortunately, we should not optimize our Data Science teams solely for productivity. However, you do this when you understand what you are producing: pins or something else, and merely strive for efficiency. The goal of assembly lines is to complete tasks. We know exactly what we want â itâs pins (as in Smithâs example), but any product or service can be mentioned where requirements fully describe all aspects of the product and its behavior. The role of employees is to fulfill these requirements as efficiently as possible.
However, the goal of Data Science is not to perform tasks. Rather, the aim is to explore and develop new strong business capabilities. Algorithmic products and services, such as recommendation systems, customer interactions, preference classification in style, size selection, fashion design, logistics optimization, seasonal trend detection, and much more cannot be developed in advance. They must be explored. There are no blueprints for reproduction; these are new opportunities with their inherent uncertainty. Coefficients, models, model types, hyperparameters, all necessary elements must be explored through experiments, trials, errors, and iterations. In a pin factory, training and design are performed in advance, prior to their production. With Data Science, you learn during the process, not before it.
At the pin factory, where training is prioritized, we do not wait for or desire employees to improvise on any product attribute, except to enhance production efficiency. Task specialization makes sense as it leads to process efficiency and production consistency (without altering the final product).
However, when the product is still evolving and the goal is learning, specialization hinders our objectives in the following instances:
1. It increases coordination costs.
That is, the costs that accumulate over time spent on communication, discussion, justification, and prioritizing the work that needs to be done. These costs scale superlinearly with the number of people involved. (As J. Richard Hackman taught us, the number of relationships r grows similarly to the function of the number of members n according to this equation: r = (n ^ 2 - n) / 2. And each relationship reveals a certain amount of cost ratio). When data analysts are organized by functions, at every stage, with every change, every service transfer, etc., many specialists are required, which increases coordination costs. For example, statistical modeling specialists wanting to experiment with new features will need to coordinate their actions with data processing engineers who complement datasets every time they want to try something new. Similarly, each newly trained model means that the model developer will need someone to coordinate their actions for deployment. Coordination costs act as a fee for iteration, making them more difficult and expensive, and with a higher likelihood of abandoning the research. This can hinder learning.
2. This complicates wait times.
Even more daunting than the costs of coordination is the time lost between work shifts. While coordination costs are typically measured in hoursâtime spent on meetings, discussions, and project reviewsâwaiting time is usually measured in days, weeks, or even months! Aligning the schedules of functional specialists is challenging since each specialist must be allocated to several projects. A one-hour meeting to discuss changes can take several weeks to align workflows. Once changes are agreed upon, scheduling the actual work in the context of multiple other projects also occupies the specialists' working hours. Tasks related to code fixes or research that could be completed in just a few hours or days may end up taking much longer before resources become available. Until then, iteration and learning come to a halt.
3. This narrows the context.
Division of labor can artificially restrict learning by rewarding individuals for staying within their specialization. For example, a research scientist who is expected to remain within their functional area will focus their energy on experiments with different types of algorithms: regression, neural networks, random forest, and so on. While a good algorithm choice can lead to incremental improvements, generally much more can be gained from other activities, such as integrating new data sources. Similarly, this helps in developing a model that utilizes every bit of explanatory power inherent in the data. However, its strength may lie in modifying the objective function or relaxing certain constraints. This is difficult to see or achieve when its work is limited. As the specialist scientist focuses on optimizing algorithms, they have much less chance to engage in anything else, even if it offers significant benefits.
Let's identify the signs that appear when data science teams operate like a pin factory (for example, in simple status updates): "waiting for changes to the data pipeline" and "waiting for ML Eng resources," which are common blockers. However, I believe the more dangerous influence lies in what you do not notice, as you cannot regret what you do not know yet. Flawless execution of requirements and complacency achieved through process efficiency can obscure the truth that organizations are not familiar with the learning benefits they are missing out on.
The solution to this issue, of course, lies in eliminating the pin factory approach. To foster learning and iteration, data science roles should be general, with broad responsibilities that do not depend on a technical functionâthis means organizing data specialists to be optimized for learning. This requires hiring "full-stack specialists"âgeneralists capable of performing various functions: from conception to modeling, from implementation to measurement. It is important to note that I do not suggest that hiring full-stack specialists should reduce the number of employees. Rather, I will just assume that when they are organized differently, their incentives are better aligned with the benefits of learning and efficiency. For example, you have a three-person team with three business qualities. In a pin factory, each specialist would dedicate a third of their time to each professional task, as no one else could perform their work. In a full-stack setup, each versatile employee is fully dedicated to the entire business process, scaling operations and learning.
With fewer people supporting the production cycle, coordination diminishes. A generalist smoothly transitions between functions, expanding the data pipeline to add greater volumes of data, experimenting with new features in models, deploying new versions in production for causal measurements, and repeating steps as quickly as new ideas emerge. Of course, the generalist performs various functions sequentially rather than in parallel. After all, itâs just one person. However, completing a task typically takes only a small fraction of the time needed to access another specialized resource. Thus, iteration time decreases.
Our generalist may not be as skilled as a specialist in a specific labor function, but we are not striving for functional perfection or minor gradual improvements. Rather, we aim to explore and uncover entirely new professional challenges with a gradual impact. With a holistic context for a comprehensive solution, they see opportunities that a narrowly focused specialist might miss. They have more ideas and greater possibilities. They also face failures. However, the cost of failure is low, while the benefits of learning are high. This asymmetry encourages rapid iteration and rewards learning.
It is important to note that the scale of autonomy and diversity of skills provided by a scientist working with full stacks largely depends on the reliability of the data platform they can rely on. A well-constructed data platform abstracts data processing scientists from the complexities of containerization, distributed processing, automatic failover to another resource, and other advanced computing concepts. In addition to abstraction, a robust data platform can ensure seamless connectivity to experimental infrastructure, automate monitoring and alert systems, provide automatic scaling, and visualize algorithmic results and debugging. These components are designed and built by data platform engineers; the data science specialist does not pass them to the data platform development team. It is the data science specialist who is responsible for all code applied to running the platform.
I was once intrigued by the functional division of labor using process efficiency, but through trial and error (the best way to learn), I discovered that typical roles better foster learning and innovation and provide accurate metrics: discovering and building far more business opportunities than a specialized approach. (A more effective way to learn about this organizational approach than the trial-and-error method I went through is to read Amy Edmondsonâs book, "Teaming: How Organizations Learn, Innovate, and Compete in the Knowledge Economy").
There are certain important assumptions that can make this organizational approach more or less reliable in some companies. The iteration process reduces the cost of trial and error. If the cost of failure is high, you may want to minimize them (though this is not recommended for medical applications or manufacturing). Additionally, if you are dealing with petabytes or exabytes of data, specialization in data design may be required. Likewise, if maintaining business opportunities online and their availability is more important than their improvement, functional superiority may outweigh training. Finally, the full-stack model relies on experts in the field. They are not unicorns; they can be found or trained internally. However, they are in high demand, and attracting and retaining them will require competitive financial compensation, sustainable corporate values, and engaging work. Ensure that your corporate culture can provide such conditions.
Even considering all of this, I believe that the full-stack model provides the best conditions for starting. Begin with them, and then consciously move towards functional specialization only when it is absolutely necessary.
There are also other drawbacks to functional specialization. It can lead to a loss of responsibility and passivity among workers. Smith himself criticizes the division of labor, suggesting that it leads to a dulling of talent, meaning that workers become ignorant and isolated as their roles are confined to a few repetitive tasks. While specialization may provide process efficiency, it is less likely to inspire workers.
Universal roles provide everything that stimulates job satisfaction: autonomy, mastery, and purpose. Autonomy means they are not dependent on external factors to achieve success. Mastery refers to the strong competitive advantages they hold. Purpose is the ability to make an impact on the business they create. If we can ignite a passion in people for their work and allow them to significantly influence the company, everything else will fall into place.
Source: habr.com
