Cliff Click — CTO of Cratus (IoT sensors for process improvement), founder and co-founder of several startups (including Rocket Realtime School, Neurensic, and H2O.ai) with multiple successful exits. Cliff wrote his first compiler at the age of 15 (Pascal for the TRS Z-80)! He is best known for his work on C2 in Java (the Sea of Nodes IR). This compiler showed the world that JIT can produce quality code, which was one of the factors in establishing Java as one of the main modern programming platforms. Later, Cliff helped Azul Systems build an 864-core mainframe with pure Java software that supported GC pauses on a 500-gigabyte heap within 10 milliseconds. In general, Cliff has worked on all aspects of the JVM.

 
This hub post is a lengthy interview with Cliff. We will discuss the following topics:

• Transitioning to low-level optimizations
• How to do a large refactor
• Cost model
• Learning low-level optimizations
• Practical examples of performance improvement
• Why to create your own programming language
• Career of a performance engineer
• Technical challenges
• A little about register allocation and multi-core processing
• The biggest challenge of life

The interviewers are:

• Andrei Satarin from Amazon Web Services. In his career, he has worked on various projects: testing a NewSQL distributed database at Yandex, a cloud detection system at Kaspersky Lab, a multiplayer game at Mail.ru, and a currency pricing service at Deutsche Bank. He is interested in testing large-scale backend and distributed systems.
• Vladimir Sitnikov from Netcracker. For ten years, he has worked on performance and scalability of NetCracker OS — software used by telecom operators to automate network management processes and network equipment. He is passionate about Java and Oracle Database performance. He is the author of more than a dozen performance improvements in the official PostgreSQL JDBC driver.

Transitioning to low-level optimizations

Andrei: You are a well-known figure in the world of JIT compilation, Java, and performance work in general, right? 

Cliff: That's right!

Andrei: Let's start with general questions about performance work. What are your thoughts on the choice between high-level and low-level optimizations like working at the CPU level?

Cliff: It's quite simple here. The fastest code is the one that never runs. Therefore, you should always start at a high level and work on algorithms. A better Big O notation will outshine a worse one, unless some sufficiently large constants come into play. Low-level things come last. Typically, if you have optimized the entire stack well enough and still have something interesting left – that’s when you hit the low level. But how do you start from a high level? How can you know if you've done enough work at a high level? Well… you can't. There are no ready-made recipes. You need to understand the problem, decide what you're going to do (to avoid unnecessary steps later on), and then you can pull out the profiler that can tell you something useful. At some point, you realize you’ve eliminated unnecessary things and it’s time to focus on fine-tuning the low level. This is definitely a special kind of art. A lot of people do unnecessary things but move so quickly that they don't have time to care about performance. But that is until the issue becomes critical. Usually, 99% of the time, no one cares what I’m doing until a critical thing on the path comes up that someone finds important. And that’s when everyone starts questioning you about “why it didn't work perfectly from the start.” In general, there’s always something to improve in performance. But for 99% of the time, you have no clues! You’re just trying to make something work and during that process, you realize what is important. You can never know in advance that this particular piece needs to be perfect, so essentially you have to be perfect in everything. And that’s impossible, and you don't do it that way. There are always a ton of things to fix – and that’s perfectly normal.

How to do a large refactor

Andrei: How do you work on performance? It’s a cross-cutting issue. For example, have you ever had to deal with problems arising from the intersection of a lot of existing functionality?

Cliff: I try to avoid this. If I know that performance will be an issue, I think about it before I start coding, especially regarding data structures. But often, you discover all of this much later. Then you have to resort to extreme measures and do what I call 'rewrite and conquer': you need to grab a sufficiently large chunk. A part of the code will still have to be rewritten due to performance issues or something else. Whatever the reason for rewriting code, it's almost always better to rewrite a larger chunk than a smaller one. At that moment, everyone starts to panic: 'Oh no, we can't touch so much code!' But in fact, this approach almost always works much better. You need to tackle the big problem right away, draw a large circle around it, and say: everything inside this circle, I will rewrite. The boundary is much smaller than the content inside it that needs to be replaced. And if such boundary definition allows you to do the work inside perfectly – you’re free, do whatever you want. Once you understand the problem, the rewriting process becomes much easier, so take a big bite!
At the same time, when you are rewriting a large chunk and realize that performance will be an issue, you can start worrying about it right away. Usually, this translates into simple things like 'don't copy data, manage data as simply as possible, make it smaller.' In large rewrites, there are standard ways to improve performance. And they almost always revolve around data.

Cost model

Andrei: In one of your podcasts, you talked about cost models in the context of performance. Can you explain what you meant by that?

Cliff: Certainly. I was born in an era when CPU performance was extremely important. And that era is returning again — fate is not without irony. I began my life in the days of eight-bit machines, my first computer operated with 256 bytes. Yes, bytes. Everything was very small. You had to count instructions, and as we started moving up the programming language stack, the languages took on more and more. There was Assembly, then Basic, then C, and C handled many details, like register allocation and instruction fitting. But it was all fairly straightforward, and if I pointed to an instance of a variable, I'd get a load, and the cost of that instruction was known. The hardware gives a known number of machine cycles, so the execution speed of different tasks could be calculated simply by adding up all the instructions you intended to execute. Each compare/test/branch/call/load/store could be summed, and you could say: here’s your execution time. When improving performance, you definitely pay attention to the numbers corresponding to those small hot loops. 
But as soon as you switch to Java, Python, and similar languages, you quickly move away from low-level hardware. What is the cost of calling a getter in Java? If JIT in HotSpot got it right, inlined, it will be a load, but if it didn’t – it will be a function call. Since the call lies in a hot loop, it will cancel all other optimizations in that loop. Therefore, the actual cost will be much higher. And you immediately lose the ability to look at a piece of code and understand what it would take to execute it in terms of CPU clock cycles, memory used, and cache. All of this becomes interesting only if you really delve into performance.
We find ourselves in a situation where processor speeds have hardly increased for a decade. The old days are back! You can no longer rely on good single-thread performance. However, if you suddenly venture into parallel computing – it’s incredibly complex; everyone looks at you like you’re James Bond. Tenfold speedups usually occur in places where someone missed something. Parallelism requires a lot of effort. To achieve that tenfold speedup, you need to understand the cost model. What and how much it costs. And for that, you need to understand how the language fits onto the underlying hardware.
Martin Thompson found an excellent word for his blog Mechanical SympathyYou need to understand what the hardware is going to do, how it will do it, and why it does what it does at all. By using this understanding, it's quite easy to start counting instructions and figuring out where execution time is going. If you lack the relevant training, you're just searching for a black cat in a dark room. I constantly see people optimizing performance who have no idea what on earth they are doing. They struggle tremendously and don't really make any progress. And when I take the same piece of code, apply a few small hacks, and achieve a fivefold or tenfold speedup, they say: well, that's not fair, we already knew you were better. It's astonishing. What was I saying… the cost model is about what code you write and how fast it runs on average in the big picture.

Andrei: And how do you keep such a volume in your head? Is it achieved through a lot of experience, or? Where does such experience come from?

Cliff: Well, I gained my experience the hard way. I programmed in Assembly back when you could understand every single instruction. It sounds silly, but since then, a set of Z80 instructions has remained in my mind forever. I can’t remember the names of people just a minute after a conversation, but I remember code I wrote 40 years ago. It's funny; it feels like the "idiot savant».

Learning low-level optimizations

Andrei": Is there a simpler way to get into this?

Cliff: Yes and no. The hardware we all use hasn't changed much over time. Everyone still uses x86, except for smartphones which are on Arm. Unless you're deep into hardcore embedded development, you have essentially the same setup. Alright, moving on. Instructions haven't changed in centuries either. You need to go write something in Assembly. Just a bit, but enough to start understanding. You might smile, but I'm absolutely serious. It's crucial to grasp the correspondence between the language and the hardware. After that, you should go and write a little, and create a small toy compiler for a small toy language. 'Toy' means it should be accomplished in a reasonable time. It can be super simple, but it must generate instructions. The act of generating instructions will help you understand the cost model for the bridge between the high-level code that everyone writes and the machine code that runs on the hardware. This correspondence will be etched into your mind while writing the compiler. Even the simplest compiler. After that, you can start looking at Java, where the semantic gap is much deeper, and building bridges over it is significantly more complex. In Java, it’s much harder to determine whether our bridge is good or bad, what will cause it to collapse, and what won’t. But you need some starting point when you look at code and understand: 'Aha, this getter should inline every time.' Then it turns out that sometimes it does, except when the method becomes too large, and JIT starts inlining everything. The performance of such spots can be predicted instantly. Usually, getters perform well, but then you look at big hot loops and realize that there are some function calls floating around that do who knows what. This is the problem with the widespread use of getters; the reason they don’t inline is that it’s unclear whether it’s a getter or not. If you have a super small codebase, you can just memorize it and then say: this is a getter, and this is a setter. In a large codebase, each function lives its own story, which is generally unknown to anyone. The profiler says we lost 24% of the time on some loop, and to understand what that loop does, you need to look at every function inside. It's impossible to comprehend this without studying the function, and that seriously slows down the understanding process. That’s why I don’t use getters and setters; I’ve moved on to a new level!
Where can you find a cost model? Well, you can read something, of course… But I think the best way is to take action. Make a small compiler, and that will be the best way to understand the cost model and fit it into your own mind. A small compiler suitable for programming a microwave is a task for a beginner. I mean, if you already have programming skills, that should be enough. All these things like parsing a string which will be some algebraic expression, extracting the instructions for mathematical operations in the correct order, grabbing the right values from registers – it's all done easily. And while you're doing this, it'll print itself in your brain. I think everyone knows what a compiler does. And this will give an understanding of the cost model.

Practical examples of performance improvement

Andrei: What else should you focus on when working on performance?

Cliff: Data structures. By the way, I haven't conducted those classes in a long time… Rocket School. It was fun, but it required so much effort, and I have my own life too! Anyway. So, in one of the big and interesting classes, "Where Does Your Performance Go," I gave students an example: two and a half gigabytes of fintech data were read from a CSV file, and then it was necessary to calculate the number of products sold. Regular tick market data. UDP packets converted to text format, starting from the 70s. Chicago Mercantile Exchange – various items like oil, corn, soybeans, and such. It was necessary to count these products, the number of transactions, the average amount of funds and goods flowing, etc. This is quite simple trading math: find the product code (that's 1-2 characters in a hash table), get the sum, add it to one of the transaction sets, add the volume, add the cost, and a couple of other things. Very simple math. The toy implementation was very straightforward: everything is in a file, I read the file and move through it, splitting individual records into Java strings, searching for the needed things, and summing them up according to the aforementioned math. And it works with some small speed.

With this approach, everything that is happening is clear, and parallel computing won't help here, right? It turns out that you can achieve a fivefold increase in performance just by choosing the right data structures. This even surprises seasoned programmers! In my specific case, the insight was that memory allocations should not be made in a hot loop. Well, that's not the whole truth, but generally speaking – you shouldn't allocate 'once every X' when X is sufficiently large. When X is two and a half gigabytes, you shouldn't allocate anything 'once per letter', or 'once per line', or 'once per field', nothing of that sort. This is exactly where the time goes. How does this even work? Imagine I'm making a call String.split() or BufferedReader.readLine(). Readline creates a string from a set of bytes received over the network, once for each line, for each of hundreds of millions of lines. I take this string, analyze it, and discard it. Why do I discard it? Well, I've already processed it, that's it. So for every byte read from these 2.7G, two characters will be written in the string, which means already 5.4G, and I don’t need them anymore, so they are discarded. If we look at the memory bandwidth, we load 2.7G, which goes through the memory and memory bus into the processor, and then twice as much is sent into the string in memory, and all of this is processed when creating each new string. But I need to read it; the hardware reads it, even if everything will be processed later. And I must write it because I've created a string and the caches overflowed – the cache can't hold 2.7G. As a result, for each byte read, I read two additional bytes and write two additional bytes, creating a 4:1 ratio – that's how we waste memory bandwidth. And then it turns out that if I do String.split() – I certainly won't be doing this for the last time, as there might be another 6-7 fields inside. Therefore, the classic CSV reading code followed by string parsing leads to a memory bandwidth loss in the region of 14:1 compared to what you would actually like to have. If these allocations are discarded, you can achieve a fivefold speedup.

And it's not that it's very difficult. If you look at the code from the right angle, it all becomes quite simple, as soon as you grasp the essence of the problem. You should never stop allocating memory: the issue lies in the fact that you're allocating something and it dies immediately, consuming an important resource along the way, which in this case is memory bandwidth. And all this results in a performance drop. On x86, you usually need to actively burn CPU cycles, but here you've burned through all the memory much earlier. The solution is to reduce the number of allocations. 
Another part of the problem is that if you start the profiler when the memory bandwidth runs out, right at the moment it happens, you usually expect to return to cache because it's full of garbage that you've just spawned with all these strings. Therefore, each load or store operation becomes slow, as they lead to cache misses—the entire cache has become slow, waiting for the garbage to exit. So, the profiler will only show warm random noise, spread out along the entire cycle—there won't be any distinct hot instruction or code location. Just noise. And if you look at the GC cycles, they will all be in the Young Generation and super fast—microseconds or milliseconds at most. Because all this memory dies instantly. You allocate billions of gigabytes, and it cuts them off, and cuts them off, and cuts them off again. All this happens very quickly. So you end up with cheap GC cycles, warm noise over the entire cycle, but we want a 5-fold speedup. At this point, something should click in your mind and resonate: 'Why is it like this?!' Memory bandwidth overflow isn't shown in a classic debugger; you need to run a hardware performance counters debugger to see it yourself and directly. If you don't see it directly, you might suspect it from these three symptoms. The third symptom is when you look at what you're allocating, ask the profiler, and it replies: 'You've created a billion strings, but the GC worked for free.' Once this happens, you realize that you've spawned too many objects and burned through all the memory bandwidth. There is a way to figure this out, but it's not obvious. 

The issue lies in the data structure: a bare structure underlying everything that happens, it's too large, clocking in at 2.7G on disk, so making a copy of this is highly undesirable – the goal is to load it directly from the network byte buffer into the registers, avoiding reading and writing back and forth five times. Unfortunately, Java does not provide such a library as part of the JDK by default. But isn't this trivial? Essentially, it’s 5-10 lines of code that will implement a custom buffered line loader, mimicking the behavior of the String class, while wrapping the underlying byte buffer. As a result, you almost work with strings, but what’s actually happening is that pointers to the buffer are moving without any raw bytes being copied, thus reusing the same buffers over and over again, and the operating system is happy to take care of tasks it is designed for, like hidden double buffering of these byte buffers, while you stop grinding through an endless stream of unnecessary data. By the way, you do understand that when working with GC, it's guaranteed that each memory allocation won't be visible to the processor after the last GC cycle, right? Therefore, all of this cannot be cached, leading to a guaranteed 100% miss. When working with a pointer, on x86 reading a register from memory takes 1-2 cycles, and as soon as this happens, you pay, pay, pay, because all the memory is IN NINE caches – and this constitutes the cost of memory allocation. The real cost.

In other words, data structures are the hardest things to change. Once you realize you've chosen the wrong data structure that will ultimately kill performance, it usually requires substantial work to fix it. If you don't, things will only get worse. First and foremost, you need to think about data structures; this is important. The main cost lies in bloated data structures that start being used in a way like, 'I copied data structure X into data structure Y because I like the shape of Y more.' But the copying operation (which seems cheap) actually consumes memory bandwidth, and this is where all the execution time is lost. If I have a huge JSON string and I want to turn it into a structured DOM tree of POJO or something similar, the parsing of that string and building the POJO, followed by further calls to the POJO, will incur unnecessary costs – it's quite expensive. Unless you find yourself accessing the POJO much more often than the string. As a quick alternative, you might want to try decoding the string and extracting only what you need, without converting it into any POJO. If all of this happens on a path requiring maximum performance, then no POJO for you – you need to dig directly into the string.

Why to create your own programming language

Andrei: You said that to understand the cost model, you need to write your own little language...

Cliff: Not a language, but a compiler. A language and a compiler are different things. The most important distinction is in your head. 

Andrei: By the way, as far as I know, you're experimenting with creating your own languages. Why?

Cliff: Because I can! I'm semi-retired, so it's my hobby. I've spent my life implementing other people's languages. I've also worked a lot on coding style. Moreover, I see problems in other languages. I notice there are better ways to do common tasks, and I would take advantage of them. I'm just tired of seeing issues in myself, in Java, in Python, and in any other language. Right now, I'm writing in React Native, JavaScript, and Elm as a hobby that isn't retirement but rather about being actively engaged. I'm also writing in Python and will probably continue working on machine learning for Java backends. There are many popular languages, each with interesting features. Each has its strengths, and you can try to consolidate all these features. So, I’m studying things that interest me, the behavior of languages, trying to come up with sensible semantics. So far, I'm succeeding! At the moment, I'm grappling with memory semantics because I want it to behave like in C and Java, providing a strong model and semantics for loads and stores, while also having type inference like in Haskell. I'm trying to mix Haskell-like type inference with memory working like in C and Java. I've been working on this for the last 2-3 months, for example.

Andrei: If you're building a language that takes the best aspects from others, have you considered that someone might do the opposite: take your ideas and use them?

Cliff: That's exactly how new languages emerge! Why is Java similar to C? Because C had a good syntax that everyone understood, and Java was inspired by that syntax while adding type safety, array bounds checks, garbage collection, and improving certain aspects of C. They also added their own features. But they were quite inspired, right? Everyone stands on the shoulders of giants who came before – that’s how progress is made.

Andrei: As I understand, your language will be safe with respect to memory usage. Have you considered implementing something like Rust's borrow checker? Have you looked at it, and what do you think?

Cliff: Well, I have been programming in C for ages, dealing with all that malloc and free, and manually managing lifetimes. You know, 90-95% of manually managed lifetimes have the same structure. And it's very, very painful to handle that manually. I wish the compiler could just tell me what’s going on and what I've achieved with my actions. For some tasks, the borrow checker does that out of the box. It should also automatically infer information, understand everything, and not burden me with detailing that understanding. It should at least perform local escape analysis, and only if it fails, then I should add type annotations describing the lifetime – and that kind of scheme is much more complex than a borrow checker or any existing memory checker. The choice between 'everything is fine' and 'I don't understand anything' — no, there should be something better. 
As someone who has written a lot of C code, I believe that having support for automatic lifetime management is crucial. I am also frustrated by how much memory Java uses, with the primary complaint being its garbage collection (GC). When memory is allocated in Java, you don't get back the memory that was local during the last GC cycle. In languages with more precise memory management, that isn’t the case. When you call malloc, you immediately receive memory that was just used. Typically, you do temporary things with that memory and return it right away. It immediately goes back to the malloc pool, and the next malloc cycle pulls it back out. Therefore, actual memory usage decreases to the set of live objects at any given moment, plus any leaks. If you don’t have significant leaks, most of the memory ends up in caches and the processor, and this works quickly. But it requires a lot of manual memory management using malloc and free, called in the right order and at the right time. Rust can handle this correctly on its own and in many cases can even provide greater performance, as memory consumption narrows down to just the current computations—opposed to waiting for the next GC cycle to free up memory. In the end, we found a very interesting way to enhance performance. It’s quite powerful—in fact, I have worked on similar things while processing data for fintech, which allowed for about a fivefold speedup. That’s a significant boost, especially in a world where processors aren’t getting faster, yet we continue to expect improvements.

Career of a performance engineer

Andrei: I'm also curious to ask about your career in general. You became well-known for your work on JIT in HotSpot and then moved to Azul—which is also a JVM company. But you shifted more towards hardware than software. Then suddenly you switched to Big Data and Machine Learning, and later to fraud detection. How did that happen? These are very different areas of development.

Cliff: I have been programming for quite a while and have had the opportunity to work on very different projects. And when people say, "Oh, you're the one who developed the JIT for Java!" it’s always amusing. Before that, I worked on a clone of PostScript—the language that Apple once used for its laser printers. Prior to that, I implemented the Forth language. I think my common theme has been tool development. I have spent my life creating tools that enable others to write their amazing programs. But I also worked on operating systems, drivers, kernel-level debuggers, and languages for OS development that started off simple, but gradually became more and more complex. Nevertheless, the main theme is still tool development. A significant part of my life was spent between Azul and Sun, and that time was focused on Java. However, when I delved into Big Data and Machine Learning, I put on my formal hat and said, "Oh, now we have a non-trivial problem, and a lot of interesting things and people who are doing something is happening here." This is a great path for development that is worth taking.

Yes, I really love distributed computing. My first job was during my studies in C, working on an advertising project. It involved distributed computing on Zilog Z80 chips, which gathered data for analog optical character recognition produced by a real analog analyzer. It was a cool and completely unconventional topic. However, there were issues; some parts were not recognized correctly, so it was necessary to retrieve the image and show it to a person who would read it visually and report what it said. Hence, there were data jobs, and these jobs had their own language. There was a backend that processed everything – parallel-working Z80s with running vt100 terminals – one per person. Additionally, there was a parallel programming model on Z80. A shared block of memory that all the Z80s inside a star-type configuration shared; both the backplane and half of the RAM were shared within the network, and the other half was private or used for something else. It was a meaningfully complex parallel distributed system with shared... semi-shared memory. When was that... I can't even remember, somewhere in the mid-80s. Quite a while ago. 
Yes, let's consider that 30 years is quite a long time. Tasks related to distributed computing have existed for a long time; people have been fighting with them for ages. Beowulf-clusters. Such clusters look like... For example, there is Ethernet and your fast x86 is connected to this Ethernet, and now you want to obtain fake shared memory, because back then no one could deal with coding distributed computing, it was too complex and therefore there was fake shared memory with page protection on x86, and if you wrote to this page, we informed other processors that if they accessed the same shared memory, it would need to be loaded from you, and thus something like a cache coherence protocol and software for this emerged. An interesting concept. The real problem, of course, lay elsewhere. All of this worked, but you quickly ran into performance issues, as no one understood the performance model at a sufficiently good level – what the memory access patterns were, how to ensure that nodes didn’t constantly ping each other, and so on.

In H2O, I came up with the idea that the developers themselves need to determine where parallelism exists and where it doesn't. I devised a coding model that makes writing high-performance code easy and straightforward. However, writing slow code is challenging; it tends to look poor. It takes a concerted effort to write slow code, often requiring non-standard methods. Poorly performing code is immediately noticeable. Consequently, code is typically written to execute quickly, but you have to understand what to do in the case of shared memory. All of this hinges on large arrays, and the behavior is reminiscent of non-volatile large arrays in parallel Java. Imagine two threads writing to a parallel array; one wins, and the other loses, and you don't know which is which. If they’re not volatile, the order can be anything—and that really works well. People genuinely care about the order of operations, correctly placing volatile where needed and anticipating performance issues tied to memory. Otherwise, they might just write loops from 1 to N, where N represents trillions, hoping all complex cases will automatically become parallel—this approach doesn't work. But H2O isn't Java or Scala; you can consider it 'Java minus minus' if you like. It's a very understandable programming style, akin to writing simple code in C or Java with loops and arrays. Yet, it allows you to process terabytes of memory. I still use H2O. Occasionally, I employ it in various projects—and it's still the fastest thing out there, far surpassing its competitors. If you’re doing Big Data with columnar data, it’s very challenging to beat H2O.

Technical challenges

Andrei: What has been the biggest challenge in your career?

Cliff: Are we discussing the technical or non-technical aspect of the question? I would say the biggest challenges are non-technical. 
Regarding the technical challenges, I simply overcame them. I don't even know what the biggest one was, but there were several quite interesting ones that took a lot of time and mental effort. When I joined Sun, I was confident that I would create a fast compiler, while a bunch of senior staff told me that I would never succeed. But I went down that path, wrote a compiler all the way down to the register allocator, and it was quite fast. It was just as fast as the modern C1, but back then the allocator was much slower, and looking back – it was a problem of a large data structure. I needed it to write a graphical register allocator, and I didn't grasp the dilemma between code expressiveness and speed that existed at that time and was very important. It turned out that the data structure often exceeded the cache size on x86 machines of that era, and therefore, if I initially assumed that the register allocator would take 5-10 percent of the entire JIT time, in reality, it turned out to be 50 percent.

As time passed, the compiler became clearer and more efficient, stopped generating dreadful code in more cases, and performance increasingly resembled that of a C compiler. Unless, of course, you wrote some junk that even C wouldn't speed up. If you wrote code as you would in C, you got performance akin to C in more cases. And the further we went, the more often we produced code that asymptotically matched C level, the register allocator started to resemble something complete… regardless of whether your code ran fast or slow. I continued to work on the allocator to achieve better allocations. It became slower and slower, but delivered increasingly better performance in cases where no one else could manage. I could dive into the register allocator, bury a month of work in it, and suddenly the whole code would start running 5% faster. This happened over and over, and the register allocator became something like a work of art – everyone either loved or hated it, and people from academia would ask questions like 'why is everything done this way?' linear scanning, and what is the difference. The answer remains the same: a graph-coloring allocator combined with very precise buffer code handling equals a winning tool, the best combination that no one can surpass. This is quite a subtle point. Everything else that the compiler does are relatively well-understood concepts, though also perfected to the level of art. I have always focused on aspects that should turn the compiler into a work of art. But nothing of this was extraordinary – with the exception of the register allocator. The key is to handle things carefully trim under load and, if this happens (I can explain in more detail if interested), it means that more aggressive inlining can be done without the risk of exceeding the performance graph's breaking point. Back then, there were plenty of full-fledged compilers, loaded with bells and whistles, that had register allocators, but no one could manage it like that anymore.

The problem is that if you add methods eligible for inlining, expanding the inlining area more and more, the set of used values quickly surpasses the number of registers, and you have to spill. The critical level usually occurs when the allocator gives up, and one good candidate for spilling is worth another, leading to some really wild spills. The value of inlining lies in that you lose part of the overhead, the overhead of calls and saves, you can see the values inside and further optimize them. The cost of inlining is that a large number of live values are created, and if your register allocator spills more than necessary, you immediately lose. Therefore, most allocators face the problem: when inlining crosses a certain threshold, everything starts spilling, and performance can go down the drain. Those who implement the compiler add certain heuristics: for example, to stop inlining after reaching a sufficiently large size since allocations would spoil everything. Thus, a break in performance graph forms – you inline, inline, performance slowly grows – and then bam! – it drops down rapidly like a jack, because you've inlined too much. This is how it worked until Java appeared. Java requires much more inlining, so I had to make my allocator much more aggressive to keep it aligned instead of falling, and if you inline too much – it starts to spill, but there still comes a moment of 'no more spilling'. This is an interesting observation that came to me out of nowhere, not obvious, but very rewarding. I took to aggressive inlining and it led me to places where the performance of Java and C run side by side. They are really close – I can write Java code that is significantly faster than C code and so on, but on average, in the bigger picture, they are roughly comparable. I think part of this credit goes to the register allocator which allows me to inline in a very straightforward manner. I simply inline everything I see. The question here is whether the allocator works well, resulting in reasonably functioning code. That was a big challenge: to understand all this and make it work.

A little about register allocation and multi-core processing

Vladimir: Issues like register allocation seem like an endlessly recurring topic. I'm curious, was there ever an idea that seemed promising but then failed in practice?

Cliff: Of course! Register allocation is a domain where, to tackle an NP-complete problem, you're trying to come up with some heuristics. And you can never achieve a perfect solution, right? It's just impossible. Look at Ahead of Time compilation—it also performs poorly. The discussion here is about average cases. Typical performance, so you can go and measure something that you consider to be good typical performance—after all, you're working to improve it! Register allocation is a topic entirely focused on performance. Once you have your first prototype, it works and paints what’s needed, then performance tuning begins. You need to learn how to measure effectively. Why is this important? If you have clear data, you can look at different parts and see: ah, this helped here, but there it all broke down! Good ideas emerge, you add a new heuristic, and suddenly everything starts to work slightly better on average. Or it doesn't. I have had numerous cases where we fought for five percent of performance that distinguished our development from the previous allocator. And every time it looks like this: you win some, you lose some. If you have good performance analysis tools, you can identify losing ideas and understand why they fail. Maybe it's best to leave everything as is, or perhaps take a more serious approach to fine-tuning, or go fix something else. It's a whole set of things! I made this cool hack, but I also need this, and this, and this— and their combined effect brings some improvements. And single changes can fall short. That's the nature of working on NP-complete performance problems.

Vladimir: It seems that tasks like coloring in allocators are already solved problems. Well, at least for you, based on what you’re telling us, so should it really be…

Cliff: It hasn't been resolved as such. It's up to you to turn it into a "resolved" one. There are tough problems that need solutions. Once that is done, it's time to focus on performance. You should approach this work appropriately – conduct benchmarks, gather metrics, clarify situations when rolling back to a previous version causes your old hack to work again (or vice versa, not work anymore). And don't back down until you achieve something. As I said, if there are cool ideas that didn't work, in the realm of register allocation, the ideas are almost infinite. For example, you can read scholarly publications. Although, now this field has slowed down and become clearer than in its early days. Nonetheless, an entire infinity of people works in this field, and all of their ideas are worth trying; they all await their moment. And you can't say how good they are until you try. How well they integrate with everything else in your allocator, since the allocator does many things, and some ideas might not work in your specific allocator but work perfectly in another one. The main way to win for an allocator is to pull the slow stuff out of the main path and forcibly split along the boundaries of the slow paths. Therefore, if you want to trigger GC, go down the slow path, deoptimize, throw an exception, all that sort of thing – you know those things are relatively rare. And they are indeed rare; I have checked. You do extra work, and as a result, many constraints on those slow paths disappear, but that isn't very important because they are slow, and traffic on them is infrequent. For example, a null pointer – it never occurs, right? You need to have several paths for different things, but they shouldn't interfere with the main one. 

Vladimir: What do you think about multicore systems when there are thousands of cores? Is it a useful thing?

Cliff: The success of GPUs shows that it's quite useful!

Vladimir: They are quite specialized. But what about general-purpose processors?

Cliff: Well, that was Azul's business model. The response came back in an era when people really appreciated predictable performance. Back then, writing parallel code was quite challenging. The H2O coding model scales well, but it's not a general-purpose model. It's just a bit more general than using a GPU. Are we talking about the complexity of developing such a thing or the complexity of using it? For example, an interesting lesson Azul taught me, which isn’t very obvious: small caches are just fine. 

The biggest challenge of life

Vladimir: What about non-technical challenges?

Cliff: The biggest challenge was to not be… kind and nice to people. Consequently, I often found myself in extremely conflictive situations. Those in which I knew everything was going awry, but I didn’t know how to move forward in solving these problems and couldn't handle them. Many long-standing issues, lasting for decades, arose this way. The fact that Java has both C1 and C2 compilers is a direct consequence of this. The lack of multi-level compilation in Java for a decade is also a direct outcome. It was obvious that we needed such a system, but it's not clear why it didn't exist. I had problems with one engineer… or a group of engineers. A long time ago, when I started working at Sun, I was… Well, not just then; I generally have my own opinion on everything. I believed it was true that you could simply take your truth and state it bluntly. Especially since I was often shockingly right. And if you don’t like that approach… particularly if you are obviously wrong and making mistakes… In general, few people could tolerate such a form of communication. Although some could, like me. I built my entire life on meritocratic principles. If you show me something incorrect, I'll immediately turn around and say: you are wrong. At the same time, I would of course apologize and all that, note merits if they exist at all, and take other correct actions. On the other hand, I am shockingly right a shockingly large percentage of the time. And this doesn’t work very well in relationships with people. I am not trying to be nice; I present the issue bluntly. “This will never work because one, two, and three.” And they are like: “Oh!”. There were also other consequences, which are probably better left unmentioned: for instance, those that led to a divorce from my wife and a decade of depression afterward.

A challenge is a struggle against people's perceptions of what you can or cannot do, what is important and what is not. There have been many challenges regarding coding styles. I still write a lot of code, and back then, I had to slow down because I was juggling too many tasks poorly, instead of focusing on one. Looking back, I wrote half of the Java JIT team's code, the C2 team. The next fastest coder was writing at half my speed, the next one was even slower, and it was an exponential decline. The seventh person in this lineup was very, very sluggish – that's how it always is! I touched a lot of code. I watched what everyone was writing, without exception; I scrutinized their code, reviewed each of them, and still continued to write more than any of them. This approach doesn't work very well with people. Some don't like it. And when they can't handle it, all sorts of complaints start. For example, once I was told to stop writing code because I was writing too much code, and this endangered the team; it all sounded like a joke to me: dude, if the rest of the team disappears and I keep writing code, you would only lose half the team. On the other hand, if I keep writing code and you lose half the team – that sounds like really bad management. I never thought much about it, never spoke of it, but it was still somewhere in my mind. In the back of my consciousness, a thought was spinning: 'Are you all joking?'. So, the biggest problem was me and my relationships with people. Now I understand myself much better; I led programmers for a long time, and now I directly tell people: you know, this is who I am, and you'll have to deal with me – is it okay if I stand here? And once they started handling that, everything worked. I'm neither bad nor good; I have no bad intentions or selfish ambitions; this is just my essence, and somehow we have to live with it.

AndreiRecently, there has been a lot of talk about self-awareness for introverts and soft skills in general. What can be said about that?

CliffYes, this was an understanding and a lesson I took from the divorce with my wife. What I gained from the divorce was a deeper understanding of myself. This led me to begin to understand other people as well. Understanding how this interaction works resulted in discoveries one after another. I became aware of who I am and what I represent. What I do: either I am focused on a task, or I avoid conflict, or something else – and this level of self-awareness really helps keep myself in check. After that, everything becomes much easier. One thing I've found not only in myself but also in other programmers is the inability to verbalize thoughts when you are in a state of emotional stress. For instance, you’re coding, in a flow state, and suddenly someone runs in, yelling in a panic that something has broken, and extreme measures will be taken against you. And you can’t say a word because you’re in a state of emotional stress. The knowledge gained allows you to prepare for that moment, to experience it, and to switch to a backup plan after which you can do something. So yes, when you begin to realize how all of this works – it’s a huge life-changing event. 
I couldn't find the right words myself, but I remembered the sequence of actions. The essence is that this reaction is as physical as it is verbal, and you need space. Such space, in the Zen sense. This is exactly what needs to be explained, and then step aside immediately – physically step aside. When I’m silent verbally, I can process the situation emotionally. As adrenaline reaches the brain, switching you to ‘fight or flight’ mode, you can’t say anything; no – now you’re an idiot, a punching bag, incapable of a worthy response or even of halting the attack, and the aggressor can freely attack again and again. First, you need to become yourself again, regain control, and exit the ‘fight or flight’ mode.

And for that, a verbal space is necessary. Just free space. If you must say anything at all, you can declare this and then go and actually find yourself some "space": take a walk in the park, lock yourself in the shower – it doesn’t matter. The main thing is to temporarily disconnect from that situation. As soon as you disconnect, even for a few seconds, control returns, and you start thinking clearly. "Okay, I’m not any kind of idiot, I don’t do stupid things, I’m quite a useful person." Once you've convinced yourself of this, it’s time to move to the next stage: to understand what happened. You were attacked; the attack came from an unexpected place; it was an unfair, sneaky ambush. That’s bad. The next step is to understand why the attacker needed this. Really, why? Maybe because they are furious themselves? Why are they furious? For example, because they messed up and can't take responsibility? This is how you need to carefully process the whole situation. But for this, you need room to maneuver, a verbal space. The very first step is to break off verbal contact. Move away from the discussion verbally. Cancel it, walk away as quickly as possible. If it's a phone conversation – just hang up – that's a skill I learned from dealing with my ex-wife. If the conversation is going nowhere good, just say "goodbye" and hang up. On the other end of the line: "blah blah blah," you respond: "uh-huh, bye!" and hang up. You just cut off the conversation. Five minutes later, when your ability to think rationally returns, you cool down a bit, and it becomes possible to reflect on what actually happened and what will happen next. And start formulating a thoughtful response rather than simply reacting emotionally. For me, a breakthrough in self-awareness was realizing that in the case of emotional stress, I can’t speak. Getting out of that state, thinking it through, and planning how to respond and address the issues – these are the right steps when you can’t speak. The simplest way is to escape from the situation where emotional stress manifests and just stop participating in that stress. After that, you regain the ability to think, and when you can think, the ability to speak becomes available, and so on.

By the way, in court, the opposing lawyer is trying to do this to you — now it’s clear why. Because he has the ability to overwhelm you to the point where you can’t even say your name, for example. Literally, you won’t be able to speak. If this is happening to you and you know you’ll be in a place where verbal battles are boiling, like in a courtroom, you can come with your lawyer. The lawyer will stand up for you and stop the verbal attack, and he will do it in a completely legal way, allowing you to regain your lost zen space. For example, I had to call my family a couple of times, and the judge was quite friendly about it, but the opposing lawyer was shouting and yelling at me; I couldn’t even get a word in. In such cases, using a mediator works best for me. The mediator stops all that pressure that flows over you continuously, and you find the necessary zen space, along with the ability to speak returning. This is an entire field of knowledge where you need to learn a lot, discover many things within yourself, and all of this turns into high-level strategic solutions, which differ for different people. Some people don’t have the above-mentioned problems, usually, those who are professional in sales don’t face them. All those who earn their living with words – renowned singers, poets, religious figures, and politicians – always have something to say. They don’t have these problems, but I do.

Andrei: That was… unexpected. Great, we’ve talked quite a bit, and it’s time to wrap up this interview. We will definitely meet at the conference and can continue this dialogue. See you at Hydra!

You can continue the conversation with Cliff at the Hydra 2019 conference, which will take place on July 11-12, 2019, in St. Petersburg. He will be coming with a presentation ‘The Azul Hardware Transactional Memory Experience’. Tickets can be purchased on the official website.

Source: habr.com