SRE: Performance Analysis. A Configuration Method Using a Simple Web Server in Go

Performance analysis and tuning is a powerful tool for verifying performance compliance for clients.

Performance analysis can be used to identify bottlenecks in programs, applying a scientific approach to tuning experiment verification. This article outlines a general approach to performance analysis and tuning using a web server in Go as an example.

Go is particularly well-suited for this purpose, as it has profiling tools pprof in its standard library.

Strategy

Let's create a summary list for our structural analysis. We will try to use some data for decision-making instead of making changes based on intuition or guesswork. To do this, we will proceed as follows:

• Define the optimization boundaries (requirements);
• Calculate the transactional load for the system;
• Perform the test (create data);
• Observe;
• Analyze — are all requirements met?
• Tune scientifically, formulate a hypothesis;
• Conduct an experiment to validate this hypothesis.

Architecture of a Simple HTTP Server

For this article, we will use a small HTTP server written in Golang. All code from this article can be found here.

The analyzed application is an HTTP server that queries PostgreSQL on each request. Additionally, there is Prometheus, node_exporter, and Grafana for collecting and displaying application and system metrics.

For simplicity, we will assume that for horizontal scaling (and simplifying calculations), each service and database is deployed together:

Define the objectives

At this step, we define the objective. What are we trying to analyze? How will we know when it's time to stop? In this article, we will assume we have clients, and that our service will handle 10,000 requests per second.

In Google SRE Book discusses methods for selection and modeling in detail. We will do the same, building models:

• Latency: 99% of requests must be completed in less than 60ms;
• Cost: the service must consume the smallest amount of money that seems reasonably possible to us. To achieve this, we will maximize throughput;
• Capacity planning requires understanding and documenting how many instances of the application need to be launched, including the overall scaling function, as well as how many instances are needed to meet the initial load requirements and ensure redundancy. redundancy n+1.

Latency may require optimization in addition to analysis, but bandwidth needs to be clearly analyzed. In the SRE SLO process, the latency requirement comes from the client or business, represented by the product owner. Our service will meet this obligation from the very beginning without any adjustments!

Set up a test environment

With the test environment, we can apply a controlled load to our system. Performance data for the web service will be generated for analysis.

Transactional load

This environment uses Vegeta to create a configurable HTTP request rate until stopped:

$ make load-test LOAD_TEST_RATE=50
echo "POST http://localhost:8080" | vegeta attack -body tests/fixtures/age_no_match.json -rate=50 -duration=0 | tee results.bin | vegeta report

Monitoring

During execution, a transactional load will be applied. In addition to application metrics (number of requests, response latency) and operating system metrics (memory, CPU, IOPS), application profiling will be performed to understand where issues exist, as well as how CPU time is being consumed.

Profiling

Profiling is a type of measurement that allows you to see where CPU time is spent while the application is running. It helps determine exactly where and how much CPU time is being used:

This data can be used during analysis to gain insights into wasted CPU time and unnecessary work being performed. Go (pprof) can generate profiles and visualize them as flame graphs using a standard toolkit. I will discuss their usage and setup guidance further down in the article.

Execution, monitoring, analysis.

Let's conduct an experiment. We will execute, observe, and analyze until the performance meets our satisfaction. We will randomly choose a low load value to apply for obtaining the results of the first observations. At each subsequent step, we will increase the load by a scaling factor chosen with some variation. Each load test run is executed with an adjustment of the number of requests: make load-test LOAD_TEST_RATE=X.

50 requests per second

Notice the two top graphs. The upper left shows that our application handles 50 requests per second (in its perspective), while the upper right shows the duration of each request. Both parameters help us to watch and analyze whether we are within our performance boundaries or not. The red line on the graph HTTP Request Latency indicates an SLO of 60ms. The line shows that we are well below our maximum response time.

Let's look at it from a cost perspective:

10000 requests per second / 50 requests per server = 200 servers + 1

We can still improve this figure.

500 requests per second

More interesting things start happening when the load becomes 500 requests per second:

Again, the upper left graph shows that the application records the normal load. If not, there is an issue with the server where the application is running. The response time latency graph located at the top right shows that 500 requests per second resulted in a response delay of 25-40ms. The 99th percentile still fits superbly within the SLO of 60ms set above.

From a cost perspective:

10000 requests per second / 500 requests per server = 20 servers + 1

There is still room for improvement.

1000 requests per second

Great run! The application indicates it processed 1000 requests per second, but the latency limit was breached on the SLO side. This is evident from the p99 line on the upper right graph. Although the p100 line is much higher, the actual latencies exceed the maximum of 60ms. Let's dive into profiling to find out what the application is actually doing.

Profiling

For profiling, we set the load to 1000 requests per second, then use pprof to collect data to understand where the application is spending CPU time. This can be done by activating the HTTP endpoint. pprof, after which results are saved under load using curl:

$ curl http://localhost:8080/debug/pprof/profile?seconds=29 > cpu.1000_reqs_sec_no_optimizations.prof

The results can be displayed as follows:

$ go tool pprof -http=:12345 cpu.1000_reqs_sec_no_optimizations.prof

The graph shows where and how much CPU time the application consumes. From the description by Brendan Gregg:

On the X-axis — stack profile fills sorted alphabetically (this is not time), the Y-axis shows the stack depth, counting from zero at [top]. Each rectangle represents a stack frame. The wider the frame, the more frequently it appears in the stacks. What is above is CPU work, and below are child elements. Colors usually signify nothing and are simply chosen at random to differentiate frames.

Analysis — hypothesis

For setup, we will focus on trying to find wasteful CPU time usage. We will look for the largest sources of waste and remove them. Considering that profiling reveals exactly where the application spends its CPU time, it may need to be done several times, and changes in the application’s source code will be required, as well as restarting tests and observing how performance approaches the target.

Following Brendan Gregg's recommendations, we will read the graph from top to bottom. Each line represents a stack frame (function call). The first line is the entry point of the program, the parent of all other calls (in other words, all other calls will have it in their stack). The next line already differs:

Hovering over the function name on the graph will display the total time it spent in the stack during debugging. The HTTPServe function was there 65% of the time, while other runtime functions, runtime.mcall, mstart and gc, accounted for the remaining time. An interesting fact: 5% of the total time was spent on DNS requests:

The addresses the program is looking for belong to Postgresql. Click on FindByAge:

Interestingly, the program shows that, in principle, there are three main sources that add delays: opening and closing connections, data request, and connecting to the database. The graph shows that DNS requests, opening, and closing connections account for about 13% of the total execution time.

Hypothesis: Reusing connections through a pool should reduce the time of a single HTTP request, allowing for higher throughput and lower latency..

Application Configuration - Experiment

We are updating the source code, trying to eliminate the PostgreSQL connection for each request. The first option is using a connection pool at the application level. In this experiment, we will configure the connection pool using the SQL driver for Go:

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)

if err != nil {
   return nil, err
}

Execution, Monitoring, Analysis

After restarting the test with 1000 requests per second, it is clear that the p99 latencies have normalized with an SLO of 60ms!

What about the cost?

10000 requests per second / 1000 requests per server = 10 servers + 1

Let's make it even better!

2000 requests per second

Doubling the load shows the same result, the top left graph demonstrates that the application can handle 2000 requests per second, with p100 below 60ms, and p99 meeting the SLO.

From a cost perspective:

10000 requests per second / 2000 requests per server = 5 servers + 1

3000 requests per second

Here the application can handle 3000 requests with p99 latency under 60ms. The SLO is not violated, and the cost is calculated as:

10000 requests per second / 3000 requests per server = 4 servers + 1 (the author rounded up, translator's note)

Let's try one more round of analysis.

Analysis — hypothesis

We gather and display the debugging results of the application at 3000 requests per second:

Still, 6% of the time is spent on establishing connections. Configuring the pool has improved performance, but it is still evident that the application continues to create new connections to the database.

Hypothesis: Connections, despite the pool, are still being dropped and cleaned up, therefore the application needs to reinstall them. Setting the number of pending connections to pool size should help with latency by minimizing the time the application spends creating a connection..

Application Configuration - Experiment

We try to set MaxIdleConns equal to the pool size (also described here):

db, err := sql.Open("postgres", dbConnectionString)
db.SetMaxOpenConns(8)
db.SetMaxIdleConns(8)
if err != nil {
   return nil, err
}

Execution, Monitoring, Analysis

3000 requests per second

p99 is below 60ms with a significantly lower p100!

The flame graph check shows that connection establishment is no longer noticeable! Let's examine in detail. pg(*conn).query - we also do not notice connection establishment here.

Conclusion

Performance analysis is critical for understanding whether client expectations and non-functional requirements are met. Analyzing observations against client expectations can help determine what is acceptable and what is not. Go provides efficient tools built into the standard library that make analysis straightforward and accessible.

Source: habr.com