We continue our dive into the fascinating world of troubleshooting logs. In we agreed on the meaning of basic terms and took a glimpse at the overall structure of Veeam as a unified application. The task now is to understand how log files are generated, what information is displayed in them, and why they look the way they do.
What do you think these "logs" actually are? Most people believe that the logs of any application should serve as an all-powerful entity that spends most of its time lurking somewhere in the background, only to emerge unexpectedly in shining armor to save the day. This means they should contain everything, from the tiniest errors in every component to individual database transactions. And it should immediately provide ways to correct any errors. And all of this should fit into a couple of megabytes, at most. It's just text! Text files can't possibly take up tens of gigabytes; I've heard that somewhere!
So, logs
In the real world, logs are merely an archive of diagnostic information. It's up to the developers to decide what to store, where to get the information for storage, and how detailed it should be. Some choose minimalism, storing only ON/OFF level records, while others diligently gather everything they can reach. There's also an intermediate option involving the choice of a so-called Logging Level, where you specify how detailed the information you want to store is and how much extra disk space you have =) VBR has six such levels, by the way. And believe me, you don’t want to see what happens with the most detailed logging when you have free space on your disk.
Okay. We have a general idea of what we want to log, but the legitimate question arises: where do we get this information? Some events for logging, of course, are generated by our internal processes. But what to do when interacting with external environments? To avoid descending into a chaotic mess of workarounds, Veeam tends to not reinvent the wheel. Whenever there is an existing API, built-in function, library, etc., we prefer to use those solutions before we start crafting our clever fixes. Though there are many clever fixes as well. Therefore, when analyzing logs, it's important to understand that a significant portion of errors comes from messages sent by external APIs, system calls, and other libraries. In this case, VBR's role is to forward these errors to the log files as is. The main task of the user is to learn how to understand which line comes from whom and what this 'who' is responsible for. Therefore, if the error code from the VBR log leads you to an MSDN page, that is normal and correct.
As we agreed earlier: Veeam is a so-called SQL-based application. This means that all settings, all information, and everything necessary for normal operation is stored in its database. Hence, the simple truth: what is not in the logs is likely in the database. But this is not a silver bullet: some things are found in neither the local logs of Veeam components nor in its database. Thus, one must learn to examine the host logs, local machine logs, and logs from everything involved in the backup and restore processes. There are also times when the needed information is not found anywhere at all. Such is the way.
A few examples of such APIs
This list does not aim to be exhaustively complete, so don’t look for the absolute truth within it. Its purpose is simply to show the most common third-party APIs and technologies used in our products.
Let's start with VMware.
First on the list will be vSphere API. It is used for authentication, reading the hierarchy, creating and deleting snapshots, querying information about machines, and much (very much) more. The functionality of the solution is very extensive, so I can recommend the VMware vSphere API Reference for version and . For more current versions, it's simply a matter of Googling.
VIX API. The black magic of the hypervisor, for which there is a separate . VMware API for working with files on the host without connecting to them over the network. A last-resort option when you need to put a file on a machine with no better communication channel available. It can be a pain and suffering if the file is large and the host is busy. But here the rule applies that even 56.6 Kb/s is better than 0 Kb/s. In Hyper-V, a similar feature is called PowerShell Direct. But that was only until the emergence of
vSphere Web Services API . Starting with vSphere 6.0 (approximately, since this API was first introduced in version 5.5), it is used for working with guest machines and has practically replaced VIX everywhere. Essentially, this is another API for managing vSphere. For those interested, I recommend studying the manual.
VDDK (Virtual Disk Development Kit). A library that was partially discussed in this . It is used for reading virtual disks. Long ago, it was part of VIX, but over time it was separated into its own product. However, as a successor, it uses the same error codes as VIX. Yet for some reason, the SDK does not include any description of these errors. Therefore, it was discovered through experience that VDDK errors with different codes are merely a translation from binary to decimal code. It consists of two parts – the first half contains undocumented information about the context, while the second part includes traditional VIX/VDDK errors. For example, if we see:
VDDK error: 21036749815809.Unknown error
Then we confidently convert this to hex and get 132200000001. The uninformative beginning 132200 is simply discarded, and the remainder is our error code (VDDK 1: Unknown error). Recently, there was a separate discussion on the most common VDDK errors. .
Now let's look at Windows.
Here you can find everything essential and important for us in the standard Event Viewer. But there is one catch: by long-standing tradition, Windows logs not the full text of the error, but only its number. For example, error 5 means “Access denied,” error 1722 is “The RPC server is unavailable,” and 10060 indicates “Connection timed out.” Of course, it's great if you remember the most well-known ones, but what about the previously unseen?
To ensure that life doesn't seem too sweet, errors are also stored in hexadecimal format, starting with the prefix 0x8007. For example, 0x8007000e actually means 14, Out of Memory. The reason behind this is a mystery. However, you can download the complete list of errors for free and without SMS from .
By the way, sometimes other prefixes appear, not just 0x8007. In such a sad situation, understanding HRESULT ("result handle") requires delving even deeper into for developers. In everyday life, I wouldn't advise you to do this, but if you find yourself in a tight spot or are simply curious, now you know what to do.
But the folks at Microsoft have shown some mercy and introduced a utility . This is a small piece of console joy that can translate error codes into human-readable format without using Google. It works roughly like this.
C:UsersrootDesktop>err.exe 0x54f
# for hex 0x54f / decimal 1359
ERROR_INTERNAL_ERROR winerror.h
# An internal error occurred.
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x54f
# for hex 0x54f / decimal 1359
ERROR_INTERNAL_ERROR winerror.h
# An internal error occurred.
# 2 matches found for "0x54f"A legitimate question arises: why don't we write the explanation in the logs right away, leaving these mysterious codes? The answer lies in third-party applications. When you invoke a WinAPI call yourself, decoding its response is straightforward since there's even a special WinAPI call for that. But as already mentioned, our logs capture everything that comes to us in responses. Here, to decode, one would have to constantly monitor this stream of consciousness, extract the bits with Windows errors, decode them and insert them back. To be honest, it's not the most captivating activity.
Windows File Management API is used in various file operations. Creating files, deleting, opening for writing, working with attributes, and so on.
The mentioned PowerShell Direct acts as an analog to the VIX API in the Hyper-V world. Unfortunately, it is not as flexible: it has many functional limitations, works with not every version of the host, and far from all guests.
RPC (Remote Procedure Call) I believe there is hardly anyone who has worked with Windows and hasn't encountered RPC-related errors. Contrary to popular belief, it is not a single protocol, but any client-server protocol that meets certain criteria. However, if we see an RPC error in our logs, in 90% of cases, it will be an error from Microsoft RPC, which is part of DCOM (Distributed Component Object Model). There is a vast amount of documentation available on this topic online, but a good portion of it is quite outdated. If there is a strong desire to dive into the topic, I can recommend articles. , and a long list .
The main causes of RPC errors in our logs are failed attempts at interaction between VBR components (server > proxy, for example), and most often due to communication issues.
The top error among all is The RPC server is unavailable (1722). Simply put, the client could not establish a connection with the server. There is no single answer to why this occurs, but it is generally a problem with authentication or network access to port 135. The latter is typical in infrastructures with dynamically assigned ports. There is even a . And Microsoft has a on troubleshooting.
The second most common error is: There are no more endpoints available from the endpoint mapper (1753). An RPC client or server could not assign a port. This usually occurs when the server (in our case, the guest machine) is configured to dynamically allocate ports from a narrow range that has been exhausted. If we look from the client side (in our case, the VBR server), this means our VeeamVssAgent either did not start or was not registered as an RPC interface. There is also information on this topic. .
And to complete the Top-3 RPC errors, let's remember RPC function call failed (1726). This occurs when the connection is established, but RPC requests do not process. For example, we request the status information of VSS (in case a shadow copy is being made right now, and we are trying to access it), and in response, we hear silence and receive no information.
Windows Tape Backup API is needed to work with tape libraries or drives. As I mentioned at the beginning: writing our own drivers and struggling with the support of each device is not enjoyable for us at all. That's why Veeam doesn't have any proprietary drivers. Everything goes through the standard API, which is supported by the hardware vendors themselves. That makes much more sense, right?
SMB/CIFS Everyone usually writes them together, although not many remember that CIFS (Common Internet File System) is just a private version of SMB (Server Message Block). So there is nothing wrong with generalizing these concepts. Samba, on the other hand, is the Linux/Unix implementation, and it has its own peculiarities, but I digress. What is important here is that when Veeam requests to write something via the UNC path (serverdirectory), the server uses the hierarchy of file system drivers, including mup and mrxsmb, to write to the share. Consequently, these drivers will also generate errors.
It is impossible to do without Winsock API. If something needs to be done over the network, VBR works through the Windows Socket API, commonly known as Winsock. So when we see an IP:Port pair in the log, that's it. The official documentation has a decent list of possible .
The mentioned WMI (Windows Management Instrumentation) is a kind of omnipotent API for managing everything and anything in the Windows world. For example, when working with Hyper-V, almost all requests to the host go through it. In short, it is an absolutely indispensable tool and very powerful in its capabilities. In attempts to help determine where and what is broken, the built-in tool WBEMtest.exe is very helpful.
And the last on the list, but certainly not the least in importance — VSS (Volume Shadow Storage). This topic is as inexhaustible and mysterious as the amount of documentation written about it. Shadow Copy is easiest to understand as a special type of snapshot, which is essentially what it is. Thanks to it, application-consistent backups can be made in VMware, and in Hyper-V, almost everything can be done. I plan to write a separate article with a summary on VSS, but for now, you can try reading . Just be careful, as trying to understand VSS at a glance may lead to severe headaches.
On that note, I think we can stop here. I consider the task of explaining the most basic things accomplished, so in the next chapter, we will look at the logs. But if you have any questions, feel free to voice them in the comments.
Source: habr.com
