At Veeam, we love logs. And since most of our solutions are modular, they generate quite a few logs. Given that our field is ensuring the safety of your data (i.e., peaceful sleep), the logs must not only record every little detail but do so quite comprehensively. This is necessary so that in case of an incident, it is clear how it happened, who is at fault, and what needs to be done next. It's like in criminal investigation: you never know which small detail will help you find Laura Palmer's killer.
That's why I decided to embark on a series of articles where I will sequentially explain what we log, where we store them, how not to lose our minds over their structure, and what to look for in them.
Why a series of articles and why not describe everything at once?
Simply listing where each log is located and what it contains is a rather futile endeavor. It's even scary to think about keeping that information up-to-date. Just enumerating all possible types of logs in Veeam Backup & Replication would be a table spanning several pages in small print. Moreover, it would only be relevant at the time of publication, since a new patch might introduce new logs or change the logic for stored information in existing ones, etc. Therefore, it will be much more beneficial to explain their structure and the essence of the information they contain. This will allow for better navigation than mere rote memorization of names.
So, to avoid diving headfirst into a sea of text, let's do some preparatory work in this article. Therefore, today we won't delve into the logs themselves, but will approach from a distance: we'll create a glossary and discuss Veeam's structure from the perspective of log generation.
Glossary and Jargon
First of all, I must apologize to the advocates of the purity of the Russian language and the adherents of Ozhogov's dictionary. We all love our native language, but the damn IT industry operates in English. We didn't create it; it just historically developed that way. It's not my fault, it just came on its own.
In our field, the issue of anglicisms (and jargon) has its own specifics. When innocent words like "host" or "guest" mean very specific things across one-sixth of the land, a heroic confusion and uncertainty continues on another part of the globe, with people pointing to dictionaries. And the obligatory argument is "But at our workplace...".
Additionally, there is our own terminology that pertains specifically to Veeam products, although some words and phrases have permeated the general populace. So now we will agree on what each term means, and from here on out, when I say "guest," I will mean exactly what is written in this chapter, not what you are used to at your workplace. And yes, this is not just my personal whim; these are established industry terms. It is somewhat pointless to fight against them. However, I am always up for some discussion in the comments.
Unfortunately, there are a lot of terms in our work and products, so I wonāt attempt to list them all. Only the most basic and essential ones for navigating the sea of information about backups and logs. For those interested, I can also by colleagues about tapes, where he also provided a list of terms relating to that part of the functionality.
Host: In the world of virtualization, this is a machine with a hypervisor. Physical, virtual, cloudāit's all the same. If a hypervisor (ESXi, Hyper-V, KVM, etc.) is running on something, then that "something" is called a host. Whether it's a cluster of ten racks or your laptop with one and a half virtual machinesāif a hypervisor is running, it has become a host. Because the hypervisor hosts virtual machines. Thereās even a rumor that VMware once wanted to firmly associate the word host specifically with ESXi. But they didnāt succeed.
In the modern world, the concept of "host" has almost merged with the concept of "server," which adds a certain confusion to communication, especially when it comes to Windows infrastructure. Thus, any machine hosting an interesting service can confidently be called a host. For example, in WinSock logs, everything is tagged with the word host. The classic "Host not found" is a prime example. So we should rely on context, but rememberā in the world of virtualization, a host is what hosts the guests (about which we will talk in a couple of lines below).
Among local jargon (or rather acronyms in this case), it's worth mentioning that VMware is VI, vSphere is VC, and Hyper-V is HV.
Guest: A virtual machine running on a host. There's really nothing to explain; everything is so logical and simple. However, many are keen to drag in other meanings.
Why? I don't know.
Guest OS, accordingly, is the operating system of the guest machine. And so on.
Backup/Replication Job: A pure VMware jargon term referring to a task. Backup job == Backup Job. No one has figured out a nice translation for this into Russian, so everyone just says 'job' with emphasis on the last syllable.
Yes, that's how simply they say 'job'. Even in emails, they write it like that, and everything is fine.
Various Backup jobs, Backup Tasks, etc., thanks, but no. Just 'job', and you'll be understood. The main thing is to stress the last syllable.
Backup (Backup, or for true old-schoolers, 'bakup'): Aside from the obvious (a backup copy of data stored somewhere), it also means the job itself (three lines above, if you've already forgotten), resulting in that very backup file. It's likely that English speakers are too lazy to say 'I ran my backup job' every time, so they simply say 'I ran my backup', and everyone understands each other perfectly. I suggest we support this wonderful initiative.
Consolidate: A term introduced in ESXi 5.0, an option in the snapshot menu that initiates the process of removing orphaned snapshots. That is, snapshots that physically exist but have fallen out of the displayed logical structure. Theoretically, this process should not affect the files displayed in the snapshot manager, but things happen. The essence of the consolidation process is that data from the snapshot (child disk) is written to the main (parent) disk. The process of merging disks is called a merge. If a consolidation command is issued, the entry for the snapshot may be removed from the database before the snapshot is merged and deleted. If the snapshot fails to delete for any reason, these orphaned snapshots appear. VMware has a decent KB on working with snapshots. on Habr. .
Datastore (Storage or Storage): Ā A very broad concept, but in the world of virtualization, it refers to a location where virtual machine files are stored. However, it's essential to understand the context clearly and, in case of any doubts, to clarify what exactly your interlocutor meant.Ā
Proxy: It's important to understand that Veeam Proxy is not quite the same as what we are accustomed to in the realms of the internet. Within Veeam's products, it is an entity responsible for transferring data from one location to another. To put it simply, VBR is the command server, and the proxy is its workhorse. In other words, the proxy is the machine through which traffic flows and on which VBR components are installed to help manage that traffic. For example, transferring data from one channel to another or simply attaching disks (HotAdd mode).
Repository: Ā Technically, this is simply an entry in the VBR database indicating where backups are stored and how to connect to that location. In practice, it can be anything from a simple CIFS share to a separate disk, server, or bucket in the cloud. Again, we are in context, but it is understood that the repository is merely a place where your backups reside.
Ā Snapshot: Those fond of Oxford grammar prefer to say 'snapshOt' or 'snapshOt,' but the illiterate majority wins due to sheer volume. For those unaware, this is a technology that allows for the restoration of a disk's state at a specific point in time. This is achieved either by temporarily redirecting I/O operations away from the main diskāthen it is called a RoW (Redirect on Write) snapshotāor by moving rewritten blocks from your disk to anotherāthis is known as a CoW (Copy on Write) snapshot. Thanks to the extensive capabilities of these functions, Veeam can perform its backup magic. Strictly speaking, not only it but this will be covered in upcoming releases.
In the documentation and logs of ESXi, there is confusion surrounding this term, and in the context of mentioning snapshots, you can find both the snapshots themselves and the redo log, as well as delta disks. There is no such chaos in Veeam's documentation, where a snapshot is simply a snapshot, and a redo log refers specifically to a REDO file created by an independent non-persistent disk. REDO files are deleted when the virtual machine is shut down, so confusing them with snapshots is a path to failure.
Synthetic: Synthetic backups relate to reverse incremental and forever forward backups. If you haven't encountered this term before, it is simply one of the mechanisms used to construct backup chains. However, you may also find the term Transform in the logs, which is used in the context of creating full copies from increments (synthetic full).
Task: This is the process of handling each individual machine within a job. For example, if you have a backup job that includes three machines, each machine will be processed within its own task. Thus, there will be four logs: one main log for the job and three for the tasks. However, there is an important nuance: over time, the word ātaskā has become overly ambiguous. When we talk about general logs, we mean that a task refers specifically to a VM. But there are also ātasksā on the proxy and the repository, where it can refer to a virtual disk, a virtual machine, or the entire job. So it's important not to lose the context.
Veeam %name% Service:: Ā Several services work for the success of backups, and their list can be found in the standard snap-in. Their names transparently reflect their essence, but among equals, the most important one is the Veeam Backup Service, without which the others will not function.
VSS: Technically, VSS should always refer to Microsoft Volume Shadow Copy Service. However, it is often used by many as a synonym for Application-Aware Image Processing. Which is, of course, categorically incorrect, but this is a story akin to saying 'Any SUV can be called a Jeep, and you'll be understood.'
Fantastic logs and the places they inhabit
I want to start this chapter by revealing a great secret ā what time is displayed in the logs?
Remember:
- ESXi always writes logs in UTC+0.
- vCenter logs time according to its own time zone.
- Veeam logs time and time zone based on the server it is running on.
- Only Windows events in EVTX format are not tied to anything. When opened, the time is recalculated for the machine they're opened on. This is the most convenient option, although there can be complications with it. The only significant difficulty is the difference in locales. This is practically a guaranteed path to unreadable logs. Yes, there are ways to address this, but let's not argue that everything in IT works in English, and let's agree to always set the servers to English locale. Please.Ā
Now let's talk about where the logs live and how to obtain them. In the case of VBR, there are two approaches.Ā
The first option is suitable if you don't feel like digging through a pile of files to find those related to your issue. For this, we have a separate wizard that allows you to specify a particular job and the specific time period for which you need logs. It will then automatically browse through the folders and compile everything needed into one archive. Detailed information about where to find it and how to work with it is provided in .
However, the wizard does not collect logs for all jobs, and if you need to analyze logs from the restaurant, failover, or failback, your path leads to the folder %ProgramData%/Veeam/Backup. This is the main log storage for VBR, and %ProgramData% is a hidden folder, which is normal. By the way, the default location can be re-assigned using a registry key of type REG_SZ: LogDirectory in the branch HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Backup and Replication.
On Linux machines, logs for the worker agents should be sought in /var/log/VeeamBackup/, if using a root or sudo account. If you do not have such privileges, look for logs in /tmp/VeeamBackup.Ā
For Veeam agent for %OS_name%, logs should be found in %ProgramData%/Veeam/Endpoint (or %ProgramData%/Veeam/Backup/Endpoint) and /var/log/veeam respectively.
If you are using Application-Aware Image Processing (and most likely you are), the situation becomes a bit more complicated. You will need the logs from our helper, which are stored within the virtual machine itself, as well as VSS logs. Detailed information on how and where to acquire these is provided in . And of course, there is on collecting the necessary system logs.Ā
Windows events can be conveniently collected according to . If you are using Hyper-V, it gets more complicated as you will also need all its logs from the Applications and Service Logs > Microsoft > Windows branch. However, you can always take the more straightforward approach and simply grab all objects from %SystemRoot%System32winevtLogs.
If something breaks during installation/upgrade, everything you need can be found in the %ProgramData%/Veeam/Setup/Temp folder. However, I won't hide the fact that you can find more useful information in the OS events than in these logs. The remaining interesting data lies in %Temp%, but primarily there are installation logs of accompanying software, like the database, .Net libraries, and so on. Keep in mind that Veeam is installed from an MSI, and all its components are also installed as separate MSI packages, even if this is not displayed in the GUI. Therefore, if the installation of one of the components fails, the entire VBR installation will be halted. Thus, you need to check the logs to see what exactly broke and at what moment.
And a final life hack: when you encounter an error during installation, don't rush to click OK. First, gather the logs, then click OK. This way, you'll receive the log that ends at the moment of the error, without junk at the end.
Sometimes, you need to dig into the vSphere logs. It's a thankless task, but rolling up your sleeves, you have to do even that. In the simplest case, we need the logs with events from the virtual machine, vmware.log, which are located next to its .vmx file. In a more complex case, open Google and ask where the logs are for your version of the host, as VMware loves to change that location from release to release. Here, for example, , and here for . For vCenter logs, we repeat the procedure of . But in general, we are interested in the host events logs, hostd.log, events of hosts managed by vCenter, vpxa.log, kernel logs, vmkernel.log, and authentication logs, auth.log. In the most complicated cases, the SSO log, located in the SSO folder, might also come in handy.
Bulky? Confusing? Scary? Yet, this is not even half the information our support team deals with on a daily basis. So they are indeed very cool.
Veeam Components
And as a conclusion to this introductory article, let's talk a bit about the components of Veeam Backup & Replication. Because when you're looking for the cause of the pain, it would be good to understand how the patient is structured.
As everyone is surely aware, Veeam Backup is a SQL-based application. This means that all settings, all information, and everything necessary for its normal operation is stored in its database. Specifically, in two databases if we are talking about the VBR and EM combination: VeeamBackup and VeeamBackupReporting, respectively. That's just how it is: install another application, and another database appears. This way, we don't keep all our eggs in one basket.
But for everything to work smoothly, we need a set of services and applications that will connect all components together. Just as an example, hereās how it looks in one of my labs:
Serving as the main conductor is Veeam Backup Service. It is responsible for information exchange with the databases. It also handles the initiation of all tasks, orchestrates the allocated resources, and serves as a communication center for various consoles, agents, and more. In short, itās indispensable, but that doesnāt mean it does everything by itself.
Assisting in its execution is Veeam Backup Manager. This is not a service, but rather an entity that launches jobs and monitors their execution process. The working hands of the backup service that connect to hosts, create snapshots, monitor retention, and so on.
Now, back to the list of services. Veeam Broker Service. Introduced in v9.5 (and no, it's not a crypto miner, as some initially thought). It collects information about VMware hosts and keeps it up to date. But don't rush to write angry comments claiming we're spying on you and leaking all your logins/passwords to the major. It's a bit simpler than that. When you initiate a backup, the first step is to connect to the host and refresh all data about its structure. This is quite a slow and cumbersome process. Just remember how long it takes for you to log in via the web interface, and keep in mind that only the top layer is considered there. And then you also have to expand the entire hierarchy to the required location, by the way. In a word, it's a nightmare. If you're running a dozen backups, then each job has to go through this procedure. If we're talking about large infrastructures, this process can take ten minutes or more. Therefore, it was decided to allocate a separate service for this, through which you can always obtain up-to-date information. Upon startup, it checks and scans all added infrastructure, and then tries to work only at the level of incremental changes. So even if you have a hundred backups starting simultaneously, they will all request information from our broker instead of burdening the hosts with their queries. If you're concerned about resources, our estimates suggest that around 100 MB of memory is needed for 5000 virtual machines.
Next, we have Veeam Console. Also known as Veeam Remote Console or Veeam.Backup.Shell. This is the GUI that we see in the screenshots. It's simple and straightforwardāthe console can be launched from anywhere, as long as it's Windows and has connectivity to the VBR server. The only thing to mention is that the FLR process will mount points locally (i.e., on the machine where the console is running). And the various Veeam Explorers will also run locally, as they are part of the console. But that's already leading me down a rabbit hole...
The next interesting service is Veeam Backup Catalog Data Service. In the list of services, it is known as the Veeam Guest Catalog Service. It is responsible for indexing file systems on guest machines and populating the VBRCatalog folder with this knowledge. It is used only where indexing is enabled. And it makes sense to enable it only if you have Enterprise Manager. So hereās a piece of advice: donāt enable indexing just for the sake of it if you donāt have EM. Save your nerves and support time.
Also worth mentioning from other important services is Veeam Installer Service, which facilitates the delivery and installation of necessary components on proxies, repositories, and other gateways. Essentially, it transfers the required .msi packages to the servers and conducts their installation.Ā
Veeam Data Mover ā which operates using auxiliary agents running on proxies (and beyond) for data transfer. For instance, during a backup, one agent will read files from the host's datastore, while the other meticulously writes them to the backup.
I would like to highlight an important thing that clients often react to ā the difference in versions of services and the information within the Programs and Features snap-in. Yes, the list will be the same, but the versions may not match at all. While it may look a bit messy visually, it's completely normal if everything is functioning stably. For example, the Installer Serviceās version number falls significantly behind its neighbors. Is it a disaster? No, because it doesnāt get entirely reinstalled; its DLL simply gets updated. In patch v9.5 U4, tech support experienced a nightmare: during the update, all services received new versions except for the most crucial one. In patch U4b, the transport service outpaced all others by two versions (based on the numbers). This is also normal ā a serious bug was found in it, which is why it received a bonus update compared to the others. So, in summary: the version difference CAN be an issue, but if a difference exists and everything is working correctly, itās likely meant to be that way. However, no one forbids you from checking with tech support for clarification.
These were the so-called mandatory services. There is also a whole set of auxiliary ones, such as Tape Service, Mount Service, vPowerNFS Service, and so on.
For Hyper-V, everything is pretty much the same, only there is a specific Veeam Backup Hyper-V Integration Service and its own driver for working with CBT.
And finally, let's discuss who works on virtual machines during backups. To run pre- and post-freeze scripts, create shadow copies, collect metadata, work with SQL transaction logs, and more, we use Veeam Guest Helper. And if file system indexing occurs, Veeam Guest Indexer . These are temporary services deployed during the backup and removed afterward.
In the case of Linux machines, it's much simpler due to the multitude of built-in libraries and the capabilities of the system itself. For example, indexing is done using mlocate.
That's all for now.
I wonāt keep you any longer, and a brief introduction to the inner workings of Veeam is concluded. Yes, we haven't even touched on the logs yet, but believe me, this introduction is essential to ensure that the information presented in them doesn't seem like a disjointed stream of consciousness. I plan to delve into the logs themselves only in the third article, and the next one will explain who generates the logs, what exactly is displayed in them, and why this is the case rather than anything else.
Source: habr.com
