We at Veeam love logs. And since most of our solutions are modular, they write a lot of logs. And since the scope of our activity is to ensure the safety of your data (i.e., restful sleep), then the logs should not only record every sneeze, but also do it in some detail. This is necessary so that in case of something it is clear how this “what” happened, who is to blame, and what needs to be done next. It's like in forensic science: you never know what little thing will help you find Laura Palmer's killer.
Therefore, I decided to take a swing at a series of articles, where I will sequentially talk about what we write to the logs, where we store them, how not to go crazy with their structure and what to look for inside them.
Why a series of articles and why not describe everything at once?
Simply listing which log is where and what is stored in it is a rather disastrous undertaking. And it’s scary to even think about keeping this information up to date. A simple listing of all possible types of logs in Veeam Backup & Replication is a table on several sheets in small print. Yes, and it will be relevant only at the time of publication, because. when the next patch is released, new logs may appear, the logic of the stored information in the old ones will change, etc. Therefore, it will be much more profitable to explain their structure and the essence of the information contained in them. This will allow you to better navigate the places than the banal cramming of names.
Therefore, in order not to rush headlong into the pool of text sheets, let's do some preparatory work in this article. Therefore, today we will not get into the logs themselves, but will go from afar: we will compile a glossary and discuss the Veeam structure a little in terms of generating logs.
Glossary and jargon
Here, first of all, it is worth apologizing to the champions of the purity of the Russian language and the witnesses of Ozhegov's dictionary. We all love our native language very much, but the damned IT industry operates in English. Well, we didn’t come up with it, but it happened historically. It's not my fault, he came himself (c)
In our business, the problem of anglicisms (and jargon) has its own specifics. When under innocent words like “host” or “guest” the whole world has long understood very specific things, then on ⅙ of the land, heroic confusion and staggering with poking into dictionaries continues. And the strictly obligatory argument "But at our work ...".
Plus, there is purely our terminology, which is inherent in Veeam products, although some words and phrases have gone to the people. Therefore, now we will agree on what term means what, and in the future, under the word “guest”, I will mean exactly what is written in this chapter, and not what you are used to at work. And yes, this is not my personal whim, these are well-established terms in the industry. Fighting them is somewhat pointless. Although I'm always in favor of chilling out in comments.
Unfortunately, there are a lot of terms in our work and products, so I will not try to list them all. Only the most basic and necessary information about backups and logs for survival in the sea. For those interested, I can also colleagues about tapes, where he also gave a list of terms related to that part of the functionality.
Host (Host): In the world of virtualization, this is a machine with a hypervisor. Physical, virtual, cloud - it doesn't matter. If something is running a hypervisor (ESXi, Hyper-V, KVM etc), then this “something” is called a host. Whether it's a cluster with ten racks or your laptop with a lab for one and a half virtual machines - if you launched a hypervisor, you became a host. Because the hypervisor hosts virtual machines. There is even a story that VMware at one time wanted to achieve a firm association of the word host with ESXi. But she didn't.
In the modern world, the concept of "host" has practically merged with the concept of "server", which brings some confusion to communication, especially when it comes to Windows infrastructure. So any machine that hosts some service of interest to us can be safely called a host. For example, in the WinSock logs everything is marked with the word host. The classic "Host not found" is an example of this. So we start from the context, but remember - in the world of virtualization, a host is what hosts guests (more on this in two lines below).
From local jargon (rather even acronyms, in this case), it is recalled here that VMware is VI, vSphere is VC, and Hyper-V is HV.
Guest (Guest): The virtual machine running on the host. There is nothing to explain here, everything is so logical and simple. However, many diligently drag here some other meanings.
For what? I don't know.
Guest OS, respectively, the operating system of the guest machine. And so on.
Backup/Replication Job (jobA): Pure Wim jargon, denoting some of the tasks. Backup job == Backup Job. No one has figured out how to translate it beautifully into Russian, so everyone says “JobA”. With emphasis on the last syllable.
Yes, they simply take it and say “joba”. And even in letters they write like that, and everything is fine.
All sorts of Backup jobs, Backup Tasks, etc., thanks, but no need. Just a job, and you will be understood. The main thing is to put the stress on the last syllable.
Backup (Backup, backup. For true-oldfags, backup is allowed): In addition to the obvious (a backup copy of data lying somewhere), it also means the job itself (three lines above, if you have already forgotten), as a result of which the very backup file appears. Probably, gentlemen native English speakers are too lazy to say I ran my backup job every time, so they just say I ran my backup, and everyone understands each other perfectly. I invite you to support this wonderful initiative.
Consolidate (Consolidation): A term that appeared in ESXi 5.0 An option in the snapshot menu that starts the process of deleting so-called orphaned snapshots. That is, snapshots that are physically available, but fell out of the displayed logical structure. Theoretically, this process should not affect the files displayed in the snapshot manager, but anything can happen. The essence of the consolidation process is that the data from the snapshot (child disk) is written to the main (parent) disk. The process of combining disks is called merge. If a consolidation command has been issued, then the snapshot record can be removed from the database before the snapshot is merged and deleted. And if the snapshot could not be deleted for any reason, then these same orphaned snapshots appear. About working with snapshots, VMware has . And we also somehow about them .
Datastore (Stora or storage): A very broad concept, but in the world of virtualization, it is understood as a place where virtual machine files are stored. But in any case, here you need to understand the context very clearly and, with the slightest doubt, clarify what exactly your interlocutor had in mind.
Proxy (Proxy): It is important to immediately understand that Veeam Proxy is not quite the same as what we are used to on the Internet. Within Veeam products, this is a kind of entity that deals with transferring data from one place to another. If you do not go into details, then VBR is a command and control server, and proxies are its workhorses. That is, a proxy is a machine through which traffic flows and on which VBR components are installed that help to steer this traffic. For example, to transfer data from one channel to another, or simply to cling disks to itself (HotAdd mode).
Repository (Repository): Technically, this is just an entry in the VBR database, indicating the place where the backups are stored, and how to connect to this place. In fact, it can be either just a CIFS ball or a separate disk, server or bucket in the cloud. Again, we are in context, but we understand that a repository is just a place where your backups are.
Snapshot (SnapshOt): Oxford grammar buffs prefer to say who is snapshot and who is snapshot, but the illiterate majority benefit from the larger mass. If anyone does not know, this is a technology that allows you to restore the state of a disk at a certain point in time. This is done either by temporarily redirecting I / O operations away from the main disk - then it will be called RoW (Redirect on Write) snapshot - or by moving rewritable blocks from your disk to another - this will be called CoW (Copy on Write) snapshot. It is thanks to the wide possibilities for using these functions that Veeam can work its backup magic. Strictly speaking, not only them, but this is the matter of the next releases.
There is chaos around this term in the ESXi documentation and logs, and in the context of mentioning snapshots, you can find snapshots themselves, and redo log, and even delta disk. The Veeam documentation does not contain such a tear, and a snapshot is a snapshot, and a redo log is exactly a REDO file created by an independent non-persistent disk. REDO files are deleted when the virtual machine is turned off, so confusing them with snapshots is a path to failure.
Synthetic (Synthetics): Synthetic backups are reverse incremental and forever forward backups. In case you haven't come across this term, it's just one of the mechanisms used to build a backup chain transformation. However, in the logs you can also find the concept of Transform, which is used in the framework of creating full copies from increments (synthetic full).
Task (Task): This is the process of processing each individual machine within the job. That is: you have a backup job, which includes three machines. This means that each car will be processed as part of a separate task. In total, there will be four logs: the main one for jobs and three for tasks. However, there is an important nuance here: over time, the word "task" has become unnecessarily ambiguous. When we talk about general logs, we mean that a task is exactly a VM. But there are “tasks” both on the proxy and on the repository. There it can mean a virtual disk, a virtual machine, and the entire job. That is, it is important not to lose the context.
Veeam %name% Service: For the benefit of successful backups, several services work at once, a list of which can be found in the standard equipment. Their names quite transparently reflect their essence, but among equals there is the most important one - Veeam Backup Service, without which the rest will not work.
VSS: Technically, VSS should always stand for Microsoft Volume Shadow Copy Service. In fact, it is used by many as a synonym for Application-Aware Image Processing. Which, of course, is categorically wrong, but this is a story from the category "Any SUV can be called a jeep, and you will be understood."
Fantastic logs and where they live
I want to start this chapter by revealing the great secret - what time is displayed in the logs?
Remember:
- ESXi always writes logs in UTC+0.
- vCenter keeps logs according to the time of its time zone.
- Veeam keeps logs by time and timezone of the server it is on.
- And only Windows events in EVTX format do not suffer from binding to anything. When opened, the time is recalculated for the car on which they were opened. The most convenient option, although there are difficulties with it. The only tangible difficulty is the difference in locales. This is a practically guaranteed path to unreadable logs. Yes, there are options for how to treat this, but let's just not argue with the fact that everything in IT works in English, and agree to always set the English locale on the servers. Oh please.
Now let's talk about the places where the logs live and how to get them. In the case of VBR, there are two approaches.
The first option is suitable if you are not eager to look for files in the general heap that are specifically related to your trouble. To do this, we have a separate wizard, to which you can specify a specific job and a specific period for which you need logs. Then he will go over the folders himself and put everything you need into one archive. Where to look for it and how to work with it is described in detail in .
However, the wizard does not collect the logs of all tasks and, for example, if you need to study the logs of the restor, failover or failback, your path lies in the folder %ProgramData%/Veeam/Backup. This is the main VBR logostore and %ProgramData% is a hidden folder and that's fine. By the way, the default location can be reassigned using the REG_SZ: LogDirectory type registry key in the HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Backup and Replication branch.
On Linux machines, worker agent logs should be looked for in /var/log/VeeamBackup/if using a root or sudo account. If you do not have such privileges, then look for logs in /tmp/VeeamBackup.
For Veeam agent for %OS_name% logs should be searched in %ProgramData%/Veeam/Endpoint (or %ProgramData%/Veeam/Backup/Endpoint) and /var/log/veeam respectively.
If you are using Application-Aware Image Processing (and most likely you are), then the situation becomes somewhat more complicated. You will need the logs of our helper, which are stored inside the virtual machine itself, and the VSS logs. About how and where to get this happiness, it is written in detail in . And of course there is to collect the necessary system logs.
Windows events are conveniently collected according to . If you are using Hyper-V, things get more complicated, as you will also need all of its logs from the Applications and Service Logs > Microsoft > Windows branch. Although you can always go the more stupid way and just pick up all the objects from %SystemRoot%System32winevtLogs.
If something breaks during installation/upgrade, then everything you need can be found in the %ProgramData%/Veeam/Setup/Temp folder. Although I will not hide the fact that in OS events you can find more useful information than in these logs. The rest of the interesting lies in %Temp%, but there are mainly installation logs for related software, such as the base, .Net libraries and other things. Note that Veeam is installed from msi and all of its components are also installed as separate msi packages, even if this was not shown in the GUI. Therefore, if the installation of one of the components fails, the entire VBR installation will be stopped. Therefore, you need to go into the logs and see what exactly broke and at what point.
And finally, a life hack: if you receive an error during installation, do not rush to click OK. First we take the logs, then click OK. This way you will get a log that ends at the time of the error, without garbage at the end.
And it happens that you need to get into the vSphere logs. The occupation is very ungrateful, but, having rolled up the sleeves, one has to do something else. In the simplest version, we need logs with virtual machine events vmware.log, which are located next to its .vmx file. In a more difficult case, open Google and ask where the logs for your host version are located, since VMware loves to change this place from release to release. For example, , but for . For vCenter logs, repeat the procedure . But in general, we will be interested in host event logs hostd.log, host events managed by vCenter vpxa.log, kernel logs vmkernel.log and authentication logs auth.log. Well, in the most neglected cases, the SSO log, which lies in the SSO folder, may come in handy.
Cumbersome? Confused? Scary? But this is not even half of the information that our support works with on a daily basis. So they are really, really cool.
Veeam Components
And as a conclusion to this introductory article, let's talk a little about the components of Veeam Backup & Replication. For when you are looking for the cause of pain, it would be nice to understand how the patient works.
So, as everyone probably knows, Veeam Backup is a so-called SQL-based application. That is, all settings, all information and in general everything that is only necessary for normal functioning - all this is in its database. Or rather, in two databases, if we are talking about a bunch of VBR and EM: VeeamBackup and VeeamBackupReporting, respectively. And so it happened: we put another application - another database appears. In order not to store all the eggs in one basket.
But for all this economy to work smoothly, we need a set of services and applications that will tie all the components together. Just as an example, this is what it looks like in one of my labs:
Acts as chief conductor Veeam Backup Service. It is he who is responsible for the exchange of information with the bases. He is also responsible for launching all tasks, orchestrating allocated resources and working as a sort of communications center for a variety of consoles, agents and everything else. In a word, there is definitely no way without him, but this does not mean at all that he does everything himself.
Helps him in the fulfillment of his plan Veeam Backup Manager. This is not a service, but an entity that launches jobs and monitors the process of their execution. Backup service's working hands, with which it connects to hosts, creates snapshots, monitors retention, and so on.
But back to the list of services. Veeam Broker Service. Appeared in v9.5 (and this is not a crypto miner, as some thought then). Collects information about VMware hosts and maintains its relevance. But do not immediately run to write angry comments that we are spying on you and leaking all logins / passwords to the taschmajor. Everything is somewhat simpler. When you run a backup, the first thing you need to do is connect to the host and update all data about its structure. This is a rather slow and cumbersome story. Just remember how long it takes for you to login through the web interface, and remember that only the top layer is counted there. And then you still need to open the entire hierarchy to the right place, by the way. In a word, horror. If you run a dozen backups, then each job needs to do this procedure. If we are talking about large infrastructures, then this process can take ten minutes or more. Therefore, it was decided to allocate a separate service for this, through which it will be possible to receive always up-to-date information. At startup, it checks and scans all added infrastructure, and then tries to work only at the level of incremental changes. So even if you run a hundred backups at the same time, they will all request information from our broker, and will not torment the hosts with their requests. If you are worried about resources, then according to our calculations, 5000 virtual machines need only about 100 Mb of memory.
Next we have Veeam Console. He is Veeam Remote Console, he is Veeam.Backup.Shell. This is the same GUI that we see in the screenshots. Everything is simple and obvious - the console can be launched from anywhere, as long as it is Windows and there is a connection to the VBR server. The only thing that can be said is that the FLR process will mount points locally (i.e. on the machine where the console is running). Well, assorted Veeam Explorers will also run locally, because they are part of the console. But it has already carried me into the wilds ...
Another interesting service is Veeam Backup Catalog Data Service. Known as Veeam Guest Catalog Service in the list of services. He is engaged in indexing file systems on guest machines and fills the VBRCatalog folder with this knowledge. It is used only where the indexing checkbox is enabled. And it only makes sense to enable it if you have Enterprise Manager. Therefore, advice from the bottom of my heart: do not turn on indexing just like that if you do not have EAT. Save your nerves and support time.
Also from other important services it is worth noting Veeam Installer Service, with the help of which the necessary components are delivered and installed on proxies, repositories and other gateways. In fact, it takes the necessary .msi packages to the servers and installs them.
Veeam Data Mover - with the help of auxiliary agents launched on proxies (and not only) it is engaged in shifting data. For example, when backing up, one agent will read files from the host datastore, and the second will carefully write them to the backup.
Separately, I would like to note an important thing that clients often react to - this is the difference in versions of services and information in the Programs and Features snap-in. Yes, the list will be the same, but the versions can be completely discordant. It's not very cool from a visual point of view, but it's completely normal if everything works stably. For example, for the Installer service, the version number is far behind the neighboring ones. Horror and nightmare? No, because it is not completely reinstalled, but its DLL is simply updated. In patch v9.5 U4, a technical support nightmare occurred: during the update, all services received new versions, except for the most important one. In the U4b patch, the transport service overtook all the others by as much as two versions (judging by the numbers). And this is also normal - a serious bug was found in it, so it received a bonus update relative to the rest. So to sum it up: version differences COULD be a problem, but if there is a difference and everything is working properly, then it probably should be. But no one forbids you to clarify this in technical support.
These were the so-called mandatory or Mandatory services. And there is a whole bunch of auxiliary ones, such as Tape Service, Mount Service, vPowerNFS Service and so on.
For Hyper-V, in general, everything is the same, only there is a specific Veeam Backup Hyper-V Integration Service and your own driver for working with CBT.
And in the end, let's talk about who works on virtual machines during backup. To run pre- and post-freeze scripts, to create a shadow copy, collect metadata, work with SQL transaction logs, etc. Veeam Guest Helper. And if file systems are indexed, Veeam Guest Indexer . These are temporary services deployed for the duration of the backup and removed after it.
In the case of Linux machines, everything is much simpler due to the presence of a large number of built-in libraries and the capabilities of the system itself. For example, indexing is done through mlocate.
That's all for now
I don't dare to hurt you anymore short I consider the introduction to the Veeam engine compartment to be over. Yes, we have not even come close to the dens themselves, but believe me, so that the information presented in them does not seem like an incoherent stream of consciousness, such an introduction is absolutely necessary. I plan to go to the logs themselves only in the third article, and the plan for the next one is to explain who generates the logs, what exactly is displayed in them and why exactly, and not otherwise.
Source: habr.com