{"id":97729,"date":"2020-10-21T08:42:22","date_gmt":"2020-10-21T06:42:22","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/otkuda-berutsya-logi-veeam-log-diving"},"modified":"2020-10-21T08:42:22","modified_gmt":"2020-10-21T06:42:22","slug":"otkuda-berutsya-logi-veeam-log-diving","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/otkuda-berutsya-logi-veeam-log-diving","title":{"rendered":"Where do logs come from? Veeam Log Diving","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Where<\/p>\n

We continue our dive into the fascinating world of troubleshooting through logs. In the previous article<\/a><\/noindex> we agreed on the meaning of basic terms and briefly examined the overall structure of Veeam as a unified application. The task now is to understand how log files are formed, what information they contain, and why they look the way they do.<\/p>\n

What do you think these 'logs' really are? Most people believe that logs of any application should serve as an omnipotent entity, spending most of their time somewhere in the background but appearing out of nowhere in shining armor to save the day when needed. In other words, they should include everything from minor errors in each component to individual database transactions, along with immediate suggestions on how to fix any errors. And all this should fit within a couple of megabytes, no more. After all, it\u2019s just text! Text files can\u2019t take up dozens of gigabytes; I've heard that somewhere!<\/p>\n

So, logs<\/h2>\n

In the real world, logs are merely an archive of diagnostic information. It's up to the developers to decide what to store, where to get the information for storage, and how detailed it should be. Some choose minimalism, storing only ON\/OFF level records, while others diligently gather everything they can reach. There's also an intermediate option involving the choice of a so-called Logging Level, where you specify how detailed the information you want to store is and how much extra disk space you have =) VBR has six such levels, by the way. And believe me, you don\u2019t want to see what happens with the most detailed logging when you have free space on your disk.<\/p>\n

Alright. We have a general idea of what we want to keep, but the legitimate question arises: where do we get this information? Part of the events for logging is, of course, generated by our internal processes. But what do we do when interacting with the external environment? To avoid descending into a chaotic mess of workarounds and makeshift solutions, Veeam tends to avoid reinventing the wheel. Whenever there is an existing API, built-in function, library, etc., we prefer to opt for ready-made solutions before starting to devise our own intricate answers. Although there are plenty of those too. Therefore, when analyzing logs, it's essential to understand that a lion's share of errors comes from messages from third-party APIs, system calls, and various libraries. In this case, the role of VBR is to forward these errors to log files as is. The main task for the user is to learn to understand which line belongs to whom and what this 'who' is responsible for. So, if an error code from the VBR log leads you to an MSDN page, that\u2019s normal and correct.<\/p>\n

As we agreed earlier: Veeam is a so-called SQL-based application. This means that all settings, all information, and essentially everything necessary for its normal functioning is stored in its database. Hence the simple truth: what is not in the logs is likely in the database. But this is not a silver bullet; some information is not found in either the local logs of Veeam components or its database. Therefore, one must learn to study the host logs, the local machine logs, and the logs of everything involved in the backup and restore process. Sometimes the needed information is not available anywhere at all. Such is the way. <\/p>\n

A few examples of such APIs<\/h4>\n

This list is not intended to be exhaustive, so there is no need to seek the ultimate truth within it. Its purpose is simply to show the most commonly used third-party APIs and technologies in our products.<\/p>\n

Let's start with VMware<\/strong>. <\/p>\n

First on the list will be vSphere API<\/strong>It is used for authentication, reading hierarchies, creating and deleting snapshots, requesting information about machines, and a great deal (a very great deal) more. The functionality of the solution is very broad, so I can recommend the VMware vSphere API Reference for the version. 5.5<\/u><\/a><\/noindex> and 6.0<\/u><\/a><\/noindex>. For more current versions, it's simply a matter of Googling.<\/p>\n

VIX API<\/strong>. The black magic of the hypervisor, for which there is a separate error list<\/u><\/a><\/noindex>. VMware API for working with files on the host without connecting to them over the network. A last-resort option when you need to put a file on a machine with no better communication channel available. It can be a pain and suffering if the file is large and the host is busy. But here the rule applies that even 56.6 Kb\/s is better than 0 Kb\/s. In Hyper-V, a similar feature is called PowerShell Direct. But that was only until the emergence of<\/p>\n

vSphere Web Services API<\/strong> . Starting with vSphere 6.0 (approximately, since this API was first introduced in version 5.5), it is used for working with guest machines and has practically replaced VIX everywhere. Essentially, this is another API for managing vSphere. For those interested, I recommend studying the excellent<\/u><\/a><\/noindex> manual. <\/p>\n

VDDK<\/strong> (Virtual Disk Development Kit). A library that was partially discussed in this article<\/u><\/a><\/noindex>It is used to read virtual disks. Long ago, it was part of VIX, but over time it was separated into its own product. Nevertheless, as a descendant, it uses the same error codes as VIX. However, for some reason, there is no description of these errors in the SDK itself. Therefore, it has been discovered through experience that VDDK errors with different codes are merely translations from binary to decimal code. It consists of two parts \u2013 the first half contains undocumented information about the context, and the second half features traditional VIX\/VDDK errors. For example, if we see:<\/p>\n

VDDK error: 21036749815809.Unknown error<\/code><\/p>\n

We can confidently convert this into hex and get 132200000001. The uninformative beginning 132200 is simply discarded, and the remainder will be our error code (VDDK 1: Unknown error). Recently, there was a separate discussion about the most common VDDK errors. article<\/u><\/a><\/noindex>.<\/p>\n

Now let's look at Windows<\/strong>. <\/p>\n

Here you can find everything essential and important for us in the standard Event Viewer<\/strong>But there is one catch: by longstanding tradition, Windows logs do not provide the full text of the error, only its number. For example, error 5 corresponds to 'Access denied', error 1722 stands for 'The RPC server is unavailable', and error 10060 means 'Connection timed out'. Certainly, it's great if you remember the most well-known ones, but what to do with the previously unseen errors? <\/p>\n

To ensure life isn't too sweet, errors are also stored in hexadecimal format with the prefix 0x8007. For example, 0x8007000e \u2014 this actually means 14, Out of Memory. The reason for this and who it's for remains a mystery. However, a complete list of errors can be downloaded for free without SMS from the dev center<\/u><\/a><\/noindex>.<\/p>\n

By the way, sometimes other prefixes appear, not just 0x8007. In such a sad situation, understanding HRESULT (\"result handle\") requires delving even deeper into documentation<\/u><\/a><\/noindex> for developers. In everyday life, I wouldn't advise you to do this, but if you find yourself in a tight spot or are simply curious, now you know what to do.<\/p>\n

But the folks at Microsoft have shown some mercy and introduced a utility ERR<\/u><\/a><\/noindex>. This is a small piece of console joy that can translate error codes into human-readable format without using Google. It works roughly like this.<\/p>\n

C:UsersrootDesktop>err.exe 0x54f\n# for hex 0x54f \/ decimal 1359\n  ERROR_INTERNAL_ERROR                                           winerror.h\n# An internal error occurred.\n# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x54f\n# for hex 0x54f \/ decimal 1359\n  ERROR_INTERNAL_ERROR                                           winerror.h\n# An internal error occurred.\n# 2 matches found for \"0x54f\"<\/code><\/pre>\n
A legitimate question arises: why don't we write the explanation in the logs right away, leaving these mysterious codes? The answer lies in third-party applications. When you invoke a WinAPI call yourself, decoding its response is straightforward since there's even a special WinAPI call for that. But as already mentioned, our logs capture everything that comes to us in responses. Here, to decode, one would have to constantly monitor this stream of consciousness, extract the bits with Windows errors, decode them and insert them back. To be honest, it's not the most captivating activity.<\/p>\n
Windows File Management API <\/strong>is used in various file operations. Creating files, deleting, opening for writing, working with attributes, and so on.<\/p>\n
The mentioned PowerShell Direct<\/strong> acts as an analog to the VIX API in the Hyper-V world. Unfortunately, it is not as flexible: it has many functional limitations, works with not every version of the host, and far from all guests.<\/p>\n
RPC<\/strong> (Remote Procedure Call) I don't think there's anyone who has worked with Windows and hasn't seen errors related to RPC. Contrary to popular belief, it's not a single protocol, but any client-server protocol that meets a set of parameters. If there\u2019s an RPC error in our logs, in 90% of cases it will be an error from Microsoft RPC, which is part of DCOM (Distributed Component Object Model). There's a wealth of documentation on this topic out there, but much of it is quite outdated. However, if there's a strong desire to learn about it, I can recommend articles What is RPC?<\/u><\/a><\/noindex>, How RPC Works<\/u> <\/a><\/noindex>and a long list of RPC errors<\/u><\/a><\/noindex>.<\/p>\n
The main reasons for RPC errors in our logs are failed attempts at interaction between VBR components (server > proxy, for example) and most often due to communication issues.<\/p>\n
The top of all tops is the error The RPC server is unavailable (1722). To put it simply, the client was unable to establish a connection to the server. There isn't a single answer to how and why this happens, but it's usually a problem with authentication or network access to port 135. The latter is typical for infrastructures with dynamically assigned ports. There\u2019s even separate KB<\/u><\/a><\/noindex>. And at Microsoft - comprehensive guide<\/u><\/a><\/noindex> on troubleshooting.<\/p>\n
The second most common error is: There are no more endpoints available from the endpoint mapper (1753). An RPC client or server could not assign a port. This usually occurs when the server (in our case, the guest machine) is configured to dynamically allocate ports from a narrow range that has been exhausted. If we look from the client side (in our case, the VBR server), this means our VeeamVssAgent either did not start or was not registered as an RPC interface. There is also information on this topic. separate KB<\/u><\/a><\/noindex>.<\/p>\n
And to complete the Top-3 RPC errors, let's remember RPC function call failed (1726). This occurs when the connection is established, but RPC requests do not process. For example, we request the status information of VSS (in case a shadow copy is being made right now, and we are trying to access it), and in response, we hear silence and receive no information.<\/p>\n
Windows Tape Backup API <\/strong>is needed to work with tape libraries or drives. As I mentioned at the beginning: writing our own drivers and struggling with the support of each device is not enjoyable for us at all. That's why Veeam doesn't have any proprietary drivers. Everything goes through the standard API, which is supported by the hardware vendors themselves. That makes much more sense, right?<\/p>\n
SMB\/CIFS<\/strong> Everyone habitually writes them together, although not everyone remembers that CIFS (Common Internet File System) is just a private version of SMB (Server Message Block). So, there's nothing wrong with generalizing these concepts. Samba is a Linux\/Unix implementation, and it has its own specifics, but I've digressed. What\u2019s important here is that when Veeam asks to write something via UNC path (serverdirectory), the server uses the file system driver hierarchy, including mup and mrxsmb, to write to the share. Accordingly, these drivers will also generate errors.<\/p>\n
It is impossible to do without Winsock API<\/strong>. If something needs to be done over the network, VBR works through the Windows Socket API, commonly known as Winsock. So when we see an IP:Port pair in the log, that's it. The official documentation has a decent list of possible errors<\/u><\/a><\/noindex>.<\/p>\n
The mentioned WMI<\/strong> (Windows Management Instrumentation) \u2014 a powerful API for managing everything in the Windows world. For example, when working with Hyper-V, most requests to the host occur through it. In short, it's an indispensable tool with immense capabilities. The built-in tool WBEMtest.exe is very helpful in diagnosing where and what has gone wrong.<\/p>\n
And last on the list, but certainly not the least in importance \u2014 VSS<\/strong> (Volume Shadow Storage). This topic is as inexhaustible and mysterious as the amount of documentation written about it. Shadow Copy is easiest to understand as a special type of snapshot, which is essentially what it is. Thanks to it, application-consistent backups can be made in VMware, and in Hyper-V, almost everything can be done. I plan to write a separate article with a summary on VSS, but for now, you can try reading this description<\/u><\/a><\/noindex>. Just be careful, as trying to understand VSS at a glance may lead to severe headaches.<\/p>\n
On that note, I think we can stop here. I consider the task of explaining the most basic things accomplished, so in the next chapter, we will look at the logs. But if you have any questions, feel free to voice them in the comments.<\/p>\n<\/p>\n
Source: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u043d\u0430\u0448\u0435 \u043f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u043c\u0438\u0440 \u0433\u0430\u0434\u0430\u043d… \u0442\u0440\u0430\u0431\u043b\u0448\u0443\u0442\u0438\u043d\u0433\u0430 \u043f\u043e \u043b\u043e\u0433\u0430\u043c. \u0412 \u043f\u0440\u0435\u0434\u044b\u0434\u0443\u0449\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0434\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u043b\u0438\u0441\u044c \u043e \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0438 \u0431\u0430\u0437\u043e\u0432\u044b\u0445 \u0442\u0435\u0440\u043c\u0438\u043d\u043e\u0432 \u0438 \u043e\u0434\u043d\u0438\u043c \u0433\u043b\u0430\u0437\u043a\u043e\u043c \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0435\u043b\u0438 \u043e\u0431\u0449\u0443\u044e \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u0443 Veeam, \u043a\u0430\u043a \u0435\u0434\u0438\u043d\u043e\u0433\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f. \u0417\u0430\u0434\u0430\u0447\u0430 \u043d\u0430 \u044d\u0442\u0443 — \u0440\u0430\u0437\u043e\u0431\u0440\u0430\u0442\u044c\u0441\u044f \u043a\u0430\u043a \u0444\u043e\u0440\u043c\u0438\u0440\u0443\u044e\u0442\u0441\u044f \u043b\u043e\u0433 \u0444\u0430\u0439\u043b\u044b, \u0447\u0442\u043e \u0437\u0430 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u044f \u0432 \u043d\u0438\u0445 \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0430 \u0438 \u043f\u043e\u0447\u0435\u043c\u0443 \u043e\u043d\u0438 \u0432\u044b\u0433\u043b\u044f\u0434\u044f\u0442 \u043a\u0430\u043a \u0432\u044b\u0433\u043b\u044f\u0434\u044f\u0442. \u041a\u0430\u043a \u0432\u044b \u0434\u0443\u043c\u0430\u0435\u0442\u0435, \u0447\u0442\u043e \u0432\u043e\u043e\u0431\u0449\u0435 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":97730,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-97729","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Where Do Logs Come From? Veeam Log Diving | ProHoster","description":"Continuing our exploration of the fascinating world of divination...","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/otkuda-berutsya-logi-veeam-log-diving","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u0442\u043a\u0443\u0434\u0430 \u0431\u0435\u0440\u0443\u0442\u0441\u044f \u043b\u043e\u0433\u0438? Veeam Log Diving | ProHoster","og:description":"\u041f\u0440\u043e\u0434\u043e\u043b\u0436\u0430\u0435\u043c \u043d\u0430\u0448\u0435 \u043f\u043e\u0433\u0440\u0443\u0436\u0435\u043d\u0438\u0435 \u0432 \u0443\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u043c\u0438\u0440 \u0433\u0430\u0434\u0430\u043d...","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/otkuda-berutsya-logi-veeam-log-diving","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-10-21T06:42:22+00:00","article:modified_time":"2020-10-21T06:42:22+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"97729","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 10:14:39","updated":"2022-09-30 13:30:55","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/97729","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=97729"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/97729\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/97730"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=97729"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=97729"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=97729"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}