{"id":34321,"date":"2019-10-31T21:57:38","date_gmt":"2019-10-31T18:57:38","guid":{"rendered":"https:\/\/prohoster.info\/blog\/vosstanavlivaem-virtualnye-mashiny-s-oshibochno-initsializirovannogo-datastore-istoriya-odnoj-gluposti-s-heppi-endom\/"},"modified":"2019-10-31T21:57:38","modified_gmt":"2019-10-31T18:57:38","slug":"vosstanavlivaem-virtualnye-mashiny-s-oshibochno-initsializirovannogo-datastore-istoriya-odnoj-gluposti-s-heppi-endom","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vosstanavlivaem-virtualnye-mashiny-s-oshibochno-initsializirovannogo-datastore-istoriya-odnoj-gluposti-s-heppi-endom","title":{"rendered":"Restoring virtual machines from an incorrectly initialized Datastore. A story of a mistake with a happy ending.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Disclaimer: <\/b>This note is of an entertaining nature. The concentration of useful information in it is low. It was written \"for myself.\"<\/p><\/blockquote>\n

<\/p>\n

Lyrical introduction<\/h3>\n

\nThe file dump in our organization runs on a VMware ESXi 6 virtual machine under Windows Server 2016. And it\u2019s not just a dump. It\u2019s a file exchange server between structural divisions: here lies collaborative work, project documentation, and folders from network scanners. In short, this is where all the operational life takes place.<\/p>\n

And this repository of operational life began to hang. Moreover, a guest could quietly hang itself without affecting others. It could hang along with the entire host, consequently affecting all other guest machines. It could hang itself and disrupt the client services of vSphere: that is, the processes of other guests are alive, machines work correctly and respond, but the file dump doesn\u2019t and the vSphere Client cannot connect to the host. Overall, no pattern could be identified. Hangs could occur during the day with low load. They could happen at night under zero load. They might occur at night during differential backups with moderate load. They could happen on weekends during full backups with high load. An evident degradation of the situation was observed. At first, it happened once a year, then once every six months. By the end of my patience \u2014 twice a week.
\nI suspected the RAM. But I couldn\u2019t stop the dump even on weekends and run Memtest. We were waiting for the May holidays. During the May holidays, I ran Memtest and\u2026 no errors were found.
\n<\/a><\/noindex>
\nI was astonished and decided to take a vacation. While I was on vacation, the dump did not hang once. But when I returned to work on Monday \u2014 the dump was hanging. It endured a full backup and precisely after it finished, it hung. This warm welcome back from vacation prompted me to physically move the disks from the guest machine to another host.<\/p>\n

And, although it is well known that no serious work should be done on the first day back from vacation, even though I spent the whole way to work preparing myself not to work, my outrage at yet another hang wiped my mindset and resolutions from my mind...<\/p>\n

The physical disks were moved to another host. Hot-swapped. In the storage settings on the Drives<\/i> disks appear. On the tab Datastores<\/i> there are no storages on these disks. Refresh<\/i> \u2014 do not appear. Well, of course, the first impulse is Add Storage<\/i>. The add storage wizard explains that it supports this. Of course it supports VMFS. I had no doubt. A quick glance at the wizard's messages at each step: Next, Next, Next, Finish. My gaze didn\u2019t even catch the small yellow circle with an exclamation mark at the bottom of one of the wizard's steps.<\/p>\n

At the end of the wizard, the new Datastore appeared in the list\u2026 along with the Datastores from other physical disks.<\/p>\n

I proceed to navigate the newly added Datastore, and it\u2026 is empty. Naturally, I was astonished again. 8 AM on the clock, the first 15 minutes of work after vacation, I haven't even stirred the sugar in my coffee yet. And then this happens. The first thought was \u2014 didn\u2019t pull the right disk from the 'native' host. I checked if the desired Datastore was present on the 'native' host: no, it is not present. The second thought was: 'damn!'. I'm not sure, but it seems that the third, fourth, and at least the fifth thought were the same.<\/p>\n

To dispel doubts, I quickly installed a fresh ESXi for testing, took a random disk and, while reading attentively, went through the wizard's steps. Yes. When adding a Datastore with the wizard, there is a loss of all data on the disk without the possibility of reverting the operation and recovering the data. Later, I read on one of the forums an assessment of such a wizard design: shitsome crap. And I completely agreed.<\/p>\n

Starting from the sixth thought \u2014 I began to think more constructively. Alright. Initialization takes mere seconds even for a 3Tb disk. So, this is high-level formatting. This means the partition table was simply rewritten. So, the data is still there. Now let's look for some unformat tool and voila.<\/p>\n

I boot the machine from the Strelec boot image\u2026 And discover that partition recovery programs know everything except VMFS. They know Synology partitioning, for example, but not VMFS.<\/p>\n

The range of programs is not reassuring: at best, GetDataBack and R.Saver find NTFS partitions with an intact directory structure and live file names. But that doesn't satisfy me. I need two vmdk files: one for the system disk and one for the trash file disk.<\/p>\n

And here I realize that I seem to be about to install Windows and restore from a file backup. At the same time, I remember that I had a DFS root there. And also a completely wild volume and branching system of access rights to the departmental folders. Not an option. The only time-acceptable option is to recover the system and data disk state along with all rights.<\/p>\n

Once again, searching on Google, forums, and FAQs brings back the lament of many: VMware ESXi does not provide a data recovery mechanism. All discussion threads lead to two outcomes: either someone recovered with the help of the pricey DiskInternals VMFS Recovery or an expert promoting their services came to the rescue. vmfs-tools<\/i> and dd<\/i>. The option of purchasing a DiskInternals VMFS Recovery license for $700 is not viable. Allowing an outsider from the 'territory of a potential adversary' access to corporate data is also not an option. However, it was found that UFS Explorer can read VMFS partitions as well.<\/p>\n

DiskInternals VMFS Recovery<\/h3>\n

\nA trial version was downloaded and installed. The program successfully detected an empty VMFS partition:<\/p>\n

\"Restoring
\n
\nIn strict tracking protection mode Undelete (Fast Scan)<\/i> also found the worn Datastore with folders of virtual machines<\/a> containing disks:<\/p>\n

\"Restoring
\n
\nThe preview showed that the files are alive:<\/p>\n

\"Restoring
\n
\nMounting the partition in the system was successful, but for some unknown reason, all three folders contained the same virtual machine. Of course, Murphy's law \u2014 not the one that was needed.<\/p>\n

Three lines of shame<\/b>The attempted shameless piracy of the software ended in failure. However, UFS Explorer was pirated.<\/p>\n

I have a very negative attitude towards software theft. Under no circumstances do I encourage the use of means to bypass licensing protection. \n<\/p><\/blockquote>\n

\nI was in a catastrophic position and was not proud of the measures I resorted to.<\/p>\n

UFS Explorer<\/h3>\n

\nThe disk scan showed the presence of 7 nodes. The number of nodes coincidentally matched the number of *-flat.vmdk files found by VMFS Recovery:<\/p>\n

\"Restoring
\n
\nThe comparison of file sizes and node sizes also showed a match down to the byte. Meanwhile, the names of the *-flat.vmdk files were recovered, and accordingly, their association with the virtual machines.<\/p>\n

\"Restoring
\n
\nIn general, vmdk disks in terms of ESXi consist of two files: one is the data file (-flat.vmdk) and the other is the 'physical' disk descriptor file (.vmdk). If you upload the *-flat.vmdk file from your local machine to the Datastore, ESXi will not recognize it as a valid disk file. There is an article in VMware's knowledge base about how to manually create a disk descriptor file: kb.vmware.com\/s\/article\/1002511<\/a><\/noindex>, but I didn't need to do that; I simply copied the contents of the corresponding files from the file preview area in DiskInternals VMFS Recovery:<\/p>\n

\"Restoring
\n
\nAfter 4 hours of extracting 2.5TB from UFS Explorer and 20 hours of loading it into the hypervisor's Datastore, the corrupted disk files were attached to a newly created virtual machine. The disks mounted successfully. No data loss was observed.<\/p>\n

\"Restoring
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

Disclaimer: \u0417\u0430\u043c\u0435\u0442\u043a\u0430 \u043d\u043e\u0441\u0438\u0442 \u0440\u0430\u0437\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u0445\u0430\u0440\u0430\u043a\u0442\u0435\u0440. \u0423\u0434\u0435\u043b\u044c\u043d\u0430\u044f \u043f\u043b\u043e\u0442\u043d\u043e\u0441\u0442\u044c \u043f\u043e\u043b\u0435\u0437\u043d\u043e\u0439 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438 \u0432 \u043d\u0435\u0439 \u043c\u0430\u043b\u0430. \u0411\u044b\u043b\u0430 \u043d\u0430\u043f\u0438\u0441\u0430\u043d\u0430 \u00ab\u0434\u043b\u044f \u0441\u0435\u0431\u044f\u00bb. \u041b\u0438\u0440\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0432\u0441\u0442\u0443\u043f\u043b\u0435\u043d\u0438\u0435 \u0424\u0430\u0439\u043b\u043e\u0432\u0430\u044f \u043f\u043e\u043c\u043e\u0439\u043a\u0430 \u0432 \u043d\u0430\u0448\u0435\u0439 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u0430\u0446\u0438\u0438 \u043a\u0440\u0443\u0442\u0438\u0442\u0441\u044f \u043d\u0430 \u0432\u0438\u0440\u0442\u0443\u0430\u043b\u044c\u043d\u043e\u0439 \u043c\u0430\u0448\u0438\u043d\u0435 VMware ESXi 6 \u043f\u043e\u0434 Windows Server 2016. \u0418 \u044d\u0442\u043e \u043d\u0435 \u043f\u0440\u043e\u0441\u0442\u043e \u043f\u043e\u043c\u043e\u0439\u043a\u0430. \u042d\u0442\u043e \u0441\u0435\u0440\u0432\u0435\u0440 \u0444\u0430\u0439\u043b\u043e\u0432\u043e\u0433\u043e \u043e\u0431\u043c\u0435\u043d\u0430 \u043c\u0435\u0436\u0434\u0443 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043d\u044b\u043c\u0438 \u043f\u043e\u0434\u0440\u0430\u0437\u0434\u0435\u043b\u0435\u043d\u0438\u044f\u043c\u0438: \u0442\u0443\u0442 \u0438 \u0441\u043e\u0432\u043c\u0435\u0441\u0442\u043d\u0430\u044f \u0440\u0430\u0431\u043e\u0442\u0430, \u0438 \u043f\u0440\u043e\u0435\u043a\u0442\u043d\u0430\u044f \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044f, \u0438 \u043f\u0430\u043f\u043a\u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":25881,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-34321","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47We recover virtual machines from incorrectly initialized Datastore. A story of one foolish mistake with a happy ending | ProHoster","description":"Disclaimer: This note is for entertainment purposes. The specific information density is low.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vosstanavlivaem-virtualnye-mashiny-s-oshibochno-initsializirovannogo-datastore-istoriya-odnoj-gluposti-s-heppi-endom","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0412\u043e\u0441\u0441\u0442\u0430\u043d\u0430\u0432\u043b\u0438\u0432\u0430\u0435\u043c \u0432\u0438\u0440\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0435 \u043c\u0430\u0448\u0438\u043d\u044b \u0441 \u043e\u0448\u0438\u0431\u043e\u0447\u043d\u043e \u0438\u043d\u0438\u0446\u0438\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u043e\u0433\u043e Datastore. \u0418\u0441\u0442\u043e\u0440\u0438\u044f \u043e\u0434\u043d\u043e\u0439 \u0433\u043b\u0443\u043f\u043e\u0441\u0442\u0438 \u0441 \u0445\u044d\u043f\u043f\u0438-\u044d\u043d\u0434\u043e\u043c | ProHoster","og:description":"Disclaimer: \u0417\u0430\u043c\u0435\u0442\u043a\u0430 \u043d\u043e\u0441\u0438\u0442 \u0440\u0430\u0437\u0432\u043b\u0435\u043a\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u0439 \u0445\u0430\u0440\u0430\u043a\u0442\u0435\u0440. \u0423\u0434\u0435\u043b\u044c\u043d\u0430\u044f \u043f\u043b\u043e\u0442\u043d\u043e\u0441\u0442\u044c \u043f\u043e\u043b\u0435\u0437\u043d\u043e\u0439 \u0438\u043d\u0444\u043e\u0440\u043c\u0430\u0446\u0438\u0438 \u0432 \u043d\u0435\u0439 \u043c\u0430\u043b\u0430.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/vosstanavlivaem-virtualnye-mashiny-s-oshibochno-initsializirovannogo-datastore-istoriya-odnoj-gluposti-s-heppi-endom","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:57:38+00:00","article:modified_time":"2019-10-31T18:57:38+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"34321","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-09 17:15:13","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:24:25","updated":"2026-02-09 17:15:13","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34321","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=34321"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34321\/revisions"}],"predecessor-version":[{"id":159242,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/34321\/revisions\/159242"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/25881"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=34321"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=34321"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=34321"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}