{"id":33123,"date":"2019-10-31T21:50:52","date_gmt":"2019-10-31T18:50:52","guid":{"rendered":"https:\/\/prohoster.info\/blog\/terraformer-infrastructure-to-code\/"},"modified":"2019-10-31T21:50:52","modified_gmt":"2019-10-31T18:50:52","slug":"terraformer-infrastructure-to-code","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/terraformer-infrastructure-to-code","title":{"rendered":"Terraformer \u2014 Infrastructure To Code","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Terraformer
\nI'd like to tell you about a new CLI tool I created to solve an old problem. <\/p>\n

The Problem<\/h3>\n

\nTerraform has long been a standard in the DevOps\/Cloud\/IT community. It's a very handy and useful tool for managing infrastructure as code. There are many benefits to Terraform, as well as many forks, sharp knives, and pitfalls.
\nWith Terraform, it's very convenient to create new things and then manage, modify, or delete them. But what about those with a vast cloud infrastructure that wasn't created using Terraform? Rewriting and recreating the entire cloud seems costly and unsafe.
\nI've faced this problem at two jobs; a simple example is wanting everything in git as Terraform files, but you have 250+ buckets, and writing them for Terraform manually is quite a lot.
\nThere is issue<\/a><\/noindex> Since 2014, there was a tool for Terraform that was shut down in 2016 with the hope that import functionality would be available.<\/p>\n

Basically, everything is as shown in the picture, just from right to left.
\n<\/a><\/noindex><\/p>\n

Warnings: The author has not lived in Russia for half his life and writes in Russian infrequently. Beware of spelling errors.<\/p>\n

Solutions<\/h3>\n

\n1. There are old and existing solutions for AWS. terraforming<\/a><\/noindex>. When I tried to use it for my 250+ buckets, I realized it was inadequate. AWS has long introduced many new options that terraforming doesn't recognize and it relies on Ruby. The templates look sparse.<\/a><\/noindex>. After two evenings, I sent a Pull request<\/a><\/noindex> to add more capabilities, and I realized that such a solution is not suitable at all.
\nTerraforming works by taking data from the AWS SDK and generating tf and tfstate files through templates.
\nThere are three issues:
\n1. There will always be a delay in updates.
\n2. tf files sometimes come out corrupted.
\n3. tfstate is collected separately from tf and doesn't always match.
\nIt's generally difficult to achieve a result where `terraform plan` indicates there are no changes.<\/p>\n

2. `terraform import` \u2014 an integrated command in Terraform. How does it work?
\nYou write an empty TF file with the name and type of resource, then run `terraform import` and provide the resource ID. Terraform calls the provider, retrieves the data, and creates a tfstate file.
\nThere are three issues:
\n1. We only get the tfstate file, and the tf file is empty; it must be written manually or converted from tfstate.
\n2. It can only handle one resource at a time and doesn't support all resources. And what am I supposed to do with 250+ buckets again?
\n3. You need to know the resource IDs \u2014 meaning you have to wrap this in code that retrieves the list of resources.
\nOverall, the result is partial and doesn't scale well.<\/p>\n

My solution
\n<\/h3>\n

Requirements:
\n1. The ability to create tf and tfstate files from resources. For example, to download all buckets\/security groups\/load balancers and for `terraform plan` to return that there are no changes.
\n2. Need 2 clouds: GCP + AWS.
\n3. A global solution that is easy to update each time and doesn't take 3 days of work for each resource.
\n4. Make it open source \u2014 everyone has this problem.<\/p>\n

The Go language \u2014 that's why I love it, and there is a library for creating HCL files that is used in terraform + a lot of code in terraform that can be useful.<\/p>\n

Path<\/h3>\n

\nFirst attempt
\nStarted with a simple option. Accessing the cloud through the SDK for the required resource and converting it into fields for terraform. The attempt failed immediately on the security group because I didn't like spending 1.5 days converting only the security group (and there are many resources). It's time-consuming, and fields can change\/add.<\/p>\n

Second attempt
\nBased on the idea described. here<\/a><\/noindex>. Just take and convert tfstate to tf. All the data is there and the fields are the same. How to get the full tfstate for multiple resources?? Here, the command `terraform refresh` came to the rescue. Terraform takes all the resources in tfstate and pulls the data by ID, writing everything into tfstate. So, create an empty tfstate just with names and IDs, run `terraform refresh`, and we get complete tfstate. Hooray!
\nNow let's engage in the recursive writing of a converter from tfstate to tf. For those who have never read tfstate, it is JSON, but special.
\nHere is its important part attributes<\/p>\n

 \"attributes\": {\n                            \"id\": \"default\/backend-logging-load-deployment\",\n                            \"metadata.#\": \"1\",\n                            \"metadata.0.annotations.%\": \"0\",\n                            \"metadata.0.generate_name\": \"\",\n                            \"metadata.0.generation\": \"24\",\n                            \"metadata.0.labels.%\": \"1\",\n                            \"metadata.0.labels.app\": \"backend-logging\",\n                            \"metadata.0.name\": \"backend-logging-load-deployment\",\n                            \"metadata.0.namespace\": \"default\",\n                            \"metadata.0.resource_version\": \"109317427\",\n                            \"metadata.0.self_link\": \"\\\/apis\\\/apps\\\/v1\\\/namespaces\\\/default\\\/deployments\\\/backend-logging-load-deployment\",\n                            \"metadata.0.uid\": \"300ecda1-4138-11e9-9d5d-42010a8400b5\",\n                            \"spec.#\": \"1\",\n                            \"spec.0.min_ready_seconds\": \"0\",\n                            \"spec.0.paused\": \"false\",\n                            \"spec.0.progress_deadline_seconds\": \"600\",\n                            \"spec.0.replicas\": \"1\",\n                            \"spec.0.revision_history_limit\": \"10\",\n                            \"spec.0.selector.#\": \"1\",\n<\/code><\/pre>\n
\nHere there is:
\n1. id \u2014 string
\n2. metadata \u2014 an array of size 1 containing an object with fields described below
\n3. spec \u2014 a hash of size 1 with key-value pairs
\nIn short, a fun format, everything can go deep too for several levels.<\/p>\n
                   \"spec.#\": \"1\",\n                            \"spec.0.min_ready_seconds\": \"0\",\n                            \"spec.0.paused\": \"false\",\n                            \"spec.0.progress_deadline_seconds\": \"600\",\n                            \"spec.0.replicas\": \"1\",\n                            \"spec.0.revision_history_limit\": \"10\",\n                            \"spec.0.selector.#\": \"1\",\n                            \"spec.0.selector.0.match_expressions.#\": \"0\",\n                            \"spec.0.selector.0.match_labels.%\": \"1\",\n                            \"spec.0.selector.0.match_labels.app\": \"backend-logging-load\",\n                            \"spec.0.strategy.#\": \"0\",\n                            \"spec.0.template.#\": \"1\",\n                            \"spec.0.template.0.metadata.#\": \"1\",\n                            \"spec.0.template.0.metadata.0.annotations.%\": \"0\",\n                            \"spec.0.template.0.metadata.0.generate_name\": \"\",\n                            \"spec.0.template.0.metadata.0.generation\": \"0\",\n                            \"spec.0.template.0.metadata.0.labels.%\": \"1\",\n                            \"spec.0.template.0.metadata.0.labels.app\": \"backend-logging-load\",\n                            \"spec.0.template.0.metadata.0.name\": \"\",\n                            \"spec.0.template.0.metadata.0.namespace\": \"\",\n                            \"spec.0.template.0.metadata.0.resource_version\": \"\",\n                            \"spec.0.template.0.metadata.0.self_link\": \"\",\n                            \"spec.0.template.0.metadata.0.uid\": \"\",\n                            \"spec.0.template.0.spec.#\": \"1\",\n                            \"spec.0.template.0.spec.0.active_deadline_seconds\": \"0\",\n                            \"spec.0.template.0.spec.0.container.#\": \"1\",\n                            \"spec.0.template.0.spec.0.container.0.args.#\": \"3\",<\/code><\/pre>\n
\nIn general, anyone who wants a programming task for an interview can simply ask to write a parser for this \ud83d\ude42
\nAfter many attempts to write a bug-free parser, I found part of it in the Terraform code, specifically the most important part. And everything seemed to work fine.<\/p>\n
Attempt three
\nA Terraform provider is a binary that contains code with all the resources and logic to work with cloud APIs. Each cloud has its own provider, and Terraform simply calls them via its RPC protocol between two processes.
\nI decided to call Terraform providers directly through RPC calls. It turned out beautifully and provided the ability to switch Terraform providers to newer ones, gaining new capabilities without changing the code. It also turned out that not all fields in tfstate need to be in tf, but how to find out? You can only ask the provider about that. Then began another recursive mess of building regular expressions with searching for fields within tfstate at all levels deep.<\/p>\n
In the end, I created a useful CLI tool that has a common infrastructure for all Terraform providers, and adding a new one is easy. Moreover, adding resources requires little code. Plus, there are various advantages like connections between resources. Of course, there were many different problems that can\u2019t all be described.
\nI named the little creature Terrafomer. <\/p>\n
Final<\/h3>\n
\nUsing Terraformer, we generated 500-700 thousand lines of tf + tfstate code across two clouds. We were able to take legacy systems and start managing them with Terraform, embodying the best ideas of infrastructure as code. It\u2019s simply magic when you take a huge cloud and receive it in the form of operational Terraform files via a command. Then it\u2019s just grep\/replace\/git and so on.<\/p>\n
I\u2019ve calculated and cleaned everything up, and received the necessary permissions. I released it on GitHub for everyone on Thursday (02.05.19). github.com\/GoogleCloudPlatform\/terraformer<\/a><\/noindex>
\nI've already received 600 stars, and there are 2 pull requests adding support for OpenStack and Kubernetes. The feedback has been positive. Overall, it\u2019s a useful project for people.
\nI recommend it to everyone who wants to start working with Terraform and doesn\u2019t want to rewrite everything for that. 
\nI would appreciate pull requests, issues, and stars.<\/p>\n
Demo
\n\"Terraformer<\/noindex>
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0425\u043e\u0442\u0435\u043b \u0431\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043f\u0440\u043e \u043d\u043e\u0432\u044b\u0439 CLI tool \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u044f \u043d\u0430\u043f\u0438\u0441\u0430\u043b \u0434\u043b\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u044f \u043e\u0434\u043d\u043e\u0439 \u0441\u0442\u0430\u0440\u043e\u0439 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u0430 Terraform \u0443\u0436\u0435 \u0434\u0430\u0432\u043d\u043e \u0441\u0442\u0430\u043b \u0441\u0442\u0430\u043d\u0434\u0430\u0440\u0442\u043e\u043c \u0432 Devops\/Cloud\/IT \u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u0435. \u0412\u0435\u0449\u044c \u043e\u0447\u0435\u043d\u044c \u0443\u0434\u043e\u0431\u043d\u0430\u044f \u0438 \u043f\u043e\u043b\u0435\u0437\u043d\u0430\u044f \u0447\u0442\u043e\u0431 \u0437\u0430\u043d\u0438\u043c\u0430\u0442\u0441\u044f infrastructure as code. \u0415\u0441\u0442\u044c \u043c\u043d\u043e\u0433\u043e \u043f\u0440\u0435\u043b\u0435\u0441\u0442\u0435\u0439 \u0432 Terraform \u0430 \u0442\u0430\u043a \u0436\u0435 \u043c\u043d\u043e\u0433\u043e \u0432\u0438\u043b\u043e\u043a, \u043e\u0441\u0442\u0440\u044b\u0445 \u043d\u043e\u0436\u0435\u0439 \u0438 \u0433\u0440\u0430\u0431\u043b\u0435\u0439. \u0421 Terraform \u043e\u0447\u0435\u043d\u044c \u0443\u0434\u043e\u0431\u043d\u043e \u0434\u0435\u043b\u0430\u0442\u044c \u043d\u043e\u0432\u044b\u0435 \u0432\u0435\u0449\u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24869,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-33123","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Terraformer \u2014 Infrastructure To Code | ProHoster","description":"I would like to talk about a new CLI tool that I wrote.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/terraformer-infrastructure-to-code","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Terraformer \u2014 Infrastructure To Code | ProHoster","og:description":"\u0425\u043e\u0442\u0435\u043b \u0431\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043f\u0440\u043e \u043d\u043e\u0432\u044b\u0439 CLI tool \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u044f \u043d\u0430\u043f\u0438\u0441\u0430\u043b.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/terraformer-infrastructure-to-code","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:50:52+00:00","article:modified_time":"2019-10-31T18:50:52+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"33123","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 14:02:48","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:46:23","updated":"2026-01-21 14:02:48","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=33123"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33123\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/24869"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=33123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=33123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=33123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}