{"id":84416,"date":"2020-06-07T13:42:50","date_gmt":"2020-06-07T11:42:50","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/formaty-fajlov-v-bolshih-dannyh-kratkij-likbez"},"modified":"2020-06-07T13:42:50","modified_gmt":"2020-06-07T11:42:50","slug":"formaty-fajlov-v-bolshih-dannyh-kratkij-likbez","status":"publish","type":"post","link":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/formaty-fajlov-v-bolshih-dannyh-kratkij-likbez","title":{"rendered":"Filformater i big data: en kort introduktion","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Filformater
\nWeather Deity af Remarin<\/i><\/a><\/noindex> <\/p>\n

Kommandoen Mail.ru Cloud Solutions<\/a><\/noindex> tilbyder overs\u00e6ttelsen af artiklen<\/a><\/noindex> ingeni\u00f8r Rahul Bhatia fra Clairvoyant om de forskellige filformater inden for big data, de mest almindelige funktioner i Hadoop-formater, og hvilket format der er bedst at bruge.
\n<\/a><\/noindex><\/p>\n

Hvorfor er der behov for forskellige filformater<\/h2>\n

\nEt alvorligt flaskehals i ydeevnen for applikationer, der underst\u00f8tter HDFS, s\u00e5som MapReduce og Spark, er tiden til at s\u00f8ge, l\u00e6se samt skrive data. Disse problemer forst\u00e6rkes af vanskelighederne ved at h\u00e5ndtere store datas\u00e6t, hvis vi ikke har et fast, men et udviklende skema, eller hvis der er bestemte begr\u00e6nsninger for opbevaring.<\/p>\n

Behandling af big data \u00f8ger belastningen p\u00e5 lagersystemet - Hadoop gemmer data redundantly for at opn\u00e5 fejltolerance. Udover diske belastes CPU, netv\u00e6rk, input-output systemer osv. Som datam\u00e6ngden vokser, stiger omkostningerne ved behandling og opbevaring.<\/p>\n

Forskellige filformater i Hadoop<\/a><\/noindex> er opfundet for at l\u00f8se netop disse problemer. Valget af det rigtige filformat kan give betydelige fordele:<\/p>\n

    \n
  1. Hurtigere l\u00e6setid.<\/li>\n
  2. Hurtigere skrivning.<\/li>\n
  3. Delte filer.<\/li>\n
  4. Underst\u00f8ttelse af skemaudvikling.<\/li>\n
  5. Udvidet kompressionsunderst\u00f8ttelse.<\/li>\n<\/ol>\n

    \nNogle filformater er designet til almindelig brug, andre til mere specifikke anvendelser, og nogle er udviklet med fokus p\u00e5 konkrete datakarakteristika. S\u00e5 valget er faktisk ret stort.<\/p>\n

    Filformatet Avro<\/h2>\n

    \nFor til dataserialisering <\/i>bruges i vid udstr\u00e6kning - Avro er et linie-baseret<\/i>, hvilket betyder, at det er et linjebaseret format til datalagring i Hadoop. Det gemmer skemaet i JSON-format, hvilket letter l\u00e6sningen og fortolkningen af ethvert program. Selve dataene ligger i et bin\u00e6rt format, kompakt og effektivt.<\/p>\n

    Avro's serialiseringssystem er sprogneutral. Filer kan behandles af forskellige sprog, herunder C, C++, C#, Java, Python og Ruby.<\/p>\n

    En n\u00f8glefunktion ved Avro er den p\u00e5lidelige underst\u00f8ttelse af datastrukturer, der \u00e6ndrer sig over tid, det vil sige, at de udvikler sig. Avro forst\u00e5r \u00e6ndringer i skemaet - fjernelse, tilf\u00f8jelse eller \u00e6ndring af felter.<\/p>\n

    Avro underst\u00f8tter forskellige datastrukturer. For eksempel kan du oprette en post, der indeholder et array, et enum-typ og en underpost.<\/p>\n

    \"Filformater
    \nDette format er ideelt til skrivning til en landing (overgangs) zone i datalaget (    data lake<\/a><\/noindex>, or data lake \u2014 a collection of instances for storing various types of data in addition to the data sources themselves). <\/p>\n

    So, to record in the landing zone of the data lake, this format is best suited for the following reasons:<\/p>\n

      \n
    1. Data from this zone is usually read in full for further processing by downstream systems \u2014 and the row-based format is more efficient in this case.<\/li>\n
    2. Downstream systems can easily extract schema tables from files \u2014 there's no need to store schemas separately in an external metadata storage.<\/li>\n
    3. Any change in the original schema is easily handled (schema evolution).<\/li>\n<\/ol>\n

      <\/p>\n

      Parquet file format<\/h2>\n

      \nParquet is an open-source file format for Hadoop that stores nested data structures in a flat columnar format.<\/i>.<\/p>\n

      Compared to the traditional row-based approach, Parquet is more efficient in terms of storage and performance.<\/p>\n

      This is especially useful for queries that read specific columns from a wide (with many columns) table. Due to the file format, only the necessary columns are read, minimizing input-output operations.<\/p>\n

      A brief aside-explanation<\/strong>: to better understand the Parquet file format in Hadoop, let\u2019s look at what a column-based \u2014 that is, columnar \u2014 format is. In such a format, homogeneous values of each column are stored together. <\/p>\n

      For eksempel<\/a><\/noindex>, the record includes fields ID, Name, and Department. In this case, all values of the ID column will be stored together, as will the values of the Name column, and so on. The table will look something like this:<\/p>\n

      ID<\/strong>
      \nNavn<\/strong>
      \nDepartment<\/strong><\/p>\n

      1
      \nemp1
      \nd1<\/p>\n

      2
      \nemp2
      \nd2<\/p>\n

      3
      \nemp3
      \nd3<\/p>\n

      \nIn row format, the data will be saved as follows:<\/p>\n

      1
      \nemp1
      \nd1
      \n2
      \nemp2
      \nd2
      \n3
      \nemp3
      \nd3<\/p>\n

      \nIn a columnar file format, the same data will be saved like this:<\/p>\n

      1
      \n2
      \n3
      \nemp1
      \nemp2
      \nemp3
      \nd1
      \nd2
      \nd3<\/p>\n

      \nThe columnar format is more efficient when you need to query several columns from a table. It will read only the necessary columns, as they are adjacent to each other. Thus, input-output operations are minimized.<\/p>\n

      For example, you only need the NAME column. In row format,<\/a><\/noindex> each record in the dataset needs to be loaded, parsed into fields, and then the NAME data extracted. The columnar format allows you to go directly to the Name column, as all values for that column are stored together. You won\u2019t have to scan the entire record.<\/p>\n

      P\u00e5 den m\u00e5de \u00f8ger kolonneformatet foresp\u00f8rgselsydelsen, da det kr\u00e6ver mindre tid at s\u00f8ge efter de n\u00f8dvendige kolonner, og antallet af input-output operationer reduceres, fordi der kun l\u00e6ses de n\u00f8dvendige kolonner.<\/p>\n

      En af de unikke egenskaber Parquet<\/a><\/noindex> er, at det i dette format kan opbevare data med indlejrede strukturer<\/i>. Det betyder, at i Parquet-filen kan selv indlejrede felter l\u00e6ses individuelt uden at skulle l\u00e6se alle felter i den indlejrede struktur. For at gemme indlejrede strukturer bruger Parquet en shredding and assembly-algoritme.<\/p>\n

      \"Filformater
      \nFor at forst\u00e5 Parquet filformatet i Hadoop, er det n\u00f8dvendigt at kende f\u00f8lgende termer:<\/p>\n

        \n
      1. R\u00e6kkegruppe<\/strong> (row group): en logisk vandret opdeling af data i r\u00e6kker. En r\u00e6kkegruppe best\u00e5r af et fragment af hver kolonne i datas\u00e6ttet.<\/li>\n
      2. Kolonnefragment<\/strong> (column chunk): et fragment af en specifik kolonne. Disse kolonnefragmenter lever i en bestemt r\u00e6kkegruppe og vil v\u00e6re sammenh\u00e6ngende i filen.<\/li>\n
      3. Side<\/strong> (page): kolonnefragmenter opdeles i sider, der skrives efter hinanden. Siderne har en f\u00e6lles header, s\u00e5 un\u00f8dvendige data kan springes over n\u00e5r de l\u00e6ses.<\/li>\n<\/ol>\n

        \n\"Filformater
        \nHer indeholder headeren simpelthen et magisk tal PAR1<\/i> (4 byte), som identificerer filen som en Parquet filformat.<\/p>\n

        I footer'en er der skrevet f\u00f8lgende:<\/p>\n

          \n
        1. Metadata for filen, som indeholder startkoordinaterne for metadata for hver kolonne. Ved l\u00e6sning skal man f\u00f8rst l\u00e6se metadataen for filen for at finde alle de interessante kolonnefragmenter. Derefter skal kolonnefragmenterne l\u00e6ses sekventielt. Metadata inkluderer ogs\u00e5 versionsnummeret for formatet, skemaet og eventuelle yderligere n\u00f8gle-v\u00e6rdi par.<\/li>\n
        2. L\u00e6ngden af metadata (4 byte).<\/li>\n
        3. Det magiske tal PAR1<\/i> (4 byte).<\/li>\n<\/ol>\n

          <\/p>\n

          ORC filformat<\/h2>\n

          \nOptimeret r\u00e6kke-kolonne filformat<\/i> (Optimized Row Columnar, ORC<\/a><\/noindex>) tilbyder en meget effektiv m\u00e5de at opbevare data p\u00e5 og blev udviklet for at overvinde begr\u00e6nsningerne ved andre formater. Det opbevarer data i en perfekt kompakt form, hvilket g\u00f8r det muligt at springe over un\u00f8dvendige detaljer - uden at det kr\u00e6ver opbygning af store, komplekse eller manuelt vedligeholdte indekser. <\/p>\n

          Fordele ved ORC formatet:<\/p>\n

            \n
          1. \u00c9n fil som output fra hver opgave, hvilket reducerer belastningen p\u00e5 NameNode (navneserver).<\/li>\n
          2. Underst\u00f8ttelse af Hive-datatyper, herunder DateTime, decimaler og komplekse datatyper (struct, list, map og union).<\/li>\n
          3. Samtidig l\u00e6sning af den samme fil af forskellige RecordReader-processer.<\/li>\n
          4. Mulighed for at opdele filer uden at scanne for mark\u00f8rer.<\/li>\n
          5. Vurdering af den maksimalt mulige tildeling af heap-hukommelse til l\u00e6se-\/skriveprocesser baseret p\u00e5 information i filens footer.<\/li>\n
          6. Metadata gemmes i det bin\u00e6re protokolbuffer-serialiseringsformat, som tillader tilf\u00f8jelse og fjernelse af felter.<\/li>\n<\/ol>\n

            \n\"Filformater
            \nORC gemmer samlinger af r\u00e6kker i en fil, mens de indeni samlingen lagres i kolonneformat.<\/p>\n

            ORC-filen gemmer grupper af r\u00e6kker, kaldet strimler (stripes), og supplerende information i filens footer. Postscripten i slutningen af filen indeholder komprimeringsparametre og st\u00f8rrelsen af den komprimerede footer.<\/p>\n

            Standardst\u00f8rrelsen for en stripe er 250 MB. Ved at have striber af en s\u00e5 stor st\u00f8rrelse udf\u00f8res l\u00e6sning fra HDFS mere effektivt i store, sammenh\u00e6ngende blokke.<\/p>\n

            Footeren i filen indeholder en liste over strimlerne i filen, antallet af r\u00e6kker pr. stripe og datatypen for hver kolonne. Der st\u00e5r ogs\u00e5 den resulterende v\u00e6rdi for count, min, max og sum for hver kolonne.<\/p>\n

            Footer for strimlen indeholder en katalog med placeringer for str\u00f8mmen.<\/p>\n

            R\u00e6kke-data bruges, n\u00e5r tabeller scannes.<\/p>\n

            Indexdata inkluderer minimums- og maksimumsv\u00e6rdier for hver kolonne og r\u00e6kkeplaceringer i hver kolonne. ORC-indexer bruges kun til at v\u00e6lge striber og grupper af r\u00e6kker, ikke til at besvare foresp\u00f8rgsler.<\/p>\n

            Sammenligning af forskellige filformater<\/h2>\n

            <\/p>\n

            Avro sammenlignet med Parquet<\/h3>\n

            <\/p>\n

              \n
            1. Avro er et r\u00e6kkeorienteret lagringsformat, mens Parquet lagrer data i kolonner.<\/li>\n
            2. Parquet er bedre egnet til analytiske foresp\u00f8rgsler, hvilket betyder, at l\u00e6snings- og dataprocesseringsoperationer er langt mere effektive end skrivning.<\/li>\n
            3. Skriveoperationer i Avro er mere effektive end i Parquet.<\/li>\n
            4. Avro h\u00e5ndterer skemaudvikling mere modent. Parquet underst\u00f8tter kun tilf\u00f8jelse af skemaer, mens Avro implementerer multifunktionel udvikling, dvs. tilf\u00f8jelse eller \u00e6ndring af kolonner.<\/li>\n
            5. Parquet er ideelt til at foresp\u00f8rge et delm\u00e6ngde af kolonner i en multi-kolonne tabel. Avro egner sig til ETL-operationer, hvor vi anmoder om alle kolonner.<\/li>\n<\/ol>\n

              <\/p>\n

              ORC sammenlignet med Parquet<\/h3>\n

              <\/p>\n

                \n
              1. Parquet er bedre til at gemme indlejrede data.<\/li>\n
              2. ORC er bedre til at underst\u00f8tte predicate pushdown.<\/li>\n
              3. ORC underst\u00f8tter ACID-egenskaber.<\/li>\n
              4. ORC komprimerer data bedre.<\/li>\n<\/ol>\n

                \nAndre artikler om emnet<\/strong>:<\/p>\n

                  \n
                1. Analyse af store data i skyen: hvordan virksomheder bliver datadrevne<\/a><\/noindex>.<\/li>\n
                2. En beskeden guide til databasediagrammer<\/a><\/noindex>.<\/li>\n
                3. Vores Telegram-kanal om digital transformation<\/a><\/noindex>. \n<\/li>\n<\/ol>\n

                  Kilde: habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

                  Weather Deity by Remarin \u041a\u043e\u043c\u0430\u043d\u0434\u0430 Mail.ru Cloud Solutions \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u0442 \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u0441\u0442\u0430\u0442\u044c\u0438 \u0438\u043d\u0436\u0435\u043d\u0435\u0440\u0430 \u0420\u0430\u0445\u0443\u043b\u0430 \u0411\u0445\u0430\u0442\u0438\u0438 \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 Clairvoyant \u043e \u0442\u043e\u043c, \u043a\u0430\u043a\u0438\u0435 \u0435\u0441\u0442\u044c \u0444\u043e\u0440\u043c\u0430\u0442\u044b \u0444\u0430\u0439\u043b\u043e\u0432 \u0432 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u043a\u0430\u043a\u0438\u0435 \u0441\u0430\u043c\u044b\u0435 \u0440\u0430\u0441\u043f\u0440\u043e\u0441\u0442\u0440\u0430\u043d\u0435\u043d\u043d\u044b\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u0438 \u0444\u043e\u0440\u043c\u0430\u0442\u043e\u0432 Hadoop \u0438 \u043a\u0430\u043a\u043e\u0439 \u0444\u043e\u0440\u043c\u0430\u0442 \u043b\u0443\u0447\u0448\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c. \u0417\u0430\u0447\u0435\u043c \u043d\u0443\u0436\u043d\u044b \u0440\u0430\u0437\u043d\u044b\u0435 \u0444\u043e\u0440\u043c\u0430\u0442\u044b \u0444\u0430\u0439\u043b\u043e\u0432 \u0421\u0435\u0440\u044c\u0435\u0437\u043d\u043e\u0435 \u0443\u0437\u043a\u043e\u0435 \u043c\u0435\u0441\u0442\u043e \u0432 \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0441 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u043a\u043e\u0439 HDFS, \u0442\u0430\u043a\u0438\u0445 \u043a\u0430\u043a MapReduce \u0438 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":84417,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-84416","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Filformater i store data: kort opsummering | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/formaty-fajlov-v-bolshih-dannyh-kratkij-likbez","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"da_DK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0424\u043e\u0440\u043c\u0430\u0442\u044b \u0444\u0430\u0439\u043b\u043e\u0432 \u0432 \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445: \u043a\u0440\u0430\u0442\u043a\u0438\u0439 \u043b\u0438\u043a\u0431\u0435\u0437 | ProHoster","og:url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/formaty-fajlov-v-bolshih-dannyh-kratkij-likbez","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-06-07T11:42:50+00:00","article:modified_time":"2020-06-07T11:42:50+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"84416","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 14:58:40","updated":"2022-09-28 08:24:46","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/84416","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/comments?post=84416"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/84416\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/media\/84417"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/media?parent=84416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/categories?post=84416"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/tags?post=84416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}