This is a standard (ISO, ITU-T, GOST) for describing structured information, as well as the rules for encoding that information. For me as a programmer, it's just another data serialization and representation format, alongside JSON, XML, XDR, and others. It is widely used in our everyday lives, encountered in mobile, telephone, and VoIP communications (UMTS, LTE, WiMAX, SS7, H.323), in network protocols (LDAP, SNMP, Kerberos), in everything related to cryptography (X.509, CMS, PKCS standards), on bank cards and biometric passports, and in many other areas.
This article discusses : a Python ASN.1 library actively used in cryptocurrency-related projects in .
In general, recommending ASN.1 for cryptographic tasks is not advisable: ASN.1 and its codecs are complex. This means the code will not be straightforward, which is always an additional attack vector. Just look of vulnerabilities in ASN.1 libraries. Bruce Schneier in his also advises against using this standard due to its complexity: 'The best-known TLV encoding is ASN.1, but it is incredibly complex, and we shy away from it.' Unfortunately, today we have in which , CRL, OCSP, TSP, CMP protocols, , messages , and a plethora of standards . Therefore, one must know how to work with ASN.1 if you are involved in anything related to cryptography.
ASN.1 can be encoded in various ways/codecs:
- (Basic Encoding Rules)
- (Canonical Encoding Rules)
- (Distinguished Encoding Rules)
- (Generic String Encoding Rules)
- (JSON Encoding Rules)
- LWER (Light Weight Encoding Rules)
- (Octet Encoding Rules)
- (Packed Encoding Rules)
- SER (Signalling specific Encoding Rules)
- (XML Encoding Rules)
and several others. However, in cryptographic tasks, in practice, only two are used: BER and DER. Even in signed XML documents (, ) there will still be Base64-encoded ASN.1 DER objects, just as in the JSON-oriented protocol from Let's Encrypt. It's better to understand all these codecs and coding principles of BER/CER/DER in articles and books: , , .
BER is a binary byte-oriented (for example, PER, popular in mobile communications — bit-oriented) TLV format. Each element is encoded as: a tag (Tag), identifying the type of the encoded element (integer, string, date, etc.), length (Length) of the content and the content itself (VThe BER format optionally allows omitting the length value, using a special indefinite length value and ending the message with an End-Of-Octets marker. Besides length encoding, BER has much variability in the way data types are encoded, such as:
- INTEGER, OBJECT IDENTIFIER, BIT STRING, and the length of the element may be non-normalized (not encoded in minimal form);
- BOOLEAN is true for any non-zero content;
- BIT STRING may contain 'extra' zero bits;
- BIT STRING, OCTET STRING, and all their derived string types, including date/time, can be broken into variable-length chunks, the lengths of which are not known in advance during (de)coding;
- UTCTime/GeneralizedTime can have different ways of specifying the time zone offset and 'extra' zero fractions of a second;
- DEFAULT values for SEQUENCE may be encoded, but need not be;
- Named values of the last bits in BIT STRING can be optionally left unencoded;
- SEQUENCE (OF)/SET (OF) may have an arbitrary order of elements.
Due to all of the above, encoding data in such a way that it is identical to the original form is not always possible. Therefore, a subset of rules was devised: DER—strictly regulates only one admissible encoding method, which is critical for cryptographic tasks, where, for example, changing a single bit will invalidate the signature or checksum. DER has a significant drawback: the lengths of all elements must be known in advance during encoding, which does not allow for streaming serialization of data. The CER codec is free of this drawback while similarly ensuring a unique representation of the data. Unfortunately (or happily, as we don’t face even more complex decoders?), it did not become popular. Therefore, in practice, we encounter 'mixed' use of BER and DER-encoded data. Since both CER and DER are subsets of BER, any BER decoder can process them.
Issues with pyasn1
At work, we write many Python programs related to cryptography. A few years ago, there were hardly any free libraries to choose from: either very low-level libraries that only allowed for encoding/decoding simple integers and structure headers, or this library . We lived on it for several years and were initially very satisfied, as it allows working with ASN.1 structures as high-level objects: for example, the decoded X.509 certificate object allows access to its fields through a dictionary interface: cert["tbsCertificate"]["serialNumber"] will show us the serial number of this certificate. Similarly, we can "assemble" complex objects by working with them as lists, dictionaries, and then simply call the function pyasn1.codec.der.encoder.encode to get a serialized representation of the document.
However, shortcomings, problems, and limitations became apparent. There were, and unfortunately still are, errors in pyasn1: at the time of writing this article, one of the basic types in pyasn1 — GeneralizedTime, decoded and encoded.
In our projects, to save space, we often only store the file path, offset, and length in bytes of the object we want to reference. For example, an arbitrary signed file is likely to be found in the CMS SignedData ASN.1 structure:
0 [1,3,1018] ContentInfo SEQUENCE
4 [1,1, 9] . contentType: ContentType OBJECT IDENTIFIER 1.2.840.113549.1.7.2 (id_signedData)
19-4 [0,0,1003] . content: [0] EXPLICIT [UNIV 16] ANY
19 [1,3, 999] . . DEFINED BY id_signedData: SignedData SEQUENCE
23 [1,1, 1] . . . version: CMSVersion INTEGER v3 (03)
26 [1,1, 19] . . . digestAlgorithms: DigestAlgorithmIdentifiers SET OF
[...]
47 [1,3, 769] . . . encapContentInfo: EncapsulatedContentInfo SEQUENCE
51 [1,1, 8] . . . . eContentType: ContentType OBJECT IDENTIFIER 1.3.6.1.5.5.7.12.2 (id_cct_PKIData)
65-4 [1,3, 751] . . . . eContent: [0] EXPLICIT OCTET STRING 751 bytes OPTIONAL
HERE IS THE CONTENT OF THE SIGNED FILE WITH A SIZE OF 751 BYTES
820 [1,2, 199] . . . signerInfos: SignerInfos SET OF
823 [1,2, 196] . . . . 0: SignerInfo SEQUENCE
826 [1,1, 1] . . . . . version: CMSVersion INTEGER v3 (03)
829 [0,0, 22] . . . . . sid: SignerIdentifier CHOICE subjectKeyIdentifier
[...]
956 [1,1, 64] . . . . . signature: SignatureValue OCTET STRING 64 bytes
. . . . . . C1:B3:88:BA:F8:92:1C:E6:3E:41:9B:E0:D3:E9:AF:D8
. . . . . . 47:4A:8A:9D:94:5D:56:6B:F0:C1:20:38:D2:72:22:12
. . . . . . 9F:76:46:F6:51:5F:9A:8D:BF:D7:A6:9B:FD:C5:DA:D2
. . . . . . F3:6B:00:14:A4:9D:D7:B5:E1:A6:86:44:86:A7:E8:C9
We can retrieve the original signed file with an offset of 65 bytes, with a length of 751 bytes. pyasn1 does not store this information in its decoded objects. A so-called TLVSeeker was written — a small library that allows decoding tags and lengths of objects, with which we commanded 'go to the next tag', 'dive into the tag' (entering the SEQUENCE object), 'go to the next tag', 'report your offset and length of the object we are in'. This was a 'manual' traversal of ASN.1 DER-serialized data. However, this approach could not be used with BER-serialized data, as, for example, the byte string OCTET STRING could be encoded in multiple chunks.
Another drawback for our needs with pyasn1 is the inability to determine from decoded objects whether a given field was present in the SEQUENCE or not. For instance, if the structure contains a field Field SEQUENCE OF Smth OPTIONAL, it might be completely absent in the incoming data (OPTIONAL), or it might be present but have a zero length (empty list). In general, this could not be determined. This is necessary for strict validation of incoming data. Imagine if some certificate authority issued a certificate with 'not quite' valid data from the ASN.1 schema perspective! For example, the certificate authority 'TÜRKTRUST Elektronik Sertifika Hizmet Sağlayıcısı' went beyond permissible limits in its root certificate. component length boundaries of the subject — it cannot be honestly decoded according to the schema. The DER codec requires that a field which has a value equal to DEFAULT must not be encoded during transmission — such documents exist in practice, and the first version of PyDERASN even consciously allowed such invalid (from the perspective of DER) behavior for backward compatibility.
Another limitation is the inability to easily determine in which format (BER/DER) a particular object is encoded in the structure. For example, the CMS standard states that the message is encoded in BER, but the signedAttrs field, which is used to generate the cryptographic signature, must be in DER. If we decode with DER, we will encounter issues processing the CMS itself; if we decode with BER, we won't know in what format the signedAttrs were. As a result, we will have to use TLVSeeker (of which there is no equivalent in pyasn1) to locate each of the signedAttrs fields and then individually decode them from the serialized representation using DER.
The ability to automatically process DEFINED BY fields, which occur very frequently, was highly desired by us. After decoding the ASN.1 structure, we may end up with numerous ANY fields that need to be processed according to the scheme chosen based on the OBJECT IDENTIFIER provided in the structure field. In Python code, this means writing an if statement and then calling the decoder for the ANY field.
The emergence of PyDERASN
At Atlas, we regularly submit patches upstream when we find issues or improve the open-source programs we use. We have sent several enhancements to pyasn1, but the code of pyasn1 is not the easiest to understand, and it sometimes underwent incompatible API changes that hindered us. Additionally, we are accustomed to writing tests with generative testing, which was absent in pyasn1.
One fine day, I decided that I had enough of this and it was time to try writing my own library with __slots__, offsets, and beautifully displayed blobs! Simply creating an ASN.1 codec would not be sufficient—I needed to translate all our interdependent projects onto it, which amounts to hundreds of thousands of lines of code involving extensive work with ASN.1 structures. Therefore, one of its requirements was ease of translating the current pyasn1 code. Spending my entire vacation, I wrote this library and migrated all projects to it. Since they have nearly 100% test coverage, this meant the library was fully functional.
PyDERASN similarly has nearly 100% test coverage. It employs generative testing with a wonderful library . Fuzzing was also performed with -em on 32-core machines. Despite the fact that we have virtually eliminated Python2 code, PyDERASN still maintains compatibility with it, which is why it has a single dependency. Additionally, it has been tested against .
The principle of working with it is similar to pyasn1 — working with high-level Python objects. The description of ASN.1 schemas is similar.
class TBSCertificate(Sequence):
schema = (
("version", Version(expl=tag_ctxc(0), default="v1")),
("serialNumber", CertificateSerialNumber()),
("signature", AlgorithmIdentifier()),
("issuer", Name()),
("validity", Validity()),
("subject", Name()),
("subjectPublicKeyInfo", SubjectPublicKeyInfo()),
("issuerUniqueID", UniqueIdentifier(impl=tag_ctxp(1), optional=True)),
("subjectUniqueID", UniqueIdentifier(impl=tag_ctxp(2), optional=True)),
("extensions", Extensions(expl=tag_ctxc(3), optional=True)),
)
However, PyDERASN has a semblance of strict typing. In pyasn1, if a field had the type CMSVersion(INTEGER), it could be assigned an int or INTEGER. PyDERASN strictly requires that the assigned object be exactly CMSVersion. Moreover, since we write Python3 code, we also use , which means that our functions will have clear argument types like def func(serial, contents), instead of def func(serial: CertificateSerialNumber, contents: EncapsulatedContentInfo), and PyDERASN helps maintain such code.
At the same time, PyDERASN offers very convenient allowances for this typing. pyasn1 did not allow assigning a SubjectKeyIdentifier() object to the field in SubjectKeyIdentifier().subtype(implicitTag=Tag(…)) (without the required IMPLICIT TAG), necessitating frequent copying and recreating of objects just due to changed IMPLICIT/EXPLICIT tags. PyDERASN strictly watches only the base type — it will automatically substitute the tags from the existing ASN.1 schema structure. This significantly simplifies the application code.
If an error occurs during decoding, it is not easy to understand exactly where it happened in pyasn1. For example, in the previously mentioned Turkish certificate, we would get an error like this: UTF8String (tbsCertificate:issuer:rdnSequence:3:0:value:DEFINED BY 2.5.4.10:utf8String) (at 138) unsatisfied bounds: 1 ⇐ 77 ⇐ 64 People can make mistakes when writing ASN.1 structures, and this helps to debug applications or uncover issues with encoded documents from the opposite side.
In the first version, PyDERASN did not support BER encoding. It appeared much later and still lacks support for processing UTCTime/GeneralizedTime with time zones. This will come in the future, as the project is mainly being developed in spare time.
In the first version, there was also no handling of DEFINED BY fields. A few months later, this and began to be actively used, significantly reducing application code — with a single decoding operation, it was possible to obtain the entire structure disassembled to the deepest level. For this, the schema specifies which fields 'define' what. For example, the CMS schema description:
class ContentInfo(Sequence):
schema = (
("contentType", ContentType(defines=((("content",), {
id_authenticatedData: AuthenticatedData(),
id_digestedData: DigestedData(),
id_encryptedData: EncryptedData(),
id_envelopedData: EnvelopedData(),
id_signedData: SignedData(),
}),))),
("content", Any(expl=tag_ctxc(0))),
)
indicates that if contentType contains an OID with the value id_signedData, then the content field (located in the same SEQUENCE) needs to be decoded according to the SignedData schema. Why so many parentheses? A field can 'define' several fields simultaneously, as seen in EnvelopedData structures. Defined fields are identified by a so-called decode path — it specifies the exact location of any element in all structures.
Sometimes you may not want or may not have the opportunity to immediately introduce these defines into the schema. There may be application-specific cases where OIDs and structures are known only in an external project. PyDERASN provides the ability to specify these defines right at the moment of decoding the structure:
ContentInfo().decode(data, ctx={"defines_by_path": ((
(
"content", DecodePathDefBy(id_signedData),
"certificates", any, "certificate", "tbsCertificate",
"extensions", any, "extnID",
),
((("extnValue",), {
id_ce_authorityKeyIdentifier: AuthorityKeyIdentifier(),
id_ce_basicConstraints: BasicConstraints(),
[...]
id_ru_subjectSignTool: SubjectSignTool(),
}),),
),)})
Here, we specify that in CMS SignedData for all attached certificates, decode all their extensions (AuthorityKeyIdentifier, BasicConstraints, SubjectSignTool, etc.). We indicate through the decode path which element needs to 'substitute' defines as if it had been defined in the schema.
Finally, PyDERASN has the capability to work from the for decoding ASN.1 files and offers rich . You can decode arbitrary ASN.1, or you can specify a clearly defined schema and see something like this:
Displayed information: object offset, tag length, length of length, content length, presence of EOC (end-of-octets), BER encoding flag, indefinite-length encoding flag, length and offset of EXPLICIT tag (if any), object nesting depth in structures, IMPLICIT/EXPLICIT tag value, object name by schema, its base ASN.1 type, ordinal number within SEQUENCE/SET OF, value of CHOICE (if any), human-readable name INTEGER/ENUMERATED/BIT STRING by schema, value of any base type, DEFAULT/OPTIONAL flag from the schema, indication that the object was automatically decoded as DEFINED BY and the OID responsible for that, human-readable OID.
The pretty printing system is specifically designed to generate a sequence of PP objects, which are then visualized using separate means. The screenshot shows a renderer in simple colored text. There are also renderers in JSON/HTML format, so this can be seen highlighted in the ASN.1 browser. project.
Other libraries
This was not the goal, but PyDERASN turned out to be significantly than pyasn1. For example, decoding CRL files of megabyte sizes can take such a long time that one might consider intermediate data storage formats (faster) and changing application architectures. pyasn1 decodes CRL on my laptop in over 20 minutes, whereas PyDERASN does it in just 28 seconds! There is a project , aimed at fast processing of cryptographic structures: it decodes (fully, not lazily) the same CRL in 29 seconds, but consumes nearly twice as much memory when running under Python3 (983 MiB vs 498 MiB), and 3.5 times more under Python2 (1677 MiB vs 488 MiB), while pyasn1 uses a staggering 4.3 times more (2093 MiB vs 488 MiB).
We did not consider asn1crypto, which I mentioned, because the project was still in its infancy, and we hadn't heard about it. Even now, we wouldn’t look its way, as I immediately found that it doesn't accept arbitrary formats for GeneralizedTime, and when serializing, it silently removes fractions of a second. This is acceptable for working with X.509 certificates, but not suitable in general cases.
Currently, PyDERASN is the strictest of the free Python/Go DER decoders known to me. In the encoding/asn1 library of my favorite Go OBJECT IDENTIFIER and UTCTime/GeneralizedTime strings. Sometimes strictness can hinder (primarily due to backward compatibility with old applications that no one will fix), therefore in PyDERASN during decoding you can pass relaxing checks.
The project code aims to be as simple as possible. The entire library is contained in a single file. The code is written with a focus on ease of understanding, without excessive performance optimizations and DRY code. As mentioned before, it does not support full BER decoding of UTCTime/GeneralizedTime strings, as well as REAL, RELATIVE OID, EXTERNAL, INSTANCE OF, EMBEDDED PDV, CHARACTER STRING data types. In all other cases, personally, I see no reason to use other libraries in Python.
Like all my projects, of the type , , , , PyDERASN is fully , distributed under the terms of , and is available for free download. Usage examples can be found in and in .
, , member of , Python/Go developer, chief specialist .
Source: habr.com
