IBM has introduced the CodeNet initiative, aimed at providing researchers with a dataset that allows them to experiment with machine learning techniques for creating translators between different programming languages, as well as code generators and analyzers. CodeNet includes a collection of 14 million code examples that solve 4,053 common programming problems. In total, the collection comprises about 500 million lines of code and covers 55 programming languages, including modern languages like C++, Java, Python, and Go, as well as legacy languages such as COBOL, Pascal, and FORTRAN. The project's findings are distributed under the Apache 2.0 license, and the datasets are set to be made available in the public domain.
The examples are annotated and implement identical algorithms across different programming languages. It is expected that the proposed dataset will aid in training machine learning systems and foster innovation in code translation and machine parsing, similar to how the ImageNet annotated image database facilitated the development of image recognition and computer vision systems. Various programming competitions are cited as key sources for forming the collection.
Unlike traditional translators that are based on transformation rules, machine learning systems can capture and account for the context in which code is used. When translating from one programming language to another, context is just as important as when translating between human languages. The lack of context consideration is what hinders the conversion of code from legacy languages like COBOL.
Having a large database of algorithm implementations in various languages will facilitate the creation of universal machine learning systems that manipulate code through a more abstract representation, independent of specific programming languages, instead of direct translation between particular languages. Such a system could function as a translator that converts the input code in any of the supported languages into its internal abstract representation, from which code in multiple languages can then be generated.
The system will also be capable of performing bidirectional transformations. For example, banks and government agencies still continue to use projects written in the outdated COBOL language. A translator based on a machine learning system can convert COBOL code into a Java representation and, if necessary, translate a Java fragment back into COBOL code.
In addition to language translation, CodeNet also mentions applications such as creating intelligent code search systems and automating the detection of code clones, as well as developing optimizers and automatic code correction systems. In particular, the examples presented in CodeNet are equipped with metadata describing performance test results, resulting program size, memory consumption, and state, which helps distinguish correct code from erroneous code (the collection specifically includes examples with errors, which constitute 29.5% of the total). The machine learning system can take this metadata into account to generate the most optimal code or identify regressions in the analyzed code (the system can understand if the algorithm in the provided code is implemented suboptimally or contains errors).
Source: opennet.ru
