After three and a half years of development, the first stable release of the GNU Wget2 project has been presented, which is a complete overhaul of the GNU Wget program designed for automating the recursive downloading of content. GNU Wget2 is designed and rewritten from scratch and is notable for extracting the core functionality of the web client into the libwget library, which can be used independently in applications. The utility is licensed under GPLv3+, while the library is under LGPLv3+.
Instead of gradually reworking the existing codebase, it was decided to rewrite everything from scratch and establish a separate Wget2 branch to embody ideas for restructuring, expanding functionality, and implementing changes that break compatibility. Except for the discontinuation of support for the FTP protocol and the WARC format, wget2 can act as a transparent replacement for the classic wget utility in most situations.
At the same time, wget2 has some documented behavioral differences, provides about 30 additional options, and discontinues support for several dozen options. This includes the cessation of handling options such as “—ask-password,” “—header,” “—exclude-directories,” “—ftp*,” “—warc*,” “—limit-rate,” “—relative,” and “—unlink.”
Key innovations include:
- Extracting functionality into the libwget library.
- Transitioning to a multithreaded architecture.
- The ability to establish multiple connections and download in multiple threads simultaneously. This includes the option to parallelize the download of a single file by breaking it into blocks using the “—chunk-size” option.
- Support for the HTTP/2 protocol.
- Using the HTTP header If-Modified-Since to download only changed data.
- Transitioning to the use of external bandwidth throttling tools, such as trickle.
- Support for the Accept-Encoding header, compressed data transfer, and compression algorithms including brotli, zstd, lzip, gzip, deflate, lzma, and bzip2.
- Support for TLS 1.3, the OCSP (Online Certificate Status Protocol) for checking revoked certificates, the HSTS (HTTP Strict Transport Security) mechanism for enforcing redirection to HTTPS, and HPKP (HTTP Public Key Pinning) for binding certificates.
- The ability to use GnuTLS, WolfSSL, and OpenSSL as backends for TLS.
- Support for TCP FastOpen.
- Built-in support for the Metalink format.
- Support for internationalized domain names (IDNA2008).
- The ability to work simultaneously through multiple proxies -servers (one stream will upload through one proxy, the second through another).
- Built-in support for news feeds in Atom and RSS formats (for example, to scan and download links). RSS/Atom data can be loaded from a local file or over the network.
- Support for extracting URLs from Sitemap files. Parsers are available to extract links from CSS and XML files.
- Support for the 'include' directive in configuration files and distribution of settings across multiple files (/etc/wget/conf.d/*.conf).
- Built-in DNS request caching mechanism.
- Ability to transcode content while changing the document encoding.
- Respect for the 'robots.txt' file during recursive downloads.
- Reliable write mode with fsync() called after saving data.
- Ability to resume interrupted TLS sessions, as well as cache and save TLS session parameters to a file.
- ‘--input-file’ mode for loading URLs received through standard input.
- Cookie scope check against the Public Suffix List directory to isolate different sites hosted on the same second-level domain (e.g., ‘a.github.io’ and ‘b.github.io’).
- Support for downloading streaming broadcasts in ICEcast / SHOUTcast format.
Source: opennet.ru
