ostium
In hoc articulo loquar de clarissimo Huffman algorithmo, necnon de applicatione eius in compressione data.
Quam ob rem scrinium simplex scribemus. Hoc iam dictum est sed sine exsecutione. Materia theorica hodiernae tabellae e lectionibus scientiarum computatricis scholae sumitur et in libro Robert Laforet "Data Structures et Algorithmus in Java". Ita omnia secantur!
Aliquot cogitationes
In fasciculo textu regulari, character unus cum 8 lamellis (ASCII descriptam) vel 16 (Unicode descriptam) codicatur. Deinde de ASCII descriptam videbimus. Exempli causa, sume lineam s1 = "SUSIE AIT FACILIS". Summa 22 ingenia sunt in linea, naturaliter, inclusa spatia et lineae novae characteris - 'n'. Tabella continens hanc lineam ponderabit 22*8 = 176 frena. Interrogatio statim oritur: Utrum rationale sit omnibus 8 frustula uti ad 1 characterem encode? ASCII omnibus non utimur ingenia. Quod si fecissent, rationabilior esset literae communissimae - S - quam brevissimae tribuendae, ac rarissimae litterae - T (vel U, vel 'n') - longiori codici tribuendae. Hoc est quod Huffman algorithmus consistit: necesse est invenire optionem optimalem descriptam in qua tabella minimum pondus habebit. Prorsus normale codicem longitudinum pro diversis symbolis differre — hoc enim algorithmus innititur.
Coding
Cur codicem characterem 'S' non dabis, exempli gratia, 1 paulum longum: 0 vel 1. Sit 1. Tum indoles communissima secunda - '' (spatium) - da 0. Finge te inchoatum decoctionem tuam relatum - chorda encoded s1 - et vides codicem incipit a 1. Quid ergo agis: hicne est character S, an est aliquis alius character, exempli gratia A? Ideo magna regula oritur.
Neutrum signum praepositionem alterius esse debet
Haec regula clavis est in algorithmo. Ideo codicem creans incipit a frequentia tabulae, quae significat frequentiam cuiusque symboli;
Characteres cum pluribus eventibus encoded minimus esse debet maxime frena numero. Exemplum dabo unius tabulae ex codice possibili;
Itaque nuntius encoded hoc simile erit:
10 01111 10 110 1111 00 10 010 1110 10 00 110 0110 00 110 10 00 1111 010 10 1110 01110
Singularum characterum codicem distinxi cum spatio. Hoc in fasciculo vere compresso non fiet!
Interrogatio oritur: quomodo iuvenis iste ascendit cum codice ad mensam Codicum creandam? De hoc infra dicetur.
Construendo Huffman arbore
Hoc est, ubi binae arbores quaesitae succurrunt. Noli sollicitare, investigationem inserere vel delere modos hic non indigebis. Hic est structura arboris in Java;
public class Node {
private int frequence;
private char letter;
private Node leftChild;
private Node rightChild;
...
}
class BinaryTree {
private Node root;
public BinaryTree() {
root = new Node();
}
public BinaryTree(Node root) {
this.root = root;
}
...
}
Hoc non est integrum codicem, plenum codicem infra erit.
Hic est algorithmus ad construendam arborem;
- Node objectum crea cuique ex nuntio (line s1). In casu nostro noni nodi erunt (res Node). Quisque nodi duo data agri constat: symbolum et frequentiam
- Facere obiectum arboris (BinaryTree) pro singulis Node. Nodus fit radix arboris.
- Has arbores in prioritatem queue inserere. Inferior frequentia, superior prioritas. Sic, cum extrahendo, dervo infima frequentia semper eligitur.
Deinde debes facere cyclice sequenti:
- Duas arbores e queue prioritate remove et eas natos nodis novae fac (nodi recenti sine littera creato). Frequentia novae nodi aequalis est summa frequentiis duarum arborum posteritatum.
- Hac node, hac radice arborem nodi efficiunt. Hanc arborem in prioritate queue inserere. (Cum arbor novam frequentiam habeat, in loco novo in queue maxime verisimile apparebit)
- Perge vestigia 1 et 2 donec una tantum arbor in queue relicta sit
Vide hoc algorithmum in linea s1;
Hic symbolum "lf" (linefeed) significat lineam novam, "sp" spatium est.
Quid suus 'postero?
Huffman arborem venimus. OK. Et quid ad rem? Ne id quidem gratis accipient, deinde omnes vias possibilis ab radice ad folia arboris investigare debes. Lets consentire ad designandum marginem 0 si ducit ad puerum sinistrum et 1 si ducit ad dextram. Proprie, in hac notatione, signum symboli est via a radice arboris usque ad folium quod hoc ipsum symbolum continet.
Ita versa est mensa Codicum. Nota quod si hanc tabulam consideremus, de cuiusvis symboli pondere concludere possumus - haec est longitudo codicis. Tum, in forma compressa, fasciculus originalis ponderabit: 2*3 + 2*4 + 3*3 + 6* 2 + 1* 4 + 1* 5 + 2* 4 + 4* 2 + 1*5= 65 lamellis. . Primo 176 frena appendit. Quantum ergo ad 176/65 = 2.7 tempora redegerunt! Sed hoc utopia est. Talis coefficiens abhorret obtinenda. Quare? De hoc paulo post dicetur.
Decoding
Bene, fortasse simplicissimum superest decoding. Multos ex vobis coniectare puto nos non posse simpliciter creare limam compressam sine ulla admonitus quomodo encoded - nos eam decoquere non posse! Ita, sic, difficile mihi hoc percipere, sed tabellam textilem creare debebo.txt cum tabula compressione:
01110
00
A010
E1111
I110
S10
T0110
U01111
Y1110
Tabula ingressum in forma "symbolum" "charactere". Quare sine symbolo 01110? Re vera, cum symbolo, iustum est ut instrumenta Javae utar, cum ad limam, ingenium novum - 'n' - convertitur in novam lineam (quamvis stultus sonet). Linea igitur vacua in summo est character in codice 01110. Pro codice 00, character est spatium in linea initio. Ilicet dicamus hunc modum condendi mensam nostram Khan coefficientem esse irrationabilem. Sed facile est ad intelligendum et efficiendum. Laetus ero in commentis de optimisationi tua commendatione audire.
Hanc tabulam cum hac tabula facillime decoquere facit. Memineramus quam regulam cum descriptam creando secuti sumus;
Neutrum signum praepositionem alterius esse debet
Haec ubi partes faciliores agit. Sequenter per partes legimus et, cum primum d chorda inde ducta, in lectis nexibus constans, cum characteri characteri descriptam congruit, statim novimus characterem characterem (et solum illud) esse encoded. Deinde characterem scribimus in linea decoding (linea continens nuntium decocta), lineam d reset, ac deinde fasciculum encoded lege.
Реализация
Tempus est codicem meum humiliare et scrinium scribere. Vocemus Compressor.
Incipi iterum. Imprimis scribemus Nodum genus;
public class Node {
private int frequence;//частота
private char letter;//буква
private Node leftChild;//левый потомок
private Node rightChild;//правый потомок
public Node(char letter, int frequence) { //собственно, конструктор
this.letter = letter;
this.frequence = frequence;
}
public Node() {}//перегрузка конструтора для безымянных узлов(см. выше в разделе о построении дерева Хаффмана)
public void addChild(Node newNode) {//добавить потомка
if (leftChild == null)//если левый пустой=> правый тоже=> добавляем в левый
leftChild = newNode;
else {
if (leftChild.getFrequence() <= newNode.getFrequence()) //в общем, левым потомком
rightChild = newNode;//станет тот, у кого меньше частота
else {
rightChild = leftChild;
leftChild = newNode;
}
}
frequence += newNode.getFrequence();//итоговая частота
}
public Node getLeftChild() {
return leftChild;
}
public Node getRightChild() {
return rightChild;
}
public int getFrequence() {
return frequence;
}
public char getLetter() {
return letter;
}
public boolean isLeaf() {//проверка на лист
return leftChild == null && rightChild == null;
}
}
Arbor autem;
class BinaryTree {
private Node root;
public BinaryTree() {
root = new Node();
}
public BinaryTree(Node root) {
this.root = root;
}
public int getFrequence() {
return root.getFrequence();
}
public Node getRoot() {
return root;
}
}
Queue prioritate:
import java.util.ArrayList;//да-да, очередь будет на базе списка
class PriorityQueue {
private ArrayList<BinaryTree> data;//список очереди
private int nElems;//кол-во элементов в очереди
public PriorityQueue() {
data = new ArrayList<BinaryTree>();
nElems = 0;
}
public void insert(BinaryTree newTree) {//вставка
if (nElems == 0)
data.add(newTree);
else {
for (int i = 0; i < nElems; i++) {
if (data.get(i).getFrequence() > newTree.getFrequence()) {//если частота вставляемого дерева меньше
data.add(i, newTree);//чем част. текущего, то cдвигаем все деревья на позициях справа на 1 ячейку
break;//затем ставим новое дерево на позицию текущего
}
if (i == nElems - 1)
data.add(newTree);
}
}
nElems++;//увеличиваем кол-во элементов на 1
}
public BinaryTree remove() {//удаление из очереди
BinaryTree tmp = data.get(0);//копируем удаляемый элемент
data.remove(0);//собственно, удаляем
nElems--;//уменьшаем кол-во элементов на 1
return tmp;//возвращаем удаленный элемент(элемент с наименьшей частотой)
}
}
genus quod Huffman arbore creat;
public class HuffmanTree {
private final byte ENCODING_TABLE_SIZE = 127;//длина кодировочной таблицы
private String myString;//сообщение
private BinaryTree huffmanTree;//дерево Хаффмана
private int[] freqArray;//частотная таблица
private String[] encodingArray;//кодировочная таблица
//----------------constructor----------------------
public HuffmanTree(String newString) {
myString = newString;
freqArray = new int[ENCODING_TABLE_SIZE];
fillFrequenceArray();
huffmanTree = getHuffmanTree();
encodingArray = new String[ENCODING_TABLE_SIZE];
fillEncodingArray(huffmanTree.getRoot(), "", "");
}
//--------------------frequence array------------------------
private void fillFrequenceArray() {
for (int i = 0; i < myString.length(); i++) {
freqArray[(int)myString.charAt(i)]++;
}
}
public int[] getFrequenceArray() {
return freqArray;
}
//------------------------huffman tree creation------------------
private BinaryTree getHuffmanTree() {
PriorityQueue pq = new PriorityQueue();
//алгоритм описан выше
for (int i = 0; i < ENCODING_TABLE_SIZE; i++) {
if (freqArray[i] != 0) {//если символ существует в строке
Node newNode = new Node((char) i, freqArray[i]);//то создать для него Node
BinaryTree newTree = new BinaryTree(newNode);//а для Node создать BinaryTree
pq.insert(newTree);//вставить в очередь
}
}
while (true) {
BinaryTree tree1 = pq.remove();//извлечь из очереди первое дерево.
try {
BinaryTree tree2 = pq.remove();//извлечь из очереди второе дерево
Node newNode = new Node();//создать новый Node
newNode.addChild(tree1.getRoot());//сделать его потомками два извлеченных дерева
newNode.addChild(tree2.getRoot());
pq.insert(new BinaryTree(newNode);
} catch (IndexOutOfBoundsException e) {//осталось одно дерево в очереди
return tree1;
}
}
}
public BinaryTree getTree() {
return huffmanTree;
}
//-------------------encoding array------------------
void fillEncodingArray(Node node, String codeBefore, String direction) {//заполнить кодировочную таблицу
if (node.isLeaf()) {
encodingArray[(int)node.getLetter()] = codeBefore + direction;
} else {
fillEncodingArray(node.getLeftChild(), codeBefore + direction, "0");
fillEncodingArray(node.getRightChild(), codeBefore + direction, "1");
}
}
String[] getEncodingArray() {
return encodingArray;
}
public void displayEncodingArray() {//для отладки
fillEncodingArray(huffmanTree.getRoot(), "", "");
System.out.println("======================Encoding table====================");
for (int i = 0; i < ENCODING_TABLE_SIZE; i++) {
if (freqArray[i] != 0) {
System.out.print((char)i + " ");
System.out.println(encodingArray[i]);
}
}
System.out.println("========================================================");
}
//-----------------------------------------------------
String getOriginalString() {
return myString;
}
}
Classis continens quae encodes / decodes:
public class HuffmanOperator {
private final byte ENCODING_TABLE_SIZE = 127;//длина таблицы
private HuffmanTree mainHuffmanTree;//дерево Хаффмана (используется только для сжатия)
private String myString;//исходное сообщение
private int[] freqArray;//частотаная таблица
private String[] encodingArray;//кодировочная таблица
private double ratio;//коэффициент сжатия
public HuffmanOperator(HuffmanTree MainHuffmanTree) {//for compress
this.mainHuffmanTree = MainHuffmanTree;
myString = mainHuffmanTree.getOriginalString();
encodingArray = mainHuffmanTree.getEncodingArray();
freqArray = mainHuffmanTree.getFrequenceArray();
}
public HuffmanOperator() {}//for extract;
//---------------------------------------compression-----------------------------------------------------------
private String getCompressedString() {
String compressed = "";
String intermidiate = "";//промежуточная строка(без добавочных нулей)
//System.out.println("=============================Compression=======================");
//displayEncodingArray();
for (int i = 0; i < myString.length(); i++) {
intermidiate += encodingArray[myString.charAt(i)];
}
//Мы не можем писать бит в файл. Поэтому нужно сделать длину сообщения кратной 8=>
//нужно добавить нули в конец(можно 1, нет разницы)
byte counter = 0;//количество добавленных в конец нулей (байта в полне хватит: 0<=counter<8<127)
for (int length = intermidiate.length(), delta = 8 - length % 8;
counter < delta ; counter++) {//delta - количество добавленных нулей
intermidiate += "0";
}
//склеить кол-во добавочных нулей в бинарном предаствлении и промежуточную строку
compressed = String.format("%8s", Integer.toBinaryString(counter & 0xff)).replace(" ", "0") + intermidiate;
//идеализированный коэффициент
setCompressionRatio();
//System.out.println("===============================================================");
return compressed;
}
private void setCompressionRatio() {//посчитать идеализированный коэффициент
double sumA = 0, sumB = 0;//A-the original sum
for (int i = 0; i < ENCODING_TABLE_SIZE; i++) {
if (freqArray[i] != 0) {
sumA += 8 * freqArray[i];
sumB += encodingArray[i].length() * freqArray[i];
}
}
ratio = sumA / sumB;
}
public byte[] getBytedMsg() {//final compression
StringBuilder compressedString = new StringBuilder(getCompressedString());
byte[] compressedBytes = new byte[compressedString.length() / 8];
for (int i = 0; i < compressedBytes.length; i++) {
compressedBytes[i] = (byte) Integer.parseInt(compressedString.substring(i * 8, (i + 1) * 8), 2);
}
return compressedBytes;
}
//---------------------------------------end of compression----------------------------------------------------------------
//------------------------------------------------------------extract-----------------------------------------------------
public String extract(String compressed, String[] newEncodingArray) {
String decompressed = "";
String current = "";
String delta = "";
encodingArray = newEncodingArray;
//displayEncodingArray();
//получить кол-во вставленных нулей
for (int i = 0; i < 8; i++)
delta += compressed.charAt(i);
int ADDED_ZEROES = Integer.parseInt(delta, 2);
for (int i = 8, l = compressed.length() - ADDED_ZEROES; i < l; i++) {
//i = 8, т.к. первым байтом у нас идет кол-во вставленных нулей
current += compressed.charAt(i);
for (int j = 0; j < ENCODING_TABLE_SIZE; j++) {
if (current.equals(encodingArray[j])) {//если совпало
decompressed += (char)j;//то добавляем элемент
current = "";//и обнуляем текущую строку
}
}
}
return decompressed;
}
public String getEncodingTable() {
String enc = "";
for (int i = 0; i < encodingArray.length; i++) {
if (freqArray[i] != 0)
enc += (char)i + encodingArray[i] + 'n';
}
return enc;
}
public double getCompressionRatio() {
return ratio;
}
public void displayEncodingArray() {//для отладки
System.out.println("======================Encoding table====================");
for (int i = 0; i < ENCODING_TABLE_SIZE; i++) {
//if (freqArray[i] != 0) {
System.out.print((char)i + " ");
System.out.println(encodingArray[i]);
//}
}
System.out.println("========================================================");
}
}
Genus quo facilius limam scribere facit:
import java.io.File;
import java.io.PrintWriter;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.Closeable;
public class FileOutputHelper implements Closeable {
private File outputFile;
private FileOutputStream fileOutputStream;
public FileOutputHelper(File file) throws FileNotFoundException {
outputFile = file;
fileOutputStream = new FileOutputStream(outputFile);
}
public void writeByte(byte msg) throws IOException {
fileOutputStream.write(msg);
}
public void writeBytes(byte[] msg) throws IOException {
fileOutputStream.write(msg);
}
public void writeString(String msg) {
try (PrintWriter pw = new PrintWriter(outputFile)) {
pw.write(msg);
} catch (FileNotFoundException e) {
System.out.println("Неверный путь, или такого файла не существует!");
}
}
@Override
public void close() throws IOException {
fileOutputStream.close();
}
public void finalize() throws IOException {
close();
}
}
Genus quo facilius ex lima legere facit:
import java.io.FileInputStream;
import java.io.EOFException;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.Closeable;
import java.io.File;
import java.io.IOException;
public class FileInputHelper implements Closeable {
private FileInputStream fileInputStream;
private BufferedReader fileBufferedReader;
public FileInputHelper(File file) throws IOException {
fileInputStream = new FileInputStream(file);
fileBufferedReader = new BufferedReader(new InputStreamReader(fileInputStream));
}
public byte readByte() throws IOException {
int cur = fileInputStream.read();
if (cur == -1)//если закончился файл
throw new EOFException();
return (byte)cur;
}
public String readLine() throws IOException {
return fileBufferedReader.readLine();
}
@Override
public void close() throws IOException{
fileInputStream.close();
}
}
Etiam ac consectetur inceptos:
import java.io.File;
import java.nio.charset.MalformedInputException;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
import java.nio.file.Paths;
import java.util.List;
import java.io.EOFException;
public class Main {
private static final byte ENCODING_TABLE_SIZE = 127;
public static void main(String[] args) throws IOException {
try {//указываем инструкцию с помощью аргументов командной строки
if (args[0].equals("--compress") || args[0].equals("-c"))
compress(args[1]);
else if ((args[0].equals("--extract") || args[0].equals("-x"))
&& (args[2].equals("--table") || args[2].equals("-t"))) {
extract(args[1], args[3]);
}
else
throw new IllegalArgumentException();
} catch (ArrayIndexOutOfBoundsException | IllegalArgumentException e) {
System.out.println("Неверный формат ввода аргументов ");
System.out.println("Читайте Readme.txt");
e.printStackTrace();
}
}
public static void compress(String stringPath) throws IOException {
List<String> stringList;
File inputFile = new File(stringPath);
String s = "";
File compressedFile, table;
try {
stringList = Files.readAllLines(Paths.get(inputFile.getAbsolutePath()));
} catch (NoSuchFileException e) {
System.out.println("Неверный путь, или такого файла не существует!");
return;
} catch (MalformedInputException e) {
System.out.println("Текущая кодировка файла не поддерживается");
return;
}
for (String item : stringList) {
s += item;
s += 'n';
}
HuffmanOperator operator = new HuffmanOperator(new HuffmanTree(s));
compressedFile = new File(inputFile.getAbsolutePath() + ".cpr");
compressedFile.createNewFile();
try (FileOutputHelper fo = new FileOutputHelper(compressedFile)) {
fo.writeBytes(operator.getBytedMsg());
}
//create file with encoding table:
table = new File(inputFile.getAbsolutePath() + ".table.txt");
table.createNewFile();
try (FileOutputHelper fo = new FileOutputHelper(table)) {
fo.writeString(operator.getEncodingTable());
}
System.out.println("Путь к сжатому файлу: " + compressedFile.getAbsolutePath());
System.out.println("Путь к кодировочной таблице " + table.getAbsolutePath());
System.out.println("Без таблицы файл будет невозможно извлечь!");
double idealRatio = Math.round(operator.getCompressionRatio() * 100) / (double) 100;//идеализированный коэффициент
double realRatio = Math.round((double) inputFile.length()
/ ((double) compressedFile.length() + (double) table.length()) * 100) / (double)100;//настоящий коэффициент
System.out.println("Идеализированный коэффициент сжатия равен " + idealRatio);
System.out.println("Коэффициент сжатия с учетом кодировочной таблицы " + realRatio);
}
public static void extract(String filePath, String tablePath) throws FileNotFoundException, IOException {
HuffmanOperator operator = new HuffmanOperator();
File compressedFile = new File(filePath),
tableFile = new File(tablePath),
extractedFile = new File(filePath + ".xtr");
String compressed = "";
String[] encodingArray = new String[ENCODING_TABLE_SIZE];
//read compressed file
//!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!check here:
try (FileInputHelper fi = new FileInputHelper(compressedFile)) {
byte b;
while (true) {
b = fi.readByte();//method returns EOFException
compressed += String.format("%8s", Integer.toBinaryString(b & 0xff)).replace(" ", "0");
}
} catch (EOFException e) {
}
//--------------------
//read encoding table:
try (FileInputHelper fi = new FileInputHelper(tableFile)) {
fi.readLine();//skip first empty string
encodingArray[(byte)'n'] = fi.readLine();//read code for 'n'
while (true) {
String s = fi.readLine();
if (s == null)
throw new EOFException();
encodingArray[(byte)s.charAt(0)] = s.substring(1, s.length());
}
} catch (EOFException ignore) {}
extractedFile.createNewFile();
//extract:
try (FileOutputHelper fo = new FileOutputHelper(extractedFile)) {
fo.writeString(operator.extract(compressed, encodingArray));
}
System.out.println("Путь к распакованному файлу " + extractedFile.getAbsolutePath());
}
}
Scribere readme.txt lima te ipsum :)
conclusio,
Coniecto quod omnia volui dicere. Si quid habes quod dicam de imperitia mea in emendando codice, algorithmo, vel de optimatione in genere, tum libenter scribam. Si nihil explicavi, scribes etiam. Ego amare a te audire in comment!
PS
Ita, est, adhuc hic sum, quia de coefficientis non sum oblitus. Pro chorda s1, tabella descriptam 48 bytes - multo maior quam fasciculus fons est, nec obliti sumus circa zeros additos (numerus zerorum additorum est 7) => proportio compressionis minor erit quam una: 176/ (65 + 48*8 + 7) = 0.38. Si hoc etiam animadvertistis, non est vultus tui magnus. Ita, haec exsecutio perquam inhabilis erit ad parvas tabulas. Sed quid fit magnas files? Magnitudinum tabellae sunt multo maiores quam magnitudo tabulae descriptae. Hoc est, ubi algorithmus operatur sicut debet! Verbi gratia, Archivum efficit realem (non idealizatum) coefficientem 1.46 - paene unum et dimidium! Etiam tabella Anglice putabatur.
Source: www.habr.com