Aug 26, 2017, 22:27 pm
Hierarchical CRC Lossy compression Lossless decompression with 'wild' collisions that should vary so much from original data as to be obvious fakes.
Its very heavy on the CPU usage, and I mean heavy man.
I'm going to describe the compression method and the decompression method below, the compression is really for trading cpu time vs download time.
I'm going to use CRC8 in my example which is 8 bits long.
Firstly you checksum a sector of data, say 4 bytes, then another sector. Then you checksum both sectors together, this gives a nice little 'triangle' of checksums. you call the checksum of both sectors a cluster. You continue on doing this until you run out of data. This gives around 3 bytes for every 8 bytes consumed. Finally you encode a short header of length, file extension, sector and cluster counts aswell as a checksum for the whole file.
This gives you the compression method
The decompression method would work well on a quantum computer or somthing that could execute CRC8ReversedMultipath FPGA????
You create ALL sectors possible for the sector length by brute forcing them(or by crc8reversed)
then you create a list of all sector combinations within the clusters
this should give you a list of collisions or similar files
you checksum the paths through the clusters to compare with your master checksum and output any files that match using somthing like filename.collisioncount.extension as your output file name(s).
If memory is an issue save to disk on generate and do the sectors, cluster and file weaving lightly. If your packet of data was small enough just hold the whole thing in memory.
Its very heavy on the CPU usage, and I mean heavy man.
I'm going to describe the compression method and the decompression method below, the compression is really for trading cpu time vs download time.
I'm going to use CRC8 in my example which is 8 bits long.
Firstly you checksum a sector of data, say 4 bytes, then another sector. Then you checksum both sectors together, this gives a nice little 'triangle' of checksums. you call the checksum of both sectors a cluster. You continue on doing this until you run out of data. This gives around 3 bytes for every 8 bytes consumed. Finally you encode a short header of length, file extension, sector and cluster counts aswell as a checksum for the whole file.
This gives you the compression method
The decompression method would work well on a quantum computer or somthing that could execute CRC8ReversedMultipath FPGA????
You create ALL sectors possible for the sector length by brute forcing them(or by crc8reversed)
then you create a list of all sector combinations within the clusters
this should give you a list of collisions or similar files
you checksum the paths through the clusters to compare with your master checksum and output any files that match using somthing like filename.collisioncount.extension as your output file name(s).
If memory is an issue save to disk on generate and do the sectors, cluster and file weaving lightly. If your packet of data was small enough just hold the whole thing in memory.
