Re: sha-512 efficiency

[Jordan Geoghegan <jordan%geoghegan.ca@localhost>]

On 2020-01-04 04:36, Sad Clouds wrote:
> Not tried ZFS on NetBSD yet, but I was wondering about the efficiency
> of the various hash functions. ZFS can use fletcher4, sha-256, sha-512,
> etc. in order to verify its data integrity. Some of them may or may not
> be implemented on different platforms, but the most secure hash
> function seems to be sha-512.
>
> The problem with sha-512 is that it is quite slow, even on 64-bit Intel
> CPUs. Not exactly what you want when checksumming large numbers of
> blocks of data on a file system.
>
> I'm not a cryptographer, but from my limited knowledge I assume that
> cryptographic hash function like sha-512 are designed around the
> following principles:
>
> 1. It should be extremely difficult to reverse the hash value in order
> to get the original data.
>
> 2. It should be extremely difficult to get collisions, where the same
> hash value maps to different blocks of data.
>
> So when looking at ZFS (or any other file system that checksums data) I
> would say that for the purpose of checksumming (and not encrypting data
> on disk) principle No 1 is completely unnecessary.
>
> I think principle No 2 is the most important, as the less collisions we
> get, the better we can detect data corruption/modification.
>
> And so, it makes me wonder if sha-512 is too complex, when it doesn't
> need to be. I think of hash functions as lottery ball machines, which
> mix numbers in random order. If you run the machine for 60 seconds it
> will mix all numbers pretty well, but I'm pretty sure you are not going
> to get any more randomness if you run the machine for say 6 hours. After
> a certain amount of work you reach the point of diminishing returns.
>
> Doe anyone know if this point of diminishing returns has been validated
> in practice for sha-512 vs other functions? Mathematicians can do
> theoretical analysis, but are there practical tests which show that
> after a set of mixing operations, the randomness remains static and does
> not increase any more?

SHA512 tends to be faster on 64bit processors and when hashing larger 
block sizes.

You can test your machines hashing speed using the command: "openssl 
speed sha256 sha512"

I tested this on my laptop (HP Probook 640 G4 with 8th Gen i5 running 
OpenBSD 6.6-stable) and it seems that SHA512 is faster when hashing 
block sizes larger than 1024 bytes:

probook$ openssl speed sha256 sha512
Doing sha256 for 3s on 16 size blocks: 7092346 sha256's in 3.01s
Doing sha256 for 3s on 64 size blocks: 4914063 sha256's in 3.01s
Doing sha256 for 3s on 256 size blocks: 1688215 sha256's in 3.02s
Doing sha256 for 3s on 1024 size blocks: 179917 sha256's in 3.03s
Doing sha256 for 3s on 8192 size blocks: 24795 sha256's in 3.04s
Doing sha512 for 3s on 16 size blocks: 1407346 sha512's in 3.02s
Doing sha512 for 3s on 64 size blocks: 1757133 sha512's in 3.01s
Doing sha512 for 3s on 256 size blocks: 836394 sha512's in 3.01s
Doing sha512 for 3s on 1024 size blocks: 320466 sha512's in 3.01s
Doing sha512 for 3s on 8192 size blocks: 57414 sha512's in 2.98s
LibreSSL 3.0.2
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) 
idea(int) blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes       64 bytes        256 bytes      1024 
bytes    8192 bytes
sha256         37700.18k   104485.06k   143106.97k   60803.63k 66816.00k
sha512         7456.14k     37360.97k     71135.17k     109022.32k 
157830.70k


When I run the same test on a 2nd Gen i5 desktop and an Edgerouter Lite 
[featuring a MIPS64 cpu] (both also running OpenBSD 6.6 - stable) SHA512 
is faster on block sizes over 16 bytes for both machines ( ie SHA512 is 
faster in all tests but one):

# 2nd Gen i5 Desktop:
LibreSSL 3.0.2
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) 
idea(int) blowfish(idx)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type                16 bytes      64 bytes       256 bytes      1024 
bytes     8192 bytes
sha256           34938.55k    90543.08k   169867.23k   223988.26k 247599.08k
sha512           28363.61k   112817.63k   210125.65k   320487.51k 381940.43k

# MIPS64 Edgerouter Lite:
LibreSSL 3.0.2
built on: date not available
options:bn(64,64) rc4(ptr,int) des(ptr,risc2,2,int) aes(partial) 
idea(int) blowfish(ptr)
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type                16 bytes       64 bytes     256 bytes     1024 
bytes   8192 bytes
sha256            2388.76k     5985.47k    11379.24k    14386.52k 15311.69k
sha512            2012.78k     8230.70k    13971.58k    20825.82k 24665.81k


So in short, hashing performance can vary quite largely depending on 
your machine. The only way to to know is to test for yourself, but a 
simple rule of thumb would be to assume that SHA512 will be faster for 
most workloads.