https://mokurin000.github.io • This is a web feed, also known as an Atom feed. Subscribe by copying the URL from the address bar into your newsreader. Visit About Feeds to learn more and get started. It's free. Visit website Recent posts Updated on $DATE date optimization mokurin000's blog - optimization geek blog Zola2026-06-16T07:00:00+09:00https://mokurin000.github.io/tags/optimization/atom.xml 性能接近理论上限的的大文件 hashing 2026-06-16T07:00:00+09:00 2026-06-16T07:00:00+09:00 mokurin000 https://mokurin000.github.io/blog/fasthash/ <h2 id="qian-yan">前言</h2> <p>笔者下载到 SDEZ 1.65 后，发现 7-zip SHA256 计算性能并不理想，于是本着写玩具的念头搓了一个可能是目前最快的 hash 工具。</p> <p>第一版基于 compio ，但是单纯使用 compio + 同线程 hashing 对于单个大文件顺序读的效果并不理想。</p> <p>第二版改为完全单线程，read 与 hash 负载集中，达到了 50% 理论吞吐极限。</p> <p>此时 strace 抓了一下 Linux <code>sha256sum</code> syscall ，还引入了 <code>POSIX_FADV_SEQUENTIAL</code>，对应 Windows <code>FILE_FLAG_SEQUENTIAL_SCAN</code> 。</p> <p>第三版改为 IO Thread 分离，<code>crossfire</code> 无锁 SPSC channel 与无锁 queue buffer pool，终于达到了 <code>openssl speed</code> 16k block microbench 性能的 97%。</p> <h2 id="shi-xian">实现</h2> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-keyword">use</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">fs</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">File</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">use</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">io</span><span class="z-keyword z-operator">::</span><span class="z-punctuation">{</span><span class="z-entity z-name z-type">BorrowedBuf</span><span class="z-punctuation">,</span><span class="z-entity z-name z-type"> Read</span><span class="z-keyword"> as</span><span class="z-source"> _</span><span class="z-punctuation">};</span></span> <span class="giallo-l"><span class="z-keyword">use</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">sync</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Arc</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">use</span><span class="z-entity z-name z-namespace"> crossfire</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">flavor</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Queue</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-keyword">use</span><span class="z-entity z-name z-namespace"> digest</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Digest</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">fn</span><span class="z-entity z-name z-function"> hash_file</span><span class="z-punctuation z-brackets z-angle z-rust">&lt;</span><span class="z-entity z-name z-type">D</span><span class="z-keyword z-operator">:</span><span class="z-entity z-name z-type"> Digest</span><span class="z-punctuation z-brackets z-angle z-rust">&gt;</span><span class="z-punctuation">(</span></span> <span class="giallo-l"><span class="z-storage z-modifier"> mut</span><span class="z-variable z-other z-rust"> file</span><span class="z-keyword z-operator">:</span><span class="z-entity z-name z-type"> File</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-variable z-other z-rust"> buffer_size</span><span class="z-keyword z-operator">:</span><span class="z-entity z-name z-type z-numeric z-rust"> usize</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-variable z-other z-rust"> queue_len</span><span class="z-keyword z-operator">:</span><span class="z-entity z-name z-type z-numeric z-rust"> usize</span><span class="z-punctuation">,</span></span> <span class="giallo-l"><span class="z-punctuation">)</span><span class="z-keyword z-operator"> -&gt;</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">io</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Result</span><span class="z-punctuation z-brackets z-angle z-rust">&lt;</span><span class="z-entity z-name z-type">Box</span><span class="z-punctuation z-brackets z-angle z-rust">&lt;</span><span class="z-punctuation">[</span><span class="z-entity z-name z-type z-numeric z-rust">u8</span><span class="z-punctuation">]</span><span class="z-punctuation z-brackets z-angle z-rust">&gt;&gt;</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type z-rust z-storage z-modifier"> let mut</span><span class="z-source"> ctx</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-type"> D</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">new</span><span class="z-punctuation">();</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-source"> queue</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-type"> Arc</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">new</span><span class="z-punctuation">(</span><span class="z-entity z-name z-namespace">crossfire</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">spsc</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">Array</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">new</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">queue_len</span><span class="z-punctuation">));</span></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-punctuation"> (</span><span class="z-source">readed_tx</span><span class="z-punctuation">,</span><span class="z-source"> readed_rx</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-namespace"> crossfire</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">spsc</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">bounded_blocking</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">queue_len</span><span class="z-punctuation">);</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> for</span><span class="z-source"> _</span><span class="z-keyword"> in</span><span class="z-constant z-numeric"> 0</span><span class="z-keyword z-operator">..</span><span class="z-source">queue_len</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-source"> _</span><span class="z-keyword z-operator"> =</span><span class="z-source"> queue</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">push</span><span class="z-punctuation">(</span><span class="z-entity z-name z-namespace">Vec</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">with_capacity</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">buffer_size</span><span class="z-punctuation">));</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-source"> queue_</span><span class="z-keyword z-operator"> =</span><span class="z-source"> queue</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">clone</span><span class="z-punctuation">();</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-source"> io_result</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">thread</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">spawn</span><span class="z-punctuation">(</span><span class="z-keyword">move</span><span class="z-keyword z-operator"> ||</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-keyword"> loop</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-entity z-name z-type"> Some</span><span class="z-punctuation">(</span><span class="z-storage z-modifier">mut</span><span class="z-variable z-other z-rust"> buffer</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> =</span><span class="z-source"> queue_</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">pop</span><span class="z-punctuation">()</span><span class="z-keyword"> else</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">thread</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">yield_now</span><span class="z-punctuation">();</span></span> <span class="giallo-l"><span class="z-keyword"> continue</span><span class="z-punctuation">;</span></span> <span class="giallo-l"><span class="z-punctuation"> };</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-rust z-storage z-modifier"> let mut</span><span class="z-source"> cursor</span><span class="z-keyword z-operator"> =</span><span class="z-entity z-name z-type"> BorrowedBuf</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-function">from</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">buffer</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">spare_capacity_mut</span><span class="z-punctuation">());</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source"> file</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">read_buf</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">cursor</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">unfilled</span><span class="z-punctuation">())</span><span class="z-keyword z-operator">?</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword"> if</span><span class="z-source"> cursor</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">len</span><span class="z-punctuation">()</span><span class="z-keyword z-operator"> ==</span><span class="z-constant z-numeric"> 0</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-keyword"> break</span><span class="z-entity z-name z-namespace"> std</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-namespace">io</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Result</span><span class="z-keyword z-operator">::</span><span class="z-entity z-name z-type">Ok</span><span class="z-punctuation">(());</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span><span class="z-keyword"> else</span><span class="z-punctuation"> {</span></span> <span class="giallo-l"><span class="z-storage z-type z-rust"> let</span><span class="z-source"> new_len</span><span class="z-keyword z-operator"> =</span><span class="z-source"> cursor</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">len</span><span class="z-punctuation">();</span></span> <span class="giallo-l"><span class="z-keyword"> unsafe</span><span class="z-punctuation"> {</span><span class="z-source"> buffer</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">set_len</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">new_len</span><span class="z-punctuation">) };</span></span> <span class="giallo-l"><span class="z-source"> _</span><span class="z-keyword z-operator"> =</span><span class="z-source"> readed_tx</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">send</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">buffer</span><span class="z-punctuation">);</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"><span class="z-punctuation"> });</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword z-storage z-type z-rust"> while let</span><span class="z-entity z-name z-type"> Ok</span><span class="z-punctuation">(</span><span class="z-storage z-modifier">mut</span><span class="z-source"> buffer</span><span class="z-punctuation">)</span><span class="z-keyword z-operator"> =</span><span class="z-source"> readed_rx</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">recv</span><span class="z-punctuation">() {</span></span> <span class="giallo-l"><span class="z-source"> ctx</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">update</span><span class="z-punctuation">(</span><span class="z-keyword z-operator">&amp;</span><span class="z-variable z-other z-rust">buffer</span><span class="z-punctuation">);</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source"> buffer</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">clear</span><span class="z-punctuation">();</span></span> <span class="giallo-l"><span class="z-source"> _</span><span class="z-keyword z-operator"> =</span><span class="z-source"> queue</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">push</span><span class="z-punctuation">(</span><span class="z-variable z-other z-rust">buffer</span><span class="z-punctuation">);</span></span> <span class="giallo-l"><span class="z-punctuation"> }</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-source"> io_result</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">join</span><span class="z-punctuation">()</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">expect</span><span class="z-punctuation">(</span><span class="z-punctuation z-definition z-string z-string">&quot;file io thread panicked&quot;</span><span class="z-punctuation">)</span><span class="z-keyword z-operator">?</span><span class="z-punctuation">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-type"> Ok</span><span class="z-punctuation">(</span><span class="z-source">ctx</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">finalize</span><span class="z-punctuation">()</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">to_vec</span><span class="z-punctuation">()</span><span class="z-keyword z-operator">.</span><span class="z-entity z-name z-function">into_boxed_slice</span><span class="z-punctuation">())</span></span> <span class="giallo-l"><span class="z-punctuation">}</span></span></code></pre><h2 id="shi-ji-ce-shi-shu-ju">实际测试数据</h2> <ul> <li>OS: Windows 11 x86_64 22620</li> <li>CPU: Intel Core i7-12700H</li> <li>Storage: Predator GM7000</li> <li>File size: 69.8 GiB</li> <li>Hash algorithm: SHA-256</li> </ul> <table><thead><tr><th>Command</th><th>Time</th><th>Notes</th></tr></thead><tbody> <tr><td><code>fasthash sha256 -b 1MiB file</code></td><td>36s</td><td>queue=8</td></tr> <tr><td><code>fasthash sha256 -b 256K file</code></td><td>52s</td><td>queue=8</td></tr> <tr><td><code>sha256sum file</code></td><td>67s</td><td>uutils 0.9.0</td></tr> <tr><td>hashlib*</td><td>68s</td><td>Python 3.13.5</td></tr> <tr><td><code>openssl sha256 file</code></td><td>78s</td><td>OpenSSL 3.6.0 MSVC</td></tr> <tr><td>–</td><td>85s</td><td>Nanazip 6.0.1742</td></tr> <tr><td><code>Get-FileHash file</code></td><td>96.9s</td><td>PowerShell 5.1</td></tr> <tr><td><code>certutil -hashfile file sha256</code></td><td>107s</td><td></td></tr> <tr><td><code>sha256sum file</code></td><td>123s</td><td>Microsoft coreutils</td></tr> <tr><td><code>open -r file | hash sha256sum</code></td><td>130s</td><td>NuShell 0.112.2</td></tr> </tbody></table> <h3 id="openssl-speed-evp-sha256">openssl speed -evp sha256</h3> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>Doing sha256 ops for 3s on 16 size blocks: 15350007 sha256 ops in 2.91s</span></span> <span class="giallo-l"><span>Doing sha256 ops for 3s on 64 size blocks: 13970470 sha256 ops in 2.97s</span></span> <span class="giallo-l"><span>Doing sha256 ops for 3s on 256 size blocks: 10049997 sha256 ops in 2.88s</span></span> <span class="giallo-l"><span>Doing sha256 ops for 3s on 1024 size blocks: 4545898 sha256 ops in 3.00s</span></span> <span class="giallo-l"><span>Doing sha256 ops for 3s on 8192 size blocks: 743081 sha256 ops in 2.94s</span></span> <span class="giallo-l"><span>Doing sha256 ops for 3s on 16384 size blocks: 386881 sha256 ops in 2.95s</span></span> <span class="giallo-l"><span>version: 3.6.0</span></span> <span class="giallo-l"><span>built on: Wed Oct 8 20:29:58 2025 UTC</span></span> <span class="giallo-l"><span>options: bn(64,64)</span></span> <span class="giallo-l"><span>compiler: cl /Z7 /Fdossl_static.pdb /Gs0 /GF /Gy /MD /W3 /wd4090 /nologo /O2 -DL_ENDIAN -DOPENSSL_PIC -D&quot;OPENSSL_BUILDING_OPENSSL&quot; -D&quot;OPENSSL_SYS_WIN32&quot; -D&quot;WIN32_LEAN_AND_MEAN&quot; -D&quot;UNICODE&quot; -D&quot;_UNICODE&quot; -D&quot;_CRT_SECURE_NO_DEPRECATE&quot; -D&quot;_WINSOCK_DEPRECATED_NO_WARNINGS&quot; -D&quot;NDEBUG&quot; -D_WINSOCK_DEPRECATED_NO_WARNINGS -D_WIN32_WINNT=0x0502</span></span> <span class="giallo-l"><span>CPUINFO: OPENSSL_ia32cap=0xfffaf38bffcbffff:0x184007a4239c27a9:0x00400810bc18c410:0x0000000000000000:0x0000000000000000</span></span> <span class="giallo-l"><span>The &#39;numbers&#39; are in 1000s of bytes per second processed.</span></span> <span class="giallo-l"><span>type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes</span></span> <span class="giallo-l"><span>sha256 84507.57k 301173.92k 894886.69k 1551666.52k 2072279.00k 2146423.98k</span></span></code></pre><h2 id="wei-shen-me-bu-huan-suan-fa">为什么不换算法？</h2> <p>好问题。<code>gxhash</code>/<code>rapidhash</code>/<code>xxh3</code> 这些力大砖飞或 SIMD 优化的新型 non-crypto 算法对于大文件校验是很合适的。</p> <p>虽然没有人提供。</p> <h2 id="hou-ji">后记</h2> <p>绞尽脑汁的 userspace I/O 优化，第一步应该先看看有没有合适的flag。</p> <p>内核可以在更底层根据你将要进行的读取策略，进行第一步优化。</p> <p>Repo：<a rel="external" href="https://github.com/mokurin000/fasthash">fasthash</a></p> 从 compio 到无锁 SPSC queue，逐步逼近 openssl speed 的大文件 hashing 实现