atom.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title><![CDATA[鸟窝]]></title>
  <subtitle><![CDATA[大道至简 Simplicity is the ultimate form of sophistication]]></subtitle>
  <link href="/atom.xml" rel="self"/>
  <link href="https://colobu.com/"/>
  <updated>2025-02-01T15:46:14.351Z</updated>
  <id>https://colobu.com/</id>
  
  <author>
    <name><![CDATA[smallnest]]></name>
    
  </author>
  
  <generator uri="http://zespia.tw/hexo/">Hexo</generator>
  
  <entry>
    <title><![CDATA[啥时候等到Go官方支持SIMD?]]></title>
    <link href="https://colobu.com/2025/02/01/the-state-of-simd-in-go/"/>
    <id>https://colobu.com/2025/02/01/the-state-of-simd-in-go/</id>
    <published>2025-02-01T15:39:40.000Z</published>
    <updated>2025-02-01T15:45:58.244Z</updated>
    <content type="html"><![CDATA[<p>单指令多数据流（<code>SIMD</code>，Single Instruction Multiple Data）是一种并行计算技术，允许一条指令同时处理多个数据点。SIMD在现代CPU中广泛应用，能够显著提升计算密集型任务的性能，如图像处理、机器学习、科学计算等。随着Go语言在高性能计算领域的应用逐渐增多，SIMD支持成为了开发者关注的焦点。</p>
<p>当前很多主流和新型的语言都有相应的<code>simd</code>库了，比如C++、Rust、Zig等，但Go语言的<code>simd</code>官方支持还一直在讨论中(<a href="https://github.com/golang/go/issues/67520" target="_blank" rel="external">issue#67520</a>)。Go语言的设计目标是简单性和可移植性，而SIMD的实现通常需要针对不同的硬件架构进行优化，这与Go的设计目标存在一定冲突。因此，Go语言对SIMD的支持一直备受争议。<br>最近几周这个issue的讨论有活跃起来， 希望能快点支持。</p>
<a id="more"></a>
<h2 id="1-_Go语言与SIMD的背景">1. Go语言与SIMD的背景</h2>
<h3 id="1-1_Go语言的性能追求">1.1 Go语言的性能追求</h3>
<p>Go语言以其简洁的语法、高效的并发模型和快速的编译速度赢得了广泛的应用。然而，Go在性能优化方面一直面临挑战，尤其是在需要处理大量数据的场景下。SIMD作为一种高效的并行计算技术，能够显著提升计算性能，因此Go社区对SIMD的支持呼声日益高涨。</p>
<p>如果没有 SIMD，我们就会错过很多潜在的优化。以下是可以提高日常生活场景中性能的具体事项的非详尽列表：</p>
<ul>
<li><a href="https://github.com/simdjson/simdjson" target="_blank" rel="external">simdjson</a></li>
<li><a href="https://people.csail.mit.edu/jshun/6886-s19/lectures/lecture19-1.pdf" target="_blank" rel="external">通过矢量化每秒解码数十亿个整数</a></li>
<li><a href="https://opensource.googleblog.com/2022/06/Vectorized%20and%20performance%20portable%20Quicksort.html" target="_blank" rel="external">矢量化和性能可移植的快速排序</a></li>
<li><a href="https://www.intel.com/content/www/us/en/developer/articles/technical/introduction-to-hyperscan.html" target="_blank" rel="external">Hyperscan 简介</a></li>
<li><a href="https://sourcegraph.com/blog/slow-to-simd" target="_blank" rel="external">From slow to SIMD: A Go optimization story</a></li>
<li><a href="https://gorse.io/posts/avx512-in-golang.html#convert-assembly" target="_blank" rel="external">How to Use AVX512 in Golang via C Compiler</a></li>
</ul>
<p>此外，它将使这些当前存在的软件包更具可移植性和可维护性：</p>
<ul>
<li><a href="https://github.com/minio/simdjson-go" target="_blank" rel="external">simdjson-go</a></li>
<li><a href="https://github.com/minio/sha256-simd" target="_blank" rel="external">SHA256-SIMD</a></li>
<li><a href="https://github.com/minio/md5-simd" target="_blank" rel="external">MD5-SIMD</a></li>
</ul>
<p>在这个月即将发布的Go 1.24版中，将会将内建的map使用Swiss Tables替换，而Swiss Tables针对AMD64的架构采用了<a href="https://go-review.googlesource.com/c/go/+/626277" target="_blank" rel="external">SIMD的代码</a>，这是不是Go官方代码库首次引进了SIMD的指令呢？</p>
<p>当前先前也有人实现了SIMD加速encoding/hex，<a href="https://go-review.googlesource.com/c/go/+/110195" target="_blank" rel="external">被否了</a>，当然理由也很充分：加速效果很好但请放弃吧，看起来太复杂，违背了Go简洁的初衷。<br>类似的还有<a href="https://go-review.googlesource.com/c/go/+/535838" target="_blank" rel="external">unicode/utf8: make Valid use AVX2 on amd64</a></p>
<p>其实Go官方在2023就已经在标准库crypto/sha256中使用SIMD指令了 <a href="https://go-review.googlesource.com/c/go/+/408795" target="_blank" rel="external">crypto/sha256: add sha-ni implementation</a>。</p>
<h3 id="1-2_SIMD的基本概念">1.2 SIMD的基本概念</h3>
<p>SIMD通过一条指令同时处理多个数据点，通常用于向量化计算。现代CPU（如Intel的<code>SSE/AVX</code>、ARM的<code>NEON</code>）都提供了SIMD指令集，允许开发者通过特定的指令集加速计算任务。然而，直接使用SIMD指令集通常需要编写汇编代码或使用特定的编译器内置函数，这对开发者提出了较高的要求。</p>
<h4 id="1-2-1_SIMD的核心思想">1.2.1 SIMD的核心思想</h4>
<p>SIMD的核心思想是通过一条指令同时处理多个数据点。例如，传统的标量加法指令一次只能处理两个数，而SIMD加法指令可以同时处理多个数（如4个、8个甚至更多）。这种并行化处理方式能够显著提升计算密集型任务的性能。</p>
<h4 id="1-2-2_SIMD指令集的组成">1.2.2 SIMD指令集的组成</h4>
<p>SIMD指令集通常包括以下几类指令：</p>
<ul>
<li><strong>算术运算</strong>：加法、减法、乘法、除法等。</li>
<li><strong>逻辑运算</strong>：与、或、非、异或等。</li>
<li><strong>数据搬移</strong>：加载、存储、重排数据。</li>
<li><strong>比较操作</strong>：比较多个数据点并生成掩码。</li>
<li><strong>特殊操作</strong>：如求平方根、绝对值、最大值、最小值等。</li>
</ul>
<h3 id="1-3_常见的指令集">1.3 常见的指令集</h3>
<h4 id="1-3-1_Intel的SIMD指令集">1.3.1 Intel的SIMD指令集</h4>
<h5 id="1-3-1-1_MMX（MultiMedia_eXtensions）">1.3.1.1 MMX（MultiMedia eXtensions）</h5>
<ul>
<li><strong>推出时间</strong>：1996年</li>
<li><strong>寄存器宽度</strong>：64位</li>
<li><strong>数据类型</strong>：整数（8位、16位、32位）</li>
<li><strong>特点</strong>：<ul>
<li>主要用于多媒体处理。</li>
<li>引入了8个64位寄存器（MM0-MM7）。</li>
<li>不支持浮点数运算。</li>
</ul>
</li>
</ul>
<h5 id="1-3-1-2_SSE（Streaming_SIMD_Extensions）">1.3.1.2 SSE（Streaming SIMD Extensions）</h5>
<ul>
<li><strong>推出时间</strong>：1999年</li>
<li><strong>寄存器宽度</strong>：128位</li>
<li><strong>数据类型</strong>：单精度浮点数（32位）、整数（8位、16位、32位、64位）</li>
<li><strong>特点</strong>：<ul>
<li>引入了8个128位寄存器（XMM0-XMM7）。</li>
<li>支持浮点数运算，适用于科学计算和图形处理。</li>
<li>后续版本（SSE2、SSE3、SSSE3、SSE4）增加了更多指令和功能。</li>
</ul>
</li>
</ul>
<h5 id="1-3-1-3_AVX（Advanced_Vector_Extensions）">1.3.1.3 AVX（Advanced Vector Extensions）</h5>
<ul>
<li><strong>推出时间</strong>：2011年</li>
<li><strong>寄存器宽度</strong>：256位</li>
<li><strong>数据类型</strong>：单精度浮点数（32位）、双精度浮点数（64位）、整数（8位、16位、32位、64位）</li>
<li><strong>特点</strong>：<ul>
<li>引入了16个256位寄存器（YMM0-YMM15）。</li>
<li>支持更宽的向量操作，性能进一步提升。</li>
<li>后续版本（AVX2、AVX-512）支持更复杂的操作和更宽的寄存器（512位）。</li>
</ul>
</li>
</ul>
<h5 id="1-3-1-4_AVX-512">1.3.1.4 AVX-512</h5>
<ul>
<li><strong>推出时间</strong>：2016年</li>
<li><strong>寄存器宽度</strong>：512位</li>
<li><strong>数据类型</strong>：单精度浮点数（32位）、双精度浮点数（64位）、整数（8位、16位、32位、64位）</li>
<li><strong>特点</strong>：<ul>
<li>引入了32个512位寄存器（ZMM0-ZMM31）。</li>
<li>支持更复杂的操作，如掩码操作、广播操作等。</li>
<li>主要用于高性能计算和人工智能领域。</li>
</ul>
</li>
</ul>
<h4 id="1-3-2_ARM的SIMD指令集">1.3.2 ARM的SIMD指令集</h4>
<h5 id="1-3-2-1_NEON">1.3.2.1 NEON</h5>
<ul>
<li><strong>推出时间</strong>：2005年</li>
<li><strong>寄存器宽度</strong>：128位</li>
<li><strong>数据类型</strong>：单精度浮点数（32位）、整数（8位、16位、32位、64位）</li>
<li><strong>特点</strong>：<ul>
<li>广泛应用于移动设备和嵌入式系统。</li>
<li>支持16个128位寄存器（Q0-Q15）。</li>
<li>适用于多媒体处理、信号处理等场景。</li>
</ul>
</li>
</ul>
<h5 id="1-3-2-2_SVE（Scalable_Vector_Extension）">1.3.2.2 SVE（Scalable Vector Extension）</h5>
<ul>
<li><strong>推出时间</strong>：2016年</li>
<li><strong>寄存器宽度</strong>：可变（128位至2048位）</li>
<li><strong>数据类型</strong>：单精度浮点数（32位）、双精度浮点数（64位）、整数（8位、16位、32位、64位）</li>
<li><strong>特点</strong>：<ul>
<li>支持可变长度的向量操作，适应不同的硬件配置。</li>
<li>引入了谓词寄存器（Predicate Registers），支持条件执行。</li>
<li>主要用于高性能计算和机器学习。</li>
</ul>
</li>
</ul>
<h3 id="1-4_编译器内置函数">1.4 编译器内置函数</h3>
<p>大多数现代编译器（如GCC、Clang、MSVC）提供了SIMD指令集的内置函数，开发者可以通过这些函数调用SIMD指令，而无需编写汇编代码。</p>
<h3 id="1-5_自动向量化">1.5 自动向量化</h3>
<p>一些编译器支持自动向量化功能，能够自动将标量代码转换为SIMD代码。例如，使用GCC编译以下代码时，可以启用自动向量化：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">gcc -O3 -mavx2 -o program program.c</div></pre></td></tr></table></figure>

<h2 id="2-_Go语言中的SIMD支持现状">2. Go语言中的SIMD支持现状</h2>
<h3 id="2-1_Go语言标准库的SIMD支持">2.1 Go语言标准库的SIMD支持</h3>
<p>Go语言的标准库尚未提供对SIMD的直接支持。Go语言的编译器（gc）也没有自动向量化功能，这意味着开发者无法像在C/C++中那样通过编译器自动生成SIMD代码。</p>
<p>在Issue <a href="https://github.com/golang/go/issues/67520" target="_blank" rel="external">#67520</a> 中，讨论依然磨磨唧唧，讨论时常偏离到实现的具体方式上(build tag)。</p>
<h3 id="2-2_第三方库与解决方案">2.2 第三方库与解决方案</h3>
<p>尽管Go语言标准库缺乏对SIMD的直接支持，但社区已经开发了一些第三方库和工具，帮助开发者在Go中使用SIMD指令集。在<a href="https://github.com/golang/go/issues/67520" target="_blank" rel="external">#67520</a>的讨论中，Clement Jean 也提供了一个概念化的实现方案：<a href="https://github.com/Clement-Jean/simd-go-POC" target="_blank" rel="external">simd-go-POC</a> 。</p>
<p>以下是一些第三方实现的(simd指令，不是基于simd实现的库sonic、simdjson-go等)：</p>
<h4 id="2-2-1_kelindar/simd">2.2.1 <code>kelindar/simd</code></h4>
<p><a href="https://github.com/kelindar/simd" target="_blank" rel="external">kelindar/simd</a>这个库包含一组矢量化的数学函数，它们使用 clang 编译器自动矢量化，并转换为 Go 的 PLAN9 汇编代码。对于不支持矢量化的 CPU，或此库没有为其生成代码的 CPU，也提供了通用版本。</p>
<p>目前它仅支持 AVX2，但生成 AVX512 和 SVE (for ARM) 的代码应该很容易。这个库中的大部分代码都是自动生成的，这有助于维护。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">sum := simd.SumFloat32s([]<span class="typename">float32</span><span class="number">{1</span>,<span class="number"> 2</span>,<span class="number"> 3</span>,<span class="number"> 4</span>,<span class="number"> 5</span>})</div></pre></td></tr></table></figure>

<h4 id="2-2-2_alivanz/go-simd">2.2.2 <code>alivanz/go-simd</code></h4>
<p>[alivanz/go-simd](<a href="https://github.com/alivanz/go-simd）实现了" target="_blank" rel="external">https://github.com/alivanz/go-simd）实现了</a> Go 语言的 SIMD（单指令多数据）操作，专门针对 ARM NEON 架构进行了优化。其目标是为特定的计算任务提供优化的并行处理能力。<br>下面是一个加法和乘法的例子：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"log"</span></div><div class="line"></div><div class="line">	<span class="string">"github.com/alivanz/go-simd/arm"</span></div><div class="line">	<span class="string">"github.com/alivanz/go-simd/arm/neon"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	<span class="keyword">var</span> a, b arm.Int8X8</div><div class="line">	<span class="keyword">var</span> add, mul arm.Int16X8</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt;<span class="number"> 8</span>; i++ {</div><div class="line">		a[i] = arm.Int8(i)</div><div class="line">		b[i] = arm.Int8(i * i)</div><div class="line">	}</div><div class="line">	log.Printf(<span class="string">"a = %+v"</span>, b)</div><div class="line">	log.Printf(<span class="string">"b = %+v"</span>, a)</div><div class="line">	neon.VaddlS8(&add, &a, &b)</div><div class="line">	neon.VmullS8(&mul, &a, &b)</div><div class="line">	log.Printf(<span class="string">"add = %+v"</span>, add)</div><div class="line">	log.Printf(<span class="string">"mul = %+v"</span>, mul)</div><div class="line">}</div></pre></td></tr></table></figure>

<h4 id="2-2-3_pehringer/simd">2.2.3 <code>pehringer/simd</code></h4>
<p><a href="https://github.com/pehringer/simd" target="_blank" rel="external">pehringer/simd</a> 通过 Go 汇编提供 SIMD 支持，实现了算术运算、位运算以及最大值和最小值运算。它允许进行并行的逐元素计算，从而带来 100% 到 400% 的速度提升。目前支持 AMD64 (x86_64) 和 ARM64 处理器。</p>
<h3 id="2-3_Go汇编与SIMD">2.3 Go汇编与SIMD</h3>
<p>Go语言支持通过汇编代码直接调用CPU指令集，这为SIMD的实现提供了可能。开发者可以编写Go汇编代码，调用特定的SIMD指令集（如SSE、AVX等），从而实现高性能的向量化计算。然而，编写和维护汇编代码对开发者提出了较高的要求，且代码的可移植性较差。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// 以下是一个简单的Go汇编示例，使用AVX指令集进行向量加法</span></div><div class="line">TEXT ·add(SB), <span class="number">$0</span><span class="number">-32</span></div><div class="line">    MOVQ a<span class="number">+0</span>(FP), DI</div><div class="line">    MOVQ b<span class="number">+8</span>(FP), SI</div><div class="line">    MOVQ result<span class="number">+16</span>(FP), DX</div><div class="line">    MOVQ <span class="built_in">len</span><span class="number">+24</span>(FP), CX</div><div class="line">    </div><div class="line">    TESTQ CX, CX        ; 检查长度是否<span class="number">为0</span></div><div class="line">    JZ done             ; 如果<span class="number">为0</span>直接返回</div><div class="line">    </div><div class="line">    MOVQ CX, R8         ; 保存原始长度</div><div class="line">    SHRQ <span class="number">$2</span>, CX         ; 除<span class="number">以4</span>得到循环次数</div><div class="line">    JZ remainder        ; 如果不<span class="number">足4</span>个元素，跳到处理余数</div><div class="line">    </div><div class="line">    XORQ R9, R9         ; 用于索引的计数器，<span class="number">从0</span>开始</div><div class="line">loop:</div><div class="line">    VMOVUPD (DI)(R9<span class="number">*8</span>), Y0</div><div class="line">    VMOVUPD (SI)(R9<span class="number">*8</span>), Y1</div><div class="line">    VADDPD Y0, Y1, Y0</div><div class="line">    VMOVUPD Y0, (DX)(R9<span class="number">*8</span>)</div><div class="line">    ADDQ <span class="number">$4</span>, R9</div><div class="line">    DECQ CX</div><div class="line">    JNZ loop</div><div class="line"></div><div class="line">remainder:              ; 处理剩余的元素</div><div class="line">    ANDQ <span class="number">$3</span>, R8        ; 获取余数</div><div class="line">    JZ done</div><div class="line">    ; 这里添加处理余数的代码</div><div class="line"></div><div class="line">done:</div><div class="line">    RET</div></pre></td></tr></table></figure>

<p>当然需要a,b和 result 数组的地址是对齐的，以获得最佳性能。</p>
<h2 id="结论">结论</h2>
<p>尽管Go语言目前对SIMD的支持尚不完善，但社区已经通过第三方库和汇编代码提供了一些解决方案。未来，随着Go编译器的改进和标准库的支持(相信Go官方最终会支持的)，Go语言在高性能计算领域的潜力将进一步释放。对于开发者而言，掌握SIMD技术将有助于编写更高效的Go代码，应对日益复杂的计算任务。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>单指令多数据流（<code>SIMD</code>，Single Instruction Multiple Data）是一种并行计算技术，允许一条指令同时处理多个数据点。SIMD在现代CPU中广泛应用，能够显著提升计算密集型任务的性能，如图像处理、机器学习、科学计算等。随着Go语言在高性能计算领域的应用逐渐增多，SIMD支持成为了开发者关注的焦点。</p>
<p>当前很多主流和新型的语言都有相应的<code>simd</code>库了，比如C++、Rust、Zig等，但Go语言的<code>simd</code>官方支持还一直在讨论中(<a href="https://github.com/golang/go/issues/67520" target="_blank" rel="external">issue#67520</a>)。Go语言的设计目标是简单性和可移植性，而SIMD的实现通常需要针对不同的硬件架构进行优化，这与Go的设计目标存在一定冲突。因此，Go语言对SIMD的支持一直备受争议。<br>最近几周这个issue的讨论有活跃起来， 希望能快点支持。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[DeepSeek数据库暴露？扫描一下，应该不止此一家吧!]]></title>
    <link href="https://colobu.com/2025/01/31/scan-clickhouse-service/"/>
    <id>https://colobu.com/2025/01/31/scan-clickhouse-service/</id>
    <published>2025-01-31T04:02:33.000Z</published>
    <updated>2025-01-31T04:04:52.695Z</updated>
    <content type="html"><![CDATA[<p>DeepSeek出街老火了，整个AI界都在热火朝天的讨论它。</p>
<p>同时，安全界也没闲着，来自美国的攻击使它不得不通知中国大陆以外的手机号的注册，同时大家也对它的网站和服务安全性进行了审视，这不Wiz Research就发现它们的数据库面向公网暴露并且无需任何身份即可访问。这两个域名oauth2callback.deepseek.com:9000和dev.deepseek.com:9000。</p>
<p>AI的核心技术既需要这些清北的天才去研究，产品也需要专业的人才去打磨。像DeepSeek这么专业的公司都可能出现这样的漏洞，相信互联网上这么数据库无密码暴露的实例也应该不在少数(实际只找到了2个)。</p>
<p>基于上一篇《扫描全国的公网IP要多久》，我们改造一下代码，让它使用 <code>tcp_syn</code> 的方式探测clickhopuse的9000端口。</p>
<p>首先声明，所有的技术都是为了给大家介绍使用Go语言开发底层的网络程序所做的演示，不是为了介绍安全和攻击方面的内容，所以也不会使用已经成熟的端口和IP扫描工具如zmap、rustscan、nmap、masscan、Advanced IP Scanner、Angry IP Scanner、unicornscan等工具。</p>
<p>同时，也不会追求快速，我仅仅在家中的100M的网络中，使用一台10多年前的4核Linux机器进行测试，尽可能让它能出结果。我一般晚上启动它，早上吃过早餐后来查看结果。</p>
<a id="more"></a>
<p>我想把这个实验分成两部分：</p>
<ol>
<li>寻找中国大陆暴露9000端口的公网IP</li>
<li>检查这些IP是否是部署clickhouse并可以无密码的访问</li>
</ol>
<p>接下来先介绍第一部分。</p>
<h2 id="寻找暴露端口9000的IP">寻找暴露端口9000的IP</h2>
<p>我们需要将上一篇的代码改造，让它使用TCP进行扫描，而不是ICMP扫描，而且我们只扫描9000端口。</p>
<p>为了更有效的扫描，我做了以下的优化：</p>
<ol>
<li>使用ICMP扫描出来的可用IP, 一共五千多万</li>
<li>使用tcp sync模拟TCP建联是的握手，这样目的服务器会回一个sync+ack的包</li>
<li>同时探测机自动回复一个RST, 我们也别老挂着目的服务器，怪不好意思的，及时告诉人家别等着咱了</li>
</ol>
<p>同样的，我们也定义一个<code>TCPScanner</code>结构体，用来使用TCP握手来进行探测。如果你已经阅读了前一篇文章，应该对我们实现的套路有所了解。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> fishfinding</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"net"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">	<span class="string">"time"</span></div><div class="line"></div><div class="line">	<span class="string">"github.com/kataras/golog"</span></div><div class="line">	<span class="string">"golang.org/x/net/bpf"</span></div><div class="line">	<span class="string">"golang.org/x/net/ipv4"</span></div><div class="line">)</div><div class="line"></div><div class="line"></div><div class="line"><span class="keyword">type</span> TCPScanner <span class="keyword">struct</span> {</div><div class="line">	src     net.IP</div><div class="line">	srcPort <span class="typename">int</span></div><div class="line">	dstPort <span class="typename">int</span></div><div class="line">	input   <span class="keyword">chan</span> <span class="typename">string</span></div><div class="line">	output  <span class="keyword">chan</span> <span class="typename">string</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> NewTCPScanner(srcPort, dstPort <span class="typename">int</span>, input <span class="keyword">chan</span> <span class="typename">string</span>, output <span class="keyword">chan</span> <span class="typename">string</span>) *TCPScanner {</div><div class="line">	localIP := GetLocalIP()</div><div class="line">	s := &TCPScanner{</div><div class="line">		input:   input,</div><div class="line">		output:  output,</div><div class="line">		src:     net.ParseIP(localIP).To4(),</div><div class="line">		srcPort: srcPort,</div><div class="line">		dstPort: dstPort,</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> s</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (s *TCPScanner) Scan() {</div><div class="line">	<span class="keyword">go</span> s.recv()</div><div class="line">	<span class="keyword">go</span> s.send(s.input)</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这里定义了一个<code>TCPScanner</code>结构体，它有一个<code>Scan</code>方法，用来启动接收和发送两个goroutine。接收goroutine用来接收目标服务器的回复(sync+ack 包)，发送goroutine用来发送TCP sync包。</p>
<h3 id="发送逻辑">发送逻辑</h3>
<p>发送goroutine首先通过<code>net.ListenPacket</code>创建一个原始套接字，这里使用的是<code>ip4:tcp</code>，然后发送TCP的包就可以了。</p>
<p>我并没有使用gopacket这个库来构造TCP包，而是自己构造了TCP包，因为我觉得gopacket这个库太重了，而且我只需要构造TCP包，所以自己构造一个TCP包也不是很难。</p>
<p>seq数我们使用了当前进程的PID，这样在接收到回包的时候，还可以使用这个seq数来判断是不是我们发送的回包。</p>
<p>注意这里我们要计算tcp包的checksum, 并没有利用网卡的TCP/IP Checksum Offload功能，而是自己计算checksum,原因在于我的机的网卡很古老了，没有这个功能。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (s *TCPScanner) send(input <span class="keyword">chan</span> <span class="typename">string</span>) error {</div><div class="line">	<span class="keyword">defer</span> <span class="keyword">func</span>() {</div><div class="line">		time.Sleep<span class="number">(5</span> * time.Second)</div><div class="line">		<span class="built_in">close</span>(s.output)</div><div class="line">		golog.Infof(<span class="string">"send goroutine exit"</span>)</div><div class="line">	}()</div><div class="line"></div><div class="line">	<span class="comment">// 创建原始套接字</span></div><div class="line">	conn, err := net.ListenPacket(<span class="string">"ip4:tcp"</span>, s.src.To4().String())</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		golog.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> conn.Close()</div><div class="line"></div><div class="line">	pconn := ipv4.NewPacketConn(conn)</div><div class="line">	<span class="comment">// 不接收数据</span></div><div class="line">	filter := createEmptyFilter()</div><div class="line">	<span class="keyword">if</span> assembled, err := bpf.Assemble(filter); err == <span class="constant">nil</span> {</div><div class="line">		pconn.SetBPF(assembled)</div><div class="line">	}</div><div class="line"></div><div class="line">	seq := <span class="typename">uint32</span>(os.Getpid())</div><div class="line">	<span class="keyword">for</span> ip := <span class="keyword">range</span> input {</div><div class="line">		dstIP := net.ParseIP(ip)</div><div class="line">		<span class="keyword">if</span> dstIP == <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to resolve IP address %s"</span>, ip)</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 构造 TCP SYN 包</span></div><div class="line">		tcpHeader := &TCPHeader{</div><div class="line">			Source:      <span class="typename">uint16</span>(s.srcPort), <span class="comment">// 源端口</span></div><div class="line">			Destination: <span class="typename">uint16</span>(s.dstPort), <span class="comment">// 目标端口(这里探测80端口)</span></div><div class="line">			SeqNum:      seq,</div><div class="line">			AckNum:     <span class="number"> 0</span>,</div><div class="line">			Flags:      <span class="number"> 0</span>x002, <span class="comment">// SYN</span></div><div class="line">			Window:     <span class="number"> 65535</span>,</div><div class="line">			Checksum:   <span class="number"> 0</span>,</div><div class="line">			Urgent:     <span class="number"> 0</span>,</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 计算校验和</span></div><div class="line">		tcpHeader.Checksum = tcpChecksum(tcpHeader, s.src, dstIP)</div><div class="line"></div><div class="line">		<span class="comment">// 序列化 TCP 头</span></div><div class="line">		packet := tcpHeader.Marshal()</div><div class="line"></div><div class="line">		<span class="comment">// 发送 TCP SYN 包</span></div><div class="line">		_, err = conn.WriteTo(packet, &net.IPAddr{IP: dstIP})</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to send TCP packet: %v"</span>, err)</div><div class="line">		}</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">}</div></pre></td></tr></table></figure>

<h3 id="接收逻辑">接收逻辑</h3>
<p>接收goroutine首先创建一个原始套接字，使用<code>net.ListenIP</code>，然后使用<code>ipv4.NewPacketConn</code>来创建一个<code>ipv4.PacketConn</code>，并设置<code>ipv4.FlagSrc|ipv4.FlagDst|ipv4.FlagInterface</code>，这样可以获取到源IP、目标IP和接口信息。<br>这里必须设置<code>ipv4.FlagSrc|ipv4.FlagDst|ipv4.FlagInterface</code>, 否则不能获取到目标服务器的IP。<code>pv4.FlagDst</code>到是不需要的。</p>
<p>接收到数据后，我们解析TCP头，然后判断是否是我们发送的包，如果是我们发送的包，我们就将目标IP发送到<code>output</code>通道。</p>
<p>如果是我们发送的回包，我们就判断是否是SYN+ACK包，同时判断ACK是否和我们发送的seq对应，如果是，我们就将目标IP发送到<code>output</code>通道。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (s *TCPScanner) recv() error {</div><div class="line">	<span class="keyword">defer</span> <span class="built_in">recover</span>()</div><div class="line"></div><div class="line">	<span class="comment">// 创建原始套接字</span></div><div class="line">	conn, err := net.ListenIP(<span class="string">"ip4:tcp"</span>, &net.IPAddr{IP: s.src})</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		golog.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> conn.Close()</div><div class="line"></div><div class="line">	pconn := ipv4.NewPacketConn(conn)</div><div class="line">	<span class="keyword">if</span> err := pconn.SetControlMessage(ipv4.FlagSrc|ipv4.FlagDst|ipv4.FlagInterface, <span class="constant">true</span>); err != <span class="constant">nil</span> {</div><div class="line">		golog.Fatalf(<span class="string">"set control message error: %v\n"</span>, err)</div><div class="line">	}</div><div class="line"></div><div class="line">	seq := <span class="typename">uint32</span>(os.Getpid()) +<span class="number"> 1</span></div><div class="line"></div><div class="line">	buf := <span class="built_in">make</span>([]<span class="typename">byte</span>,<span class="number"> 1024</span>)</div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		n, peer, err := conn.ReadFrom(buf)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to read: %v"</span>, err)</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="keyword">if</span> n &lt; tcpHeaderLength {</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 解析 TCP 头</span></div><div class="line">		tcpHeader := ParseTCPHeader(buf[:n])</div><div class="line"></div><div class="line">		<span class="keyword">if</span> tcpHeader.Destination != <span class="typename">uint16</span>(s.srcPort) || tcpHeader.Source != <span class="typename">uint16</span>(s.dstPort) {</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// golog.Info("peer: %s, flags: %d", peer.String(), tcpHeader.Flags)</span></div><div class="line"></div><div class="line">		<span class="comment">// 检查是否是 SYN+ACK, 同时检查ACK是否和发送的seq对应</span></div><div class="line">		<span class="keyword">if</span> tcpHeader.Flags ==<span class="number"> 0</span>x012 && tcpHeader.AckNum == seq { <span class="comment">// SYN + ACK</span></div><div class="line">			s.output &lt;- peer.String()</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>完整的代码在<a href="https://github.com/smallnest/fishfinder" target="_blank" rel="external">这里</a>。</p>
<p>最终我把可以连接端口9000的IP保存到了一个文件中，一共有970+个IP。</p>
<h2 id="检查没有身份验证clickhouse">检查没有身份验证clickhouse</h2>
<p>接下来我们要检查这些IP是否是clickhouse的服务，而且没有身份验证。</p>
<p>使用类似的方法，我们定义一个<code>ClickHouseChecker</code>结构体，用来检查这些IP是否是clickhouse的服务。</p>
<p>它会尝试使用这些IP和9000建立和clickhouse的连接，如果连接成功，并且调用<code>Ping()</code>方法成功，我们就认为这个IP是clickhouse的服务。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> fishfinding</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"context"</span></div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"runtime"</span></div><div class="line">	<span class="string">"sync"</span></div><div class="line">	<span class="string">"time"</span></div><div class="line"></div><div class="line">	<span class="string">"github.com/ClickHouse/clickhouse-go/v2"</span></div><div class="line">	_ <span class="string">"github.com/ClickHouse/clickhouse-go/v2"</span></div><div class="line">	<span class="string">"github.com/kataras/golog"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">type</span> ClickHouseChecker <span class="keyword">struct</span> {</div><div class="line">	wg   *sync.WaitGroup</div><div class="line">	port <span class="typename">int</span></div><div class="line"></div><div class="line">	input  <span class="keyword">chan</span> <span class="typename">string</span></div><div class="line">	output <span class="keyword">chan</span> <span class="typename">string</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> NewClickHouseChecker(port <span class="typename">int</span>, input <span class="keyword">chan</span> <span class="typename">string</span>, output <span class="keyword">chan</span> <span class="typename">string</span>, wg *sync.WaitGroup) *ClickHouseChecker {</div><div class="line">	s := &ClickHouseChecker{</div><div class="line">		port:   port,</div><div class="line">		input:  input,</div><div class="line">		output: output,</div><div class="line">		wg:     wg,</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> s</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (s *ClickHouseChecker) Check() {</div><div class="line">	parallel := runtime.NumCPU()</div><div class="line"></div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; parallel; i++ {</div><div class="line">		s.wg.Add<span class="number">(1</span>)</div><div class="line">		<span class="keyword">go</span> s.check()</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (s *ClickHouseChecker) check() {</div><div class="line">	<span class="keyword">defer</span> s.wg.Done()</div><div class="line"></div><div class="line">	<span class="keyword">for</span> ip := <span class="keyword">range</span> s.input {</div><div class="line">		<span class="keyword">if</span> ip == <span class="string">"splitting"</span> || ip == <span class="string">"failed"</span> {</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="keyword">if</span> isClickHouse(ip, s.port) {</div><div class="line">			s.output &lt;- ip</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> isClickHouse(ip <span class="typename">string</span>, port <span class="typename">int</span>) <span class="typename">bool</span> {</div><div class="line">	conn, err := clickhouse.Open(&clickhouse.Options{</div><div class="line">		Addr: []<span class="typename">string</span>{fmt.Sprintf(<span class="string">"%s:%d"</span>, ip, port)},</div><div class="line">		<span class="comment">// Auth: clickhouse.Auth{</span></div><div class="line">		<span class="comment">// 	Database: "default",</span></div><div class="line">		<span class="comment">// 	Username: "default",</span></div><div class="line">		<span class="comment">// 	Password: "",</span></div><div class="line">		<span class="comment">// },</span></div><div class="line">		Settings: clickhouse.Settings{</div><div class="line">			<span class="string">"max_execution_time"</span>:<span class="number"> 1</span>,</div><div class="line">		},</div><div class="line">		DialTimeout:          time.Second,</div><div class="line">		MaxOpenConns:        <span class="number"> 1</span>,</div><div class="line">		MaxIdleConns:        <span class="number"> 1</span>,</div><div class="line">		ConnMaxLifetime:      time.Duration<span class="number">(1</span>) * time.Minute,</div><div class="line">		ConnOpenStrategy:     clickhouse.ConnOpenInOrder,</div><div class="line">		BlockBufferSize:     <span class="number"> 10</span>,</div><div class="line">		MaxCompressionBuffer:<span class="number"> 1024</span>,</div><div class="line">	})</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		golog.Errorf(<span class="string">"open %s:%d failed: %v"</span>, ip, port, err)</div><div class="line">		<span class="keyword">return</span> <span class="constant">false</span></div><div class="line">	}</div><div class="line"></div><div class="line">	ctx, cancel := context.WithTimeout(context.Background(), time.Second)</div><div class="line">	<span class="keyword">defer</span> cancel()</div><div class="line">	err = conn.Ping(ctx)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		golog.Warnf(<span class="string">"ping %s:%d failed: %v"</span>, ip, port, err)</div><div class="line">		<span class="keyword">return</span> <span class="constant">false</span></div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">return</span> <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>实际扫描下来，几乎所有的IP的9000端口都连接超时或者不是clickhouse服务，只有4个IP是clickhouse服务，但是需要身份验证。，报错<code>default: Authentication failed: password is incorrect, or there is no user with such name.</code></p>
<p>挺好的一件事情，至少公网暴露的clickhouse服务都是需要身份验证的。</p>
<p>当然也有可能是clickhouse的服务端配置了IP白名单，只允许内网访问，这样的话我们就无法访问了。也可能是clickhouse的端口改成了其他端口，我们无法访问。</p>
<h2 id="有必要扫描一下全网的IP和它们的9000端口了">有必要扫描一下全网的IP和它们的9000端口了</h2>
<p>使用既有的程序即可。我们先拉取全网的网段信息。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">wget -c -O- http://ftp.apnic.net/stats/apnic/delegated-apnic-latest | awk -F <span class="string">'|'</span> <span class="string">'/ipv4/ {print $4 "/" 32-log($5)/log(2)}'</span> | cat &gt; ipv4.txt</div></pre></td></tr></table></figure>

<p>先用<code>icmp_scan</code>扫描一下公网课访问的IP地址：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">......</div><div class="line"></div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">250.221</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.1</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">240.91</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.10</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.15</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.11</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.115</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> <span class="number">223.255</span>.<span class="number">233.100</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> send goroutine <span class="keyword">exit</span></div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">03</span>:<span class="number">56</span> total: <span class="number">884686592</span>, alive: <span class="number">15500888</span>, time: <span class="number">2</span>h35m28.<span class="number">930123788</span>s</div></pre></td></tr></table></figure>

<p>一共8亿多个IP，可以ping的通的有1500多万个，耗时2小时扫描完。</p>
<blockquote>
<p>根据网友在上一篇的留言反馈，光美国就有8亿多个IP。<br>我问deepseek,全球有37亿个IP，美国有9亿个，这个数量才合理，我自己扫描的8亿要远远少于这个数量。而且活跃的IP我感觉应该远远大于1500多万。<br>但是这些不重要了，我要做的就是能扫描到可以免密登录的clickhouse服务,看看这些IP里面有没有。</p>
</blockquote>
<p>接下来我们使用<code>tcp_scan</code>扫描这些IP的9000端口：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line">......</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">222.126</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">219.60</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.220</span>.<span class="number">171.218</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.221</span>.<span class="number">238.176</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">235.26</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">225.240</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">225.208</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> <span class="number">223.197</span>.<span class="number">219.139</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> send goroutine <span class="keyword">exit</span></div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">08</span>:<span class="number">47</span> total: <span class="number">15500890</span>, alive: <span class="number">3953</span>, time: <span class="number">2</span>m41.<span class="number">23585658</span>s</div></pre></td></tr></table></figure>

<p>在这1500多万个IP中，有3953个IP的9000端口是可以访问的，但是都需要验证能不能进行clickhouse的操作，我们需要进一步检查。</p>
<p>接下来我们使用<code>clickhouse_check</code>检查这些IP是否是clickhouse服务：</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line">......</div><div class="line"></div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">222.126</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">53494</span>-&gt;<span class="number">223.197</span>.<span class="number">222.126</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">219.60</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">49718</span>-&gt;<span class="number">223.197</span>.<span class="number">219.60</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.221</span>.<span class="number">238.176</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">56662</span>-&gt;<span class="number">223.221</span>.<span class="number">238.176</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">235.26</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">47676</span>-&gt;<span class="number">223.197</span>.<span class="number">235.26</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping send:<span class="number">9000</span> failed: dial tcp: lookup send on <span class="number">127.0</span>.<span class="number">0.53</span>:<span class="number">53</span>: server misbehaving</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping total::<span class="number">9000</span> failed: dial tcp: address total::<span class="number">9000</span>: too many colons <span class="keyword">in</span> address</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">225.240</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">55342</span>-&gt;<span class="number">223.197</span>.<span class="number">225.240</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">225.208</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">43300</span>-&gt;<span class="number">223.197</span>.<span class="number">225.208</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[WARN] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> ping <span class="number">223.197</span>.<span class="number">219.139</span>:<span class="number">9000</span> failed: <span class="built_in">read</span>: <span class="built_in">read</span> tcp <span class="number">192.168</span>.<span class="number">1.5</span>:<span class="number">57552</span>-&gt;<span class="number">223.197</span>.<span class="number">219.139</span>:<span class="number">9000</span>: i/o timeout</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">31</span> <span class="number">11</span>:<span class="number">47</span> total: <span class="number">2</span>, time: <span class="number">4</span>m20.<span class="number">744235925</span>s</div></pre></td></tr></table></figure>

<p>4分钟完成。最终还是真的发现有两个IP的9000端口是clickhouse服务，而且不需要密码验证。</p>
<p>类似的我们还可以验证Redis、Mysql等服务的安全性。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>DeepSeek出街老火了，整个AI界都在热火朝天的讨论它。</p>
<p>同时，安全界也没闲着，来自美国的攻击使它不得不通知中国大陆以外的手机号的注册，同时大家也对它的网站和服务安全性进行了审视，这不Wiz Research就发现它们的数据库面向公网暴露并且无需任何身份即可访问。这两个域名oauth2callback.deepseek.com:9000和dev.deepseek.com:9000。</p>
<p>AI的核心技术既需要这些清北的天才去研究，产品也需要专业的人才去打磨。像DeepSeek这么专业的公司都可能出现这样的漏洞，相信互联网上这么数据库无密码暴露的实例也应该不在少数(实际只找到了2个)。</p>
<p>基于上一篇《扫描全国的公网IP要多久》，我们改造一下代码，让它使用 <code>tcp_syn</code> 的方式探测clickhopuse的9000端口。</p>
<p>首先声明，所有的技术都是为了给大家介绍使用Go语言开发底层的网络程序所做的演示，不是为了介绍安全和攻击方面的内容，所以也不会使用已经成熟的端口和IP扫描工具如zmap、rustscan、nmap、masscan、Advanced IP Scanner、Angry IP Scanner、unicornscan等工具。</p>
<p>同时，也不会追求快速，我仅仅在家中的100M的网络中，使用一台10多年前的4核Linux机器进行测试，尽可能让它能出结果。我一般晚上启动它，早上吃过早餐后来查看结果。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[趁着假期， 快速了解 Go io/fs 包]]></title>
    <link href="https://colobu.com/2025/01/30/some-notes-about-go-io-fs-package/"/>
    <id>https://colobu.com/2025/01/30/some-notes-about-go-io-fs-package/</id>
    <published>2025-01-29T16:44:01.000Z</published>
    <updated>2025-01-29T17:30:00.416Z</updated>
    <content type="html"><![CDATA[<p>Go 语言的 <code>io/fs</code> 包是 Go 1.16 版本引入的一个标准库包，它定义了文件系统的抽象接口。这个包提供了一种统一的方式来访问<strong>不同类型的文件系统</strong>，包括本地文件系统、内存文件系统、zip 文件等。</p>
<a id="more"></a>
<h3 id="io/fs_包的主要作用"><code>io/fs</code> 包的主要作用</h3>
<ul>
<li><strong>抽象文件系统：</strong> <code>io/fs</code> 包定义了一组接口，用于描述文件系统的基本操作，如打开文件、读取目录等。通过这些接口，我们可以编写与具体文件系统无关的代码。</li>
<li><strong>统一访问方式：</strong> 无论底层文件系统是什么类型，只要实现了 <code>io/fs</code> 包定义的接口，就可以使用相同的代码进行访问。</li>
<li><strong>提高代码可测试性：</strong> 通过使用 <code>io/fs</code> 包，我们可以方便地mock文件系统，从而提高代码的可测试性。</li>
</ul>
<h3 id="io/fs_包的核心接口"><code>io/fs</code> 包的核心接口</h3>
<ul>
<li><strong><code>fs.FS</code>：</strong> 表示一个文件系统，定义了打开文件的方法 <code>Open</code>。</li>
<li><strong><code>fs.File</code>：</strong> 表示一个打开的文件，定义了读取、写入、关闭等方法。</li>
<li><strong><code>fs.FileInfo</code>：</strong> 表示文件的元信息，包括文件名、大小、修改时间等。</li>
<li><code>fs.DirEntry</code> 接口表示一个目录项，它可以是文件或子目录。</li>
<li><code>fs.FileInfo</code> 接口表示文件的元信息。</li>
<li><code>fs.FileMode</code> 类型表示文件的权限和类型，它是一个位掩码。</li>
</ul>
<p>还有一些基于<code>fs.FS</code>、<code>fs.File</code>等接口扩展的一些接口：</p>
<ul>
<li><code>fs.GlobFS</code> 接口扩展了 <code>fs.FS</code> 接口，增加了 <code>Glob(pattern string) ([]string, error)</code> 方法。该方法允许使用通配符模式匹配文件和目录。</li>
<li><code>fs.ReadDirFS</code> 接口也扩展了 <code>fs.FS</code> 接口，增加了 <code>ReadDir(name string) ([]fs.DirEntry, error)</code> 方法。该方法用于读取指定目录下的所有文件和子目录。</li>
<li><code>fs.ReadDirFile</code> 接口扩展了 <code>fs.File</code> 接口，增加了 <code>ReadDir(n int) ([]fs.DirEntry, error)</code> 方法。这个接口主要用于读取目录文件中的内容，返回一个 <code>fs.DirEntry</code> 列表。它通常用于实现了 <code>fs.ReadDirFS</code> 的文件系统。</li>
<li><code>fs.ReadFileFS</code> 接口扩展了 <code>fs.FS</code> 接口，增加了 <code>ReadFile(name string) ([]byte, error)</code> 方法。这个接口允许直接读取指定文件的全部内容，返回字节切片。 它提供了一种更便捷的方式来读取文件内容，避免了先打开文件再读取的步骤。</li>
<li><code>fs.StatFS</code> 接口也扩展了 <code>fs.FS</code> 接口，增加了 <code>Stat(name string) (fs.FileInfo, error)</code> 方法。该方法用于获取指定文件的元信息，返回一个 <code>fs.FileInfo</code> 对象。</li>
<li><code>fs.SubFS</code> 接口也扩展了 <code>fs.FS</code> 接口，增加了 <code>Sub(dir string) (fs.FS, error)</code> 方法。该方法用于创建一个新的文件系统，它表示原始文件系统的一个子目录。这在需要限制访问文件系统的特定部分时非常有用。</li>
<li><code>fs.WalkDirFunc</code> 类型定义了一个函数签名，用于 <code>fs.WalkDir</code> 函数的回调。</li>
</ul>
<h3 id="io/fs_包的应用场景"><code>io/fs</code> 包的应用场景</h3>
<ul>
<li><strong>访问不同类型的文件系统：</strong> 可以使用相同的代码访问本地文件系统、内存文件系统、zip 文件等。</li>
<li><strong>测试代码：</strong> 可以方便地mock文件系统，从而提高代码的可测试性。</li>
<li><strong>嵌入资源：</strong> 可以将静态资源嵌入到程序中，并使用 <code>io/fs</code> 包进行访问。</li>
</ul>
<h3 id="示例代码">示例代码</h3>
<h4 id="示例代码一：fs-FS_接口">示例代码一：<code>fs.FS</code> 接口</h4>
<p><code>fs.FS</code> 接口是 <code>io/fs</code> 包的核心，它表示一个文件系统。最常见的实现是 <code>os.DirFS</code>，它表示本地文件系统的一个目录。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	<span class="comment">// 创建一个表示当前目录的文件系统</span></div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	<span class="comment">// 打开一个文件</span></div><div class="line">	f, err := fsys.Open(<span class="string">"README.md"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> f.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 读取文件内容</span></div><div class="line">	data := <span class="built_in">make</span>([]<span class="typename">byte</span>,<span class="number"> 100</span>)</div><div class="line">	n, err := f.Read(data)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	fmt.Println(<span class="typename">string</span>(data[:n]))</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 <code>os.DirFS</code> 创建一个文件系统，然后使用 <code>fsys.Open</code> 方法打开一个文件并读取其内容。</p>
<h4 id="示例代码二：fs-File_接口">示例代码二：<code>fs.File</code> 接口</h4>
<p><code>fs.File</code> 接口表示一个打开的文件，它提供了读取、写入、关闭等方法。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	f, err := fsys.Open(<span class="string">"README.md"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> f.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 获取文件信息</span></div><div class="line">	info, err := f.Stat()</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	fmt.Println(<span class="string">"File size:"</span>, info.Size())</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 f.Stat 方法获取文件的元信息。</p>
<h4 id="示例代码三：fs-DirEntry_接口">示例代码三：<code>fs.DirEntry</code> 接口</h4>
<p><code>fs.DirEntry</code> 接口表示一个目录项，它可以是文件或子目录。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	entries, err := fs.ReadDir(fsys, <span class="string">"."</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">for</span> _, entry := <span class="keyword">range</span> entries {</div><div class="line">		fmt.Println(<span class="string">"Name:"</span>, entry.Name())</div><div class="line">		fmt.Println(<span class="string">"Is directory:"</span>, entry.IsDir())</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 <code>fs.ReadDir</code> 函数读取目录中的所有条目，并使用 <code>entry.Name</code> 和 <code>entry.IsDir</code> 方法获取条目的名称和类型。</p>
<h4 id="示例代码四：fs-GlobFS_接口">示例代码四：<code>fs.GlobFS</code> 接口</h4>
<p><code>fs.GlobFS</code> 接口扩展了 <code>fs.FS</code> 接口，增加了 <code>Glob</code> 方法，允许使用通配符模式匹配文件和目录。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	<span class="keyword">if</span> globFS, ok := fsys.(fs.GlobFS); ok {</div><div class="line">		matches, err := globFS.Glob(<span class="string">"*.go"</span>)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			log.Fatal(err)</div><div class="line">		}</div><div class="line"></div><div class="line">		fmt.Println(<span class="string">"Go files:"</span>, matches)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 fs.Glob 函数查找所有以 .go 结尾的文件。</p>
<h4 id="示例代码五：fs-ReadDirFS_接口">示例代码五：<code>fs.ReadDirFS</code> 接口</h4>
<p><code>fs.ReadDirFS</code> 接口也扩展了 <code>fs.FS</code> 接口，增加了 <code>ReadDir</code> 方法，用于读取指定目录下的所有文件和子目录。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	<span class="keyword">if</span> readDirFS, ok := fsys.(fs.ReadDirFS); ok {</div><div class="line">		entries, err := readDirFS.ReadDir(<span class="string">"."</span>)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			log.Fatal(err)</div><div class="line">		}</div><div class="line"></div><div class="line">		fmt.Println(<span class="string">"Directory contents:"</span>)</div><div class="line">		<span class="keyword">for</span> _, entry := <span class="keyword">range</span> entries {</div><div class="line">			fmt.Println(entry.Name())</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 <code>fs.ReadDir</code> 函数读取目录中的所有条目。</p>
<h4 id="示例代码六：fs-SubFS_接口">示例代码六：<code>fs.SubFS</code> 接口</h4>
<p><code>fs.SubFS</code> 接口也扩展了 <code>fs.FS</code> 接口，增加了 <code>Sub</code> 方法，用于创建一个新的文件系统，它表示原始文件系统的一个子目录。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	<span class="keyword">if</span> subFS, ok := fsys.(fs.SubFS); ok {</div><div class="line">		sub, err := subFS.Sub(<span class="string">"subdir"</span>)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			log.Fatal(err)</div><div class="line">		}</div><div class="line"></div><div class="line">		fmt.Println(<span class="string">"Sub directory contents:"</span>)</div><div class="line">		entries, err := fs.ReadDir(sub, <span class="string">"."</span>)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			log.Fatal(err)</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="keyword">for</span> _, entry := <span class="keyword">range</span> entries {</div><div class="line">			fmt.Println(entry.Name())</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 <code>fs.Sub</code> 函数创建一个表示子目录的文件系统，并读取其内容。</p>
<h4 id="示例代码七：fs-WalkDirFunc_接口">示例代码七：<code>fs.WalkDirFunc</code> 接口</h4>
<p><code>fs.WalkDirFunc</code> 类型定义了一个函数签名，用于 <code>fs.WalkDir</code> 函数的回调。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	fsys := os.DirFS(<span class="string">"."</span>)</div><div class="line"></div><div class="line">	err := fs.WalkDir(fsys, <span class="string">"."</span>, <span class="keyword">func</span>(path <span class="typename">string</span>, d fs.DirEntry, err error) error {</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			<span class="keyword">return</span> err</div><div class="line">		}</div><div class="line"></div><div class="line">		fmt.Println(<span class="string">"Walking:"</span>, path)</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	})</div><div class="line"></div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 fs.WalkDir 函数遍历目录，并使用 fs.WalkDirFunc 函数打印每个文件和目录的路径。</p>
<h3 id="那些有趣的文件系统">那些有趣的文件系统</h3>
<h4 id="内存文件系统">内存文件系统</h4>
<p>内存文件系统是一种虚拟文件系统，它将文件存储在内存中而不是磁盘上。内存文件系统通常用于临时存储数据，或者用于测试和调试目的。<br>这种文件系统速度非常快，但数据在程序退出后会丢失。Go 语言的 <code>testing/fstest</code> 包提供了一个 <code>MapFS</code> 包，可以方便地创建内存文件系统。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">	<span class="string">"os"</span></div><div class="line">	<span class="string">"testing/fstest"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	<span class="comment">// 创建一个内存文件系统</span></div><div class="line">	fsys := fstest.MapFS{</div><div class="line">		<span class="string">"file1.txt"</span>: {Data: []<span class="typename">byte</span>(<span class="string">"Hello, world!"</span>)},</div><div class="line">		<span class="string">"dir1/file2.txt"</span>: {Data: []<span class="typename">byte</span>(<span class="string">"This is file2."</span>)},</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="comment">// 打开一个文件</span></div><div class="line">	f, err := fsys.Open(<span class="string">"file1.txt"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> f.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 读取文件内容</span></div><div class="line">	data := <span class="built_in">make</span>([]<span class="typename">byte</span>,<span class="number"> 100</span>)</div><div class="line">	n, err := f.Read(data)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	fmt.Println(<span class="typename">string</span>(data[:n]))</div><div class="line"></div><div class="line">    <span class="comment">// 遍历文件系统</span></div><div class="line">    err = fs.WalkDir(fsys, <span class="string">"."</span>, <span class="keyword">func</span>(path <span class="typename">string</span>, d fs.DirEntry, err error) error {</div><div class="line">        <span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">            <span class="keyword">return</span> err</div><div class="line">        }</div><div class="line">        fmt.Println(<span class="string">"Walking:"</span>, path)</div><div class="line">        <span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">    })</div><div class="line">    <span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">        log.Fatal(err)</div><div class="line">    }</div><div class="line"></div><div class="line">}</div></pre></td></tr></table></figure>

<p>也有一些第三方的库实现了内存文件系统，比如<a href="https://github.com/psanford/memfs" target="_blank" rel="external">psanford/memfs</a>,这是一个它的例子：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line"></div><div class="line">	<span class="string">"github.com/psanford/memfs"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	rootFS := memfs.New()</div><div class="line"></div><div class="line">	err := rootFS.MkdirAll(<span class="string">"dir1/dir2"</span>,<span class="number"> 0777</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		<span class="built_in">panic</span>(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	err = rootFS.WriteFile(<span class="string">"dir1/dir2/f1.txt"</span>, []<span class="typename">byte</span>(<span class="string">"incinerating-unsubstantial"</span>),<span class="number"> 0755</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		<span class="built_in">panic</span>(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	err = fs.WalkDir(rootFS, <span class="string">"."</span>, <span class="keyword">func</span>(path <span class="typename">string</span>, d fs.DirEntry, err error) error {</div><div class="line">		fmt.Println(path)</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	})</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		<span class="built_in">panic</span>(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	content, err := fs.ReadFile(rootFS, <span class="string">"dir1/dir2/f1.txt"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		<span class="built_in">panic</span>(err)</div><div class="line">	}</div><div class="line">	fmt.Printf(<span class="string">"%s\n"</span>, content)</div><div class="line">}</div></pre></td></tr></table></figure>

<h4 id="嵌入式文件系统">嵌入式文件系统</h4>
<p>嵌入式文件系统将文件嵌入到程序中，这样可以方便地将静态资源打包到程序中。Go 语言标准库提供了一个 <code>embed</code> 包，可以方便地创建嵌入式文件系统。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"embed"</span></div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">//go:embed static</span></div><div class="line"><span class="keyword">var</span> staticFiles embed.FS</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	<span class="comment">// 打开一个嵌入的文件</span></div><div class="line">	f, err := staticFiles.Open(<span class="string">"static/file1.txt"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> f.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 读取文件内容</span></div><div class="line">	data := <span class="built_in">make</span>([]<span class="typename">byte</span>,<span class="number"> 100</span>)</div><div class="line">	n, err := f.Read(data)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line"></div><div class="line">	fmt.Println(<span class="typename">string</span>(data[:n]))</div><div class="line"></div><div class="line"></div><div class="line">    <span class="comment">// 遍历嵌入式文件系统</span></div><div class="line">    err = fs.WalkDir(staticFiles, <span class="string">"static"</span>, <span class="keyword">func</span>(path <span class="typename">string</span>, d fs.DirEntry, err error) error {</div><div class="line">        <span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">            <span class="keyword">return</span> err</div><div class="line">        }</div><div class="line">        fmt.Println(<span class="string">"Walking:"</span>, path)</div><div class="line">        <span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">    })</div><div class="line">    <span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">        log.Fatal(err)</div><div class="line">    }</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>这个例子展示了如何使用</code>embed.FS<code>类型创建一个嵌入式文件系统，并使用</code>staticFiles.Open` 方法打开一个嵌入的文件。</p>
<h4 id="云存储文件系统">云存储文件系统</h4>
<p>有一些第三方库提供了将 S3 存储桶挂载为本地文件系统的功能，这样我们就可以像访问本地文件一样访问 S3 文件。例如，<code>go-cloud</code> 库就提供了对多种云存储服务的统一访问接口，包括 S3。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"context"</span></div><div class="line">	<span class="string">"fmt"</span></div><div class="line">	<span class="string">"io/fs"</span></div><div class="line">	<span class="string">"log"</span></div><div class="line"></div><div class="line">	<span class="string">"gocloud.dev/blob"</span></div><div class="line">	_ <span class="string">"gocloud.dev/blob/gcs"</span> <span class="comment">// 引入 GCS 驱动，如果使用其他云存储服务，请引入相应的驱动</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	<span class="comment">// 设置 S3 存储桶 URL</span></div><div class="line">	bucketURL := <span class="string">"gs://my-bucket"</span></div><div class="line"></div><div class="line">	<span class="comment">// 创建一个 blob.Bucket</span></div><div class="line">	bucket, err := blob.OpenBucket(context.Background(), bucketURL)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> bucket.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 创建一个 fs.FS</span></div><div class="line">	fsys := blob.NewFS(bucket)</div><div class="line"></div><div class="line">	<span class="comment">// 现在可以使用 fsys 进行文件操作</span></div><div class="line">	err = fs.WalkDir(fsys, <span class="string">"."</span>, <span class="keyword">func</span>(path <span class="typename">string</span>, d fs.DirEntry, err error) error {</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			<span class="keyword">return</span> err</div><div class="line">		}</div><div class="line">		fmt.Println(<span class="string">"Walking:"</span>, path)</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	})</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这个例子展示了如何使用 <code>gocloud.dev/blob</code> 包将 google GCS 存储桶挂载为本地文件系统，并使用 <code>fs.WalkDir</code> 函数遍历存储桶中的文件。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>Go 语言的 <code>io/fs</code> 包是 Go 1.16 版本引入的一个标准库包，它定义了文件系统的抽象接口。这个包提供了一种统一的方式来访问<strong>不同类型的文件系统</strong>，包括本地文件系统、内存文件系统、zip 文件等。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/tags/Go/"/>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[扫描全国的公网IP需要多久？]]></title>
    <link href="https://colobu.com/2025/01/27/how-long-to-scan-all-IPs-of-cn/"/>
    <id>https://colobu.com/2025/01/27/how-long-to-scan-all-IPs-of-cn/</id>
    <published>2025-01-26T16:38:47.000Z</published>
    <updated>2025-01-26T16:42:06.652Z</updated>
    <content type="html"><![CDATA[<p>自从加入百度负责物理网络的监控业务之后，我大部分的都是编写各种各样额度底层的网络程序。业余时间我也是编写一些有趣的网络程序，不仅仅是兴趣，也是为未来的某个业务探索一下技术方案。</p>
<p>而且这次，我想知道，就在我这一个10年前的小mini机器4核机器上，在家庭网络中扫描全国(中国大陆)的所有的公网IP地址需要多少时间。</p>
<p>利用它，我可以知道和全国各省市的运营商、云服务商的联通情况。有没有运营商的出口故障以及IP已没有被运营商或者有关部门劫持。</p>
<p>TL;DR: 一共扫描了<strong>3亿</strong>个地址(343142912)，当前ping的通的IP <strong>592万</strong>个(5923768)，耗时<strong>1小时</strong>(1h2m57.973755197s)。</p>
<p>这次我重构了以前的一个扫描公网IP的程序。先前的程序使用gopacket收发包，也使用gopacket组装包。但是gopacket很讨厌的的一个地方是它依赖libpcap库，没有办法在禁用CGO的情况下。</p>
<p>事实上利用Go的扩展包icmp和ipv4,我们完全可以不使用gopacket实现这个功能，本文介绍具体的实现。</p>
<p>程序的全部代码在：<a href="https://github.com/smallnest/fishfinder" target="_blank" rel="external">https://github.com/smallnest/fishfinder</a></p>
<a id="more"></a>
<h2 id="程序的主要架构">程序的主要架构</h2>
<p>程序使用ICMP协议进行探测。</p>
<p>首先它启动一个goroutine解析全国的IP地址。IP地址文件每一行都是一个网段，它对每一个网段解析成一组IP地址，把这组IP地址扔进input channel。</p>
<p>一个发送goroutine从input通道中接收IP地址，然后组装成ICMP echo包发送给每一个IP地址，它只负责发送，发送完所有的地址就返回。</p>
<p>一个接收goroutine处理接收到的ICMP reply 回包，并将结果写入到output channel中。</p>
<p>主程序不断的从output中接收已经有回包的IP并打印到日志中，直到所有的IP都处理完就退出。</p>
<p><img src="flow.png" alt=""></p>
<p>这里涉及到并发编程的问题，几个goroutine怎么协调：</p>
<ul>
<li>IP解析和任务分发goroutine和发送goroutine通过input通讯。分发goroutine处理完所有的IP后，就会关闭input通知发送goroutine。</li>
<li>发送goroutine得知input关闭，就知道已经处理完所有的IP,发送完最后的IP后把output关闭。</li>
<li>接收goroutine往output发送接收到回包的IP, 如果output关闭，再往output发送就会panic,程序中捕获了panic。不过还没到这一步主程序应该就退出了。</li>
<li>主程序从output读取IP, 一旦output关闭，主程序就打印统计信息后推出。</li>
</ul>
<blockquote>
<p>如果你对Go并发编程有疑问，可以阅读极客时间上的《Go并发编程实战课》专栏，或者图书《深入理解Go并发编程》。<br>如果你是Rust程序员，不就我会推出《Go并发编程实战课》姊妹专栏，专门介绍Rust并发编程。<br>如果你对网络编程感兴趣，今年我还想推出《深入理解网络编程》的专栏或者图书，如果你感兴趣，欢迎和我探讨。</p>
</blockquote>
<p>主程序的代码如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"flag"</span></div><div class="line">	<span class="string">"time"</span></div><div class="line"></div><div class="line">	<span class="string">"github.com/kataras/golog"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">var</span> (</div><div class="line">	protocol = flag.String(<span class="string">"p"</span>, <span class="string">"icmp"</span>, <span class="string">"The protocol to use (icmp, tcp or udp)"</span>)</div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">// 嵌入ip.sh</span></div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">	flag.Parse()</div><div class="line"></div><div class="line">	input := <span class="built_in">make</span>(<span class="keyword">chan</span> []<span class="typename">string</span>,<span class="number"> 1024</span>)</div><div class="line">	output := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="typename">string</span>,<span class="number"> 1024</span>)</div><div class="line">	scanner := NewICMPScanner(input, output)</div><div class="line"></div><div class="line">	<span class="keyword">var</span> total <span class="typename">int</span></div><div class="line">	<span class="keyword">var</span> alive <span class="typename">int</span></div><div class="line"></div><div class="line">	golog.Infof(<span class="string">"start scanning"</span>)</div><div class="line"></div><div class="line">	start := time.Now()</div><div class="line">	<span class="comment">// 将待探测的IP发送给send goroutine</span></div><div class="line">	<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">		lines := readIPList()</div><div class="line">		<span class="keyword">for</span> _, line := <span class="keyword">range</span> lines {</div><div class="line">			ips := cidr2IPList(line)</div><div class="line">			input &lt;- ips</div><div class="line">			total += <span class="built_in">len</span>(ips)</div><div class="line">		}</div><div class="line">		<span class="built_in">close</span>(input)</div><div class="line">	}()</div><div class="line"></div><div class="line">	<span class="comment">// 启动 send goroutine</span></div><div class="line">	scanner.Scan()</div><div class="line"></div><div class="line">	<span class="comment">// 接收 send goroutine 发送的结果, 直到发送之后5秒结束</span></div><div class="line">	<span class="keyword">for</span> ip := <span class="keyword">range</span> output {</div><div class="line">		golog.Infof(<span class="string">"%s is alive"</span>, ip)</div><div class="line">		alive++</div><div class="line">	}</div><div class="line"></div><div class="line">	golog.Infof(<span class="string">"total: %d, alive: %d, time: %v"</span>, total, alive, time.Since(start))</div><div class="line">}</div></pre></td></tr></table></figure>

<p>接下来介绍三个三个主要goroutine的逻辑。</p>
<h2 id="公网IP获取以及任务分发">公网IP获取以及任务分发</h2>
<p>首先你需要到互联网管理中心下载中国大陆所有的注册的IP网段，这是从亚太互联网络信息中心下载的公网IP信息，实际上可以探测全球的IP,这里以中国大陆的公网IP为例。</p>
<p>通过下面的代码转换成网段信息：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="shebang">#!/bin/bash</span></div><div class="line"></div><div class="line">wget -c -O- http://ftp.apnic.net/stats/apnic/delegated-apnic-latest | awk -F <span class="string">'|'</span> <span class="string">'/CN/&&/ipv4/ {print $4 "/" 32-log($5)/log(2)}'</span> | cat &gt; ipv4.txt</div></pre></td></tr></table></figure>

<p>ipv4.txt文件中是一行行的网段：</p>
<figure class="highlight txt"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="number">1.0</span><span class="number">.1</span><span class="number">.0</span>/<span class="number">24</span></div><div class="line"><span class="number">1.0</span><span class="number">.2</span><span class="number">.0</span>/<span class="number">23</span></div><div class="line"><span class="number">1.0</span><span class="number">.8</span><span class="number">.0</span>/<span class="number">21</span></div><div class="line"><span class="number">1.0</span><span class="number">.32</span><span class="number">.0</span>/<span class="number">19</span></div><div class="line"><span class="number">1.1</span><span class="number">.0</span><span class="number">.0</span>/<span class="number">24</span></div><div class="line"><span class="number">1.1</span><span class="number">.2</span><span class="number">.0</span>/<span class="number">23</span></div><div class="line"><span class="number">1.1</span><span class="number">.4</span><span class="number">.0</span>/<span class="number">22</span></div><div class="line"><span class="keyword">...</span></div></pre></td></tr></table></figure>

<p>数据量不大，我们全读取进来(如果太多的话我们就流式读取了)。<br>解析每一行的网段，转换成IP地址列表，然后发送给input通道。<br>等处理完就把inpout通道关闭。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">	lines := readIPList()</div><div class="line">	<span class="keyword">for</span> _, line := <span class="keyword">range</span> lines {</div><div class="line">		ips := cidr2IPList(line)</div><div class="line">		input &lt;- ips</div><div class="line">		total += <span class="built_in">len</span>(ips)</div><div class="line">	}</div><div class="line">	<span class="built_in">close</span>(input)</div><div class="line">}()</div></pre></td></tr></table></figure>

<h2 id="发送逻辑">发送逻辑</h2>
<p>我使用了<code>ICMPScanner</code>结构体来管理发送和接收的逻辑。看名字你也可以猜测到我们将来还可以使用TCP/UDP等协议进行探测。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> ICMPScanner <span class="keyword">struct</span> {</div><div class="line">	src net.IP</div><div class="line"></div><div class="line">	input  <span class="keyword">chan</span> []<span class="typename">string</span></div><div class="line">	output <span class="keyword">chan</span> <span class="typename">string</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// 调大缓存区</span></div><div class="line"><span class="comment">// sysctl net.core.rmem_max</span></div><div class="line"><span class="comment">// sysctl net.core.wmem_max</span></div><div class="line"></div><div class="line"><span class="keyword">func</span> NewICMPScanner(input <span class="keyword">chan</span> []<span class="typename">string</span>, output <span class="keyword">chan</span> <span class="typename">string</span>) *ICMPScanner {</div><div class="line">	localIP := getLocalIP()</div><div class="line">	s := &ICMPScanner{</div><div class="line">		input:  input,</div><div class="line">		output: output,</div><div class="line">		src:    net.ParseIP(localIP),</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> s</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (s *ICMPScanner) Scan() {</div><div class="line">	<span class="keyword">go</span> s.recv()</div><div class="line">	<span class="keyword">go</span> s.send(s.input)</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>send</code>方法负责发送ICMP包，<code>recv</code>方法负责接收ICMP包。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// send sends a single ICMP echo request packet for each ip in the input channel.</span></div><div class="line"><span class="keyword">func</span> (s *ICMPScanner) send(input <span class="keyword">chan</span> []<span class="typename">string</span>) error {</div><div class="line">	<span class="keyword">defer</span> <span class="keyword">func</span>() {</div><div class="line">		time.Sleep<span class="number">(5</span> * time.Second)</div><div class="line">		<span class="built_in">close</span>(s.output)</div><div class="line">		golog.Infof(<span class="string">"send goroutine exit"</span>)</div><div class="line">	}()</div><div class="line"></div><div class="line">	id := os.Getpid() &<span class="number"> 0</span>xffff</div><div class="line"></div><div class="line">	<span class="comment">// 创建 ICMP 连接</span></div><div class="line">	conn, err := icmp.ListenPacket(<span class="string">"ip4:icmp"</span>, s.src.String())</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> conn.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 不负责接收数据</span></div><div class="line">	filter := createEmptyFilter()</div><div class="line">	<span class="keyword">if</span> assembled, err := bpf.Assemble(filter); err == <span class="constant">nil</span> {</div><div class="line">		conn.IPv4PacketConn().SetBPF(assembled)</div><div class="line">	}</div><div class="line"></div><div class="line">    ... <span class="comment">// 先忽略，后面再介绍</span></div><div class="line"></div><div class="line">	<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>send</code>方法中，我们首先创建一个ICMP连接，我通过icmp包创建了一个连接，然后设置了一个BPF过滤器，过滤掉我们不关心的包。<br>这是一个技巧，这个连接我们不关心接收到的包，只关心发送的包，所以我们设置了一个空的过滤器。</p>
<p>这个设计本来是为了将来的性能扩展做准备，可以创建多个连接用来更快的发送。不过目前我们只使用一个连接，所以这个连接其实可以和接收goroutine共享，目前的设计还是发送和接收使用各自的连接。</p>
<p>接下来就是发送的逻辑了，也就是上面省略的部分：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line">seq := <span class="typename">uint16</span><span class="number">(0</span>)</div><div class="line"><span class="keyword">for</span> ips := <span class="keyword">range</span> input {</div><div class="line">	<span class="keyword">for</span> _, ip := <span class="keyword">range</span> ips {</div><div class="line">		dst, err := net.ResolveIPAddr(<span class="string">"ip"</span>, ip)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Fatalf(<span class="string">"failed to resolve IP address: %v"</span>, err)</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 构造 ICMP 报文</span></div><div class="line">		msg := &icmp.Message{</div><div class="line">			Type: ipv4.ICMPTypeEcho,</div><div class="line">			Code:<span class="number"> 0</span>,</div><div class="line">			Body: &icmp.Echo{</div><div class="line">				ID:   id,</div><div class="line">				Seq:  <span class="typename">int</span>(seq),</div><div class="line">				Data: []<span class="typename">byte</span>(<span class="string">"Hello, are you there!"</span>),</div><div class="line">			},</div><div class="line">		}</div><div class="line">		msgBytes, err := msg.Marshal(<span class="constant">nil</span>)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to marshal ICMP message: %v"</span>, err)</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 发送 ICMP 报文</span></div><div class="line">		_, err = conn.WriteTo(msgBytes, dst)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to send ICMP message: %v"</span>, err)</div><div class="line">		}</div><div class="line">		seq++</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>发送循环从input通道中读取IP地址，然后构造ICMP echo报文，发送到目标地址。</p>
<ul>
<li>从 input channel 读取 IP 列表</li>
<li>对每个 IP 执行以下操作：<ol>
<li>解析 IP 地址</li>
<li>构造 ICMP echo 请求报文</li>
<li>序列化报文</li>
<li>发送到目标地址</li>
</ol>
</li>
</ul>
<p>icmp报文中的ID我们设置为进程的PID，在接收的时候可以用来判断是否是我们发送的回包。</p>
<h2 id="接收逻辑">接收逻辑</h2>
<p>接收逻辑比较简单，我们只需要接收ICMP回包，然后解析出IP地址，然后发送到output通道。</p>
<p>首先我们创建一个ICMP连接，然后循环接收ICMP回包，解析出IP地址，然后发送到output通道。</p>
<p>我们只需处理ICMPTypeEchoReply类型的回包，然后判断ID是否是我们发送的ID，如果是就把对端的IP发送到output通道。</p>
<p>我们通过ID判断回包针对我们的场景就足够了，不用再判断seq甚至payload信息。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (s *ICMPScanner) recv() error {</div><div class="line">	<span class="keyword">defer</span> <span class="built_in">recover</span>()</div><div class="line"></div><div class="line">	id := os.Getpid() &<span class="number"> 0</span>xffff</div><div class="line"></div><div class="line">	<span class="comment">// 创建 ICMP 连接</span></div><div class="line">	conn, err := icmp.ListenPacket(<span class="string">"ip4:icmp"</span>, <span class="string">"0.0.0.0"</span>)</div><div class="line">	<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">		log.Fatal(err)</div><div class="line">	}</div><div class="line">	<span class="keyword">defer</span> conn.Close()</div><div class="line"></div><div class="line">	<span class="comment">// 接收 ICMP 报文</span></div><div class="line">	reply := <span class="built_in">make</span>([]<span class="typename">byte</span>,<span class="number"> 1500</span>)</div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		n, peer, err := conn.ReadFrom(reply)</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			log.Fatal(err)</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 解析 ICMP 报文</span></div><div class="line">		msg, err := icmp.ParseMessage(protocolICMP, reply[:n])</div><div class="line">		<span class="keyword">if</span> err != <span class="constant">nil</span> {</div><div class="line">			golog.Errorf(<span class="string">"failed to parse ICMP message: %v"</span>, err)</div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// 打印结果</span></div><div class="line">		<span class="keyword">switch</span> msg.Type {</div><div class="line">		<span class="keyword">case</span> ipv4.ICMPTypeEchoReply:</div><div class="line">			echoReply, ok := msg.Body.(*icmp.Echo)</div><div class="line">			<span class="keyword">if</span> !ok {</div><div class="line">				<span class="keyword">continue</span></div><div class="line">			}</div><div class="line">			<span class="keyword">if</span> echoReply.ID == id {</div><div class="line">				s.output &lt;- peer.String()</div><div class="line">			}</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>可以看到，200行代码基本就可以我们扫描全国公网IP的程序了。你也可以尝试扫描一下全球的IP地址，看看需要多少时间。</p>
<p>对了，下面是我运行这个程序的输出:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">...</span></div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">26</span> <span class="number">22</span>:<span class="number">01</span> <span class="number">223.255</span><span class="number">.236</span><span class="number">.221</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">26</span> <span class="number">22</span>:<span class="number">01</span> <span class="number">223.255</span><span class="number">.252</span><span class="number">.9</span> is alive</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">26</span> <span class="number">22</span>:<span class="number">01</span> send goroutine exit</div><div class="line">[INFO] <span class="number">2025</span>/<span class="number">01</span>/<span class="number">26</span> <span class="number">22</span>:<span class="number">01</span> total: <span class="number">343142912</span>, alive: <span class="number">5923768</span>, time: 1h2m57.973755197s</div></pre></td></tr></table></figure>

]]></content>
    <summary type="html">
    <![CDATA[<p>自从加入百度负责物理网络的监控业务之后，我大部分的都是编写各种各样额度底层的网络程序。业余时间我也是编写一些有趣的网络程序，不仅仅是兴趣，也是为未来的某个业务探索一下技术方案。</p>
<p>而且这次，我想知道，就在我这一个10年前的小mini机器4核机器上，在家庭网络中扫描全国(中国大陆)的所有的公网IP地址需要多少时间。</p>
<p>利用它，我可以知道和全国各省市的运营商、云服务商的联通情况。有没有运营商的出口故障以及IP已没有被运营商或者有关部门劫持。</p>
<p>TL;DR: 一共扫描了<strong>3亿</strong>个地址(343142912)，当前ping的通的IP <strong>592万</strong>个(5923768)，耗时<strong>1小时</strong>(1h2m57.973755197s)。</p>
<p>这次我重构了以前的一个扫描公网IP的程序。先前的程序使用gopacket收发包，也使用gopacket组装包。但是gopacket很讨厌的的一个地方是它依赖libpcap库，没有办法在禁用CGO的情况下。</p>
<p>事实上利用Go的扩展包icmp和ipv4,我们完全可以不使用gopacket实现这个功能，本文介绍具体的实现。</p>
<p>程序的全部代码在：<a href="https://github.com/smallnest/fishfinder" target="_blank" rel="external">https://github.com/smallnest/fishfinder</a></p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/tags/Go/"/>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构: 四叉堆，不是普通的二叉堆]]></title>
    <link href="https://colobu.com/2024/11/18/go-internal-ds-4-ary-heap/"/>
    <id>https://colobu.com/2024/11/18/go-internal-ds-4-ary-heap/</id>
    <published>2024-11-18T14:47:50.000Z</published>
    <updated>2024-11-18T15:00:08.681Z</updated>
    <content type="html"><![CDATA[<p>Go语言中Timer以及相关的Ticker、time.After、time.AfterFunc 等定时器最终是以四叉堆的数据形式存放的。</p>
<p>全局的 timer 堆也经历过三个阶段的重要升级。</p>
<ul>
<li>Go 1.9 版本之前，所有的计时器由全局唯一的四叉堆维护，goroutine间竞争激烈。</li>
<li>Go 1.10 - 1.13，全局使用 64 个四叉堆维护全部的计时器，通过分片减少了竞争的压力，但是本质上还是没有解决 1.9 版本之前的问题</li>
<li>Go 1.14 版本之后，每个 P 单独维护一个四叉堆，避免了goroutine的竞争。 (后面我们再介绍 per-P 的数据结构)</li>
</ul>
<p>常见的堆(heap)常常以二叉堆的形式实现。可是为什么Go timer使用四叉堆呢？</p>
<a id="more"></a>
<p>以最小堆为例，下图展示了二叉堆和四叉堆的区别：</p>
<p><img src="bheap-vs-d-ary-heap.png" alt=""></p>
<ul>
<li>二叉堆：每个节点最多有2个子节点；四叉堆：每个节点最多有4个子节点</li>
<li>在相同节点数下，四叉堆的高度更低，约为二叉堆的一半（log₄n vs log₂n）</li>
<li>对于最小堆来说， 父节点的值小于等于子节点的值。</li>
</ul>
<p>父节点和子节点的索引计算也略有不同。二叉堆的父子索引如下：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">parent = (i - <span class="number">1</span>) // <span class="number">2</span></div><div class="line">left_child = <span class="number">2</span> * i + <span class="number">1</span></div><div class="line">right_child = <span class="number">2</span> * i + <span class="number">2</span></div></pre></td></tr></table></figure>

<p>四叉堆的父子索引如下：</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">parent = (i - <span class="number">1</span>) // <span class="number">4</span></div><div class="line">first_child = <span class="number">4</span> * i + <span class="number">1</span></div><div class="line">last_child = <span class="number">4</span> * i + <span class="number">4</span></div></pre></td></tr></table></figure>

<p>他们的操作时间复杂度:<br><img src="b-vs-4.png" alt=""></p>
<p>因为四叉树的高度相对更低，所以四叉堆适合数据量特别大，需要减少树的高度的场景， Go的timer很久以前(11年前)就使用四叉树来实现Timer的保存，当然Go开发者也是根据测试结果选择了四叉树，最早的这个提交可以查看: <a href="https://codereview.appspot.com/13094043/#ps1" target="_blank" rel="external">## code review 13094043: time: make timers heap 4-ary (Closed)</a></p>
<p>在Go的运行时中，四叉堆的实现在 <code>src/runtime/time.go</code> 文件中，可以查看源码实现。<code>timers</code>数据结构代表Timer的集合，每个P都有一个timers实例，用于维护当前P的所有Timer。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// A timers is a per-P set of timers.</span></div><div class="line"><span class="keyword">type</span> timers <span class="keyword">struct</span> {</div><div class="line">    <span class="comment">// 互斥锁保护timers; 虽然timers是每个P的，但是调度器可以访问另一个P的timers，所以我们必须锁定。</span></div><div class="line">	mu mutex</div><div class="line"></div><div class="line">    <span class="comment">// heap是一组计时器，按heap[i].when排序。这就是一个四叉堆，虽然没有明确的说明。</span></div><div class="line">    <span class="comment">// 必须持有锁才能访问这个堆。</span></div><div class="line">	heap []timerWhen</div><div class="line"></div><div class="line">    <span class="comment">// len是heap的长度的原子副本。</span></div><div class="line">	<span class="built_in">len</span> atomic.Uint32</div><div class="line"></div><div class="line">    <span class="comment">// zombies是堆中标记为删除的计时器的数量。</span></div><div class="line">	zombies atomic.Int32</div><div class="line"></div><div class="line">	raceCtx <span class="typename">uintptr</span></div><div class="line"></div><div class="line">    <span class="comment">// minWhenHeap是最小的heap[i].when值(= heap[0].when)。</span></div><div class="line">    <span class="comment">// wakeTime方法使用minWhenHeap和minWhenModified来确定下一个唤醒时间。</span></div><div class="line">    <span class="comment">// 如果minWhenHeap = 0，表示堆中没有计时器。</span></div><div class="line">	minWhenHeap atomic.Int64</div><div class="line"></div><div class="line">    <span class="comment">// minWhenModified是具有timerModified位设置的计时器的最小heap[i].when的下界。</span></div><div class="line">    <span class="comment">// 如果minWhenModified = 0，表示堆中没有timerModified计时器。</span></div><div class="line">	minWhenModified atomic.Int64</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> timerWhen <span class="keyword">struct</span> {</div><div class="line">	timer *timer</div><div class="line">	when  <span class="typename">int64</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (ts *timers) lock() {</div><div class="line">	lock(&ts.mu)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> (ts *timers) unlock() {</div><div class="line">	ts.<span class="built_in">len</span>.Store(<span class="typename">uint32</span>(<span class="built_in">len</span>(ts.heap)))</div><div class="line">	unlock(&ts.mu)</div><div class="line">}</div></pre></td></tr></table></figure>


<p>同时<code>Timer</code>结构体还引用了<code>Timers</code>, 这叫你中有我，我中有你，这样的设计是为了方便Timer的管理，Timer的创建、删除、执行都是通过Timers来实现的。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> timer <span class="keyword">struct</span> {</div><div class="line">	mu mutex</div><div class="line">	astate atomic.Uint8 </div><div class="line">	state  <span class="typename">uint8</span>       </div><div class="line">	isChan <span class="typename">bool</span>       </div><div class="line">	blocked <span class="typename">uint32</span></div><div class="line"></div><div class="line"></div><div class="line">	when   <span class="typename">int64</span></div><div class="line">	period <span class="typename">int64</span></div><div class="line">	f      <span class="keyword">func</span>(arg any, seq <span class="typename">uintptr</span>, delay <span class="typename">int64</span>)</div><div class="line">	arg    any</div><div class="line">	seq    <span class="typename">uintptr</span></div><div class="line"></div><div class="line">	ts *timers <span class="comment">// 注意这里</span></div><div class="line"></div><div class="line">	sendLock mutex</div><div class="line">	isSending atomic.Int32</div><div class="line">}</div></pre></td></tr></table></figure>

<p>我们来看看对这个堆操作的一些方法。</p>
<p><code>timerHeapN</code>定义了堆是四叉堆，也就是每个节点最多有4个子节点。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">const</span> timerHeapN =<span class="number"> 4</span></div></pre></td></tr></table></figure>

<p>堆常用的辅助方法就是<code>siftUp</code>和<code>siftDown</code>，分别用于上浮和下沉操作。</p>
<p>下面是上浮的方法，我把一些跟踪检查的代码去掉了。整体看代码还是比较简单的，就是不停的上浮，直到找到合适的位置。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// siftUp将位置i的计时器在堆中合适的位置，通过将其向堆的顶部移动。</span></div><div class="line"><span class="keyword">func</span> (ts *timers) siftUp(i <span class="typename">int</span>) {</div><div class="line">	heap := ts.heap</div><div class="line">	<span class="keyword">if</span> i &gt;= <span class="built_in">len</span>(heap) {</div><div class="line">		badTimer()</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 注意下面两行我们保存了当前i的计时器和它的when值</span></div><div class="line">	tw := heap[i] </div><div class="line">	when := tw.when</div><div class="line">	<span class="keyword">if</span> when &lt;=<span class="number"> 0</span> {</div><div class="line">		badTimer()</div><div class="line">	}</div><div class="line">	<span class="keyword">for</span> i &gt;<span class="number"> 0</span> {</div><div class="line">		p := <span class="typename">int</span>(<span class="typename">uint</span>(i<span class="number">-1</span>) / timerHeapN) <span class="comment">// 父节点 (i-1)/4</span></div><div class="line">		<span class="keyword">if</span> when &gt;= heap[p].when { <span class="comment">// 如果父节点的when &lt;= 当前节点的when，那么就不需要再上浮了</span></div><div class="line">			<span class="keyword">break</span></div><div class="line">		}</div><div class="line">		heap[i] = heap[p] <span class="comment">// 父节点下沉到当前的i</span></div><div class="line">		i = p <span class="comment">// i指向父节点, 继续循环上浮检查</span></div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 如果发生了上浮，那么最后将tw放到上浮到的合适位置</span></div><div class="line">	<span class="keyword">if</span> heap[i].timer != tw.timer {</div><div class="line">		heap[i] = tw</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>类似的，下面是下沉的方法：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// siftDown将位置i的计时器放在堆中的正确位置，通过将其向堆的底部移动。</span></div><div class="line"><span class="keyword">func</span> (ts *timers) siftDown(i <span class="typename">int</span>) {</div><div class="line">	heap := ts.heap</div><div class="line">	n := <span class="built_in">len</span>(heap)</div><div class="line">	<span class="keyword">if</span> i &gt;= n {</div><div class="line">		badTimer()</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 如果已经是叶子节点，不用下沉了</span></div><div class="line">	<span class="keyword">if</span> i*timerHeapN<span class="number">+1</span> &gt;= n {</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 保存当前i的计时器和when值</span></div><div class="line">	tw := heap[i]</div><div class="line">	when := tw.when</div><div class="line">	<span class="keyword">if</span> when &lt;=<span class="number"> 0</span> {</div><div class="line">		badTimer()</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 从左子节点开始，找到最小的when值，然后将当前节点下沉到这个位置</span></div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		leftChild := i*timerHeapN +<span class="number"> 1</span> <span class="comment">// 左子节点</span></div><div class="line">		<span class="keyword">if</span> leftChild &gt;= n {</div><div class="line">			<span class="keyword">break</span></div><div class="line">		}</div><div class="line">		w := when</div><div class="line">		c :=<span class="number"> -1</span></div><div class="line">		<span class="keyword">for</span> j, tw := <span class="keyword">range</span> heap[leftChild:min(leftChild+timerHeapN, n)] { <span class="comment">// 从左子节点开始遍历子节点,找到小于当前w的最小的子节点</span></div><div class="line">			<span class="keyword">if</span> tw.when &lt; w {</div><div class="line">				w = tw.when</div><div class="line">				c = leftChild + j</div><div class="line">			}</div><div class="line">		}</div><div class="line">		<span class="keyword">if</span> c &lt;<span class="number"> 0</span> { <span class="comment">// 如果没有找到比当前节点更小的子节点，那么就不用下沉了</span></div><div class="line">			<span class="keyword">break</span></div><div class="line">		}</div><div class="line"></div><div class="line">        <span class="comment">// 将当前节点下沉到最小的子节点</span></div><div class="line">		heap[i] = heap[c]</div><div class="line">		i = c</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 如果发生了下沉，那么最后将tw放到下沉到的合适位置</span></div><div class="line">	<span class="keyword">if</span> heap[i].timer != tw.timer {</div><div class="line">		heap[i] = tw</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>比上浮略微复杂，因为需要在兄弟节点中找到最小的节点，然后将当前节点下沉到这个位置。</p>
<p>对于一个任意的slice,我们可以把它初始化为一个四叉堆，方法如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (ts *timers) initHeap() {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(ts.heap) &lt;=<span class="number"> 1</span> {</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">    </div><div class="line">    <span class="comment">// 从最后一个非叶子节点开始，依次下沉</span></div><div class="line">	<span class="keyword">for</span> i := <span class="typename">int</span>(<span class="typename">uint</span>(<span class="built_in">len</span>(ts.heap<span class="number">)-1</span><span class="number">-1</span>) / timerHeapN); i &gt;=<span class="number"> 0</span>; i-- {</div><div class="line">		ts.siftDown(i)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>当然timers还有一些辅助timer处理的一些方法，很多和四叉堆没有关系了，我就不一一介绍了，我主要介绍几个和四叉堆相关的方法。</p>
<blockquote>
<p>这里吐槽一下，这个time.go文件中代码组织很乱，timer和timers的方法都穿插在一起。理论应该是timer方法和timers方法分开，这样更清晰。或者把timers抽取到一个单独的文件中。</p>
</blockquote>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (ts *timers) deleteMin() {</div><div class="line">    <span class="comment">// 得到堆顶元素</span></div><div class="line">	t := ts.heap<span class="number">[0</span>].timer</div><div class="line">	<span class="keyword">if</span> t.ts != ts {</div><div class="line">		throw(<span class="string">"wrong timers"</span>)</div><div class="line">	}</div><div class="line">	t.ts = <span class="constant">nil</span> <span class="comment">// 将timer的ts置为nil，自此和ts一别两宽，再无瓜葛</span></div><div class="line"></div><div class="line">    <span class="comment">// 将最后一个元素设置为堆顶</span></div><div class="line">	last := <span class="built_in">len</span>(ts.heap) -<span class="number"> 1</span></div><div class="line">	<span class="keyword">if</span> last &gt;<span class="number"> 0</span> {</div><div class="line">		ts.heap<span class="number">[0</span>] = ts.heap[last]</div><div class="line">	}</div><div class="line">	ts.heap[last] = timerWhen{} <span class="comment">// 将最后一个元素置为空</span></div><div class="line">	ts.heap = ts.heap[:last] <span class="comment">// 缩减slice,剔除最后的空元素</span></div><div class="line">	<span class="keyword">if</span> last &gt;<span class="number"> 0</span> { <span class="comment">// 将堆顶元素下沉</span></div><div class="line">		ts.siftDown<span class="number">(0</span>)</div><div class="line">	}</div><div class="line">	ts.updateMinWhenHeap()</div><div class="line">	<span class="keyword">if</span> last ==<span class="number"> 0</span> {</div><div class="line">		ts.minWhenModified.Store<span class="number">(0</span>)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>增加一个timer到堆中：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (ts *timers) addHeap(t *timer) {</div><div class="line">	<span class="keyword">if</span> netpollInited.Load() ==<span class="number"> 0</span> {</div><div class="line">		netpollGenericInit()</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">if</span> t.ts != <span class="constant">nil</span> {</div><div class="line">		throw(<span class="string">"ts set in timer"</span>)</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 设置timer的ts为当前的timers,从此执子之手，笑傲江湖</span></div><div class="line">	t.ts = ts</div><div class="line">    <span class="comment">// 添加到最后</span></div><div class="line">	ts.heap = <span class="built_in">append</span>(ts.heap, timerWhen{t, t.when})</div><div class="line">	ts.siftUp(<span class="built_in">len</span>(ts.heap) -<span class="number"> 1</span>) <span class="comment">// 上浮它到合适的位置</span></div><div class="line">	<span class="keyword">if</span> t == ts.heap<span class="number">[0</span>].timer {</div><div class="line">		ts.updateMinWhenHeap()</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<h2 id="n叉堆">n叉堆</h2>
<p><strong><em>d-ary</em> 堆</strong>或 <strong><em>d-heap</em></strong> 是一种优先队列数据结构，是二进制堆的泛化，其中节点有d个子节点而不是 2 个子节点。因此，二进制堆是2堆，<strong>而三元堆</strong>是3堆。根据 Tarjan 和 Jensen 等人的说法，d-ary堆是由 Donald B. Johnson 1975 年发明的。</p>
<p>此数据结构允许比二进制堆更快地执行降低优先级操作(因为深度更浅了)，但代价是删除最小操作速度较慢。这种权衡导致算法的运行时间更长，其中降低优先级操作比删除最小操作更常见。此外，d-ary堆比二进制堆具有更好的内存缓存行为，尽管理论上最坏情况下的运行时间更长，但它们在实践中运行得更快。与二进制堆一样，d-ary堆是一种就地数据结构，除了在堆中存储项目数组所需的存储空间外，它不使用任何额外的存储空间。</p>
<p>在Go生态圈已经有相应的库实现这个数据结构，比如<a href="https://github.com/ahrav/go-d-ary-heap" target="_blank" rel="external">ahrav/go-d-ary-heap</a>,所以如果你有类似场景的需求，或者想对比测试，你可以使用这个库。</p>
<p>导入库:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">import</span> <span class="string">"github.com/ahrav/go-d-ary-heap"</span></div></pre></td></tr></table></figure>

<p>下面的例子是创建三叉最小堆和四叉最大堆的例子:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">    <span class="string">"fmt"</span></div><div class="line">    <span class="string">"github.com/ahrav/go-d-ary-heap"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">    <span class="comment">// Create a min-heap for integers with a branching factor of 3.</span></div><div class="line">    minHeap := heap.NewHeap[<span class="typename">int</span>]<span class="number">(3</span>, <span class="keyword">func</span>(a, b <span class="typename">int</span>) <span class="typename">bool</span> { <span class="keyword">return</span> a &lt; b })</div><div class="line"></div><div class="line">    <span class="comment">// Create a max-heap for integers with a branching factor of 4.</span></div><div class="line">    maxHeap := heap.NewHeap[<span class="typename">int</span>]<span class="number">(4</span>, <span class="keyword">func</span>(a, b <span class="typename">int</span>) <span class="typename">bool</span> { <span class="keyword">return</span> a &gt; b })</div><div class="line">}</div></pre></td></tr></table></figure>

<p>往堆中增加元素：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">minHeap.Push<span class="number">(10</span>)</div><div class="line">minHeap.Push<span class="number">(5</span>)</div><div class="line">minHeap.Push<span class="number">(15</span>)</div><div class="line"></div><div class="line">maxHeap.Push<span class="number">(10</span>)</div><div class="line">maxHeap.Push<span class="number">(5</span>)</div><div class="line">maxHeap.Push<span class="number">(15</span>)</div></pre></td></tr></table></figure>

<p>从堆中移除最值:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">fmt.Println(minHeap.Pop()) <span class="comment">// Outputs: 5</span></div><div class="line">fmt.Println(maxHeap.Pop()) <span class="comment">// Outputs: 15</span></div></pre></td></tr></table></figure>

<p>返回但是不移除最值:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">fmt.Println(minHeap.Peek()) <span class="comment">// Assuming more elements were added, outputs the smallest</span></div><div class="line">fmt.Println(maxHeap.Peek()) <span class="comment">// Assuming more elements were added, outputs the largest</span></div></pre></td></tr></table></figure>

]]></content>
    <summary type="html">
    <![CDATA[<p>Go语言中Timer以及相关的Ticker、time.After、time.AfterFunc 等定时器最终是以四叉堆的数据形式存放的。</p>
<p>全局的 timer 堆也经历过三个阶段的重要升级。</p>
<ul>
<li>Go 1.9 版本之前，所有的计时器由全局唯一的四叉堆维护，goroutine间竞争激烈。</li>
<li>Go 1.10 - 1.13，全局使用 64 个四叉堆维护全部的计时器，通过分片减少了竞争的压力，但是本质上还是没有解决 1.9 版本之前的问题</li>
<li>Go 1.14 版本之后，每个 P 单独维护一个四叉堆，避免了goroutine的竞争。 (后面我们再介绍 per-P 的数据结构)</li>
</ul>
<p>常见的堆(heap)常常以二叉堆的形式实现。可是为什么Go timer使用四叉堆呢？</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[HeapMap, 一个混合功能的数据结构Go语言实现]]></title>
    <link href="https://colobu.com/2024/11/17/heapmap/"/>
    <id>https://colobu.com/2024/11/17/heapmap/</id>
    <published>2024-11-17T09:17:13.000Z</published>
    <updated>2024-11-17T09:18:38.769Z</updated>
    <content type="html"><![CDATA[<p>今天在准备《秘而不宣》系列下一篇文章时，思绪飘散了，突然想到使用 Heap 的功能再加 HashTable (Map) 的功能，可以构造一种新的数据结构，然后把我聚合程序中的数据聚合数据结构替换掉，总之思绪翩翩。然后在网上搜了一下，这种数据结构其实早就有了，名字叫 <code>HeapMap</code>。</p>
<a id="more"></a>
<p><code>HeapMap</code> (也叫做 <code>PriorityMap</code>) 是一种结合了<strong>堆</strong>和<strong>哈希映射</strong>的数据结构，常用于需要按键排序并进行高效查找的场景。它可以在优先级队列的基础上，使用哈希映射来提供快速访问和更新。<code>HeapMap</code> 在实现过程中利用堆的有序性和哈希表的快速查找能力，以支持<strong>按键排序</strong>和<strong>常数时间查找</strong>。</p>
<p>Go 语言支付 Rob Pike 在他的 <a href="https://users.ece.utexas.edu/~adnan/pike.html" target="_blank" rel="external">Rob Pike&#39;s 5 Rules of Programming</a> 第 5 条就指出：</p>
<blockquote>
<ul>
<li>Data dominates. If you&#39;ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.<br>数据为王。如果你选择了合适的数据结构并进行了良好的组织，算法通常会变得显而易见。<em>编程的核心在于数据结构，而非算法</em>。</li>
</ul>
</blockquote>
<p>所以，如果在合适的场景下，针对它的特点，使用 HeapMap 会取得事半功倍的效果。</p>
<h3 id="HeapMap_的主要特点"><code>HeapMap</code> 的主要特点</h3>
<ol>
<li><strong>堆的特点</strong>：<code>HeapMap</code> 内部通过堆来维护键的顺序，可以快速获取最小或最大键。堆提供了插入和删除堆顶元素的 <code>O(log n)</code> 时间复杂度。</li>
<li><strong>哈希映射的特点</strong>：<code>HeapMap</code> 同时使用哈希映射以支持快速查找。哈希映射的查找、插入、删除等操作在理想情况下时间复杂度为 <code>O(1)</code>。</li>
<li><strong>用途</strong>：<code>HeapMap</code> 适合需要<em>频繁按键排序和快速查找</em>的场景，比如带有优先级的缓存、调度系统、任务优先队列等。</li>
</ol>
<h3 id="HeapMap_的基本结构"><code>HeapMap</code> 的基本结构</h3>
<ul>
<li><strong>堆（Heap）</strong>：用来维持按键的顺序，堆可以是最小堆或最大堆，根据具体需求决定。</li>
<li><strong>哈希映射（Map）</strong>：用来存储每个键值对，并支持通过键快速查找元素。</li>
</ul>
<p>你使用一个 <code>container/heap</code> + <code>map</code> 很容易实现一个 <code>HeapMap</code>, 其实我们没必要自己去写一个重复的轮子了，网上其他语言比如 Rust、Java 都有现成的实现，Go 语言中也有一个很好的实现：<a href="https://github.com/nemars/heapmap" target="_blank" rel="external">nemars/heapmap</a></p>
<h3 id="HeapMap_的实现"><code>HeapMap</code> 的实现</h3>
<p><code>nemars/heapmap</code> 这个库是去年增加到 github 中的，我是第一个 star 它的人。我们看看它是怎么实现的。</p>
<h4 id="结构体定义">结构体定义</h4>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Entry[K comparable, V, P any] <span class="keyword">struct</span> {</div><div class="line">	Key      K</div><div class="line">	Value    V</div><div class="line">	Priority P</div><div class="line">	index    <span class="typename">int</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> heapmap[K comparable, V, P any] <span class="keyword">struct</span> {</div><div class="line">	h pq[K, V, P]</div><div class="line">	m <span class="keyword">map</span>[K]*Entry[K, V, P]</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>Entry</code> 代表这个数据结构中的一个节点 (元素、条目) , 它包含 key、value 值，还有优先级，index 记录它在堆的实现数组中的索引。</p>
<p><code>heapmap</code> 代表 <code>HeapMap</code> 的实现，它包含两个字段，第一个字段其实就是 <code>Heap</code> 的实现，为了方便实现泛型，它就自己实现了一个堆。第二个字段就是一个 map 对象了。</p>
<h4 id="典型的方法">典型的方法</h4>
<p>数据结构定义清楚了，那就就可以实现它的方法了。它实现了一些便利的方法，我们值关注几个实现就好了。</p>
<h5 id="Len_方法">Len 方法</h5>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (hm *heapmap[K, V, P]) Len() <span class="typename">int</span> {</div><div class="line">	<span class="keyword">return</span> <span class="built_in">len</span>(hm.m)</div><div class="line">}</div></pre></td></tr></table></figure>

<p>读取<code>h</code>字段或者<code>m</code>字段的长度都可以。</p>
<h5 id="Peek_方法">Peek 方法</h5>
<p>返回root元素。<br>最小堆就是返回最小的元素，最大堆就是返回最大的元素。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (hm *heapmap[K, V, P]) Peek() (Entry[K, V, P], <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">if</span> hm.Empty() {</div><div class="line">		<span class="keyword">return</span> Entry[K, V, P]{}, <span class="constant">false</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> *hm.h.entries<span class="number">[0</span>], <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<h5 id="Pop_方法">Pop 方法</h5>
<p>弹出root元素。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (hm *heapmap[K, V, P]) Pop() (Entry[K, V, P], <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">if</span> hm.Empty() {</div><div class="line">		<span class="keyword">return</span> Entry[K, V, P]{}, <span class="constant">false</span></div><div class="line">	}</div><div class="line">	e := *heap.Pop(&hm.h).(*Entry[K, V, P])</div><div class="line">	<span class="built_in">delete</span>(hm.m, e.Key)</div><div class="line">	<span class="keyword">return</span> e, <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>注意涉及到元素的删除操作，要同时删除 map 中的元素。</p>
<h5 id="Push_方法_(Set_方法)">Push 方法 (Set 方法)</h5>
<p>其实作者没有实现 Push 方法，而是使用Set 方法来实现的。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (hm *heapmap[K, V, P]) Set(key K, value V, priority P) {</div><div class="line">	<span class="keyword">if</span> e, ok := hm.m[key]; ok {</div><div class="line">		e.Value = value</div><div class="line">		e.Priority = priority</div><div class="line">		heap.Fix(&hm.h, e.index)</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	e := &Entry[K, V, P]{</div><div class="line">		Key:      key,</div><div class="line">		Value:    value,</div><div class="line">		Priority: priority,</div><div class="line">	}</div><div class="line">	heap.Push(&hm.h, e)</div><div class="line">	hm.m[key] = e</div><div class="line">}</div></pre></td></tr></table></figure>

<p>Set方法有两个功能。如果元素的Key已经存在，那么就是更新元素，并且根据优先级进行调整。<br>如果元素的Key不存在，那么就是插入元素。</p>
<h5 id="Get_方法">Get 方法</h5>
<p>Get 方法就是获取任意的元素。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"></div><div class="line"><span class="keyword">func</span> (hm *heapmap[K, V, P]) Get(key K) (Entry[K, V, P], <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">if</span> e, ok := hm.m[key]; ok {</div><div class="line">		<span class="keyword">return</span> *e, <span class="constant">true</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> Entry[K, V, P]{}, <span class="constant">false</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>有一点你需要注意的是，这个数据结构不是线程安全的，如果你需要线程安全的话，你可以使用 <code>sync.Mutex</code>/<code>sync.RWMutex</code> 来保护它。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>今天在准备《秘而不宣》系列下一篇文章时，思绪飘散了，突然想到使用 Heap 的功能再加 HashTable (Map) 的功能，可以构造一种新的数据结构，然后把我聚合程序中的数据聚合数据结构替换掉，总之思绪翩翩。然后在网上搜了一下，这种数据结构其实早就有了，名字叫 <code>HeapMap</code>。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构 CacheLinePad：精细化优化]]></title>
    <link href="https://colobu.com/2024/11/17/go-internal-ds-cacheline/"/>
    <id>https://colobu.com/2024/11/17/go-internal-ds-cacheline/</id>
    <published>2024-11-17T08:19:01.000Z</published>
    <updated>2024-11-17T09:16:18.533Z</updated>
    <content type="html"><![CDATA[<p>在现代多核处理器中，高效的缓存机制极大地提升了程序性能，而“伪共享”问题却常常导致缓存机制的低效。</p>
<a id="more"></a>
<h2 id="1-_背景">1. 背景</h2>
<blockquote>
<p>cacheline 本文中有时又叫做 缓存行</p>
</blockquote>
<p>在现代多核处理器中，三级缓存通常分为三级：L1、L2 和 L3，每一级缓存的大小、速度和共享方式都不同：</p>
<ul>
<li><p><strong>L1 缓存</strong>：这是速度最快的缓存，通常每个 CPU 核心都有独立的 L1 缓存。L1 缓存分为两个部分：一个用于存储指令（L1I），另一个用于存储数据（L1D）。L1 缓存的容量一般较小（通常 32KB - 64KB），但是读取速度极快，以极低的延迟为 CPU 核心提供服务。</p>
</li>
<li><p><strong>L2 缓存</strong>：L2 缓存通常比 L1 缓存大一些，容量一般在 256KB - 1MB 左右，每个 CPU 核心通常也会有独立的 L2 缓存。虽然 L2 缓存的访问速度比 L1 缓存稍慢，但它仍然显著快于主存。</p>
</li>
<li><p><strong>L3 缓存</strong>：这是三级缓存中容量最大的，通常在 8MB - 64MB 或更大。L3 缓存往往由所有 CPU 核心共享，并且主要用于减少核心之间的数据传输延迟。L3 缓存的读取速度比 L1、L2 缓存慢，但相对主存依然较快。对于多核处理器，L3 缓存是多核心之间协作的重要纽带。<br><img src="network-1.png" alt=""></p>
</li>
</ul>
<p>CPU缓存将数据划分成若干个 <code>cacheline</code>，使得 CPU 访问特定数据时，能以 cacheline 为单位加载或存储数据。<code>cacheline</code> 的大小通常是固定的，x86 架构中常见的 <code>cacheline</code> 大小是 64 字节，而 Apple M 系列等一些 ARM 架构处理器上可能达到 128 字节。</p>
<p>在 CPU 执行程序时，若数据在某级缓存中命中，整个 <code>cacheline</code> 会从该缓存加载到寄存器中；若数据不在 L1 缓存中，则会依次查找 L2、L3 缓存，并最终在主存中查找并加载到缓存。由于 <code>cacheline</code> 是缓存操作的基本单位，每次数据传输都是以 <code>cacheline</code> 为最小粒度的。</p>
<p>比如在 mac mini  m2 机器是，我们可以查看此 CPU 的缓存行大小为 128 字节：<br><img src="network-2.png" alt=""></p>
<p>Linux 下可以查看另外一台机器的各级别缓存行大小为 64 字节：<br><img src="network-3.png" alt=""></p>
<h3 id="1-1_伪共享_(False_Sharing)">1.1 伪共享 (False Sharing)</h3>
<p><strong>伪共享</strong> 是指多个线程访问同一个 cache line 中的不同变量时，导致频繁的缓存失效（cache invalidation），从而大大降低程序性能。伪共享通常在多线程编程中发生，因为在多个线程中，如果两个或多个线程操作的变量在同一个 cache line 中，但它们并没有真正的共享关系，每个线程对其变量的写操作会导致其他线程的缓存失效。这样，CPU 核心会不断地将数据写回并重新加载，产生了不必要的资源浪费。</p>
<p>设有两个线程，各自操作两个独立的变量 <code>x</code> 和 <code>y</code>：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Data <span class="keyword">struct</span> {</div><div class="line">    x <span class="typename">int64</span> <span class="comment">// 线程A更新的变量</span></div><div class="line">    y <span class="typename">int64</span> <span class="comment">// 线程B更新的变量</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>如果变量 <code>x</code> 和 <code>y</code> 位于同一个 cache line 中，那么线程 A 更新 <code>x</code> 后，线程 B 也会因为缓存失效而重新加载 <code>y</code>，尽管 B 实际上并未使用 <code>x</code> 的值。这种情况下，虽然两个变量并没有直接共享，但每次写操作都会导致另一方的缓存失效，从而形成了伪共享。</p>
<h3 id="1-2_如何避免伪共享？">1.2 如何避免伪共享？</h3>
<p>伪共享会对性能产生严重影响，但可以通过以下几种方法来优化：</p>
<ol>
<li><strong>变量对齐（Padding）</strong>：将每个变量扩展至一个完整的 <code>cacheline</code>，以防止多个线程访问同一个 <code>cacheline</code>。例如，可以在变量之间添加填充数据来分隔不同的 <code>cacheline</code> (假定 CPU 缓存行是 64 字节)：</li>
</ol>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Data <span class="keyword">struct</span> {</div><div class="line">    x <span class="typename">int64</span>           <span class="comment">// 线程A更新的变量</span></div><div class="line">    _ <span class="number">[7</span>]<span class="typename">int64</span>        <span class="comment">// 填充7个int64以对齐至64字节的cache line大小</span></div><div class="line">    y <span class="typename">int64</span>           <span class="comment">// 线程B更新的变量</span></div><div class="line">}</div></pre></td></tr></table></figure>

<ol start="2">
<li><strong>将变量分散到不同的结构体中</strong>：对于经常被多个线程更新的变量，可以考虑将它们分散到不同的结构体，避免同一结构体被多个线程同时频繁更新。</li>
<li><strong>使用原子变量</strong>：在某些情况下，可以使用原子变量进行更新。虽然这不会彻底消除伪共享，但可以减少缓存一致性带来的开销。</li>
<li><strong>绑定 CPU 核心（CPU Affinity）</strong>：可以将线程绑定到指定的 CPU 核心上，从而减少多个线程同时访问同一块缓存的数据的几率。</li>
</ol>
<h3 id="1-3_单线程的缓存行污染问题">1.3 单线程的缓存行污染问题</h3>
<p>虽然单线程不会出现伪共享的问题，但是单线程程序仍然有一些缓存优化的空间：</p>
<ul>
<li><strong>避免缓存行污染</strong>：在单线程程序中，如果频繁访问的变量分布在不同的 cache line 上，会导致缓存频繁更替，增加缓存开销。优化时可以将频繁使用的数据集中在同一个 cache line 内，减少 CPU 从内存加载数据的频率。</li>
<li><strong>数据布局优化</strong>：对于单线程程序，也可以通过调整数据的内存布局，让程序更好地利用缓存。将经常一起访问的数据放在连续的内存中，以提高缓存命中率。<br>比如下面一个测试，</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"testing"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">// NonAlignedStruct 未对齐的结构体，补充后占24个字节</span></div><div class="line"><span class="keyword">type</span> NonAlignedStruct <span class="keyword">struct</span> {</div><div class="line">	a <span class="typename">byte</span> <span class="comment">// 1字节,补齐7字节</span></div><div class="line">	b <span class="typename">int64</span> <span class="comment">// 8字节</span></div><div class="line">	c <span class="typename">byte</span> <span class="comment">// 1字节,补齐7字节</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// AlignedStruct 已对齐的结构体，补充后占16个字节</span></div><div class="line"><span class="keyword">type</span> AlignedStruct <span class="keyword">struct</span> {</div><div class="line">	b <span class="typename">int64</span> <span class="comment">// 8字节</span></div><div class="line">	a <span class="typename">byte</span> <span class="comment">// 1字节</span></div><div class="line">	c <span class="typename">byte</span> <span class="comment">// 1字节</span></div><div class="line">	_ <span class="number">[6</span>]<span class="typename">byte</span> <span class="comment">// 填充6个字节，总共16个字节</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">const</span> arraySize =<span class="number"> 1024</span> *<span class="number"> 1024</span></div><div class="line"></div><div class="line"><span class="keyword">var</span> (</div><div class="line">	nonAlignedArray [arraySize]NonAlignedStruct</div><div class="line">	alignedArray    [arraySize]AlignedStruct</div><div class="line">	result          <span class="typename">int64</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="comment">// 初始化数组</span></div><div class="line"><span class="keyword">func</span> init() {</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; arraySize; i++ {</div><div class="line">		nonAlignedArray[i] = NonAlignedStruct{</div><div class="line">			a: <span class="typename">byte</span>(i),</div><div class="line">			b: <span class="typename">int64</span>(i),</div><div class="line">			c: <span class="typename">byte</span>(i),</div><div class="line">		}</div><div class="line">		alignedArray[i] = AlignedStruct{</div><div class="line">			a: <span class="typename">byte</span>(i),</div><div class="line">			b: <span class="typename">int64</span>(i),</div><div class="line">			c: <span class="typename">byte</span>(i),</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// BenchmarkNonAligned 测试未对齐结构体的性能</span></div><div class="line"><span class="keyword">func</span> BenchmarkNonAligned(b *testing.B) {</div><div class="line">	<span class="keyword">var</span> sum <span class="typename">int64</span></div><div class="line">	b.ResetTimer()</div><div class="line"></div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">		<span class="keyword">for</span> j :=<span class="number"> 0</span>; j &lt; arraySize; j++ {</div><div class="line">			sum += nonAlignedArray[j].b <span class="comment">// 读取未对齐结构体的字段</span></div><div class="line">		}</div><div class="line">	}</div><div class="line">	result = sum <span class="comment">// 防止编译器优化</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// BenchmarkAligned 测试已对齐结构体的性能</span></div><div class="line"><span class="keyword">func</span> BenchmarkAligned(b *testing.B) {</div><div class="line">	<span class="keyword">var</span> sum <span class="typename">int64</span></div><div class="line">	b.ResetTimer()</div><div class="line"></div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">		<span class="keyword">for</span> j :=<span class="number"> 0</span>; j &lt; arraySize; j++ {</div><div class="line">			sum += alignedArray[j].b <span class="comment">// 读取已对齐结构体的字段</span></div><div class="line">		}</div><div class="line">	}</div><div class="line">	result = sum <span class="comment">// 防止编译器优化</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p><img src="network-4.png" alt=""></p>
<p>可以看到读取对齐的结构体性能要远远好于未对齐的结构体。</p>
<p><img src="network-5.png" alt=""><br><img src="network-6.png" alt=""></p>
<p>很多高性能的库都会采用 CacheLine 优化的数据结构，比如 Java 生态圈知名的 LMAX Disruptor。 Go 标准库中也有类似的优化，让我们一起来看看它的实现和应用场景。</p>
<h2 id="2-_Go_运行时中的_CacheLine">2. Go 运行时中的 CacheLine</h2>
<h3 id="2-1_运行时中的_CacheLinePad">2.1 运行时中的 CacheLinePad</h3>
<p>我们支持，Go 语言支持不同的 CPU 架构，不同的 CPU 架构的缓存行的大小也可能不同，Go 语言是如何统一的呢？<br>方法很简单，就是针对不同的 CPU 架构，定义不同大小的缓存行。</p>
<p>首先定义统一的结构和变量:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// CacheLinePad 用来填充结构体，避免伪共享</span></div><div class="line"><span class="keyword">type</span> CacheLinePad <span class="keyword">struct</span>{ _ [CacheLinePadSize]<span class="typename">byte</span> }</div><div class="line"></div><div class="line"><span class="comment">// CacheLineSize 是 CPU 的缓存行大小，不同的 CPU 架构可能不同.</span></div><div class="line"><span class="comment">// 目前 Go 运行时没有检测真实的缓存行大小，所以代码实现使用每个 GOARCH 的常量 CacheLinePadSize 作为近似值。</span></div><div class="line"><span class="keyword">var</span> CacheLineSize <span class="typename">uintptr</span> = CacheLinePadSize</div></pre></td></tr></table></figure>

<p>然后针对不同的 CPU 架构定义不同的缓存行大小。<br>比如arm64的CPU, 文件<code>go/src/internal/cpu/cpu_arm64.go</code>中定义了缓存行大小为128字节：</p>
<figure class="highlight go"><figcaption><span>go/src/internal/cpu/cpu_arm64.go</span></figcaption><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// CacheLinePadSize is used to prevent false sharing of cache lines.</span></div><div class="line"><span class="comment">// We choose 128 because Apple Silicon, a.k.a. M1, has 128-byte cache line size.</span></div><div class="line"><span class="comment">// It doesn't cost much and is much more future-proof.</span></div><div class="line"><span class="keyword">const</span> CacheLinePadSize =<span class="number"> 128</span></div></pre></td></tr></table></figure>

<p>比如64bit的龙芯， 缓存行大小是64字节，文件<code>go/src/internal/cpu/cpu_loong64.go</code>中定义了缓存行大小为64字节：</p>
<figure class="highlight go"><figcaption><span>go/src/internal/cpu/cpu_loong64.go</span></figcaption><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// CacheLinePadSize is used to prevent false sharing of cache lines.</span></div><div class="line"><span class="comment">// We choose 64 because Loongson 3A5000 the L1 Dcache is 4-way 256-line 64-byte-per-line.</span></div><div class="line"><span class="keyword">const</span> CacheLinePadSize =<span class="number"> 64</span></div></pre></td></tr></table></figure>

<p>又比如x86和amd64的CPU， 缓存行大小是64字节，文件<code>go/src/internal/cpu/cpu_x86.go</code>中定义了缓存行大小为64字节：</p>
<figure class="highlight go"><figcaption><span>go/src/internal/cpu/cpu_x86.go</span></figcaption><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="comment">//go:build 386 || amd64</span></div><div class="line"></div><div class="line"><span class="keyword">package</span> cpu</div><div class="line"></div><div class="line"><span class="keyword">const</span> CacheLinePadSize =<span class="number"> 64</span></div></pre></td></tr></table></figure>

<p>所以Go运行时是根据它支持的不同的 CPU 架构，定义不同的缓存行大小，以此来避免伪共享问题。</p>
<p>但是这个数据结构是定义在Go运行时<code>internal</code>库中，不对外暴露，那么我们怎么用的？</p>
<h3 id="2-2_golang-org/x/sys/cpu">2.2 golang.org/x/sys/cpu</h3>
<p>没关系，Go的扩展库<code>golang.org/x/sys/cpu</code>中提供了<code>CacheLinePad</code>的定义，我们可以直接使用。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> CacheLinePad <span class="keyword">struct</span>{ _ [cacheLineSize]<span class="typename">byte</span> }</div></pre></td></tr></table></figure>

<p>它的实现和Go运行时中的一样，只是把<code>CacheLinePad</code>暴露出来了，所以我们可以在自己的项目中直接使用。</p>
<h3 id="2-3_Go运行时中的应用场景">2.3 Go运行时中的应用场景</h3>
<p>在这个系列的上一篇文章中，我们介绍了<code>treap</code>, <code>treap</code>使用在<code>semTable</code>中，<code>semTable</code>是Go运行时中的一个数据结构，用来管理<code>semaphore</code>的等待队列。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> semaRoot <span class="keyword">struct</span> {</div><div class="line">	lock  mutex</div><div class="line">	treap *sudog        <span class="comment">// root of balanced tree of unique waiters.</span></div><div class="line">	nwait atomic.Uint32 <span class="comment">// Number of waiters. Read w/o the lock.</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">var</span> semtable semTable</div><div class="line"></div><div class="line"><span class="comment">// Prime to not correlate with any user patterns.</span></div><div class="line"><span class="keyword">const</span> semTabSize =<span class="number"> 251</span></div><div class="line"></div><div class="line"><span class="keyword">type</span> semTable [semTabSize]<span class="keyword">struct</span> {</div><div class="line">	root semaRoot</div><div class="line">	pad  [cpu.CacheLinePadSize - unsafe.Sizeof(semaRoot{})]<span class="typename">byte</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>等并发读取<code>semTable</code>时，由于<code>semTable</code>中的<code>root</code>是一个<code>semaRoot</code>结构体，<code>semaRoot</code>中有<code>mutex</code>，<code>treap</code>等字段，这些字段可能会被不同的CPU核心同时访问，导致伪共享问题。<br>为了解决伪共享问题，它增加了一个<code>Pad</code>字段，补齐字段的大小到<code>CacheLineSize</code>，这样就可以避免伪共享问题。当然这里可以确定<code>semaRoot</code>的大小不会超过一个<code>CacheLineSize</code>。</p>
<p><code>mheap</code> 结构体中展示了另外一种场景，将部分字段使用<code>CacheLinePad</code>隔开， 避免<code>arenas</code>字段和上面的字段之间的伪共享问题。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> mheap <span class="keyword">struct</span> {</div><div class="line">	_ sys.NotInHeap</div><div class="line"></div><div class="line"></div><div class="line">	lock mutex</div><div class="line"></div><div class="line">	pages pageAlloc <span class="comment">// page allocation data structure</span></div><div class="line"></div><div class="line">	sweepgen <span class="typename">uint32</span> <span class="comment">// sweep generation, see comment in mspan; written during STW</span></div><div class="line"></div><div class="line">	allspans []*mspan <span class="comment">// all spans out there</span></div><div class="line"></div><div class="line">	pagesInUse         atomic.Uintptr <span class="comment">// pages of spans in stats mSpanInUse</span></div><div class="line">	pagesSwept         atomic.Uint64  <span class="comment">// pages swept this cycle</span></div><div class="line">	pagesSweptBasis    atomic.Uint64  <span class="comment">// pagesSwept to use as the origin of the sweep ratio</span></div><div class="line">	sweepHeapLiveBasis <span class="typename">uint64</span>         <span class="comment">// value of gcController.heapLive to use as the origin of sweep ratio; written with lock, read without</span></div><div class="line">	sweepPagesPerByte  <span class="typename">float64</span>        <span class="comment">// proportional sweep ratio; written with lock, read without</span></div><div class="line"></div><div class="line">	reclaimIndex atomic.Uint64</div><div class="line"></div><div class="line">	reclaimCredit atomic.Uintptr</div><div class="line"></div><div class="line">	_ cpu.CacheLinePad <span class="comment">// prevents false-sharing between arenas and preceding variables</span></div><div class="line"></div><div class="line">	</div><div class="line">	arenas <span class="number">[1</span> &lt;&lt; arenaL1Bits]*<span class="number">[1</span> &lt;&lt; arenaL2Bits]*heapArena</div><div class="line"></div><div class="line">    ...</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>go/src/runtime/stack.go</code>中<code>stackpool</code>结构体中也使用了<code>CacheLinePad</code>，展示了另外一种用法：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> stackpool [_NumStackOrders]<span class="keyword">struct</span> {</div><div class="line">	item stackpoolItem</div><div class="line">	_    [(cpu.CacheLinePadSize - unsafe.Sizeof(stackpoolItem{})%cpu.CacheLinePadSize) % cpu.CacheLinePadSize]<span class="typename">byte</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>因为item的大小不确定，可能小于一个<code>CacheLineSize</code>，也可能大于一个<code>CacheLineSize</code>，所以这里对<code>CacheLinePad</code>求余，只需补充一个小于<code>CacheLineSize</code>的字节即可。</p>
<p>一般软件开发中，我们不需要关心这些细节，但是当我们需要优化性能时，了解这些底层的实现，可以帮助我们更好的理解和优化程序。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>在现代多核处理器中，高效的缓存机制极大地提升了程序性能，而“伪共享”问题却常常导致缓存机制的低效。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构 Treap：随机化的二叉搜索树]]></title>
    <link href="https://colobu.com/2024/11/17/go-internal-ds-treap/"/>
    <id>https://colobu.com/2024/11/17/go-internal-ds-treap/</id>
    <published>2024-11-17T08:19:00.000Z</published>
    <updated>2024-11-17T08:58:11.942Z</updated>
    <content type="html"><![CDATA[<p><code>treap</code> 是一棵二叉树，它同时维护二叉搜索树 （BST） 和堆的属性, 所以由此得名 (tree + heap   ⇒  treap)。</p>
<p>从形式上讲，treap （tree + heap） 是一棵二叉树，其节点包含两个值，一个 <em>key</em> 和一个 <em>priority</em>，这样 <strong>key</strong> 保持 BST 属性，<strong>priority</strong> 是一个保持 heap 属性的随机值（至于是最大堆还是最小堆并不重要）。相对于其他的平衡二叉搜索树，treap的特点是实现简单，且能基本实现随机平衡的结构。属于弱平衡树。</p>
<p><code>treap</code> 由 Raimund Siedel 和 Cecilia Aragon 于 1989 年提出。</p>
<p>treap 通常也被称为“笛卡尔树”，因为它很容易嵌入到笛卡尔平面中：<br><img src="Pasted-image-20241026111752.png" alt=""></p>
<p>具体来说，<code>treap</code> 是一种在二叉树中存储键值对 <code>(X,Y)</code> 的数据结构，其特点是：按 <code>X</code> 值满足二叉搜索树的性质，同时按 <code>Y</code> 值满足二叉堆的性质。如果树中某个节点包含值 <code>(X₀,Y₀)</code>，那么：</p>
<ul>
<li>左子树中所有节点的X值都满足 <code>X ≤ X₀</code> (BST 属性)</li>
<li>右子树中所有节点的X值都满足 <code>X₀ ≤ X</code> (BST 属性)</li>
<li>左右子树中所有节点的Y值都满足 Y ≤ Y₀ （堆属性。这里以最大堆为例）</li>
</ul>
<p>在这种实现中，  X是键（同时也是存储在 Treap 中的值），并且  Y称为<strong>优先级</strong>。如果没有优先级，则 treap 将是一个常规的二叉搜索树。</p>
<p>优先级（前提是每个节点的优先级都不相同）的特殊之处在于：它们可以确定性地决定树的最终结构（不会受到插入数据顺序的影响）。这一点是可以通过相关定理来证明的。<br>这里有个巧妙的设计：如果我们随机分配这些优先级值，就能在平均情况下得到一棵比较平衡的树（避免树退化成链表）。这样就能保证主要操作（如查找、插入、删除等）的时间复杂度保持在 O(log N) 水平。<br>正是因为这种随机分配优先级的特点，这种数据结构也被称为&quot;随机二叉搜索树&quot;。</p>
<p><img src="Pasted-image-20241026113542.png" alt=""></p>
<p>Treap维护堆性质的方法用到了旋转，且只需要进行两种旋转操作，因此编程复杂度较红黑树、AVL树要小一些。</p>
<p>红黑树的操作：<br><strong>插入</strong><br><em>以最大堆为例</em><br>给节点随机分配一个优先级，先和二叉搜索树的插入一样，先把要插入的点插入到一个叶子上，然后跟维护堆一样进行以下操作：</p>
<ol>
<li>如果当前节点的优先级比父节点大就进行2. 或3. 的操作</li>
<li>如果当前节点是父节点的左子叶就右旋</li>
<li>如果当前节点是父节点的右子叶就左旋。</li>
</ol>
<p><strong>删除</strong></p>
<p>因为 treap满足堆性质，所以只需要把要删除的节点旋转到叶节点上，然后直接删除就可以了。具体的方法就是每次找到优先级最大的子叶，向与其相反的方向旋转，直到那个节点被旋转到了叶节点，然后直接删除。</p>
<p><strong>查找</strong></p>
<p>和一般的二叉搜索树一样，但是由于 treap的随机化结构，Treap中查找的期望复杂度是 <code>O(logn)</code></p>
<p>以上是 treap 数据结构的背景知识，如果你想了解更多而关于 treap 的知识，你可以参考</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Treap" target="_blank" rel="external">https://en.wikipedia.org/wiki/Treap</a></li>
<li><a href="https://medium.com/carpanese/a-visual-introduction-to-treap-data-structure-part-1-6196d6cc12ee" target="_blank" rel="external">https://medium.com/carpanese/a-visual-introduction-to-treap-data-structure-part-1-6196d6cc12ee</a></li>
<li><a href="https://cp-algorithms.com/data_structures/treap.html" target="_blank" rel="external">https://cp-algorithms.com/data_structures/treap.html</a></li>
</ul>
<h2 id="Go_运行时的_treap_和用途">Go 运行时的 treap 和用途</h2>
<p>在 Go 运行时 <a href="https://github.com/golang/go/blob/master/src/runtime/sema.go#L40" target="_blank" rel="external">sema.go#semaRoot</a> 中，定义了一个数据结构 <code>semaRoot</code>:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> semaRoot <span class="keyword">struct</span> {</div><div class="line">	lock  mutex</div><div class="line">	treap *sudog        <span class="comment">// 不重复的等待者(goroutine)的平衡树(treap)的根节点</span></div><div class="line">	nwait atomic.Uint32 <span class="comment">// 等待者(goroutine)的数量</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> sudog <span class="keyword">struct</span> {</div><div class="line">	g *g</div><div class="line"></div><div class="line">	next *sudog</div><div class="line">	prev *sudog</div><div class="line">	elem unsafe.Pointer <span class="comment">// data element (may point to stack)</span></div><div class="line"></div><div class="line">	acquiretime <span class="typename">int64</span></div><div class="line">	releasetime <span class="typename">int64</span></div><div class="line">	ticket      <span class="typename">uint32</span></div><div class="line"></div><div class="line">	isSelect <span class="typename">bool</span></div><div class="line">	success <span class="typename">bool</span></div><div class="line"></div><div class="line">	waiters <span class="typename">uint16</span></div><div class="line"></div><div class="line">	parent   *sudog <span class="comment">// semaRoot binary tree</span></div><div class="line">	waitlink *sudog <span class="comment">// g.waiting list or semaRoot</span></div><div class="line">	waittail *sudog <span class="comment">// semaRoot</span></div><div class="line">	c        *hchan <span class="comment">// channel</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>这是Go语言互斥锁(Mutex)底层实现中的关键数据结构，用于管理等待获取互斥锁的goroutine队列。我们已经知道，在获取 <code>sync.Mutex</code> 时，如果锁已经被其它 goroutine 获取，那么当前请求锁的 goroutine 会被 block 住，就会被放入到这样一个数据结构中 (所以你也知道这个数据结构中的 goroutine 都是唯一的，不重复)。</p>
<p><code>semaRoot</code> 保存了一个平衡树，树中的 <code>sudog</code> 节点都有不同的地址 <code>(s.elem)</code> ,每个 <code>sudog</code> 可能通过 <code>s.waitlink</code> 指向一个链表，该链表包含等待相同地址的其他 <code>sudog</code>。对具有相同地址的 <code>sudog</code> 内部链表的操作时间复杂度都是O(1).。扫描顶层semaRoot列表的时间复杂度是 <code>O(log n)</code>,其中 <code>n</code> 是具有被阻塞goroutine的不同地址的数量（这些地址会散列到给定的semaRoot）。</p>
<p><code>semaRoot</code> 的 <code>treap *sudog</code> 其实就是一个 treap, 我们来看看它的实现。</p>
<h2 id="增加一个元素（入队）">增加一个元素（入队）</h2>
<p>增加一个等待的goroutine(<code>sudog</code>)到 <code>semaRoot</code> 的 <code>treap</code> 中，如果 <code>lifo</code> 为 <code>true</code>，则将 <code>s</code> 替换到 <code>t</code> 的位置，否则将 <code>s</code> 添加到 <code>t</code> 的等待列表的末尾。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (root *semaRoot) queue(addr *<span class="typename">uint32</span>, s *sudog, lifo <span class="typename">bool</span>) {</div><div class="line">	<span class="comment">// 设置这个要加入的节点</span></div><div class="line">	s.g = getg()</div><div class="line">	s.elem = unsafe.Pointer(addr)</div><div class="line">	s.next = <span class="constant">nil</span></div><div class="line">	s.prev = <span class="constant">nil</span></div><div class="line">	s.waiters =<span class="number"> 0</span></div><div class="line"></div><div class="line">	<span class="keyword">var</span> last *sudog</div><div class="line">	pt := &root.treap</div><div class="line">	<span class="comment">// 从根节点开始</span></div><div class="line">	<span class="keyword">for</span> t := *pt; t != <span class="constant">nil</span>; t = *pt { <span class="comment">// ①</span></div><div class="line">		<span class="comment">// 如果地址已经在列表中,则加入到这个地址的链表中</span></div><div class="line">		<span class="keyword">if</span> t.elem == unsafe.Pointer(addr) {</div><div class="line">			<span class="comment">// 如果地址已经在列表中，并且指定了先入后出flag,这是一个替换操作</span></div><div class="line">			<span class="keyword">if</span> lifo {</div><div class="line">				<span class="comment">// 替换操作</span></div><div class="line">				*pt = s</div><div class="line">				s.ticket = t.ticket</div><div class="line">				... <span class="comment">// 把t的各种信息复制给s</span></div><div class="line">			} <span class="keyword">else</span> {</div><div class="line">				<span class="comment">// 增加到到等待列表的末尾</span></div><div class="line">				<span class="keyword">if</span> t.waittail == <span class="constant">nil</span> {</div><div class="line">					t.waitlink = s</div><div class="line">				} <span class="keyword">else</span> {</div><div class="line">					t.waittail.waitlink = s</div><div class="line">				}</div><div class="line">				t.waittail = s</div><div class="line">				s.waitlink = <span class="constant">nil</span></div><div class="line">				<span class="keyword">if</span> t.waiters<span class="number">+1</span> !=<span class="number"> 0</span> {</div><div class="line">					t.waiters++</div><div class="line">				}</div><div class="line">			}</div><div class="line">			<span class="keyword">return</span></div><div class="line">		}</div><div class="line">		last = t</div><div class="line">		<span class="comment">// 二叉搜索树查找</span></div><div class="line">		<span class="keyword">if</span> <span class="typename">uintptr</span>(unsafe.Pointer(addr)) &lt; <span class="typename">uintptr</span>(t.elem) { <span class="comment">// ②</span></div><div class="line">			pt = &t.prev</div><div class="line">		} <span class="keyword">else</span> {</div><div class="line">			pt = &t.next</div><div class="line">		}</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="comment">// 为新节点设置ticket.这个ticket是一个随机值，作为随机堆的优先级，用于保持treap的平衡。</span></div><div class="line">	s.ticket = cheaprand() |<span class="number"> 1</span> <span class="comment">// ③</span></div><div class="line">	s.parent = last</div><div class="line">	*pt = s</div><div class="line"></div><div class="line">	<span class="comment">// 根据优先级(ticket)旋转以保持treap的平衡</span></div><div class="line">	<span class="keyword">for</span> s.parent != <span class="constant">nil</span> && s.parent.ticket &gt; s.ticket { <span class="comment">// ④</span></div><div class="line">		<span class="keyword">if</span> s.parent.prev == s {</div><div class="line">			root.rotateRight(s.parent) <span class="comment">// ⑤</span></div><div class="line">		} <span class="keyword">else</span> {</div><div class="line">			<span class="keyword">if</span> s.parent.next != s {</div><div class="line">				<span class="built_in">panic</span>(<span class="string">"semaRoot queue"</span>)</div><div class="line">			}</div><div class="line">			root.rotateLeft(s.parent) <span class="comment">// ⑥</span></div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>① 是遍历 treap 的过程，当然它是通过搜索二叉树的方式实现。 <code>addr</code>就是我们一开始讲的treap的key，也就是 <code>s.elem</code>。<br>首先检查 <code>addr</code> 已经在 treap 中，如果存在，那么就把 <code>s</code> 加入到 <code>addr</code> 对应的 <code>sudog</code> 链表中，或者替换掉 <code>addr</code> 对应的 <code>sudog</code>。</p>
<p>这个<code>addr</code>, 如果对于<code>sync.Mutex</code>来说，就是 <code>Mutex.sema</code>字段的地址。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Mutex <span class="keyword">struct</span> {</div><div class="line">	state <span class="typename">int32</span></div><div class="line">	sema  <span class="typename">uint32</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>所以对于阻塞在同一个<code>sync.Mutex</code>上的goroutine，他们的<code>addr</code>是相同的，所以他们会被加入到同一个<code>sudog</code>链表中。<br>如果是不同的<code>sync.Mutex</code>锁，他们的<code>addr</code>是不同的，那么他们会被加入到这个treap不同的节点。</p>
<p>进而，你可以知道，这个<code>rootSema</code>是维护多个<code>sync.Mutex</code>的等待队列的，可以快速找到不同的<code>sync.Mutex</code>的等待队列,也可以维护同一个<code>sync.Mutex</code>的等待队列。<br>这给了我们启发，如果你有类似的需求，可以参考这个实现。</p>
<p>③就是设置这个节点的优先级，它是一个随机值，用于保持treap的平衡。这里有个技巧就是总是把优先级最低位设置为1，这样保证优先级不为0.因为优先级经常和0做比较，我们将最低位设置为1，就表明优先级已经设置。</p>
<p>④ 就是将这个新加入的节点旋转到合适的位置，以保持treap的平衡。这里的旋转操作就是上面提到的左旋和右旋。稍后看。</p>
<h2 id="移除一个元素（出队）">移除一个元素（出队）</h2>
<p>对应的，还有出对的操作。这个操作就是从treap中移除一个节点，这个节点就是一个等待的goroutine(<code>sudog</code>)。</p>
<p><code>dequeue</code> 搜索并找到在<code>semaRoot</code>中第一个因<code>addr</code>而阻塞的<code>goroutine</code>。<br>比如需要唤醒一个goroutine, 让它继续执行(比如直接将锁交给它，或者唤醒它去争抢锁)。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (root *semaRoot) dequeue(addr *<span class="typename">uint32</span>) (found *sudog, now, tailtime <span class="typename">int64</span>) {</div><div class="line">	ps := &root.treap</div><div class="line">	s := *ps</div><div class="line">	<span class="keyword">for</span> ; s != <span class="constant">nil</span>; s = *ps { <span class="comment">// ①， 二叉搜索树查找</span></div><div class="line">		<span class="keyword">if</span> s.elem == unsafe.Pointer(addr) { <span class="comment">// ②</span></div><div class="line">			<span class="keyword">goto</span> Found</div><div class="line">		}</div><div class="line">		<span class="keyword">if</span> <span class="typename">uintptr</span>(unsafe.Pointer(addr)) &lt; <span class="typename">uintptr</span>(s.elem) {</div><div class="line">			ps = &s.prev</div><div class="line">		} <span class="keyword">else</span> {</div><div class="line">			ps = &s.next</div><div class="line">		}</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> <span class="constant">nil</span>,<span class="number"> 0</span>,<span class="number"> 0</span></div><div class="line"></div><div class="line">Found: <span class="comment">// ③</span></div><div class="line">	...</div><div class="line">	<span class="keyword">if</span> t := s.waitlink; t != <span class="constant">nil</span> { <span class="comment">// ④</span></div><div class="line">		*ps = t</div><div class="line">		...</div><div class="line">	} <span class="keyword">else</span> { <span class="comment">// ⑤</span></div><div class="line">		<span class="comment">// 旋转s到叶节点，以便删除</span></div><div class="line">		<span class="keyword">for</span> s.next != <span class="constant">nil</span> || s.prev != <span class="constant">nil</span> {</div><div class="line">			<span class="keyword">if</span> s.next == <span class="constant">nil</span> || s.prev != <span class="constant">nil</span> && s.prev.ticket &lt; s.next.ticket {</div><div class="line">				root.rotateRight(s)</div><div class="line">			} <span class="keyword">else</span> {</div><div class="line">				root.rotateLeft(s)</div><div class="line">			}</div><div class="line">		}</div><div class="line"></div><div class="line">		<span class="comment">// ⑤ 删除s</span></div><div class="line">		<span class="keyword">if</span> s.parent != <span class="constant">nil</span> {</div><div class="line">			<span class="keyword">if</span> s.parent.prev == s {</div><div class="line">				s.parent.prev = <span class="constant">nil</span></div><div class="line">			} <span class="keyword">else</span> {</div><div class="line">				s.parent.next = <span class="constant">nil</span></div><div class="line">			}</div><div class="line">		} <span class="keyword">else</span> {</div><div class="line">			root.treap = <span class="constant">nil</span></div><div class="line">		}</div><div class="line">		tailtime = s.acquiretime</div><div class="line">	}</div><div class="line">	... <span class="comment">// 清理s的不需要的信息</span></div><div class="line">	<span class="keyword">return</span> s, now, tailtime</div><div class="line">}</div></pre></td></tr></table></figure>

<p>① 是遍历 treap 的过程，当然它是通过搜索二叉树的方式实现。 <code>addr</code>就是我们一开始讲的treap的key，也就是 <code>s.elem</code>。如果找到了，就跳到 <code>Found</code> 标签。如果没有找到，就返回 <code>nil</code>。</p>
<p>④是检查这个地址上是不是有多个等待的goroutine，如果有，就把这个节点替换成链表中的下一个节点。把这个节点从treap中移除并返回。<br>如果就一个goroutine，那么把这个移除掉后，需要旋转treap，直到这个节点被旋转到叶节点，然后删除这个节点。</p>
<p>这里的旋转操作就是上面提到的左旋和右旋。</p>
<h2 id="左旋_rotateLeft">左旋 rotateLeft</h2>
<p><code>rotateLeft</code> 函数将以 <code>x</code> 为根的子树左旋，使其变为 <code>y</code> 为根的子树。<br>左旋之前的结构为 <code>(x a (y b c))</code>，旋转后变为 <code>(y (x a b) c)</code>。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (root *semaRoot) rotateLeft(x *sudog) {</div><div class="line">	<span class="comment">// p -&gt; (x a (y b c))</span></div><div class="line">	p := x.parent</div><div class="line">	y := x.next</div><div class="line">	b := y.prev</div><div class="line"></div><div class="line">	y.prev = x <span class="comment">// ①</span></div><div class="line">	x.parent = y <span class="comment">// ②</span></div><div class="line">	x.next = b <span class="comment">// ③</span></div><div class="line">	<span class="keyword">if</span> b != <span class="constant">nil</span> {</div><div class="line">		b.parent = x <span class="comment">// ④</span></div><div class="line">	}</div><div class="line"></div><div class="line">	y.parent = p <span class="comment">// ⑤</span></div><div class="line">	<span class="keyword">if</span> p == <span class="constant">nil</span> {</div><div class="line">		root.treap = y <span class="comment">// ⑥</span></div><div class="line">	} <span class="keyword">else</span> <span class="keyword">if</span> p.prev == x { <span class="comment">// ⑦</span></div><div class="line">		p.prev = y</div><div class="line">	} <span class="keyword">else</span> {</div><div class="line">		<span class="keyword">if</span> p.next != x {</div><div class="line">			throw(<span class="string">"semaRoot rotateLeft"</span>)</div><div class="line">		}</div><div class="line">		p.next = y</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>具体步骤：</p>
<ul>
<li>将 <code>y</code> 设为 <code>x</code> 的父节点(②)，<code>x</code> 设为 <code>y</code> 的左子节点(①)。</li>
<li>将 <code>b</code> 设为 <code>x</code> 的右子节点(③)，并更新其父节点为 <code>x</code>(④)。</li>
<li>更新 <code>y</code> 的父节点为 <code>p</code>(⑤)，即 <code>x</code> 的原父节点。如果 <code>p</code> 为 nil，则 y 成为新的树根(⑥)。</li>
<li>根据 <code>y</code> 是 <code>p</code> 的左子节点还是右子节点，更新对应的指针(⑦)。</li>
</ul>
<p><img src="Pasted-image-20241026130741.png" alt=""><br>左旋为<br><img src="Pasted-image-20241026130908.png" alt=""></p>
<h2 id="右旋_rotateRight">右旋 rotateRight</h2>
<p>rotateRight 旋转以节点 y 为根的树。<br>将 <code>(y (x a b) c)</code> 变为 <code>(x a (y b c))</code>。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (root *semaRoot) rotateRight(y *sudog) {</div><div class="line">	<span class="comment">// p -&gt; (y (x a b) c)</span></div><div class="line">	p := y.parent</div><div class="line">	x := y.prev</div><div class="line">	b := x.next</div><div class="line"></div><div class="line">	x.next = y <span class="comment">// ①</span></div><div class="line">	y.parent = x <span class="comment">// ②</span></div><div class="line">	y.prev = b <span class="comment">// ③</span></div><div class="line">	<span class="keyword">if</span> b != <span class="constant">nil</span> {</div><div class="line">		b.parent = y <span class="comment">// ④</span></div><div class="line">	}</div><div class="line"></div><div class="line">	x.parent = p <span class="comment">// ⑤</span></div><div class="line">	<span class="keyword">if</span> p == <span class="constant">nil</span> {</div><div class="line">		root.treap = x <span class="comment">// ⑥</span></div><div class="line">	} <span class="keyword">else</span> <span class="keyword">if</span> p.prev == y { <span class="comment">// ⑦</span></div><div class="line">		p.prev = x</div><div class="line">	} <span class="keyword">else</span> {</div><div class="line">		<span class="keyword">if</span> p.next != y {</div><div class="line">			throw(<span class="string">"semaRoot rotateRight"</span>)</div><div class="line">		}</div><div class="line">		p.next = x</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>具体步骤：</p>
<ul>
<li>将 y 设为 x 的右子节点(①), x 设为 y 的父节点(②)</li>
<li>将 b 设为 y 的左子节点(③)，并更新其父节点为 y(④)</li>
<li>更新 x 的父节点为 p(⑤)，即 y 的原父节点。如果 p 为 nil，则 x 成为新的树根(⑥)</li>
<li>根据 x 是 p 的左子节点还是右子节点，更新对应的指针(⑦)</li>
</ul>
<p><img src="Pasted-image-20241026132048.png" alt=""><br>右旋为<br><img src="Pasted-image-20241026132245.png" alt=""></p>
<p>理解了左旋和右旋，你就理解了出队代码中这一段为什么把当前节点旋转到叶结点中了：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// 旋转s到叶节点，以便删除</span></div><div class="line"><span class="keyword">for</span> s.next != <span class="constant">nil</span> || s.prev != <span class="constant">nil</span> {</div><div class="line">	<span class="keyword">if</span> s.next == <span class="constant">nil</span> || s.prev != <span class="constant">nil</span> && s.prev.ticket &lt; s.next.ticket {</div><div class="line">		root.rotateRight(s)</div><div class="line">	} <span class="keyword">else</span> {</div><div class="line">		root.rotateLeft(s)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>整体上看，treap这个数据结构确实简单可维护。左旋和右旋的代码量很少，结合图看起来也容易理解。 出入队的代码也很简单，只是简单的二叉搜索树的操作，加上旋转操作。</p>
<p>这是我介绍的Go秘而不宣的数据结构第三篇，希望你喜欢。你还希望看到Go运行时和标准库中的哪些数据结构呢，欢迎留言。</p>
<p>我会不定期的从关注者列表并点赞文章的同学中选出一位，送出版商和出版社的老师赠送的书，欢迎参与。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p><code>treap</code> 是一棵二叉树，它同时维护二叉搜索树 （BST） 和堆的属性, 所以由此得名 (tree + heap   ⇒  treap)。</p>
<p>从形式上讲，treap （tree + heap） 是一棵二叉树，其节点包含两个值，一个 <e]]>
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构 BitVec, 资源优化方法之位向量]]></title>
    <link href="https://colobu.com/2024/11/17/go-internal-ds-bitvec/"/>
    <id>https://colobu.com/2024/11/17/go-internal-ds-bitvec/</id>
    <published>2024-11-17T08:18:48.000Z</published>
    <updated>2024-11-17T08:58:20.629Z</updated>
    <content type="html"><![CDATA[<p>位图(bitmap)是一种优雅而高效的数据结构,它巧妙地利用了计算机最底层的位运算能力。你可以把它想象成一个巨大的开关阵列,每个开关只有打开和关闭两种状态 —— 这就是位图的本质。每一位都可以独立控制,却又可以通过位运算实现群体操作。</p>
<p>在实际应用中,位图的威力令人惊叹。设想你需要在海量数据中查找重复的数字,传统的哈希表或数组都会占用大量内存。而位图却能巧妙地用一个比特位标记一个数字的出现情况,极大地压缩了存储空间。在处理<strong>10亿个不重复的整数</strong>时,位图仅需要<strong>125MB内存</strong>,相比其他数据结构动辄需要几个GB,效率提升显著。</p>
<p>位图的运用也体现在我们日常使用的数据库系统中。数据库会用位图索引来加速查询,尤其是对于性别、状态这样的枚举字段,一个位图就能快速定位满足条件的记录。比如在电商系统中,快速筛选出&quot;在售且有库存&quot;的商品,位图索引可以通过简单的位与运算瞬间得出结果。</p>
<p>在大规模系统的权限控制中,位图也显示出其独特魅力。用户的各项权限可以编码到不同的位上,判断权限时只需一条位运算指令,既高效又直观。比如一个CMS系统,可以用一个32位的整数表示用户的全部权限状态,包括读、写、管理等多个维度。</p>
<p><strong>布隆过滤器</strong>更是位图思想的精妙应用。它用多个哈希函数在位图上标记数据,能够以极小的内存代价判断一个元素是否可能存在。这在网页爬虫、垃圾邮件过滤等场景下广泛应用。虽然可能有小概率的误判,但在实际应用中往往是可以接受的权衡。</p>
<p>正是由于以上特点,位图在处理<strong>海量数据、状态标记、数据压缩、快速统计</strong>等场景中表现出色。它用最简单的方式解决了最复杂的问题,这正是计算机科学之美的体现。</p>
<a id="more"></a>
<p><code>BitVec</code> 和 <code>BitMap</code> 类似，只是关注点有些不同。<strong>BitVec</strong>更像是位操作的抽象数据类型,它强调的是向量化的位运算操作。比如在Rust语言中, <a href="https://crates.io/crates/bitvec" target="_blank" rel="external">bitvec</a> 提供了一系列方便的接口来进行位操作。而<strong>Bitmap</strong>则更强调其作为&quot;图&quot;的特性,通常用固定大小的位数组来表示集合中元素的存在性。</p>
<p>BitVec 具有以下的优势：</p>
<ul>
<li><strong>空间效率高</strong> - 每个比特位只占用1位(bit)空间,可以表示0或1两种状态</li>
<li><strong>快速的位运算</strong> - 支持AND、OR、XOR等位运算操作,性能很高，甚至可以利用 SIMD 加速</li>
<li><strong>随机访问快</strong> - 可以O(1)时间定位到任意位置的比特位</li>
<li><strong>紧凑存储</strong> - 一个字节(byte)可以存储8个比特位的信息</li>
<li><strong>内存占用小</strong> - 对于数据量大但状态简单的场景很节省内存</li>
</ul>
<h2 id="Go_内部实现的_BitVec">Go 内部实现的 BitVec</h2>
<p>在 Go 运行时的内部， <a href="https://github.com/golang/go/blob/989eed28497cde7145958985f50bb3dd6ab698b6/src/cmd/compile/internal/bitvec/bv.go#L21" target="_blank" rel="external">cmd/compile/internal/bitvec</a> 实现了一个位向量数据结构 <code>BitVec</code>，在 ssa 活跃性分析中使用(bvecSet 封装了 BitVec)。在 <a href="https://github.com/golang/go/blob/master/src/runtime/stack.go#L595" target="_blank" rel="external">runtime/stack.go</a> 中实现了 <code>bitvector</code> 并在内存管理中使用。</p>
<p>我们重点看 <code>BitVec</code>, 它的方法比较全。</p>
<p>BitVec 的结构体定义如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> BitVec <span class="keyword">struct</span> {</div><div class="line">	N <span class="typename">int32</span>    <span class="comment">// 这个向量中包含的bit数</span></div><div class="line">	B []<span class="typename">uint32</span> <span class="comment">// 保存这些bit所需的数组</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> New(n <span class="typename">int32</span>) BitVec {</div><div class="line">	nword := (n + wordBits -<span class="number"> 1</span>) / wordBits <span class="comment">// 计算保存这些bit所需的最少的数组</span></div><div class="line">	<span class="keyword">return</span> BitVec{n, <span class="built_in">make</span>([]<span class="typename">uint32</span>, nword)}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>然后定义了一批位操作的方法：</p>
<ul>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.And" target="_blank" rel="external">func (dst BitVec) And(src1, src2 BitVec)</a> ：对两个位向量进行与操作，结果放入到 dst 位向量中</li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.AndNot" target="_blank" rel="external">func (dst BitVec) AndNot(src1, src2 BitVec)</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Clear" target="_blank" rel="external">func (bv BitVec) Clear()</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Copy" target="_blank" rel="external">func (dst BitVec) Copy(src BitVec)</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Count" target="_blank" rel="external">func (bv BitVec) Count() int</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Eq" target="_blank" rel="external">func (bv1 BitVec) Eq(bv2 BitVec) bool</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Get" target="_blank" rel="external">func (bv BitVec) Get(i int32) bool</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.IsEmpty" target="_blank" rel="external">func (bv BitVec) IsEmpty() bool</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Next" target="_blank" rel="external">func (bv BitVec) Next(i int32) int32</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Not" target="_blank" rel="external">func (bv BitVec) Not()</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Or" target="_blank" rel="external">func (dst BitVec) Or(src1, src2 BitVec)</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Set" target="_blank" rel="external">func (bv BitVec) Set(i int32)</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.String" target="_blank" rel="external">func (bv BitVec) String() string</a></li>
<li><a href="https://pkg.go.dev/cmd/compile/internal/bitvec@go1.23.2#BitVec.Unset" target="_blank" rel="external">func (bv BitVec) Unset(i int32)</a></li>
</ul>
<blockquote>
<p>这里可以看到 Go 内部实现也有一些&quot;不规范&quot;的方法，这些 Receiver 的名字不一致，叫做了 dst、bv、bv 1 三种名称，看起来是有深意的。dst 代表操作最后存储的位向量。不过 bv 1 就有点说不过去了，虽然也能理解，为了和参数中的 bv 2 保持一致。</p>
</blockquote>
<p><img src="Pasted-image-20241103115255.png" alt=""></p>
<p>我们可以挑几个方法看它的实现。</p>
<p>比如 <code>And</code> 方法:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (dst BitVec) And(src1, src2 BitVec) {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(src1.B) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	_, _ = dst.B[<span class="built_in">len</span>(src1.B<span class="number">)-1</span>], src2.B[<span class="built_in">len</span>(src1.B<span class="number">)-1</span>] <span class="comment">// hoist bounds checks out of the loop</span></div><div class="line"></div><div class="line">	<span class="keyword">for</span> i, x := <span class="keyword">range</span> src1.B {</div><div class="line">		dst.B[i] = x & src2.B[i]</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>就是求两个位向量的交集，这里用到了位运算 <code>&amp;</code>。逐个元素进行位与操作，然后存储到 dst 中。</p>
<blockquote>
<p>可以看到如果使用SIMD指令，这里的性能会有很大的提升。</p>
</blockquote>
<p>再比如<code>Not</code>方法:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (bv BitVec) Not() {</div><div class="line">	<span class="keyword">for</span> i, x := <span class="keyword">range</span> bv.B {</div><div class="line">		bv.B[i] = ^x</div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> bv.N%wordBits !=<span class="number"> 0</span> {</div><div class="line">		bv.B[<span class="built_in">len</span>(bv.B<span class="number">)-1</span>] &=<span class="number"> 1</span>&lt;&lt;<span class="typename">uint</span>(bv.N%wordBits) -<span class="number"> 1</span> <span class="comment">// clear bits past N in the last word</span></div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这里是对位向量取反，用到了位运算 <code>^</code>。然后对最后一个元素进行了特殊处理，清除了多余的位。<br>这里这一句<code>bv.B[len(bv.B)-1] &amp;= 1&lt;&lt;uint(bv.N%wordBits) - 1</code>可能难以理解，其实是为了清除最后一个元素中多余的位，这里的 <code>1&lt;&lt;uint(bv.N%wordBits) - 1</code> 就是一个掩码，用来清除多余的位。</p>
<p>再比如<code>Count</code>方法:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (bv BitVec) Count() <span class="typename">int</span> {</div><div class="line">	n :=<span class="number"> 0</span></div><div class="line">	<span class="keyword">for</span> _, x := <span class="keyword">range</span> bv.B {</div><div class="line">		n += bits.OnesCount32(x)</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> n</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这里是统计位向量中 1 的个数，用到了 <code>bits.OnesCount32</code> 方法，这个方法是一个快速计算Uint32中bit为1的个数的方法。</p>
<p>这里的实现都是比较简单的，但是在实际应用中，位向量的操作是非常高效的，可以用来解决很多问题。</p>
<p>如果你的项目中有这种需求，比如你要实现一个布隆过滤器/布谷鸟过滤器，或者你要实现一个高效的权限控制系统，那么位向量是一个非常好的选择。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>位图(bitmap)是一种优雅而高效的数据结构,它巧妙地利用了计算机最底层的位运算能力。你可以把它想象成一个巨大的开关阵列,每个开关只有打开和关闭两种状态 —— 这就是位图的本质。每一位都可以独立控制,却又可以通过位运算实现群体操作。</p>
<p>在实际应用中,位图的威力令人惊叹。设想你需要在海量数据中查找重复的数字,传统的哈希表或数组都会占用大量内存。而位图却能巧妙地用一个比特位标记一个数字的出现情况,极大地压缩了存储空间。在处理<strong>10亿个不重复的整数</strong>时,位图仅需要<strong>125MB内存</strong>,相比其他数据结构动辄需要几个GB,效率提升显著。</p>
<p>位图的运用也体现在我们日常使用的数据库系统中。数据库会用位图索引来加速查询,尤其是对于性别、状态这样的枚举字段,一个位图就能快速定位满足条件的记录。比如在电商系统中,快速筛选出&quot;在售且有库存&quot;的商品,位图索引可以通过简单的位与运算瞬间得出结果。</p>
<p>在大规模系统的权限控制中,位图也显示出其独特魅力。用户的各项权限可以编码到不同的位上,判断权限时只需一条位运算指令,既高效又直观。比如一个CMS系统,可以用一个32位的整数表示用户的全部权限状态,包括读、写、管理等多个维度。</p>
<p><strong>布隆过滤器</strong>更是位图思想的精妙应用。它用多个哈希函数在位图上标记数据,能够以极小的内存代价判断一个元素是否可能存在。这在网页爬虫、垃圾邮件过滤等场景下广泛应用。虽然可能有小概率的误判,但在实际应用中往往是可以接受的权衡。</p>
<p>正是由于以上特点,位图在处理<strong>海量数据、状态标记、数据压缩、快速统计</strong>等场景中表现出色。它用最简单的方式解决了最复杂的问题,这正是计算机科学之美的体现。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构 runq, 难怪运行时调度那么好]]></title>
    <link href="https://colobu.com/2024/10/20/go-internal-ds-runq/"/>
    <id>https://colobu.com/2024/10/20/go-internal-ds-runq/</id>
    <published>2024-10-20T04:17:47.000Z</published>
    <updated>2024-10-20T04:19:42.045Z</updated>
    <content type="html"><![CDATA[<p>首先，让我们先来回顾 Go 运行时的 GPM 模型。这方面的介绍网上的资料都非常非常多了，但是我们也不妨回顾一下：</p>
<blockquote>
<p>GPM模型中的G代表goroutine。每个goroutine只占用几KB的内存,可以轻松创建成千上万个。G包含了goroutine的栈、指令指针和其他信息,如阻塞channel的等待队列等。</p>
<p>P代表processor,可以理解为一个抽象的CPU核心。P的数量默认等于实际的CPU核心数,但可以通过环境变量进行调整。P维护了一个本地的goroutine队列,还负责执行goroutine并管理与之关联的上下文信息。</p>
<p>M代表machine,是操作系统线程。一个M必须绑定一个P才能执行goroutine。当一个M阻塞时,运行时会创建一个新的M或者复用一个空闲的M来保证P的数量总是等于GOMAXPROCS的值,从而充分利用CPU资源。</p>
<p>在这个模型中,P扮演了承上启下的角色。它连接了G和M,实现了用户层级的goroutine到操作系统线程的映射。这种设计允许Go在用户空间进行调度,避免了频繁的系统调用,大大提高了并发效率。</p>
<p>调度过程中,当一个goroutine被创建时,它会被放到P的本地队列或全局队列中。如果P的本地队列已满,一些goroutine会被放到全局队列。当P执行完当前的goroutine后,会优先从本地队列获取新的goroutine来执行。如果本地队列为空,P会尝试从全局队列或其他P的队列中偷取goroutine。</p>
<p>这种工作窃取(work-stealing)算法确保了负载的动态平衡。当某个P的本地队列为空时,它可以从其他P的队列中窃取一半的goroutine,这有效地平衡了各个P之间的工作负载。</p>
</blockquote>
<a id="more"></a>
<p><img src="gpm.png" alt=""></p>
<p>Go 运行时这么做，主要还是减少 P 之间对获取 goroutine 之间的竞争。本地队列 runq 主要由持有它的 P 进行读写，只有在&quot;被偷&quot;的情况下，才可能有&quot;数据竞争&quot;的问题，而这种情况发生概率较少，所以它设计了一个高效的 <code>runq</code> 数据结构来应对这么场景。实际看起来和上面介绍的 PoolDequeue 有异曲同工之妙。</p>
<blockquote>
<p>本文还会介绍 global queue 等数据结构，但不是本文的重点。</p>
</blockquote>
<h2 id="runq">runq</h2>
<p>在运行时中 <code>P</code> 是一个复杂的数据结构，下面列出了本文关注的它的几个字段:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// 一个goroutine的指针</span></div><div class="line"><span class="keyword">type</span> guintptr <span class="typename">uintptr</span></div><div class="line"></div><div class="line"><span class="comment">//go:nosplit</span></div><div class="line"><span class="keyword">func</span> (gp guintptr) ptr() *g { <span class="keyword">return</span> (*g)(unsafe.Pointer(gp)) }</div><div class="line"></div><div class="line"><span class="comment">//go:nosplit</span></div><div class="line"><span class="keyword">func</span> (gp *guintptr) set(g *g) { *gp = guintptr(unsafe.Pointer(g)) }</div><div class="line"></div><div class="line"><span class="comment">//go:nosplit</span></div><div class="line"><span class="keyword">func</span> (gp *guintptr) cas(old, <span class="built_in">new</span> guintptr) <span class="typename">bool</span> {</div><div class="line">	<span class="keyword">return</span> atomic.Casuintptr((*<span class="typename">uintptr</span>)(unsafe.Pointer(gp)), <span class="typename">uintptr</span>(old), <span class="typename">uintptr</span>(<span class="built_in">new</span>))</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> p <span class="keyword">struct</span> {</div><div class="line">	id          <span class="typename">int32</span></div><div class="line">	status      <span class="typename">uint32</span> <span class="comment">// one of pidle/prunning/...</span></div><div class="line">	link        puintptr</div><div class="line">	schedtick   <span class="typename">uint32</span>     <span class="comment">// incremented on every scheduler call</span></div><div class="line">	syscalltick <span class="typename">uint32</span>     <span class="comment">// incremented on every system call</span></div><div class="line">	sysmontick  sysmontick <span class="comment">// last tick observed by sysmon</span></div><div class="line">	m           muintptr   <span class="comment">// back-link to associated m (nil if idle)</span></div><div class="line">	mcache      *mcache</div><div class="line">	pcache      pageCache</div><div class="line">	raceprocctx <span class="typename">uintptr</span></div><div class="line"></div><div class="line">	deferpool    []*_defer <span class="comment">// pool of available defer structs (see panic.go)</span></div><div class="line">	deferpoolbuf <span class="number">[32</span>]*_defer</div><div class="line"></div><div class="line">	<span class="comment">// Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen.</span></div><div class="line">	goidcache    <span class="typename">uint64</span></div><div class="line">	goidcacheend <span class="typename">uint64</span></div><div class="line"></div><div class="line">	<span class="comment">// 本地运行的无锁循环队列</span></div><div class="line">	runqhead <span class="typename">uint32</span></div><div class="line">	runqtail <span class="typename">uint32</span></div><div class="line">	runq     <span class="number">[256</span>]guintptr</div><div class="line"></div><div class="line">	<span class="comment">// 如果非nil，是一个可优先运行的G</span></div><div class="line">	runnext guintptr</div><div class="line"></div><div class="line">	...</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>runq</code> 是一个无锁循环队列，由数组实现，它的长度是 256，这个长度是固定的，不会动态调整。<code>runqhead</code> 和 <code>runqtail</code> 分别是队列的头和尾，<code>runqhead</code> 指向队列的头部，<code>runqtail</code> 指向队列的尾部。<br><code>runq</code> 数组的每个元素是一个 <code>guintptr</code> 类型，它是一个 <code>uintptr</code> 类型的别名，用来存储 <code>g</code> 的指针。</p>
<p><code>runq</code> 的操作主要是 <code>runqput</code>、<code>runqputslow</code>、<code>runqputbatch</code>、<code>runqget</code>、<code>runqdrain</code>、<code>runqgrab</code>、<code>runqsteal</code>等方法。</p>
<p>接下来我们捡重点的方法看一下它是怎么实现高效额度并发读写的.</p>
<h3 id="runqput">runqput</h3>
<p><code>runqput</code> 方法是向 <code>runq</code> 中添加一个 <code>g</code> 的方法，它是一个无锁的操作，不会阻塞。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqput 尝试将 g 放到本地可运行队列上。</span></div><div class="line"><span class="comment">// 如果 next 为 false，runqput 将 g 添加到可运行队列的尾部。</span></div><div class="line"><span class="comment">// 如果 next 为 true，runqput 将 g 放在 pp.runnext 位置。</span></div><div class="line"><span class="comment">// 如果可运行队列已满，runnext 将 g 放到全局队列上。</span></div><div class="line"><span class="comment">// 只能由拥有 P 的所有者执行。</span></div><div class="line"><span class="keyword">func</span> runqput(pp *p, gp *g, next <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">if</span> !haveSysmon && next {</div><div class="line">        <span class="comment">// 如果没有 sysmon，我们必须完全避免 runnext，否则会导致饥饿。</span></div><div class="line">		next = <span class="constant">false</span></div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> randomizeScheduler && next && randn<span class="number">(2</span>) ==<span class="number"> 0</span> {</div><div class="line">        <span class="comment">// 如果随机调度器打开，我们有一半的机会避免运行 runnext</span></div><div class="line">		next = <span class="constant">false</span></div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 如果 next 为 true，优先处理 runnext</span></div><div class="line">    <span class="comment">// 将当前的goroutine放到 runnext 中, 如果原来runnext中有goroutine, 则将其放到runq中</span></div><div class="line">	<span class="keyword">if</span> next {</div><div class="line">	retryNext:</div><div class="line">		oldnext := pp.runnext</div><div class="line">		<span class="keyword">if</span> !pp.runnext.cas(oldnext, guintptr(unsafe.Pointer(gp))) {</div><div class="line">			<span class="keyword">goto</span> retryNext</div><div class="line">		}</div><div class="line">		<span class="keyword">if</span> oldnext ==<span class="number"> 0</span> {</div><div class="line">			<span class="keyword">return</span></div><div class="line">		}</div><div class="line">		<span class="comment">// Kick the old runnext out to the regular run queue.</span></div><div class="line">		gp = oldnext.ptr()</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 重点来了，将goroutine放入runq中</span></div><div class="line">retry:</div><div class="line">	h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// ①</span></div><div class="line">	t := pp.runqtail</div><div class="line">	<span class="keyword">if</span> t-h &lt; <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq)) { <span class="comment">// ② 如果队列未满</span></div><div class="line">		pp.runq[t%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))].set(gp) <span class="comment">// ③ 将goroutine放入队列</span></div><div class="line">		atomic.StoreRel(&pp.runqtail, t<span class="number">+1</span>) <span class="comment">// ④ 更新队尾</span></div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> runqputslow(pp, gp, h, t) { <span class="comment">// ⑤ 如果队列满了，调用runqputslow 尝试将goroutine放入全局队列</span></div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	<span class="comment">// 如果队列未满，上面的操作应该已经成功返回，否则重试</span></div><div class="line">	<span class="keyword">goto</span> retry</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>runqput</code> 方法的实现非常简单，它首先判断是否需要优先处理 <code>runnext</code>，如果需要，就将 <code>g</code> 放到 <code>runnext</code> 中，然后再将 <code>g</code> 放到 <code>runq</code> 中。<br><code>runq</code> 的操作是无锁的，它通过 <code>atomic</code> 包提供的原子操作来实现。<br>这里使用的内部的更精细化的原子操作，这个也是我后面专门有一篇文章来讲解的。你现在大概把①、④ 理解为<code>Load</code>、<code>Store</code>操作即可。</p>
<p>②、⑤ 分别处理本地队列未满和队列已满的情况，如果队列未满，就将 <code>g</code> 放到队列中，然后更新队尾；如果队列已满，就调用 <code>runqputslow</code> 方法，将 <code>g</code> 放到全局队列中。</p>
<p>③ 处直接将 <code>g</code> 放到队列中，这是因为只有当前的 <code>P</code> 才能操作 <code>runq</code>，所以不会有并发问题。<br>同时我们也可以看到，我们总是往尾部插入, <code>t</code>总是一直增加的， 取余操作保证了循环队列的特性。</p>
<p><code>runqputslow</code> 会把本地队列中的一半的 <code>g</code> 放到全局队列中，包括当前要放入的 <code>g</code>。一旦涉及到全局队列，就会有一定的竞争，Go运行时使用了一把锁来控制并发，所以 <code>runqputslow</code> 方法是一个慢路径，是性能的瓶颈点。</p>
<h3 id="runqputbatch">runqputbatch</h3>
<p><code>func runqputbatch(pp *p, q *gQueue, qsize int)</code> 是批量往本地队列中放入 <code>g</code> 的方法，比如它从其它 <code>P</code> 那里偷来一批 <code>g</code> ，需要放到本地队列中，就会调用这个方法。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqputbatch 尝试将 q 上的所有 G 放到本地可运行队列上。</span></div><div class="line"><span class="comment">// 如果队列已满，它们将被放到全局队列上；在这种情况下，这将暂时获取调度器锁。</span></div><div class="line"><span class="comment">// 只能由拥有 P 的所有者执行。</span></div><div class="line"><span class="keyword">func</span> runqputbatch(pp *p, q *gQueue, qsize <span class="typename">int</span>) {</div><div class="line">	h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// ①</span></div><div class="line">	t := pp.runqtail</div><div class="line">	n := <span class="typename">uint32</span><span class="number">(0</span>)</div><div class="line">	<span class="keyword">for</span> !q.empty() && t-h &lt; <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq)) { <span class="comment">// ② 放入的批量goroutine非空， 并且本地队列还足以放入</span></div><div class="line">		gp := q.pop()</div><div class="line">		pp.runq[t%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))].set(gp)</div><div class="line">		t++</div><div class="line">		n++</div><div class="line">	}</div><div class="line">	qsize -= <span class="typename">int</span>(n)</div><div class="line"></div><div class="line">	<span class="keyword">if</span> randomizeScheduler { <span class="comment">// ③ 随机调度器, 随机打乱</span></div><div class="line">		off := <span class="keyword">func</span>(o <span class="typename">uint32</span>) <span class="typename">uint32</span> {</div><div class="line">			<span class="keyword">return</span> (pp.runqtail + o) % <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))</div><div class="line">		}</div><div class="line">		<span class="keyword">for</span> i := <span class="typename">uint32</span><span class="number">(1</span>); i &lt; n; i++ {</div><div class="line">			j := cheaprandn(i +<span class="number"> 1</span>)</div><div class="line">			pp.runq[off(i)], pp.runq[off(j)] = pp.runq[off(j)], pp.runq[off(i)]</div><div class="line">		}</div><div class="line">	}</div><div class="line"></div><div class="line">	atomic.StoreRel(&pp.runqtail, t) <span class="comment">// ④ 更新队尾</span></div><div class="line">	<span class="keyword">if</span> !q.empty() {</div><div class="line">		lock(&sched.lock)</div><div class="line">		globrunqputbatch(q, <span class="typename">int32</span>(qsize))</div><div class="line">		unlock(&sched.lock)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>①获取队列头,使用原子操作获取队头。</p>
<blockquote>
<p>它下面一行是获取队尾的值，你可以思考下为什么不需要使用<code>atomic.LoadAcq</code>。</p>
</blockquote>
<p>② 逐个的将 <code>g</code> 放到队列中，直到放完或者放满。</p>
<p>如果是随机调度器，则使用混淆算法将队列中的 <code>g</code> 随机打乱。</p>
<p>最后如果队列还有剩余的 <code>g</code>，则调用 <code>globrunqputbatch</code> 方法，将剩余的 <code>g</code> 放到全局队列中。</p>
<h3 id="runqget">runqget</h3>
<p><code>runqget</code> 方法是从 <code>runq</code> 中获取一个 <code>g</code> 的方法，它是一个无锁的操作，不会阻塞。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqget 从本地可运行队列中获取一个 G。</span></div><div class="line"><span class="comment">// 如果 inheritTime 为 true，gp 应该继承当前时间片的剩余时间。</span></div><div class="line"><span class="comment">// 否则，它应该开始一个新的时间片。</span></div><div class="line"><span class="comment">// 只能由拥有 P 的所有者执行。</span></div><div class="line"><span class="keyword">func</span> runqget(pp *p) (gp *g, inheritTime <span class="typename">bool</span>) {</div><div class="line">	next := pp.runnext</div><div class="line">    <span class="comment">// 如果有 runnext，优先处理 runnext</span></div><div class="line">	<span class="keyword">if</span> next !=<span class="number"> 0</span> && pp.runnext.cas(next,<span class="number"> 0</span>) { <span class="comment">// ①</span></div><div class="line">		<span class="keyword">return</span> next.ptr(), <span class="constant">true</span></div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// ② 获取队头</span></div><div class="line">		t := pp.runqtail</div><div class="line">		<span class="keyword">if</span> t == h { <span class="comment">// ③ 队列为空</span></div><div class="line">			<span class="keyword">return</span> <span class="constant">nil</span>, <span class="constant">false</span></div><div class="line">		}</div><div class="line">		gp := pp.runq[h%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))].ptr() <span class="comment">// ④ 获取队头的goroutine</span></div><div class="line">		<span class="keyword">if</span> atomic.CasRel(&pp.runqhead, h, h<span class="number">+1</span>) { <span class="comment">// ⑤ 更新队头</span></div><div class="line">			<span class="keyword">return</span> gp, <span class="constant">false</span></div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>① 如果有 <code>runnext</code>，则优先处理 <code>runnext</code>，将 <code>runnext</code> 中的 <code>g</code> 取出来。</p>
<p>② 获取队列头。 如果 ③ 队列为空，直接返回。</p>
<p>④ 获取队头的 <code>g</code>，这就是要读取的 <code>g</code>。</p>
<p>⑤ 更新队头，这里使用的是 <code>atomic.CasRel</code> 方法，它是一个原子的 <code>Compare-And-Swap</code> 操作，用来更新队头。</p>
<p>可以看到这里只使用到了队列头<code>runqhead</code>。</p>
<h3 id="runqdrain">runqdrain</h3>
<p><code>runqdrain</code> 方法是从 <code>runq</code> 中获取所有的 <code>g</code> 的方法，它是一个无锁的操作，不会阻塞。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqdrain 从 pp 的本地可运行队列中获取所有的 G 并返回。</span></div><div class="line"><span class="comment">// 只能由拥有 P 的所有者执行。</span></div><div class="line"><span class="keyword">func</span> runqdrain(pp *p) (drainQ gQueue, n <span class="typename">uint32</span>) {</div><div class="line">	oldNext := pp.runnext</div><div class="line">	<span class="keyword">if</span> oldNext !=<span class="number"> 0</span> && pp.runnext.cas(oldNext,<span class="number"> 0</span>) {</div><div class="line">		drainQ.pushBack(oldNext.ptr()) <span class="comment">// ① 将 runnext 中的goroutine放入队列</span></div><div class="line">		n++</div><div class="line">	}</div><div class="line"></div><div class="line">retry:</div><div class="line">	h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// ② 获取队头</span></div><div class="line">	t := pp.runqtail</div><div class="line">	qn := t - h</div><div class="line">	<span class="keyword">if</span> qn ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> qn &gt; <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq)) { <span class="comment">// ③ 居然超出队列的长度了？</span></div><div class="line">		<span class="keyword">goto</span> retry</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">if</span> !atomic.CasRel(&pp.runqhead, h, h+qn) { <span class="comment">// ④ 更新队头</span></div><div class="line">		<span class="keyword">goto</span> retry</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// ⑤ 将队列中的goroutine放入队列drainQ中</span></div><div class="line">	<span class="keyword">for</span> i := <span class="typename">uint32</span><span class="number">(0</span>); i &lt; qn; i++ {</div><div class="line">		gp := pp.runq[(h+i)%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))].ptr()</div><div class="line">		drainQ.pushBack(gp)</div><div class="line">		n++</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span></div><div class="line">}</div></pre></td></tr></table></figure>

<h3 id="runqgrab">runqgrab</h3>
<p><code>runqgrab</code> 方法是从 <code>runq</code> 中获取一半的 <code>g</code> 的方法，它是一个无锁的操作，不会阻塞。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqgrab 从 pp 的本地可运行队列中获取一半的 G 并返回。</span></div><div class="line"><span class="comment">// Batch 是一个环形缓冲区，从 batchHead 开始。</span></div><div class="line"><span class="comment">// 返回获取的 goroutine 数量。</span></div><div class="line"><span class="comment">// 可以由任何 P 执行。</span></div><div class="line"><span class="keyword">func</span> runqgrab(pp *p, batch *<span class="number">[256</span>]guintptr, batchHead <span class="typename">uint32</span>, stealRunNextG <span class="typename">bool</span>) <span class="typename">uint32</span> {</div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// load-acquire, synchronize with other consumers</span></div><div class="line">		t := atomic.LoadAcq(&pp.runqtail) <span class="comment">// load-acquire, synchronize with the producer</span></div><div class="line">		n := t - h</div><div class="line">		n = n - n<span class="number">/2</span> <span class="comment">// ① 取一半的goroutine</span></div><div class="line">		<span class="keyword">if</span> n ==<span class="number"> 0</span> {</div><div class="line">			<span class="keyword">if</span> stealRunNextG {</div><div class="line">                <span class="comment">// ② 如果要偷取runnext中的goroutine</span></div><div class="line">				<span class="keyword">if</span> next := pp.runnext; next !=<span class="number"> 0</span> {</div><div class="line">					<span class="keyword">if</span> pp.status == _Prunning {</div><div class="line">                        <span class="comment">// ② 如果要偷取runnext中的goroutine，这里会sleep一会</span></div><div class="line">						<span class="keyword">if</span> !osHasLowResTimer {</div><div class="line">							usleep<span class="number">(3</span>)</div><div class="line">						} <span class="keyword">else</span> {</div><div class="line">							osyield()</div><div class="line">						}</div><div class="line">					}</div><div class="line">					<span class="keyword">if</span> !pp.runnext.cas(next,<span class="number"> 0</span>) {</div><div class="line">						<span class="keyword">continue</span></div><div class="line">					}</div><div class="line">					batch[batchHead%<span class="typename">uint32</span>(<span class="built_in">len</span>(batch))] = next</div><div class="line">					<span class="keyword">return</span><span class="number"> 1</span></div><div class="line">				}</div><div class="line">			}</div><div class="line">			<span class="keyword">return</span><span class="number"> 0</span></div><div class="line">		}</div><div class="line">		<span class="keyword">if</span> n &gt; <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq)<span class="number">/2</span>) { <span class="comment">// ③ 如果要偷取的goroutine数量超过一半, 重试</span></div><div class="line">			<span class="keyword">continue</span></div><div class="line">		}</div><div class="line"></div><div class="line">        <span class="comment">// ④ 将队列中至多一半的goroutine放入batch中</span></div><div class="line">		<span class="keyword">for</span> i := <span class="typename">uint32</span><span class="number">(0</span>); i &lt; n; i++ {</div><div class="line">			g := pp.runq[(h+i)%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))]</div><div class="line">			batch[(batchHead+i)%<span class="typename">uint32</span>(<span class="built_in">len</span>(batch))] = g</div><div class="line">		}</div><div class="line">		<span class="keyword">if</span> atomic.CasRel(&pp.runqhead, h, h+n) { <span class="comment">// ⑤ 更新队头</span></div><div class="line">			<span class="keyword">return</span> n</div><div class="line">		}</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>① 取一半的 <code>g</code>，这里是一个简单的算法，取一半的 <code>g</code>。</p>
<p>② 如果要偷取 <code>runnext</code> 中的 <code>g</code>，则会尝试偷取 <code>runnext</code> 中的 <code>g</code>。</p>
<p>③ 如果要偷取的 <code>g</code> 数量超过一半，则重试。</p>
<p>④ 将队列中至多一半的 <code>g</code> 放入 <code>batch</code> 中。</p>
<p>⑤ 更新队头，这里使用的是 <code>atomic.CasRel</code> 方法，它是一个原子的 <code>Compare-And-Swap</code> 操作，用来更新队头。</p>
<h3 id="runqsteal">runqsteal</h3>
<p><code>runqsteal</code> 方法是从其它 <code>P</code> 的 <code>runq</code> 中偷取 <code>g</code> 的方法，它是一个无锁的操作，不会阻塞。它的实现如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// runqsteal 从 p2 的本地可运行队列中偷取一半的 G 并返回。</span></div><div class="line"><span class="comment">// 如果 stealRunNextG 为 true，它还会尝试偷取 runnext 中的 G。</span></div><div class="line"><span class="keyword">func</span> runqsteal(pp, p2 *p, stealRunNextG <span class="typename">bool</span>) *g {</div><div class="line">	t := pp.runqtail</div><div class="line">	n := runqgrab(p2, &pp.runq, t, stealRunNextG) <span class="comment">// ① 从p2中偷取一半的goroutine</span></div><div class="line">	<span class="keyword">if</span> n ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	}</div><div class="line">	n--</div><div class="line">	gp := pp.runq[(t+n)%<span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq))].ptr() <span class="comment">// ② 获取偷取的一个goroutine</span></div><div class="line">	<span class="keyword">if</span> n ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> gp</div><div class="line">	}</div><div class="line">	h := atomic.LoadAcq(&pp.runqhead) <span class="comment">// ③ 获取队头</span></div><div class="line">	<span class="keyword">if</span> t-h+n &gt;= <span class="typename">uint32</span>(<span class="built_in">len</span>(pp.runq)) { <span class="comment">// ④ 如果队列满了，重置队列</span></div><div class="line">		throw(<span class="string">"runqsteal: runq overflow"</span>)</div><div class="line">	}</div><div class="line">	atomic.StoreRel(&pp.runqtail, t+n) <span class="comment">// ⑤ 更新队尾</span></div><div class="line">	<span class="keyword">return</span> gp</div><div class="line">}</div></pre></td></tr></table></figure>

<p>它实际使用了 <code>runqgrab</code> 方法来偷取 <code>g</code>，然后再从 <code>runq</code> 中取出一个 <code>g</code>。</p>
<p>以上就是<code>runq</code>的主要操作，它针对Go调度器的特点，设计了一套特定的队列操作的函数，这些函数都是无锁的，不会阻塞，保证了高效的并发读写。</p>
<h2 id="gQueue_和_gList"><code>gQueue</code> 和 <code>gList</code></h2>
<p><code>gQueue</code> 和 <code>gList</code> 是 Go 运行时中的两个队列，它们都是用来存储 <code>g</code> 的，但是它们的实现方式不同。</p>
<p><code>gQueue</code>是一个G的双端队列，可以从首尾增加gp, 通过g.schedlink链接。一个G只能在一个gQueue或gList上。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> gQueue <span class="keyword">struct</span> {</div><div class="line">	head guintptr</div><div class="line">	tail guintptr</div><div class="line">}</div><div class="line"><span class="keyword">func</span> (q *gQueue) empty() <span class="typename">bool</span> {</div><div class="line">	<span class="keyword">return</span> q.head ==<span class="number"> 0</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// push 将gp添加到q的头部。</span></div><div class="line"><span class="keyword">func</span> (q *gQueue) push(gp *g) {</div><div class="line">	gp.schedlink = q.head</div><div class="line">	q.head.set(gp)</div><div class="line">	<span class="keyword">if</span> q.tail ==<span class="number"> 0</span> {</div><div class="line">		q.tail.set(gp)</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// pushBack 增加gp到q的尾部。</span></div><div class="line"><span class="keyword">func</span> (q *gQueue) pushBack(gp *g) {</div><div class="line">	gp.schedlink =<span class="number"> 0</span></div><div class="line">	<span class="keyword">if</span> q.tail !=<span class="number"> 0</span> {</div><div class="line">		q.tail.ptr().schedlink.set(gp)</div><div class="line">	} <span class="keyword">else</span> {</div><div class="line">		q.head.set(gp)</div><div class="line">	}</div><div class="line">	q.tail.set(gp)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// q2的所有G添加到q的尾部。之后不能再使用q2。</span></div><div class="line"><span class="keyword">func</span> (q *gQueue) pushBackAll(q2 gQueue) {</div><div class="line">	<span class="keyword">if</span> q2.tail ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span></div><div class="line">	}</div><div class="line">	q2.tail.ptr().schedlink =<span class="number"> 0</span></div><div class="line">	<span class="keyword">if</span> q.tail !=<span class="number"> 0</span> {</div><div class="line">		q.tail.ptr().schedlink = q2.head</div><div class="line">	} <span class="keyword">else</span> {</div><div class="line">		q.head = q2.head</div><div class="line">	}</div><div class="line">	q.tail = q2.tail</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// pop 移除并返回队列q的头部。如果q为空，则返回nil。</span></div><div class="line"><span class="keyword">func</span> (q *gQueue) pop() *g {</div><div class="line">	gp := q.head.ptr()</div><div class="line">	<span class="keyword">if</span> gp != <span class="constant">nil</span> {</div><div class="line">		q.head = gp.schedlink</div><div class="line">		<span class="keyword">if</span> q.head ==<span class="number"> 0</span> {</div><div class="line">			q.tail =<span class="number"> 0</span></div><div class="line">		}</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> gp</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// popList 将所有的元素从队列q中取出并返回一个gList。</span></div><div class="line"><span class="keyword">func</span> (q *gQueue) popList() gList {</div><div class="line">	stack := gList{q.head}</div><div class="line">	*q = gQueue{}</div><div class="line">	<span class="keyword">return</span> stack</div><div class="line">}</div></pre></td></tr></table></figure>

<p>而<code>gList</code>是一个G的链表，通过g.schedlink链接。一个G只能在一个gQueue或gList上。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> gList <span class="keyword">struct</span> {</div><div class="line">	head guintptr</div><div class="line">}</div><div class="line"><span class="keyword">func</span> (l *gList) empty() <span class="typename">bool</span> {</div><div class="line">	<span class="keyword">return</span> l.head ==<span class="number"> 0</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// push 将gp添加到l的头部。</span></div><div class="line"><span class="keyword">func</span> (l *gList) push(gp *g) {</div><div class="line">	gp.schedlink = l.head</div><div class="line">	l.head.set(gp)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// pushAll 将q中的所有G添加到l的头部。</span></div><div class="line"><span class="keyword">func</span> (l *gList) pushAll(q gQueue) {</div><div class="line">	<span class="keyword">if</span> !q.empty() {</div><div class="line">		q.tail.ptr().schedlink = l.head</div><div class="line">		l.head = q.head</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// pop 移除并返回l的头部。如果l为空，则返回nil。</span></div><div class="line"><span class="keyword">func</span> (l *gList) pop() *g {</div><div class="line">	gp := l.head.ptr()</div><div class="line">	<span class="keyword">if</span> gp != <span class="constant">nil</span> {</div><div class="line">		l.head = gp.schedlink</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> gp</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这是常规的数据结构中链表的实现，你可以和教科书中的介绍和实现做对比，看看书本中的内容如何应用到显示的工程中的。</p>
<h2 id="global_runq">global runq</h2>
<p>一个全局的<code>runq</code>用来处理太多的goroutine, 在本地<code>runq</code>中的goroutine太少的情况下，从全局队列中偷取goroutine。<br>主要用来处理P中goroutine不均的情况。</p>
<p>因为它直接使用一把锁(<code>sched.lock</code>)，而不是lock-free的数据结构，所以代码阅读和理解起来会相对简单一些。这里就不详细介绍了</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div></pre></td><td class="code"><pre><div class="line"></div><div class="line"><span class="keyword">var</span> (</div><div class="line">	sched      schedt</div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">type</span> schedt <span class="keyword">struct</span> {</div><div class="line">	...</div><div class="line">	<span class="comment">// Global runnable queue.</span></div><div class="line">	runq     gQueue</div><div class="line">	runqsize <span class="typename">int32</span></div><div class="line">    ...</div><div class="line">}</div><div class="line"><span class="keyword">func</span> globrunqput(gp *g) {</div><div class="line">	assertLockHeld(&sched.lock) <span class="comment">// 保证锁被持有</span></div><div class="line"></div><div class="line">	sched.runq.pushBack(gp)</div><div class="line">	sched.runqsize++</div><div class="line">}</div><div class="line"></div><div class="line"></div><div class="line"><span class="keyword">func</span> globrunqputhead(gp *g) {</div><div class="line">	assertLockHeld(&sched.lock) <span class="comment">// 保证锁被持有</span></div><div class="line"></div><div class="line">	sched.runq.push(gp)</div><div class="line">	sched.runqsize++</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> globrunqputbatch(batch *gQueue, n <span class="typename">int32</span>) {</div><div class="line">	assertLockHeld(&sched.lock) <span class="comment">// 保证锁被持有</span></div><div class="line"></div><div class="line">	sched.runq.pushBackAll(*batch)</div><div class="line">	sched.runqsize += n</div><div class="line">	*batch = gQueue{}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> globrunqget(pp *p, max <span class="typename">int32</span>) *g {</div><div class="line">	assertLockHeld(&sched.lock) <span class="comment">// 保证锁被持有</span></div><div class="line"></div><div class="line">	<span class="keyword">if</span> sched.runqsize ==<span class="number"> 0</span> { <span class="comment">// 如果全局队列为空</span></div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	}</div><div class="line"></div><div class="line">	n := sched.runqsize/gomaxprocs +<span class="number"> 1</span> <span class="comment">// 从全局队列中获取goroutine的数量</span></div><div class="line">	<span class="keyword">if</span> n &gt; sched.runqsize {</div><div class="line">		n = sched.runqsize</div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> max &gt;<span class="number"> 0</span> && n &gt; max { <span class="comment">// 如果max大于0，取最小值</span></div><div class="line">		n = max</div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> n &gt; <span class="typename">int32</span>(<span class="built_in">len</span>(pp.runq))<span class="number">/2</span> { <span class="comment">// 如果要获取的goroutine数量超过一半，只取一半，不贪婪</span></div><div class="line">		n = <span class="typename">int32</span>(<span class="built_in">len</span>(pp.runq)) /<span class="number"> 2</span></div><div class="line">	}</div><div class="line"></div><div class="line">	sched.runqsize -= n</div><div class="line"></div><div class="line">	gp := sched.runq.pop() <span class="comment">// 从全局队列中获取一个goroutine</span></div><div class="line">	n--</div><div class="line">	<span class="keyword">for</span> ; n &gt;<span class="number"> 0</span>; n-- { <span class="comment">// 从全局队列中获取n-1个goroutine</span></div><div class="line">		gp1 := sched.runq.pop()</div><div class="line">		runqput(pp, gp1, <span class="constant">false</span>) <span class="comment">// 将goroutine放入本地队列</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> gp <span class="comment">// 返回获取的goroutine</span></div><div class="line">}</div></pre></td></tr></table></figure>

]]></content>
    <summary type="html">
    <![CDATA[<p>首先，让我们先来回顾 Go 运行时的 GPM 模型。这方面的介绍网上的资料都非常非常多了，但是我们也不妨回顾一下：</p>
<blockquote>
<p>GPM模型中的G代表goroutine。每个goroutine只占用几KB的内存,可以轻松创建成千上万个。G包含了goroutine的栈、指令指针和其他信息,如阻塞channel的等待队列等。</p>
<p>P代表processor,可以理解为一个抽象的CPU核心。P的数量默认等于实际的CPU核心数,但可以通过环境变量进行调整。P维护了一个本地的goroutine队列,还负责执行goroutine并管理与之关联的上下文信息。</p>
<p>M代表machine,是操作系统线程。一个M必须绑定一个P才能执行goroutine。当一个M阻塞时,运行时会创建一个新的M或者复用一个空闲的M来保证P的数量总是等于GOMAXPROCS的值,从而充分利用CPU资源。</p>
<p>在这个模型中,P扮演了承上启下的角色。它连接了G和M,实现了用户层级的goroutine到操作系统线程的映射。这种设计允许Go在用户空间进行调度,避免了频繁的系统调用,大大提高了并发效率。</p>
<p>调度过程中,当一个goroutine被创建时,它会被放到P的本地队列或全局队列中。如果P的本地队列已满,一些goroutine会被放到全局队列。当P执行完当前的goroutine后,会优先从本地队列获取新的goroutine来执行。如果本地队列为空,P会尝试从全局队列或其他P的队列中偷取goroutine。</p>
<p>这种工作窃取(work-stealing)算法确保了负载的动态平衡。当某个P的本地队列为空时,它可以从其他P的队列中窃取一半的goroutine,这有效地平衡了各个P之间的工作负载。</p>
</blockquote>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go中秘而不宣的数据结构 spmc, 10倍性能于 channel]]></title>
    <link href="https://colobu.com/2024/10/20/go-internal-ds-spmc/"/>
    <id>https://colobu.com/2024/10/20/go-internal-ds-spmc/</id>
    <published>2024-10-20T04:14:44.000Z</published>
    <updated>2024-10-20T04:19:51.828Z</updated>
    <content type="html"><![CDATA[<p>Go 标准库和运行中中，有一些专门针对特定场景优化的数据结构，这些数据结构并没有暴露出来，这个系列就是逐一介绍这些数据结构。</p>
<p>这一次给大家介绍的就是一个 lock-free、高性能的单生产者多消费者的队列：<code>PoolDequeue</code> 和 <code>PoolChain</code>。<br>到底是一个还是两个呢？<br>主要是 <code>PoolDequeue</code>, 它是一个固定尺寸，使用 ringbuffer (环形队列) 方式实现的队列。<br><code>PoolChain</code> 是在它的基础上上，实现的一个动态尺寸的队列。</p>
<a id="more"></a>
<p>生产者消费者模式是常见的一种并发模式，根据生产者的数量和消费者的数量，可以分为四种情况：</p>
<ul>
<li>单生产者-单消费者模式: spsc</li>
<li>单生产者-多消费者模式: spmc</li>
<li>多生产者-单消费者模式: mpsc</li>
<li>多生产者-多消费者模式: mpmc</li>
</ul>
<p>Channel 基本上可以看做是一种多生产者多消费者模式的队列。可以同时允许多个生产者发送数据，有可以允许多个消费者消费数据，它也可以应用在其他模式的场景，比如 rpc 包中的 oneshot 模式、通知情况下的的单生产者多消费者模式、rpc 和服务端单连接通讯时的消息处理，就是多生产者单消费者模式。</p>
<p>但是 Go 标准库的 sync 包下，有一个针对单生产者多消费者的数据结构，它是一个 lock-free 的数据结构，针对这个场景做了优化，被使用在 <code>sync.Pool</code> 中。</p>
<p><code>sync.Pool</code> 采用了一种类似 Go 运行时调度的机制，针对每个 p 有一个 <code>private</code> 的数据，同时还有一个 <code>shared</code> 的数据，如果在本地 <code>private</code>、<code>shared</code> 中没有数据，就去其他 P 对应的 <code>shared</code> 去偷取。难么同时可能有多个 P 偷取同一个 <code>shared</code>, 这是多消费者。</p>
<p>同时对 <code>shared</code> 的写只有它隶属的 p 执行 <code>Put</code> 的时候才会发生：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line">l, _ := p.pin()</div><div class="line"><span class="keyword">if</span> l.private == <span class="constant">nil</span> {</div><div class="line">	l.private = x</div><div class="line">} <span class="keyword">else</span> {</div><div class="line">	l.shared.pushHead(x)</div><div class="line">}</div><div class="line">runtime_procUnpin()</div></pre></td></tr></table></figure>

<p>这有属于单生产者模式。<code>sync.Pool</code> 使用了 <code>PoolDequeue</code> 和 <code>PoolChain</code> 来做优化。</p>
<p>首先我们先来了解 <code>poolDequeue</code>。</p>
<h2 id="poolDequeue">poolDequeue</h2>
<p><code>poolDequeue</code> 是一个 lock-free 的数据结构，必然会使用 <code>atomic</code>, 同时它要求必须使用单生产者，否则会有并发问题。消费者可以是并发多个，当然你用一个也没问题。</p>
<p>其中，生产者可以使用下面的方法：</p>
<ul>
<li><strong>pushHead</strong>: 在队列头部新增加一个数据。如果队列满了，增加失败</li>
<li><strong>popHead</strong>： 在队列头部弹出一个数据。生产者总是弹出新增加的数据，除非队列为空</li>
</ul>
<p>消费者可以使用下面的一个方法：</p>
<ul>
<li><strong>popTail</strong>: 从队尾处弹出一个数据，除非队列为空。所以消费者总是消费最老的数据，这也正好符合大部分的场景</li>
</ul>
<p>接下来就是分析代码了，有点枯燥，你可以跳过。</p>
<h3 id="代码分析">代码分析</h3>
<p>首先我们看这个struct的定义：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> poolDequeue <span class="keyword">struct</span> {</div><div class="line">	headTail atomic.Uint64</div><div class="line">	vals []eface</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这里有两个重要的字段：</p>
<ul>
<li><code>headTail</code>： 一个 <code>atomic.Uint64</code> 类型的字段，它的高 32 位是 <code>head</code>，低 32 位是 <code>tail</code>。<code>head</code> 是下一个要填充的位置，<code>tail</code> 是最老的数据的位置。</li>
<li><code>vals</code>： 一个 <code>eface</code> 类型的切片，它是一个环形队列，大小必须是 2 的幂次方。</li>
</ul>
<p>生产者增加数据的逻辑如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (d *poolDequeue) pushHead(val any) <span class="typename">bool</span> {</div><div class="line">	ptrs := d.headTail.Load()</div><div class="line">	head, tail := d.unpack(ptrs)</div><div class="line">	<span class="keyword">if</span> (tail+<span class="typename">uint32</span>(<span class="built_in">len</span>(d.vals)))&<span class="number">(1</span>&lt;&lt;dequeueBits<span class="number">-1</span>) == head {</div><div class="line">		<span class="comment">// 队列满</span></div><div class="line">		<span class="keyword">return</span> <span class="constant">false</span></div><div class="line">	}</div><div class="line">	slot := &d.vals[head&<span class="typename">uint32</span>(<span class="built_in">len</span>(d.vals<span class="number">)-1</span>)]</div><div class="line"></div><div class="line">    <span class="comment">// 检查 head slot 是否被 popTail 释放</span></div><div class="line">	typ := atomic.LoadPointer(&slot.typ)</div><div class="line">	<span class="keyword">if</span> typ != <span class="constant">nil</span> {</div><div class="line">        <span class="comment">// 另一个 goroutine 正在清理 tail，所以队列还是满的</span></div><div class="line">		<span class="keyword">return</span> <span class="constant">false</span></div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 如果值为空，那么设置一个特殊值</span></div><div class="line">	<span class="keyword">if</span> val == <span class="constant">nil</span> {</div><div class="line">		val = dequeueNil(<span class="constant">nil</span>)</div><div class="line">	}</div><div class="line">    <span class="comment">// 队列头是空的，将数据写入 slot</span></div><div class="line">	*(*any)(unsafe.Pointer(slot)) = val <span class="comment">// ①</span></div><div class="line"></div><div class="line">    <span class="comment">// 增加 head，这样 popTail 就可以消费这个 slot 了</span></div><div class="line">    <span class="comment">// 同时也是一个 store barrier，保证了 slot 的写入</span></div><div class="line">	d.headTail.Add<span class="number">(1</span> &lt;&lt; dequeueBits)</div><div class="line">	<span class="keyword">return</span> <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>① 处会有并发问题吗？万一有两个 goroutine 同时执行到这里，会不会有问题？这里没有问题，因为要求只有一个生产者，不会有另外一个goroutine同时写这个槽位。</p>
<p>注意它还实现了<code>pack</code>和<code>unpack</code>方法，用于将 <code>head</code> 和 <code>tail</code> 打包到一个 <code>uint64</code> 中，或者从 <code>uint64</code> 中解包出 <code>head</code> 和 <code>tail</code>。</p>
<p>消费者消费数据的逻辑如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (d *poolDequeue) popTail() (any, <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">var</span> slot *eface</div><div class="line">	<span class="keyword">for</span> { <span class="comment">// ②</span></div><div class="line">		ptrs := d.headTail.Load()</div><div class="line">		head, tail := d.unpack(ptrs)</div><div class="line">		<span class="keyword">if</span> tail == head {</div><div class="line">			<span class="comment">// 队列为空</span></div><div class="line">			<span class="keyword">return</span> <span class="constant">nil</span>, <span class="constant">false</span></div><div class="line">		}</div><div class="line"></div><div class="line">        <span class="comment">// 确认头部和尾部（用于我们之前的推测性检查），并递增尾部。如果成功，那么我们就拥有了尾部的插槽。</span></div><div class="line">		ptrs2 := d.pack(head, tail<span class="number">+1</span>)</div><div class="line">		<span class="keyword">if</span> d.headTail.CompareAndSwap(ptrs, ptrs2) {</div><div class="line">			<span class="comment">// 成功读取了一个 slot</span></div><div class="line">			slot = &d.vals[tail&<span class="typename">uint32</span>(<span class="built_in">len</span>(d.vals<span class="number">)-1</span>)]</div><div class="line">			<span class="keyword">break</span></div><div class="line">		}</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="comment">// 剩下来就是读取槽位的值</span></div><div class="line">	val := *(*any)(unsafe.Pointer(slot))</div><div class="line">	<span class="keyword">if</span> val == dequeueNil(<span class="constant">nil</span>) { <span class="comment">// 如果本身就存储的nil</span></div><div class="line">		val = <span class="constant">nil</span></div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 释放 slot，这样 pushHead 就可以继续写入这个 slot 了</span></div><div class="line">	slot.val = <span class="constant">nil</span> <span class="comment">// ③</span></div><div class="line">	atomic.StorePointer(&slot.typ, <span class="constant">nil</span>) <span class="comment">// ④</span></div><div class="line">	</div><div class="line">	<span class="keyword">return</span> val, <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>② 处是一个 for 循环，这是一个自旋的过程，直到成功读取到一个 slot 为止。在有大量的goroutine的时候，这里可能会是一个瓶颈点，但是少量的消费者应该还不算大问题。</p>
<p>③ 和 ④ 处是释放 slot 的过程，这样生产者就可以继续写入这个 slot 了。</p>
<p>生产者还可以调用<code>popHead</code>方法，用来弹出刚刚压入还没有消费的数据:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (d *poolDequeue) popHead() (any, <span class="typename">bool</span>) {</div><div class="line">	<span class="keyword">var</span> slot *eface</div><div class="line">	<span class="keyword">for</span> {</div><div class="line">		ptrs := d.headTail.Load()</div><div class="line">		head, tail := d.unpack(ptrs)</div><div class="line">		<span class="keyword">if</span> tail == head {</div><div class="line">			<span class="comment">// 队列为空</span></div><div class="line">			<span class="keyword">return</span> <span class="constant">nil</span>, <span class="constant">false</span></div><div class="line">		}</div><div class="line"></div><div class="line">        <span class="comment">// 确认头部和尾部（用于我们之前的推测性检查），并递减头部。如果成功，那么我们就拥有了头部的插槽。</span></div><div class="line">		head--</div><div class="line">		ptrs2 := d.pack(head, tail)</div><div class="line">		<span class="keyword">if</span> d.headTail.CompareAndSwap(ptrs, ptrs2) {</div><div class="line">            <span class="comment">// 成功取回了一个 slot</span></div><div class="line">			slot = &d.vals[head&<span class="typename">uint32</span>(<span class="built_in">len</span>(d.vals<span class="number">)-1</span>)]</div><div class="line">			<span class="keyword">break</span></div><div class="line">		}</div><div class="line">	}</div><div class="line"></div><div class="line">	val := *(*any)(unsafe.Pointer(slot))</div><div class="line">	<span class="keyword">if</span> val == dequeueNil(<span class="constant">nil</span>) {</div><div class="line">		val = <span class="constant">nil</span></div><div class="line">	}</div><div class="line">	</div><div class="line">    <span class="comment">// 释放 slot，这样 pushHead 就可以继续写入这个 slot 了</span></div><div class="line">	*slot = eface{}</div><div class="line">	<span class="keyword">return</span> val, <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>这是一个固定大小的队列，如果队列满了，生产者就会失败。这个队列的大小是 2 的幂次方，这样可以用 <code>&amp;</code> 来取模，而不用 <code>%</code>，这样可以提高性能。</p>
<h2 id="PoolChain">PoolChain</h2>
<p><code>PoolChain</code> 是在 <code>PoolDequeue</code> 的基础上实现的一个动态尺寸的队列，它的实现和 <code>PoolDequeue</code> 类似，只是增加了一个 <code>headTail</code> 的链表，用于存储多个 <code>PoolDequeue</code>。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> poolChain <span class="keyword">struct</span> {</div><div class="line">    <span class="comment">// head 是生产者用来push的 poolDequeue。只有生产者访问，所以不需要同步</span></div><div class="line">	head *poolChainElt</div><div class="line"></div><div class="line">    <span class="comment">// tail 是消费者用来pop的 poolDequeue。消费者访问，所以需要原子操作</span></div><div class="line">	tail atomic.Pointer[poolChainElt]</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> poolChainElt <span class="keyword">struct</span> {</div><div class="line">	poolDequeue</div><div class="line"></div><div class="line">    <span class="comment">// next由生产者原子写入，消费者原子读取。它只能从nil转换为非nil。</span></div><div class="line">    <span class="comment">// prev由消费者原子写入，生产者原子读取。它只能从非nil转换为nil。</span></div><div class="line">	next, prev atomic.Pointer[poolChainElt]</div><div class="line">}</div></pre></td></tr></table></figure>

<p>考虑到文章中代码过多，大家就会感觉很枯燥了，我就不具体展示代码了，你可以在 <a href="https://github.com/golang/go/blob/master/src/sync/poolqueue.go#L220-L302" target="_blank" rel="external">https://github.com/golang/go/blob/master/src/sync/poolqueue.go#L220-L302</a> 查看具体的实现。<br>整体的思想就是将多个<code>poolDequeue</code>串联起来，生产者在<code>head</code>处增加数据，消费者在<code>tail</code>处消费数据，当<code>tail</code>的<code>poolDequeue</code>为空时，就从<code>head</code>处获取一个<code>poolDequeue</code>。<br>当<code>head</code>满了的时候，就增加一个新的<code>poolDequeue</code>。<br>这样就实现了动态尺寸的队列。</p>
<p><code>sync.Pool</code>中就是使用的<code>PoolChain</code>来实现的，它是一个单生产者多消费者的队列，可以同时有多个消费者消费数据，但是只有一个生产者生产数据。</p>
<p>为了能将这个数据结构暴露出来使用，我把相关的代码复制到 <a href="https://github.com/smallnest/exp/blob/master/gods/poolqueue.go" target="_blank" rel="external">https://github.com/smallnest/exp/blob/master/gods/poolqueue.go</a> , 增加了单元测试和性能测试的代码。</p>
<p>你可以学到这个方法，使用类似的技术，创建一个 look-free 无线长度的 byte buffer。在一些 Go 的网络优化库中就使用这种方法，避免频繁的 grow 和 copy 既有数据。</p>
<h2 id="与channel的性能比较">与channel的性能比较</h2>
<p>我们来看一下<code>poolDequeue</code>、<code>PoolChain</code>和<code>channel</code>的性能对比。<br>我们使用一个goroutine进行写入，10个goroutine进行读取：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> gods</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">	<span class="string">"sync"</span></div><div class="line">	<span class="string">"testing"</span></div><div class="line">)</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkPoolDequeue(b *testing.B) {</div><div class="line">	<span class="keyword">const</span> size =<span class="number"> 1024</span></div><div class="line">	pd := NewPoolDequeue(size)</div><div class="line">	<span class="keyword">var</span> wg sync.WaitGroup</div><div class="line"></div><div class="line">	<span class="comment">// Producer</span></div><div class="line">	<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">		<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">			pd.PushHead(i)</div><div class="line">		}</div><div class="line">		wg.Done()</div><div class="line">	}()</div><div class="line"></div><div class="line">	<span class="comment">// Consumers</span></div><div class="line">	numConsumers :=<span class="number"> 10</span></div><div class="line">	wg.Add(numConsumers +<span class="number"> 1</span>)</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; numConsumers; i++ {</div><div class="line">		<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">			<span class="keyword">for</span> {</div><div class="line">				<span class="keyword">if</span> _, ok := pd.PopTail(); !ok {</div><div class="line">					<span class="keyword">break</span></div><div class="line">				}</div><div class="line">			}</div><div class="line">			wg.Done()</div><div class="line">		}()</div><div class="line">	}</div><div class="line"></div><div class="line">	wg.Wait()</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkPoolChain(b *testing.B) {</div><div class="line">	pc := NewPoolChain()</div><div class="line">	<span class="keyword">var</span> wg sync.WaitGroup</div><div class="line"></div><div class="line">	<span class="comment">// Producer</span></div><div class="line">	<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">		<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">			pc.PushHead(i)</div><div class="line">		}</div><div class="line">		wg.Done()</div><div class="line">	}()</div><div class="line"></div><div class="line">	<span class="comment">// Consumers</span></div><div class="line">	numConsumers :=<span class="number"> 10</span></div><div class="line">	wg.Add(numConsumers +<span class="number"> 1</span>)</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; numConsumers; i++ {</div><div class="line">		<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">			<span class="keyword">for</span> {</div><div class="line">				<span class="keyword">if</span> _, ok := pc.PopTail(); !ok {</div><div class="line">					<span class="keyword">break</span></div><div class="line">				}</div><div class="line">			}</div><div class="line">			wg.Done()</div><div class="line">		}()</div><div class="line">	}</div><div class="line"></div><div class="line">	wg.Wait()</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkChannel(b *testing.B) {</div><div class="line">	ch := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">interface</span>{},<span class="number"> 1024</span>)</div><div class="line">	<span class="keyword">var</span> wg sync.WaitGroup</div><div class="line"></div><div class="line">	<span class="comment">// Producer</span></div><div class="line">	<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">		<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">			ch &lt;- i</div><div class="line">		}</div><div class="line">		<span class="built_in">close</span>(ch)</div><div class="line">		wg.Done()</div><div class="line">	}()</div><div class="line"></div><div class="line">	<span class="comment">// Consumers</span></div><div class="line">	numConsumers :=<span class="number"> 10</span></div><div class="line">	wg.Add(numConsumers +<span class="number"> 1</span>)</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; numConsumers; i++ {</div><div class="line">		<span class="keyword">go</span> <span class="keyword">func</span>() {</div><div class="line">			<span class="keyword">for</span> <span class="keyword">range</span> ch {</div><div class="line">			}</div><div class="line">			wg.Done()</div><div class="line">		}()</div><div class="line">	}</div><div class="line"></div><div class="line">	wg.Wait()</div><div class="line">}</div></pre></td></tr></table></figure>

<p>运行这个benchmark,我们可以看到<code>poolDequeue</code>和<code>PoolChain</code>的性能要比<code>channel</code>高很多，大约是<code>channel</code>的10倍。<br><code>poolDequeue</code> 比 <code>PoolChain</code> 要好一些，性能是后者的两倍。<br><img src="benchmark.png" alt=""></p>
]]></content>
    <summary type="html">
    <![CDATA[<p>Go 标准库和运行中中，有一些专门针对特定场景优化的数据结构，这些数据结构并没有暴露出来，这个系列就是逐一介绍这些数据结构。</p>
<p>这一次给大家介绍的就是一个 lock-free、高性能的单生产者多消费者的队列：<code>PoolDequeue</code> 和 <code>PoolChain</code>。<br>到底是一个还是两个呢？<br>主要是 <code>PoolDequeue</code>, 它是一个固定尺寸，使用 ringbuffer (环形队列) 方式实现的队列。<br><code>PoolChain</code> 是在它的基础上上，实现的一个动态尺寸的队列。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[在 Rust 中同时支持异步和同步代码]]></title>
    <link href="https://colobu.com/2024/08/28/rust-async-sync/"/>
    <id>https://colobu.com/2024/08/28/rust-async-sync/</id>
    <published>2024-08-28T00:30:43.000Z</published>
    <updated>2024-08-28T00:32:15.564Z</updated>
    <content type="html"><![CDATA[<p>来，过路人，请坐到我身边来，听老衲讲一讲我对 Rust 过分要求的故事。</p>
<a id="more"></a>
<h2 id="介绍">介绍</h2>
<p>想象一下,你打算用Rust创建一个新库。这个库的唯一功能就是封装一个你需要的公共API, 比如 <a href="https://developer.spotify.com/documentation/web-api/" target="_blank" rel="external">Spotify API或者</a> <a href="https://www.arangodb.com/" target="_blank" rel="external">ArangoDB</a> 之类的数据库。这并不是造火箭，你也不是在发明什么新东西或者处理复杂的算法,所以你认为这应该相对简单直接。</p>
<p>你决定用异步方式实现这个库。你的库中大部分工作都涉及执行HTTP请求,主要是I/O操作,所以使用异步是有道理的(而且,这也是Rust圈里现在的潮流)。你开始编码,几天后就准备好了v0.1.0版本。当 <code>cargo publish</code> 成功完成并将你的作品上传到 <code>crates.io</code> 时,你暗自得意地想: &quot;不错嘛&quot;。</p>
<p>几天过去了,你在GitHub上收到了一个新通知。有人提了一个问题:</p>
<blockquote>
<p><strong>我如何同步使用这个库？</strong></p>
<p>我的项目不使用异步,因为对我的需求来说太复杂了。我想尝试你的新库,但不确定怎么轻松地使用它。我不想在代码中到处使用 <code>block_on(endpoint())</code>。。我见过像 <a href="https://crates.io/crates/reqwest" target="_blank" rel="external"><code>reqwest</code></a> 这样的 crate导出一个 <a href="https://docs.rs/reqwest/0.11.4/reqwest/blocking/index.html" target="_blank" rel="external"><code>blocking</code>模块</a>，提供完全相同的功能,你能不能也这么做?</p>
</blockquote>
<p>从底层来看,这听起来是个很复杂的任务。为异步代码(需要像 <a href="https://crates.io/crates/tokio" target="_blank" rel="external"><code>tokio</code></a> 这样的运行时、awaiting future、pinning等)和普通的同步代码提供一个通用接口?好吧,既然他们提出请求的态度很好,也许我们可以试试。毕竟,代码中唯一的区别就是 <code>async</code> 和 <code>await</code> 关键字的出现,因为你没有做什么花哨的事情。</p>
<p>好吧，这或多或少就是crate 发生的事情 <a href="https://crates.io/crates/rspotify" target="_blank" rel="external"><code>rspotify</code></a> ，我曾经和它的创建者 <a href="https://github.com/ramsayleung/" target="_blank" rel="external">Ramsay</a> 一起维护它。对于那些不知道的人来说，它是 Spotify Web API 的一个包装器。对不了解的人来说,这是一个Spotify Web API的封装。说明一下,我最终确实实现了这个功能,尽管不如我希望的那么干净利落;我会在Rspotify系列的这篇新文章中试图解释这个<a href="https://nullderef.com/series/rspotify" target="_blank" rel="external">情况</a>。</p>
<h2 id="第一种方法">第一种方法</h2>
<p>为了提供更多背景信息，Rspotify 的客户端大致如下：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">struct</span> Spotify { <span class="comment">/* ... */</span> }</div><div class="line"></div><div class="line"><span class="keyword">impl</span> Spotify {</div><div class="line">    async <span class="function"><span class="keyword">fn</span> <span class="title">some_endpoint</span></span>(&<span class="keyword">self</span>, param: String) -&gt; SpotifyResult&lt;String&gt; {</div><div class="line">        <span class="keyword">let</span> <span class="keyword">mut</span> params = HashMap::new();</div><div class="line">        params.insert(<span class="string">"param"</span>, param);</div><div class="line"></div><div class="line">        <span class="keyword">self</span>.http.get(<span class="string">"/some-endpoint"</span>, params).await</div><div class="line">    }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>本质上，我们需要让 <code>some_endpoint</code> 同时支持异步和阻塞两种使用方式。这里的关键问题是，当你有几十个端点时，你该如何实现这一点？而且，你怎样才能让用户在异步和同步之间轻松切换呢？</p>
<h3 id="老掉牙的复制粘贴大法">老掉牙的复制粘贴大法</h3>
<p>这是最初实现的方法。它相当简单，而且确实能用。你只需要把常规的客户端代码复制到 Rspotify 的一个新的 <a href="https://github.com/ramsayleung/rspotify/tree/v0.9/src/blocking" target="_blank" rel="external"><code>blocking</code>模块</a>里。<a href="https://docs.rs/reqwest" target="_blank" rel="external"><code>reqwest</code></a>（我们用的 HTTP 客户端）和 <code>reqwest::blocking</code> 共用一个接口，所以我们可以在新模块里手动删掉 <code>async</code> 或 <code>.await</code> 这样的关键字，然后把 <code>reqwest</code> 的导入改成 <code>reqwest::blocking</code>。</p>
<p>这样一来，Rspotify 的用户只需要用 <code>rspotify::blocking::Client</code> 替代 <code>rspotify::Client</code>，瞧！他们的代码就变成阻塞式的了。这会让只用异步的用户的二进制文件变大，所以我们可以把它放在一个叫 <code>blocking</code> 的特性开关后面，大功告成。</p>
<p>不过，问题后来就变得明显了。整个 crate 的一半代码都被复制了一遍。添加或修改一个端点就意味着要写两遍或删两遍所有东西。</p>
<p>除非你把所有东西都测试一遍，否则没法确保两种实现是等效的。这主意倒也不坏，但说不定你连测试都复制粘贴错了呢！那可怎么办？可怜的代码审查员得把同样的代码读两遍，确保两边都没问题 —— 这听起来简直就是人为错误的温床。</p>
<p>根据我们的经验，这确实大大拖慢了 Rspotify 的开发进度，尤其是对于不习惯这种折腾的新贡献者来说。作为 Rspotify 的一个新晋且热情的维护者，我开始<a href="https://github.com/ramsayleung/rspotify/issues/112" target="_blank" rel="external">研究其他可能的解决方案</a>。</p>
<h3 id="召唤_block_on">召唤 <code>block_on</code></h3>
<p><a href="https://github.com/ramsayleung/rspotify/pull/120" target="_blank" rel="external">第二种方法</a>是把所有东西都在异步那边实现。然后，你只需为阻塞接口做个包装，在内部调用 <code>block_on</code>。<code>block_on</code> 会运行 future 直到完成，本质上就是把它变成同步的。你仍然需要复制方法的定义，但实现只需写一次：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">mod</span> blocking {</div><div class="line">    <span class="keyword">struct</span> Spotify(super::Spotify);</div><div class="line"></div><div class="line">    <span class="keyword">impl</span> Spotify {</div><div class="line">        <span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>(&<span class="keyword">self</span>, param: String) -&gt; SpotifyResult&lt;String&gt; {</div><div class="line">            runtime.block_on(async move {</div><div class="line">                <span class="keyword">self</span>.<span class="number">0</span>.endpoint(param).await</div><div class="line">            })</div><div class="line">        }</div><div class="line">    }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>请注意，为了调用<code>block_on</code>，您首先必须在端点方法中创建某种运行时。例如，使用<a href="https://crates.io/crates/tokio" target="_blank" rel="external"><code>tokio</code></a> ：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">let</span> <span class="keyword">mut</span> runtime = tokio::runtime::Builder::new()</div><div class="line">    .basic_scheduler()</div><div class="line">    .enable_all()</div><div class="line">    .build()</div><div class="line">    .unwrap();</div></pre></td></tr></table></figure>

<p>这就引出了一个问题：我们是应该在每次调用端点时都初始化运行时，还是有办法共享它呢？我们可以把它保存为一个全局变量（<em>呃，真恶心</em>），或者更好的方法是，我们可以把运行时保存在 <code>Spotify</code> 结构体中。但是由于它需要对运行时的可变引用，你就得用 <code>Arc&lt;Mutex&lt;T&gt;&gt;</code> 把它包起来，这样一来就完全扼杀了客户端的并发性。正确的做法是使用 Tokio 的 <a href="https://docs.rs/tokio/latest/tokio/runtime/struct.Handle.html" target="_blank" rel="external"><code>Handle</code></a>，大概是这样的：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">use</span> tokio::runtime::Runtime;</div><div class="line"></div><div class="line">lazy_static! { <span class="comment">// You can also use `once_cell`</span></div><div class="line">    <span class="keyword">static</span> <span class="keyword">ref</span> RT: Runtime = Runtime::new().unwrap();</div><div class="line">}</div><div class="line"></div><div class="line"><span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>(&<span class="keyword">self</span>, param: String) -&gt; SpotifyResult&lt;String&gt; {</div><div class="line">    RT.handle().block_on(async move {</div><div class="line">        <span class="keyword">self</span>.<span class="number">0</span>.endpoint(param).await</div><div class="line">    })</div><div class="line">}</div></pre></td></tr></table></figure>

<p>虽然使用 handle 确实让我们的阻塞客户端更快了<a href="https://nullderef.com/blog/rust-async-sync/#block-on-perf" target="_blank" rel="external">^1</a>，但还有一种性能更高的方法。如果你感兴趣的话，这正是 <code>reqwest</code> 自己采用的方法。简单来说，它会生成一个线程，这个线程调用 <code>block_on</code> 来等待一个装有任务的通道 [^2] (<a href="https://nullderef.com/blog/rust-async-sync/#block-on-channels" target="_blank" rel="external">https://nullderef.com/blog/rust-async-sync/#block-on-channels</a>) [^3] (<a href="https://nullderef.com/blog/rust-async-sync/#block-on-reqwest)。" target="_blank" rel="external">https://nullderef.com/blog/rust-async-sync/#block-on-reqwest)。</a></p>
<p>不幸的是，这个解决方案仍然有相当大的开销。你需要引入像 <code>futures</code> 或 <code>tokio</code> 这样的大型依赖，并将它们包含在你的二进制文件中。所有这些，就是为了...最后还是写出阻塞代码。所以这不仅在运行时有成本，在编译时也是如此。这在我看来就是不对劲。</p>
<p>而且你仍然有不少重复代码，即使只是定义，积少成多也是个问题。<code>reqwest</code> 是一个巨大的项目，可能负担得起他们的 <code>blocking</code> 模块的开销。但对于像 <code>rspotify</code> 这样不那么流行的 crate 来说，这就难以实现了。</p>
<h3 id="复制_crate">复制 crate</h3>
<p>另一种可能的解决方法是，正如 features 文档所建议的那样，创建独立的 crate。我们可以有 <code>rspotify-sync</code> 和 <code>rspotify-async</code>，用户可以根据需要选择其中一个作为依赖，甚至如果需要的话可以两个都用。问题是 —— 又来了 —— 我们究竟该如何生成这两个版本的 crate 呢？即使使用 Cargo 的一些技巧，比如为每个 crate 准备一个 <code>Cargo.toml</code> 文件（这种方法本身就很不方便），我也无法在<a href="https://github.com/ramsayleung/rspotify/pull/253" target="_blank" rel="external">不复制粘贴</a>整个 crate 的情况下做到这一点。</p>
<p>采用这种方法，我们甚至无法使用过程宏，因为你不能在宏中凭空创建一个新的 crate。我们可以定义一种文件格式来编写 Rust 代码的模板，以便替换代码中的某些部分，比如 <code>async</code>/<code>.await</code>。但这听起来完全超出了我们的范畴。</p>
<h2 id="最终版是：maybe_async_crate">最终版是：<code>maybe_async</code> crate</h2>
<p><a href="https://github.com/ramsayleung/rspotify/pull/129" target="_blank" rel="external">第三次尝试</a>基于一个名为 <a href="https://crates.io/crates/maybe_async" target="_blank" rel="external"><code>maybe_async</code></a> 的 crate。我记得当初发现它时，天真地以为这就是完美的解决方案。</p>
<p>总之，这个 crate 的思路是，你可以用一个过程宏自动移除代码中的 <code>async</code> 和 <code>.await</code>，本质上就是把复制粘贴的方法自动化了。举个例子：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#[maybe_async::maybe_async]</span></div><div class="line">async <span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>() { <span class="comment">/* stuff */</span> }</div></pre></td></tr></table></figure>

<p>生成以下代码：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#[cfg(not(feature = "is_sync"))]</span></div><div class="line">async <span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>() { <span class="comment">/* stuff */</span> }</div><div class="line"></div><div class="line"><span class="preprocessor">#[cfg(feature = "is_sync")]</span></div><div class="line"><span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>() { <span class="comment">/* stuff with `.await` removed */</span> }</div></pre></td></tr></table></figure>

<p>你可以通过在编译 crate 时切换 <code>maybe_async/is_sync</code> 特性来配置是要异步还是阻塞代码。这个宏适用于函数、trait 和 <code>impl</code> 块。如果某个转换不像简单地移除 <code>async</code> 和 <code>.await</code> 那么容易，你可以用 <code>async_impl</code> 和 <code>sync_impl</code> 过程宏来指定自定义实现。它处理得非常好，我们在 Rspotify 中已经使用它一段时间了。</p>
<p>事实上，它效果如此之好，以至于我让 Rspotify 变成了HTTP 客户端无关的，这比异步/同步无关更加灵活。这使我们能够支持多种 HTTP 客户端，比如 <a href="https://crates.io/crates/reqwest" target="_blank" rel="external"><code>reqwest</code></a> 和 <a href="https://crates.io/crates/ureq" target="_blank" rel="external"><code>ureq</code></a> ，而不用管客户端是异步的还是同步的。</p>
<p>如果你有 <code>maybe_async</code>，实现<em>HTTP 客户端无关</em>并不是很难。你只需要为 <a href="https://github.com/ramsayleung/rspotify/blob/89b37219a2230cdcf08c4cfd2ebe46d64902f03d/rspotify-http/src/common.rs#L46" target="_blank" rel="external">HTTP 客户端</a>定义一个 trait，然后为你想支持的每个客户端实现它：</p>
<p><em>一段代码胜过千言万语。（你可以在这里找到 Rspotify 的 <a href="https://github.com/ramsayleung/rspotify/blob/master/rspotify-http/src/reqwest.rs#L97" target="_blank" rel="external"><code>reqwest</code>客户端</a>的完整源代码， <code>ureq</code> 也可以在<a href="https://github.com/ramsayleung/rspotify/blob/master/rspotify-http/src/ureq.rs#L56" target="_blank" rel="external">这里</a>找到 ）</em></p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#[maybe_async]</span></div><div class="line"><span class="keyword">trait</span> <span class="title">HttpClient</span> {</div><div class="line">    async <span class="function"><span class="keyword">fn</span> <span class="title">get</span></span>(&<span class="keyword">self</span>) -&gt; String;</div><div class="line">}</div><div class="line"></div><div class="line"><span class="preprocessor">#[sync_impl]</span></div><div class="line"><span class="keyword">impl</span> HttpClient <span class="keyword">for</span> UreqClient {</div><div class="line">    <span class="function"><span class="keyword">fn</span> <span class="title">get</span></span>(&<span class="keyword">self</span>) -&gt; String { ureq::get(<span class="comment">/* ... */</span>) }</div><div class="line">}</div><div class="line"></div><div class="line"><span class="preprocessor">#[async_impl]</span></div><div class="line"><span class="keyword">impl</span> HttpClient <span class="keyword">for</span> ReqwestClient {</div><div class="line">    async <span class="function"><span class="keyword">fn</span> <span class="title">get</span></span>(&<span class="keyword">self</span>) -&gt; String { reqwest::get(<span class="comment">/* ... */</span>).await }</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">struct</span> SpotifyClient&lt;Http: HttpClient&gt; {</div><div class="line">    http: Http</div><div class="line">}</div><div class="line"></div><div class="line"><span class="preprocessor">#[maybe_async]</span></div><div class="line"><span class="keyword">impl</span>&lt;Http: HttpClient&gt; SpotifyClient&lt;Http&gt; {</div><div class="line">    async <span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>(&<span class="keyword">self</span>) { <span class="keyword">self</span>.http.get(<span class="comment">/* ... */</span>) }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>然后，我们可以进一步扩展，让用户通过在他们的 <code>Cargo.toml</code> 中设置特性标志来选择他们想要使用的客户端。比如，如果启用了 <code>client-ureq</code>，由于 <code>ureq</code> 是同步的，它就会启用 <code>maybe_async/is_sync</code>。这样一来，就会移除 <code>async</code>/<code>.await</code> 和 <code>#[async_impl]</code> 块，Rspotify 客户端内部就会使用 <code>ureq</code> 的实现。</p>
<p>这个解决方案避免了我之前提到的所有缺点：</p>
<ul>
<li>完全没有代码重复</li>
<li>无论是在运行时还是编译时都没有额外开销。如果用户想要一个阻塞客户端，他们可以使用 <code>ureq</code>，这样就不会引入 <code>tokio</code> 及其相关依赖</li>
<li>对用户来说很容易理解；只需在 <code>Cargo.toml</code> 中配置一个标志</li>
</ul>
<p>不过，先停下来想几分钟，试试看你能不能找出为什么不应该这么做。实际上，我给你9个月时间，这就是我花了多长时间才意识到问题所在...</p>
<h3 id="问题">问题</h3>
<p><img src="https://nullderef.com/blog/rust-async-sync/preview.jpg" alt="预览"></p>
<p>嗯，问题在于 Rust 中的特性必须是<strong>叠加的</strong>：&quot;启用一个特性不应该禁用功能，而且通常应该可以安全地启用任意组合的特性&quot;。当依赖树中出现重复的 crate 时，Cargo 可能会合并该 crate 的特性，以避免多次编译同一个 crate。<a href="https://doc.rust-lang.org/cargo/reference/features.html#feature-unification" target="_blank" rel="external">如果您想了解更多详细信息，参考资料对此进行了很好的解释</a>。</p>
<p>这种优化意味着互斥的特性可能会破坏依赖树。在我们的情况下，<code>maybe_async/is_sync</code> 是一个由 <code>client-ureq</code> 启用的 切换特性。所以如果你试图同时启用 <code>client-reqwest</code> 来编译，它就会失败，因为 <code>maybe_async</code> 将被配置为生成同步函数签名。不可能有一个 crate 直接或间接地同时依赖于同步和异步的 Rspotify，而且根据 Cargo 参考文档，<code>maybe_async</code> 的整个概念目前是错误的。</p>
<h3 id="新特性解析器_v2">新特性解析器 v2</h3>
<p>一个常见的误解是，这个问题可以通过&quot;特性解析器v2&quot;来修复，<a href="https://doc.rust-lang.org/cargo/reference/features.html#feature-resolver-version-2" target="_blank" rel="external">参考文档也对此进行了很好的解释</a>。从2021版本开始，这个新版本已经默认启用了，但你也可以在之前的版本的 <code>Cargo.toml</code> 中指定使用它。这个新版本除了其他改进，还在一些特殊情况下避免了特性的统一，但不包括我们的情况：</p>
<ul>
<li>对于当前未在构建的目标，启用在平台特定依赖项上的特性会被忽略。</li>
<li>构建依赖和过程宏不会与普通依赖共享特性。</li>
<li>除非构建需要它们的目标（如测试或示例），否则开发依赖不会激活特性。</li>
</ul>
<p>为了以防万一，我自己尝试复现了这个问题，结果确实如我所料。<a href="https://github.com/marioortizmanero/resolver-v2-conflict" target="_blank" rel="external">这个代码库</a>是一个特性冲突的例子，在任何特性解析器下都会出错。</p>
<h3 id="其他失败">其他失败</h3>
<p>有一些 crate也存在这个问题：</p>
<ul>
<li><p><a href="https://crates.io/crates/arangors" target="_blank" rel="external"><code>arangors</code></a> 和 <a href="https://crates.io/crates/aragog" target="_blank" rel="external"><code>aragog</code></a> ：ArangoDB 的包装器。两者都用于 <code>maybe_async</code> 在异步和同步之间切换（<code>arangors</code> 事实上，的作者是同一个人）<a href="https://nullderef.com/blog/rust-async-sync/#arangors-error" target="_blank" rel="external">^5</a> [^6] (<a href="https://nullderef.com/blog/rust-async-sync/#aragog-error)。" target="_blank" rel="external">https://nullderef.com/blog/rust-async-sync/#aragog-error)。</a></p>
</li>
<li><p><a href="https://crates.io/crates/inkwell" target="_blank" rel="external"><code>inkwell</code></a> ：LLVM 的包装器。它支持多个版本的 LLVM，但彼此之间不兼容<a href="https://nullderef.com/blog/rust-async-sync/#inkwell-error" target="_blank" rel="external">[7]</a>。</p>
</li>
<li><p><a href="https://crates.io/crates/k8s-openapi" target="_blank" rel="external"><code>k8s-openapi</code></a> ：Kubernetes 的包装器，与 <code>inkwell</code> <a href="https://nullderef.com/blog/rust-async-sync/#k8s-error" target="_blank" rel="external">^8</a>存在同样的问题。</p>
</li>
</ul>
<h3 id="修复_maybe_async">修复 <code>maybe_async</code></h3>
<p>随着这个 crate 开始变得流行起来，有人在 <code>maybe_async</code> 中提出了这个问题，解释了情况并展示了一个修复方案：<br> <a href="https://github.com/fMeow/maybe-async-rs/issues/6" target="_blank" rel="external">async 和 sync 在同一程序中 fMeow/maybe-async-rs #6</a></p>
<p><code>maybe_async</code> 现在会有两个特性标志：<code>is_sync</code> 和 <code>is_async</code>。这个 crate 会以同样的方式生成函数，但会在标识符后面添加 <code>_sync</code> 或 <code>_async</code> 后缀，这样就不会冲突了。例如：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#[maybe_async::maybe_async]</span></div><div class="line">async <span class="function"><span class="keyword">fn</span> <span class="title">endpoint</span></span>() { <span class="comment">/* stuff */</span> }</div></pre></td></tr></table></figure>

<p>现在将生成以下代码：</p>
<figure class="highlight rust"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#[cfg(feature = "is_async")]</span></div><div class="line">async <span class="function"><span class="keyword">fn</span> <span class="title">endpoint_async</span></span>() { <span class="comment">/* stuff */</span> }</div><div class="line"></div><div class="line"><span class="preprocessor">#[cfg(feature = "is_sync")]</span></div><div class="line"><span class="function"><span class="keyword">fn</span> <span class="title">endpoint_sync</span></span>() { <span class="comment">/* stuff with `.await` removed */</span> }</div></pre></td></tr></table></figure>

<p>然而，这些后缀会引入噪音，所以我在想是否有可能以更符合人体工程学的方式来实现。我fork了<code>maybe_async</code>并尝试了一下，你可以在这一系列评论中读到更多相关内容。总的来说，这太复杂了，我最终放弃了。</p>
<p>修复这个边缘情况的唯一方法就是让Rspotify对所有人的可用性变差。但我认为，同时依赖异步和同步版本的人可能很少；实际上我们还没有收到任何人的抱怨。与<code>reqwest</code>不同，<code>rspotify</code>是一个&quot;高级&quot;库，所以很难想象它会在一个依赖树中出现多次。</p>
<p>也许我们可以向Cargo的开发者寻求帮助？</p>
<h3 id="官方支持">官方支持</h3>
<p>虽然不是官方的，但 Rust 中可以进一步探索的另一种有趣方法是<a href="https://sans-io.readthedocs.io/" target="_blank" rel="external">“Sans I/O”</a>。这是一个 Python 协议，它抽象了网络协议（如 HTTP）的使用，从而最大限度地提高了可重用性。Rust 中现有的一个示例是 <a href="https://github.com/EmbarkStudios/tame-oidc" target="_blank" rel="external"><code>tame-oidc</code></a>。</p>
<p>Rspotify 远不是第一个遇到这个问题的项目，所以阅读之前的相关讨论可能会很有趣：</p>
<ul>
<li><a href="https://github.com/rust-lang/rfcs/pull/2962" target="_blank" rel="external">这个现已关闭的 Rust 编译器 RFC</a> 添加 <code>oneof</code> 配置谓词（类似 <code>#[cfg(any(…))]</code>）来支持互斥特性。这只是让在别无选择的情况下拥有冲突特性变得更容易，但特性仍应该是严格叠加的。</li>
<li>前一个 RFC 在 Cargo 本身允许互斥特性的背景下引发了<a href="https://internals.rust-lang.org/t/pre-rfc-cargo-mutually-exclusive-features/13182/27" target="_blank" rel="external">一些讨论</a>，尽管有一些有趣的信息，但并没有取得太大进展。</li>
<li><a href="https://github.com/rust-lang/cargo/issues/2980" target="_blank" rel="external">Cargo 中的这个问题</a> 解释了 Windows API 的类似情况。讨论包括更多示例和解决方案想法，但还没有被 Cargo 采纳。</li>
<li><a href="https://github.com/rust-lang/cargo/issues/4803" target="_blank" rel="external">Cargo 中的另一个问题</a> 要求提供一种方法来轻松测试和构建不同标志组合。如果特性是严格叠加的，那么 <code>cargo test --all-features</code> 将涵盖所有情况。但如果不是，用户就必须用多个特性标志组合运行命令，这相当麻烦。非官方的 <a href="https://github.com/taiki-e/cargo-hack" target="_blank" rel="external"><code>cargo-hack</code></a> 已经可以实现这一点。</li>
<li>一种完全不同的方法 <a href="https://blog.rust-lang.org/inside-rust/2023/02/23/keyword-generics-progress-report-feb-2023.html" target="_blank" rel="external">基于关键字泛型倡议</a>。这似乎是解决这个问题的最新尝试，但仍处于&quot;探索&quot;阶段， <a href="https://blog.rust-lang.org/inside-rust/2022/07/27/keyword-generics.html#q-is-there-an-rfc-available-to-read" target="_blank" rel="external">截至目前还没有可用的 RFC</a>。</li>
</ul>
<p>根据<a href="https://github.com/rust-lang/rfcs/pull/2962#issuecomment-664656377" target="_blank" rel="external">这条旧评论</a>，这不是 Rust 团队已经否决的东西；它仍在讨论中。</p>
<p>虽然是非官方的，但另一个可以在 Rust 中进一步探索的另一种有趣方法是 <a href="https://sans-io.readthedocs.io/" target="_blank" rel="external">“Sans I/O”</a>。这是一种 Python 协议，它在我们的案例中抽象了 HTTP 等网络协议的使用，从而最大化了可重用性。Rust 中现有的一个例子是 <a href="https://github.com/EmbarkStudios/tame-oidc" target="_blank" rel="external"><code>tame-oidc</code></a>。</p>
<h2 id="结论">结论</h2>
<p>我们目前面临以下选择：</p>
<ul>
<li>忽视 Cargo 参考。我们可以假设没有人会同时使用 Rspotify 的同步和异步版本。</li>
<li>修复 <code>maybe_async</code> 并为我们库中的每个端点添加 <code>_async</code> 和 <code>_sync</code> 后缀。</li>
<li>放弃支持异步和同步代码。这已经变成了一团糟，我们没有足够的人力来处理，而且它影响了 Rspotify 的其他部分。问题是一些依赖 rspotify 的 crate，如 <code>ncspot</code> 或 <code>spotifyd</code> 是阻塞的，而其他如 <code>spotify-tui</code> 使用异步，所以我不确定他们会怎么想。</li>
</ul>
<p>我知道这是我给自己强加的问题。我们可以直接说&quot;不。我们只支持异步&quot;或&quot;不。我们只支持同步&quot;。虽然有用户对能够使用两者感兴趣，但有时你就是得说不。如果这样一个特性变得如此复杂，以至于你的整个代码库变成一团糟，而你没有足够的工程能力来维护它，那这就是你唯一的选择。如果有人真的很在意，他们可以直接 fork 这个 crate 并将其转换为同步版本供自己使用。</p>
<p>毕竟，大多数 API 封装库等只支持异步或阻塞代码中的一种。例如，<a href="https://crates.io/crates/serenity" target="_blank" rel="external"><code>serenity</code></a> （Discord API）、<a href="https://crates.io/crates/sqlx" target="_blank" rel="external"><code>sqlx</code></a> （SQL 工具包）和 <a href="https://crates.io/crates/teloxide" target="_blank" rel="external"><code>teloxide</code></a> （Telegram API）是仅异步的，而且它们非常流行。。</p>
<p>尽管有时候很沮丧，但我并不后悔花了这么多时间兜圈子试图让异步和同步都能工作。我最初为 Rspotify 做贡献就是为了_学习<em>。我没有截止日期，也没有压力，我只是想在空闲时间尝试改进 Rust 中的一个库。而且我确实</em>学到了_很多；希望在读完这篇文章后，你也是如此。</p>
<p>也许今天的教训是，我们应该记住 Rust 毕竟是一种低级语言，有些事情如果不引入大量复杂性是不可能实现的。无论如何，我期待 Rust 团队将来如何解决这个问题。</p>
<p>那么你怎么看？如果你是 Rspotify 的维护者，你会怎么做？如果你愿意，可以在下面留言。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>来，过路人，请坐到我身边来，听老衲讲一讲我对 Rust 过分要求的故事。</p>
]]>
    
    </summary>
    
  </entry>
  
  <entry>
    <title><![CDATA[四种字符串和bytes互相转换方式的性能比较]]></title>
    <link href="https://colobu.com/2024/08/13/string-bytes-benchmark/"/>
    <id>https://colobu.com/2024/08/13/string-bytes-benchmark/</id>
    <published>2024-08-13T14:16:34.000Z</published>
    <updated>2024-08-13T14:25:42.549Z</updated>
    <content type="html"><![CDATA[<p>昨天公司群中同事提到 Go 1.22 中 string 和 bytes 的互转不需要再用 unsafe 那个包了，直接转就可以。我翻看了 Go 1.22 的 release notes 没找到相应的介绍，但是大家提到了 kubernetes 的 <a href="https://github.com/kubernetes/kubernetes/issues/124656" target="_blank" rel="external">issue</a> 中有这个说法：</p>
<blockquote>
<p>As of go 1.22, for string to bytes conversion, we can replace the usage of <code>unsafe.Slice(unsafe.StringData(s), len(s))</code> with type casting <code>[]bytes(str)</code>, without the worry of losing performance.</p>
<p>As of go 1.22, string to bytes conversion <code>[]bytes(str)</code> is faster than using the <code>unsafe</code> package. Both methods have 0 memory allocation now.</p>
</blockquote>
<blockquote>
<p>自 Go 1.22 起，对于 string 到 bytes 的转换，我们可以用类型转换 <code>[]bytes(str)</code> 来替换 <code>unsafe.Slice(unsafe.StringData(s), len(s))</code> 的用法，而不用担心性能损失。<br>自 Go 1.22 起，string 到 bytes 的转换 <code>[]bytes(str)</code> 比使用 unsafe 包更快。现在两种方法都不会有内存分配。</p>
</blockquote>
<p>这个说法让我很好奇，但是我还是想验证一下这个说法。</p>
<p>注意，这个说法只谈到了 string 到 bytes 的转换，并没有提到 bytes 到 string 的转换，这篇文章也会关注这两者的互转。</p>
<p>首先，让我们看看几种 string 和 bytes 的转换方式，然后我们再写 benchmark 比较它们之间的性能。<br><a id="more"></a></p>
<h2 id="一、强转">一、强转</h2>
<p>字符串和 bytes 之间可以强制转换，编译器会内部处理。代码如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> toRawBytes(s <span class="typename">string</span>) []<span class="typename">byte</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(s) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> []<span class="typename">byte</span>(s)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> toRawString(b []<span class="typename">byte</span>) <span class="typename">string</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(b) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="string">""</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> <span class="typename">string</span>(b)</div><div class="line">}</div></pre></td></tr></table></figure>

<p>这里我们做了一点点优化，处理空 string或者 bytes 的情况。</p>
<h2 id="二、传统_unsafe_方式">二、传统 unsafe 方式</h2>
<p>reflect 包中定义了 <code>SliceHeader</code> 和 <code>StringHeader</code>, 分别对应 slice 和 string 的数据结构</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> SliceHeader <span class="keyword">struct</span> {</div><div class="line">	Data <span class="typename">uintptr</span></div><div class="line">	Len  <span class="typename">int</span></div><div class="line">	Cap  <span class="typename">int</span></div><div class="line">}</div><div class="line"><span class="keyword">type</span> StringHeader <span class="keyword">struct</span> {</div><div class="line">	Data <span class="typename">uintptr</span></div><div class="line">	Len  <span class="typename">int</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>我们按照这种数据结构，可以实现 string 和 bytes 的互转。我们暂且把它叫做 <code>reflect</code> 方式吧，虽然下面的代码没有用到 <code>reflect</code> 包，但是实际我们是按照 <code>reflect</code> 包中的这两个数据结构进行转换的:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> toReflectBytes(s <span class="typename">string</span>) []<span class="typename">byte</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(s) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	}</div><div class="line"></div><div class="line">	x := (*<span class="number">[2</span>]<span class="typename">uintptr</span>)(unsafe.Pointer(&s))</div><div class="line">	h := <span class="number">[3</span>]<span class="typename">uintptr</span>{x<span class="number">[0</span>], x<span class="number">[1</span>], x<span class="number">[1</span>]}</div><div class="line">	<span class="keyword">return</span> *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&h))</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> toReflectString(b []<span class="typename">byte</span>) <span class="typename">string</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(b) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="string">""</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> *(*<span class="typename">string</span>)(unsafe.Pointer(&b))</div><div class="line">}</div></pre></td></tr></table></figure>

<h2 id="三、新型_unsafe_方式">三、新型 unsafe 方式</h2>
<p>我在两年前的文章<a href="https://colobu.com/2022/09/06/string-byte-convertion/" target="_blank" rel="external">与日俱进，在 Go 1.20 中这种高效转换的方式又变了</a>介绍了新的 unsafe 方式，reflect 包中的 <code>SliceHeader</code> 和 <code>StringHeader</code> 准备废弃了。让我们看看这种新的转换方式：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> toBytes(s <span class="typename">string</span>) []<span class="typename">byte</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(s) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="constant">nil</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> unsafe.Slice(unsafe.StringData(s), <span class="built_in">len</span>(s))</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> toString(b []<span class="typename">byte</span>) <span class="typename">string</span> {</div><div class="line">	<span class="keyword">if</span> <span class="built_in">len</span>(b) ==<span class="number"> 0</span> {</div><div class="line">		<span class="keyword">return</span> <span class="string">""</span></div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> unsafe.String(unsafe.SliceData(b), <span class="built_in">len</span>(b))</div><div class="line">}</div></pre></td></tr></table></figure>

<p>利用 <code>unsafe.Slice</code> 、<code>unsafe.String</code>、<code>unsafe.StringData</code> 和 <code>unsafe.SliceData</code> 完成 Slice 和 String 的转换以及底层数据的指针的获取。</p>
<h2 id="四、kubernetes_的实现">四、kubernetes 的实现</h2>
<p>在 k8s 中，使用的是下面方式的优化的转换：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> toK8sBytes(s <span class="typename">string</span>) []<span class="typename">byte</span> {</div><div class="line">	<span class="keyword">return</span> *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&s))</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> toK8sString(b []<span class="typename">byte</span>) <span class="typename">string</span> {</div><div class="line">	<span class="keyword">return</span> *(*<span class="typename">string</span>)(unsafe.Pointer(&b))</div><div class="line">}</div></pre></td></tr></table></figure>

<p>可以看到，相对于传统 unsafe 方式，k8s 的实现更简洁，并没有为<code>toBytes</code>临时构造3元素的数组，而是直接将 string 和 bytes 的指针进行转换。</p>
<p>string不是只包含两个字段么？slice不是包含三个字段么？<code>toK8sBytes</code>返回的[]byte的cap是怎么确定的呢？ 最后我们再分析这个问题，现在先把这几个实现的性能搞清楚。 </p>
<h2 id="性能比较">性能比较</h2>
<p>我们分别对这几种实现进行 benchmark，看看它们之间的性能差异。<br>使用一个简单的字符串和它对应的bytes, 分别进行 string 到 bytes 、 bytes 到 string 的转换。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> s = <span class="string">"hello, world"</span></div><div class="line"><span class="keyword">var</span> bts = []<span class="typename">byte</span>(<span class="string">"hello, world"</span>)</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkStringToBytes(b *testing.B) {</div><div class="line">	<span class="keyword">var</span> fns = <span class="keyword">map</span>[<span class="typename">string</span>]<span class="keyword">func</span>(<span class="typename">string</span>) []<span class="typename">byte</span>{</div><div class="line">		<span class="string">"强制转换"</span>:  toRawBytes,</div><div class="line">		<span class="string">"传统转换"</span>:  toReflectBytes,</div><div class="line">		<span class="string">"新型转换"</span>:  toBytes,</div><div class="line">		<span class="string">"k8s转换"</span>: toK8sBytes,</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">for</span> name, fn := <span class="keyword">range</span> fns {</div><div class="line">		b.Run(name, <span class="keyword">func</span>(b *testing.B) {</div><div class="line">			<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">				bts = fn(s)</div><div class="line">			}</div><div class="line">		})</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkBytesToString(b *testing.B) {</div><div class="line">	<span class="keyword">var</span> fns = <span class="keyword">map</span>[<span class="typename">string</span>]<span class="keyword">func</span>([]<span class="typename">byte</span>) <span class="typename">string</span>{</div><div class="line">		<span class="string">"强制转换"</span>:  toRawString,</div><div class="line">		<span class="string">"传统转换"</span>:  toReflectString,</div><div class="line">		<span class="string">"新型转换"</span>:  toString,</div><div class="line">		<span class="string">"k8s转换"</span>: toK8sString,</div><div class="line">	}</div><div class="line"></div><div class="line">	<span class="keyword">for</span> name, fn := <span class="keyword">range</span> fns {</div><div class="line">		b.Run(name, <span class="keyword">func</span>(b *testing.B) {</div><div class="line">			<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">				s = fn(bts)</div><div class="line">			}</div><div class="line">		})</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>在Mac mini M2上运行，go1.22.6 darwin/arm64，结果如下：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="attribute">goos</span>: <span class="string">darwin</span></div><div class="line"><span class="attribute">goarch</span>: <span class="string">arm64</span></div><div class="line"><span class="attribute">pkg</span>: <span class="string">github.com/smallnest/study/str2bytes</span></div><div class="line"></div><div class="line"><span class="gradle">BenchmarkStringToBytes<span class="regexp">/强制转换-8              	78813638	        14.73 ns/</span>op	      <span class="number">16</span> B<span class="regexp">/op	       1 allocs/</span>op</span></div><div class="line">BenchmarkStringToBytes<span class="regexp">/传统转换-8              	599346962	         2.010 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkStringToBytes<span class="regexp">/新型转换-8              	624976126	         1.929 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkStringToBytes<span class="regexp">/k8s转换-8             	887370499	         1.211 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div></pre></td></tr></table></figure>

<p>string 转 bytes性能最好的是k8s方案， 新型转换和传统转换性能差不多，新型方案略好，强制转换性能最差。</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"></div><div class="line">BenchmarkBytesToString<span class="regexp">/强制转换-8              	92011309	        12.68 ns/</span>op	      <span class="number">16</span> B<span class="regexp">/op	       1 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/传统转换-8              	815922964	         1.471 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/新型转换-8              	624965414	         1.922 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/k8s转换-8             	1000000000	         1.194 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div></pre></td></tr></table></figure>

<p>而对于 bytes 转 string，k8s方案性能最好，传统转换次之，新型转换性能再次之，强制转换性能非常不好。</p>
<p>在Linux amd64上运行，go1.22.0 linux/amd64，结果如下：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line">goos: linux</div><div class="line">goarch: amd64</div><div class="line">pkg: test</div><div class="line">cpu: Intel(R) Xeon(R) Platinum</div><div class="line">BenchmarkStringToBytes<span class="regexp">/强制转换-2                 	30606319	        42.02 ns/</span>op	      <span class="number">16</span> B<span class="regexp">/op	       1 allocs/</span>op</div><div class="line">BenchmarkStringToBytes<span class="regexp">/传统转换-2                 	315913948	         3.779 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkStringToBytes<span class="regexp">/新型转换-2                 	411972518	         2.753 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkStringToBytes<span class="regexp">/k8s转换-2                	449640819	         2.770 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line"></div><div class="line"></div><div class="line">BenchmarkBytesToString<span class="regexp">/强制转换-2                 	38716465	        29.18 ns/</span>op	      <span class="number">16</span> B<span class="regexp">/op	       1 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/传统转换-2                 	458832459	         2.593 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/新型转换-2                 	439537762	         2.762 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div><div class="line">BenchmarkBytesToString<span class="regexp">/k8s转换-2                	478885546	         2.375 ns/</span>op	       <span class="number">0</span> B<span class="regexp">/op	       0 allocs/</span>op</div></pre></td></tr></table></figure>

<p>整体上看，k8s方案、传统转换、新型转换性能都挺好，强制转换性能最差。k8s在bytes转string上性能最好。</p>
<h2 id="性能分析">性能分析</h2>
<p>等等，kubernates的讨论中，不是说Go1.22中string到bytes的转换可以直接用<code>[]byte(str)</code>了么？为什么这里的性能测试中，强制转换为什么性能那么差呢？</p>
<p>同时你也可以看到，强制转换每个op都会有一次内存分配:<code>1 allocs/op</code>,这严重影响了它的性能。</p>
<p>如果我们编写两个benchmark测试函数, 如下:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> BenchmarkStringToBytesRaw(b *testing.B) {</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">		_ = toRawBytes(s)</div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> BenchmarkBytesToStringRaw(b *testing.B) {</div><div class="line">	<span class="keyword">for</span> i :=<span class="number"> 0</span>; i &lt; b.N; i++ {</div><div class="line">		_ = toRawString(bts)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>执行:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="label">goos:</span> darwin</div><div class="line"><span class="label">goarch:</span> arm64</div><div class="line"><span class="label">pkg:</span> github.com/smallnest/study/str2bytes</div><div class="line">BenchmarkStringToBytesRaw-<span class="number">8</span>   	<span class="number">1000000000</span>	         <span class="number">0.2921</span> ns/op	       <span class="number">0</span> B/op	       <span class="number">0</span> allocs/op</div><div class="line">BenchmarkBytesToStringRaw-<span class="number">8</span>   	<span class="number">506502222</span>	         <span class="number">2.363</span> ns/op	       <span class="number">0</span> B/op	       <span class="number">0</span> allocs/op</div></pre></td></tr></table></figure>

<p>你会发现一个令人诧异的事情，强制转换的性能非常好，没有额外的内存分配(零拷贝)，设置字符串转换为bytes好太多。</p>
<p>这是咋回事呢？</p>
<p>当然聪明的你就会想到这个肯定是编译器做了优化，通过内联，把toRawBytes的函数调用展开了，这个好处是发现s</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div></pre></td><td class="code"><pre><div class="line"><span class="comment"># go test -gcflags="-m=2" -bench Raw -benchmem</span></div><div class="line"><span class="keyword">...</span></div><div class="line">./convert_test.go:<span class="number">48</span>:<span class="number">6</span>: can inline toRawBytes with cost <span class="number">10</span> as: func(string) []byte { <span class="keyword">if</span> len(s) == <span class="number">0</span> { <span class="keyword">return</span> nil }; <span class="keyword">return</span> ([]byte)(s) }</div><div class="line">./convert_test.go:<span class="number">55</span>:<span class="number">6</span>: can inline toRawString with cost <span class="number">10</span> as: func([]byte) string { <span class="keyword">if</span> len(b) == <span class="number">0</span> { <span class="keyword">return</span> <span class="string">""</span> }; <span class="keyword">return</span> string(b) }</div><div class="line"><span class="keyword">...</span></div><div class="line">./convert_test.go:<span class="number">101</span>:<span class="number">17</span>: ([]byte)(s) does not escape</div><div class="line">./convert_test.go:<span class="number">101</span>:<span class="number">17</span>: zero-copy string-&gt;[]byte conversion</div><div class="line"><span class="keyword">...</span></div></pre></td></tr></table></figure>

<p>通过<code>-gcflags=&quot;-m=2&quot;</code>, 我们可以观察内联和逃逸分析的结果，可以看到编译器优化了强制转换的函数，将string转换为bytes的操作优化为零拷贝。</p>
<p>而上一节我们的benchmark中，<code>bts = toRawBytes(s)</code>这个操作，会导致<code>([]byte)(s)</code>逃逸到堆上，这样就会有一次内存分配，并且性能底下。</p>
<p>所以你现在情况了，Go1.22确实对强制转换做了优化，但是这个优化是通过编译器的内联和逃逸分析来实现的，并不是所有的场景都能够优化到零拷贝。</p>
<p>谁能在编写代码的时候注意到这个优化呢，甚至准确的判断能否避免逃逸？所以可能在现阶段，我们还是会通过其他三种方式进行优化。</p>
<p>貌似Go 1.23会进一步优化，参考这个CL: <a href="https://github.com/golang/go/commit/925d2fb36c8e4c9c0e6e240a1621db36c34e5d31" target="_blank" rel="external">cmd/compile: restore zero-copy string-&gt;[]byte optimization</a></p>
<h2 id="k8s实现的问题">k8s实现的问题</h2>
<p>一开始，我们留了一个问题：<code>toK8sBytes</code>返回的[]byte的cap是多少？</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> toK8sBytes(s <span class="typename">string</span>) []<span class="typename">byte</span> {</div><div class="line">	<span class="keyword">return</span> *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&s))</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>len</code>是明确的，字段对应字符串的len字段，但是<code>cap</code>是多少呢？字符串可是没有<code>cap</code>字段的。</p>
<p>我们可以通过下面的代码来验证:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> Test_toK8sBytes(t *testing.T) {</div><div class="line">	a := *(*<span class="number">[3</span>]<span class="typename">int64</span>)(unsafe.Pointer(&s))</div><div class="line">	fmt.Printf(<span class="string">"%d, %d, %d\n"</span>, a<span class="number">[0</span>], a<span class="number">[1</span>], a<span class="number">[2</span>])</div><div class="line"></div><div class="line">	b := *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&s))</div><div class="line">	fmt.Printf(<span class="string">"%d, %d, %d\n"</span>, unsafe.SliceData(b), <span class="built_in">len</span>(b), <span class="built_in">cap</span>(b))</div><div class="line">}</div></pre></td></tr></table></figure>

<p>首先我们强制获取三个字段，第一个字段应该是字符串底层数据的指针。第二个字段是字符串的长度，第三个字段是什么呢？<br>同样我进行强制转换成slice of byte, 然后打印slice的底层数据指针，长度和容量。</p>
<p>输出结果如下(每次运行可能会得到不同的结果):</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">4375580047, 12, 4375914624</div><div class="line">4375580047, 12, 4375914624</div></pre></td></tr></table></figure>

<p>可以看到两者的结果是一致的，第一个值就是底层数据指针，第二个值是长度12，第三个啥也不是，就取得的内存中的值，随机的，并不是容量12。</p>
<p>所以通过这种方式转换的slice，其容量是不确定的，这个是一个问题，可能会导致一些问题，比如slice的append操作。</p>
<p>1、如果得到的slice的容量那么大，我们是不是尽情的append数据呢？</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">b := *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&s))</div><div class="line">fmt.Printf(<span class="string">"%d, %d, %d\n"</span>, unsafe.SliceData(b), <span class="built_in">len</span>(b), <span class="built_in">cap</span>(b))</div><div class="line"></div><div class="line">b = <span class="built_in">append</span>(b, <span class="string">'!'</span>)</div></pre></td></tr></table></figure>

<p>运行上面的测试会导致panic:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">unexpected fault address <span class="number">0x105020dfb</span></div><div class="line">fatal error: fault</div><div class="line">[signal SIGBUS: bus error code=<span class="number">0x1</span> addr=<span class="number">0x105020dfb</span> pc=<span class="number">0x10501ee98</span>]</div></pre></td></tr></table></figure>

<p>2、如果修改返回的bytes, 共享底层数据的原始string是不是也会发生变化？</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">b := *(*[]<span class="typename">byte</span>)(unsafe.Pointer(&s))</div><div class="line">fmt.Printf(<span class="string">"%d, %d, %d\n"</span>, unsafe.SliceData(b), <span class="built_in">len</span>(b), <span class="built_in">cap</span>(b))</div><div class="line">b<span class="number">[0</span>] = <span class="string">'H'</span></div></pre></td></tr></table></figure>

<p>运行上面的测试，会导致string的值<code>s</code>发生变化吗? 答案是不会，运行这段代码依然会导致panic&quot;</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">unexpected fault address <span class="number">0x104f1cdcf</span></div><div class="line">fatal error: fault</div><div class="line">[signal SIGBUS: bus error code=<span class="number">0x1</span> addr=<span class="number">0x104f1cdcf</span> pc=<span class="number">0x104f1ae74</span>]</div></pre></td></tr></table></figure>

<p>3、如果修改原始的bytes, 返回的string是不是也会发生变化？<br>我们知道，字符串是不可变的，所以这个问题的答案是？<br>测试代码如下：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">c := *(*<span class="typename">string</span>)(unsafe.Pointer(&bts))</div><div class="line">fmt.Printf(<span class="string">"%s\n"</span>, c)</div><div class="line">bts<span class="number">[0</span>] = <span class="string">'H'</span></div><div class="line">fmt.Printf(<span class="string">"%s\n"</span>, c)</div></pre></td></tr></table></figure>

<p>原始的bytes bts发生变化，返回的string c会发生变化吗？上面的代码打印出修改前后同一个字符串的值：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div></pre></td><td class="code"><pre><div class="line">hello, world</div><div class="line">Hello, world</div></pre></td></tr></table></figure>

<p>哈，字符串也变成了&quot;可变&quot;的了。</p>
<h2 id="总结">总结</h2>
<p>Go 1.22中，string和bytes的互转在部分场景(未逃逸的情况)下做了优化，实现了零拷贝，性能优秀，但是并不是所有的场景都能优化到零拷贝，所以我们、可以再等等，再等几个版本优化完全后再替换传统的互转方式。</p>
<p>在字符串和bytes互转的情况下，我们要确定bytes是不是可变的，这样会避免意外的情况发生，否则不妨采用强制转换的方式，安全第一。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>昨天公司群中同事提到 Go 1.22 中 string 和 bytes 的互转不需要再用 unsafe 那个包了，直接转就可以。我翻看了 Go 1.22 的 release notes 没找到相应的介绍，但是大家提到了 kubernetes 的 <a href="https://github.com/kubernetes/kubernetes/issues/124656" target="_blank" rel="external">issue</a> 中有这个说法：</p>
<blockquote>
<p>As of go 1.22, for string to bytes conversion, we can replace the usage of <code>unsafe.Slice(unsafe.StringData(s), len(s))</code> with type casting <code>[]bytes(str)</code>, without the worry of losing performance.</p>
<p>As of go 1.22, string to bytes conversion <code>[]bytes(str)</code> is faster than using the <code>unsafe</code> package. Both methods have 0 memory allocation now.</p>
</blockquote>
<blockquote>
<p>自 Go 1.22 起，对于 string 到 bytes 的转换，我们可以用类型转换 <code>[]bytes(str)</code> 来替换 <code>unsafe.Slice(unsafe.StringData(s), len(s))</code> 的用法，而不用担心性能损失。<br>自 Go 1.22 起，string 到 bytes 的转换 <code>[]bytes(str)</code> 比使用 unsafe 包更快。现在两种方法都不会有内存分配。</p>
</blockquote>
<p>这个说法让我很好奇，但是我还是想验证一下这个说法。</p>
<p>注意，这个说法只谈到了 string 到 bytes 的转换，并没有提到 bytes 到 string 的转换，这篇文章也会关注这两者的互转。</p>
<p>首先，让我们看看几种 string 和 bytes 的转换方式，然后我们再写 benchmark 比较它们之间的性能。<br>]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[没有什么不可能：修改 Go 结构体的私有字段]]></title>
    <link href="https://colobu.com/2024/08/08/access-the-unexported-fields/"/>
    <id>https://colobu.com/2024/08/08/access-the-unexported-fields/</id>
    <published>2024-08-08T15:12:59.000Z</published>
    <updated>2024-08-08T15:14:22.360Z</updated>
    <content type="html"><![CDATA[<p>在Go语言中,结构体(struct)中的字段如果是私有的,只能在定义该结构体的同一个包内访问。这是为了实现数据的封装和信息隐藏,提高代码的健壮性和安全性。</p>
<p>但是在某些情况下,我们可能需要在外部包中访问或修改结构体的私有字段。这时,我们可以使用 Go 语言提供的反射(reflect)机制来实现这一功能。</p>
<p>即使我们能够实现访问，这些字段你没有办法修改，如果尝试通过反射设置这些私有字段的值，会 panic。</p>
<p>甚至有时，我们通过反射设置一些变量或者字段的值的时候，会 panic, 报错 <code>panic: reflect: reflect.Value.Set using unaddressable value</code>。</p>
<p>在本文中，你将了解到：</p>
<ol>
<li>如何通过 hack 的方式访问外部结构体的私有字段</li>
<li>如何通过 hack 的方式设置外部结构体的私有字段</li>
<li>如何通过 hack 的方式设置 unaddressable 的值</li>
</ol>
<a id="more"></a>
<p>首先我先介绍通过反射设置值遇到的 unaddressable 的困境。</p>
<h2 id="通过反射设置一个变量的值">通过反射设置一个变量的值</h2>
<p>如果你使用过反射设置值的变量，你可能熟悉下面的代码，而且这个代码工作正常：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> x =<span class="number"> 47</span></div><div class="line"></div><div class="line">v := reflect.ValueOf(&x).Elem()</div><div class="line">fmt.Printf(<span class="string">"原始值: %d, CanSet: %v\n"</span>, v.Int(), v.CanSet()) <span class="comment">// 47, false</span></div><div class="line">v.Set(reflect.ValueOf<span class="number">(50</span>))</div></pre></td></tr></table></figure>

<p>注意这里传入给 <code>reflect.ValueOf</code> 的是 x 的指针 <code>&amp;x</code>, 所以这个 Value 值是 <code>addresable</code> 的，我们可以进行赋值。</p>
<p>如果把 <code>&amp;x</code> 替换成 <code>x</code>, 我们再尝试运行：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> x =<span class="number"> 47</span></div><div class="line"></div><div class="line">v := reflect.ValueOf(x)</div><div class="line">fmt.Printf(<span class="string">"Original value: %d, CanSet: %v\n"</span>, v.Int(), v.CanSet()) <span class="comment">// 47, false</span></div><div class="line">v.Set(reflect.ValueOf<span class="number">(50</span>))</div></pre></td></tr></table></figure>

<p>可以看到panic:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">Original value: <span class="number">47</span>, CanSet: <span class="keyword">false</span></div><div class="line">panic: reflect: reflect.Value.Set using unaddressable value</div><div class="line"></div><div class="line">goroutine <span class="number">1</span> [running]:</div><div class="line">reflect.flag.mustBeAssignableSlow(<span class="number">0</span>x1400012c410?)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">272</span> +<span class="number">0</span>x74</div><div class="line">reflect.flag.mustBeAssignable(...)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">259</span></div><div class="line">reflect.Value.Set({<span class="number">0</span>x104e13e40?, <span class="number">0</span>x104e965b8?, <span class="number">0</span>x104dec7e6?}, {<span class="number">0</span>x104e13e40?, <span class="number">0</span>x104e0ada0?, <span class="number">0</span>x2?})</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">2319</span> +<span class="number">0</span>x58</div><div class="line">main.setUnaddressableValue()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">27</span> +<span class="number">0</span>x1c0</div><div class="line">main.main()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">18</span> +<span class="number">0</span>x1c</div><div class="line">exit status <span class="number">2</span></div></pre></td></tr></table></figure>

<p>文章最后我会介绍如何通过 hack 的方式解决这个问题。</p>
<p>接下来我再介绍访问私有字段的问题。</p>
<h2 id="访问外部包的结构体的私有字段">访问外部包的结构体的私有字段</h2>
<p>我们先准备一个 <code>model</code> 包，在它之下定义了两个结构体：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> model</div><div class="line"></div><div class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</div><div class="line">	Name <span class="typename">string</span></div><div class="line">	age  <span class="typename">int</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> NewPerson(name <span class="typename">string</span>, age <span class="typename">int</span>) Person {</div><div class="line">	<span class="keyword">return</span> Person{</div><div class="line">		Name: name,</div><div class="line">		age:  age, <span class="comment">// unexported field</span></div><div class="line">	}</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> Teacher <span class="keyword">struct</span> {</div><div class="line">	Name <span class="typename">string</span></div><div class="line">	Age  <span class="typename">int</span> <span class="comment">// exported field</span></div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">func</span> NewTeacher(name <span class="typename">string</span>, age <span class="typename">int</span>) Teacher {</div><div class="line">	<span class="keyword">return</span> Teacher{</div><div class="line">		Name: name,</div><div class="line">		Age:  age,</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p>注意<code>Person</code>的<code>age</code>字段是私有的，<code>Teacher</code>的<code>Age</code>字段是公开的。</p>
<p>在我们的<code>main</code>函数中，你不能访问<code>Person</code>的<code>age</code>字段：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">package</span> main;</div><div class="line"></div><div class="line"><span class="keyword">import</span> (</div><div class="line">    <span class="string">"fmt"</span></div><div class="line">    <span class="string">"reflect"</span></div><div class="line">    <span class="string">"unsafe"</span></div><div class="line"></div><div class="line">    <span class="string">"github.com/smallnest/private/model"</span></div><div class="line">)   </div><div class="line"></div><div class="line"><span class="keyword">func</span> main() {</div><div class="line">    p := model.NewPerson(<span class="string">"Alice"</span>,<span class="number"> 30</span>)</div><div class="line">    fmt.Printf(<span class="string">"Person: %+v\n"</span>, p)</div><div class="line"></div><div class="line">    <span class="comment">// fmt.Println(p.age) // error: p.age undefined (cannot refer to unexported field or method age)</span></div><div class="line"></div><div class="line">    t := model.NewTeacher(<span class="string">"smallnest"</span>,<span class="number"> 18</span>)</div><div class="line">    fmt.Printf(<span class="string">"Teacher: %+v\n"</span>, t) <span class="comment">// Teacher: {Name:Alice Age:30}</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>那么真的就无法访问了吗？也不一定，我们可以通过反射的方式访问私有字段:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line">p := model.NewPerson(<span class="string">"Alice"</span>,<span class="number"> 30</span>)</div><div class="line"></div><div class="line">age := reflect.ValueOf(p).FieldByName(<span class="string">"age"</span>)</div><div class="line">fmt.Printf(<span class="string">"原始值: %d, CanSet: %v\n"</span>, age.Int(), age.CanSet()) <span class="comment">// 30, false</span></div></pre></td></tr></table></figure>

<p>运行这个程序，可以看到我们获得了这个私有字段<code>age</code>的值:</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">原始值: <span class="number">30</span>, CanSet: <span class="literal">false</span></div></pre></td></tr></table></figure>

<p>这样我们就绕过了Go语言的访问限制，访问了私有字段。</p>
<h2 id="设置结构体的私有字段">设置结构体的私有字段</h2>
<p>但是如果我们尝试修改这个私有字段的值，会 panic:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">age.SetInt<span class="number">(50</span>)</div></pre></td></tr></table></figure>

<p>或者</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">age.Set(reflect.ValueOf<span class="number">(50</span>))</div></pre></td></tr></table></figure>

<p>报错信息：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line">原始值: <span class="number">30</span>, CanSet: <span class="keyword">false</span></div><div class="line">panic: reflect: reflect.Value.SetInt using value obtained using unexported field</div><div class="line"></div><div class="line">goroutine <span class="number">1</span> [running]:</div><div class="line">reflect.flag.mustBeAssignableSlow(<span class="number">0</span>x2?)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">269</span> +<span class="number">0</span>xb4</div><div class="line">reflect.flag.mustBeAssignable(...)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">259</span></div><div class="line">reflect.Value.SetInt({<span class="number">0</span>x1050ac0c0?, <span class="number">0</span>x14000118f20?, <span class="number">0</span>x1050830a8?}, <span class="number">0</span>x32)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">2398</span> +<span class="number">0</span>x44</div><div class="line">main.setUnexportedField()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">37</span> +<span class="number">0</span>x1a0</div><div class="line">main.main()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">18</span> +<span class="number">0</span>x1c</div><div class="line">exit status <span class="number">2</span></div></pre></td></tr></table></figure>

<p>实际上，<code>reflect.Value</code>的<code>Set</code>方法会做一系列的检查，包括检查是否是<code>addressable</code>的，以及是否是exported的字段:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (v Value) Set(x Value) {</div><div class="line">	v.mustBeAssignable()</div><div class="line">	x.mustBeExported() <span class="comment">// do not let unexported x leak</span></div><div class="line">	...</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>v.mustBeAssignable()</code>检查是否是<code>addressable</code>的，而且是exported的字段:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> (f flag) mustBeAssignable() {</div><div class="line">	<span class="keyword">if</span> f&flagRO !=<span class="number"> 0</span> || f&flagAddr ==<span class="number"> 0</span> {</div><div class="line">		f.mustBeAssignableSlow()</div><div class="line">	}</div><div class="line">}</div><div class="line"><span class="keyword">func</span> (f flag) mustBeAssignableSlow() {</div><div class="line">	<span class="keyword">if</span> f ==<span class="number"> 0</span> {</div><div class="line">		<span class="built_in">panic</span>(&ValueError{valueMethodName(), Invalid})</div><div class="line">	}</div><div class="line">	<span class="comment">// Assignable if addressable and not read-only.</span></div><div class="line">	<span class="keyword">if</span> f&flagRO !=<span class="number"> 0</span> {</div><div class="line">		<span class="built_in">panic</span>(<span class="string">"reflect: "</span> + valueMethodName() + <span class="string">" using value obtained using unexported field"</span>)</div><div class="line">	}</div><div class="line">	<span class="keyword">if</span> f&flagAddr ==<span class="number"> 0</span> {</div><div class="line">		<span class="built_in">panic</span>(<span class="string">"reflect: "</span> + valueMethodName() + <span class="string">" using unaddressable value"</span>)</div><div class="line">	}</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>f&amp;flagRO == 0</code> 代表是可写的（<code>exported</code>），<code>f&amp;flagAddr != 0</code> 代表是<code>addressable</code>的,当这两个条件任意一个不满足时，就会报错。</p>
<p>既然我们明白了它检查的原理，我们就可以通过 hack 的方式绕过这个检查，设置私有字段的值。我们还是要使用<code>unsafe</code>代码。</p>
<p>这里我们以标准库的<code>sync.Mutex</code>结构体为例， <code>sync.Mutex</code>包含两个字段，这两个字段都是私有的：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Mutex <span class="keyword">struct</span> {</div><div class="line">    state <span class="typename">int32</span></div><div class="line">    sema  <span class="typename">uint32</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>正常情况下你只能通过<code>Mutex.Lock</code>和<code>Mutex.Unlock</code>来间接的修改这两个字段。</p>
<p>现在我们演示通过 hack 的方式修改<code>Mutex</code>的<code>state</code>字段的值：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> setPrivateField() {</div><div class="line">	<span class="keyword">var</span> mu sync.Mutex</div><div class="line">	mu.Lock()</div><div class="line"></div><div class="line">	field := reflect.ValueOf(&mu).Elem().FieldByName(<span class="string">"state"</span>)</div><div class="line">	state := field.Interface().(*<span class="typename">int32</span>)</div><div class="line">	fmt.Println(*state) <span class="comment">// ❶ </span></div><div class="line"></div><div class="line">	flagField := reflect.ValueOf(&field).Elem().FieldByName(<span class="string">"flag"</span>)</div><div class="line">	flagPtr := (*<span class="typename">uintptr</span>)(unsafe.Pointer(flagField.UnsafeAddr()))</div><div class="line"></div><div class="line">	<span class="comment">// 修改flag字段的值</span></div><div class="line">	*flagPtr &= ^<span class="typename">uintptr</span>(flagRO) <span class="comment">// ❷</span></div><div class="line">	field.Set(reflect.ValueOf(<span class="typename">int32</span><span class="number">(0</span>)))</div><div class="line"></div><div class="line">	mu.Lock() <span class="comment">// ❸</span></div><div class="line">	fmt.Println(*state)</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">type</span> flag <span class="typename">uintptr</span></div><div class="line"></div><div class="line"><span class="keyword">const</span> (</div><div class="line">	flagKindWidth        =<span class="number"> 5</span> <span class="comment">// there are 27 kinds</span></div><div class="line">	flagKindMask    flag =<span class="number"> 1</span>&lt;&lt;flagKindWidth -<span class="number"> 1</span></div><div class="line">	flagStickyRO    flag =<span class="number"> 1</span> &lt;&lt;<span class="number"> 5</span></div><div class="line">	flagEmbedRO     flag =<span class="number"> 1</span> &lt;&lt;<span class="number"> 6</span></div><div class="line">	flagIndir       flag =<span class="number"> 1</span> &lt;&lt;<span class="number"> 7</span></div><div class="line">	flagAddr        flag =<span class="number"> 1</span> &lt;&lt;<span class="number"> 8</span></div><div class="line">	flagMethod      flag =<span class="number"> 1</span> &lt;&lt;<span class="number"> 9</span></div><div class="line">	flagMethodShift      =<span class="number"> 10</span></div><div class="line">	flagRO          flag = flagStickyRO | flagEmbedRO</div><div class="line">)</div></pre></td></tr></table></figure>

<p>❶ 处我们已经介绍过了，访问私有字段的值，这里会打印出1<br>❶ 处我们清除了<code>flag</code>字段的<code>flagRO</code>标志位，这样就不会报<code>reflect: reflect.Value.SetInt using value obtained using unexported field</code>错误了<br>❸ 处不会导致二次加锁带来的死锁，因为<code>state</code>字段的值已经被修改为0了，所以不会阻塞。最后打印结果还是1</p>
<p>这样我们就可以实现了修改私有字段的值了。</p>
<h2 id="使用unexported字段的Value设置公开字段">使用unexported字段的Value设置公开字段</h2>
<p>看<code>reflect.Value.Set</code>的源码，我们可以看到它会检查参数的值是否<code>unexported</code>，如果是，就会报错,下面就是一个例子：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> setUnexportedField2() {</div><div class="line">	alice := model.NewPerson(<span class="string">"Alice"</span>,<span class="number"> 30</span>)</div><div class="line">	bob := model.NewTeacher(<span class="string">"Bob"</span>,<span class="number"> 40</span>)</div><div class="line"></div><div class="line">	bobAgent := reflect.ValueOf(&bob).Elem().FieldByName(<span class="string">"Age"</span>)</div><div class="line"></div><div class="line">	aliceAge := reflect.ValueOf(&alice).Elem().FieldByName(<span class="string">"age"</span>)</div><div class="line"></div><div class="line">	bobAgent.Set(aliceAge) <span class="comment">// ❹</span></div><div class="line"></div><div class="line">}</div></pre></td></tr></table></figure>

<p>注意❹处，我们尝试把<code>alice</code>的私有字段<code>age</code>的值赋值给<code>bob</code>的公开字段<code>Age</code>，这里会报错：</p>
<figure class="highlight shell"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div></pre></td><td class="code"><pre><div class="line"><span class="attribute">panic</span>: <span class="string">reflect: reflect.Value.Set using value obtained using unexported field</span></div><div class="line"></div><div class="line"><span class="gradle">goroutine <span class="number">1</span> [running]:</span></div><div class="line">reflect.flag.mustBeExportedSlow(<span class="number">0</span>x1400012a000?)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">250</span> +<span class="number">0</span>x70</div><div class="line">reflect.flag.mustBeExported(...)</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">241</span></div><div class="line">reflect.Value.Set({<span class="number">0</span>x102773a60?, <span class="number">0</span>x1400012a028?, <span class="number">0</span>x60?}, {<span class="number">0</span>x102773a60?, <span class="number">0</span>x1400012a010?, <span class="number">0</span>x1027002b8?})</div><div class="line">	<span class="regexp">/usr/</span>local<span class="regexp">/go/</span>src<span class="regexp">/reflect/</span>value.go:<span class="number">2320</span> +<span class="number">0</span>x88</div><div class="line">main.setUnexportedField2()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">50</span> +<span class="number">0</span>x168</div><div class="line">main.main()</div><div class="line">	<span class="regexp">/Users/</span>smallnest<span class="regexp">/workspace/</span>study<span class="regexp">/private/m</span>ain.go:<span class="number">18</span> +<span class="number">0</span>x1c</div><div class="line">exit status <span class="number">2</span></div></pre></td></tr></table></figure>

<p>原因<code>alice</code>的<code>age</code>值被识别为私有字段，它是不能用来赋值给公开字段的。</p>
<p>有了上一节的经验，我们同样可以绕过这个检查，实现这个赋值：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> setUnexportedField2() {</div><div class="line">	alice := model.NewPerson(<span class="string">"Alice"</span>,<span class="number"> 30</span>)</div><div class="line">	bob := model.NewTeacher(<span class="string">"Bob"</span>,<span class="number"> 40</span>)</div><div class="line"></div><div class="line">	bobAgent := reflect.ValueOf(&bob).Elem().FieldByName(<span class="string">"Age"</span>)</div><div class="line"></div><div class="line">	aliceAge := reflect.ValueOf(&alice).Elem().FieldByName(<span class="string">"age"</span>)</div><div class="line">	<span class="comment">// 修改flag字段的值</span></div><div class="line">	flagField := reflect.ValueOf(&aliceAge).Elem().FieldByName(<span class="string">"flag"</span>)</div><div class="line">	flagPtr := (*<span class="typename">uintptr</span>)(unsafe.Pointer(flagField.UnsafeAddr()))</div><div class="line">	*flagPtr &= ^<span class="typename">uintptr</span>(flagRO) <span class="comment">// ❺</span></div><div class="line"></div><div class="line">	bobAgent.Set(reflect.ValueOf<span class="number">(50</span>))</div><div class="line">	bobAgent.Set(aliceAge) <span class="comment">// ❻</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>❺ 处我们修改了<code>aliceAge</code>的<code>flag</code>字段，去掉了<code>flagRO</code>标志位，这样就不会报错了,❻处我们成功的把<code>alice</code>的私有字段<code>age</code>的值赋值给<code>bob</code>的公开字段<code>Age</code>。</p>
<p>这样我们就可以实现了使用私有字段的值给其他Value值进行赋值了。</p>
<h2 id="给unaddressable的值设置值">给unaddressable的值设置值</h2>
<p>回到最初的问题，我们尝试给一个unaddressable的值设置值，会报错。</p>
<p>结合上面的hack手段，我们也可以绕过限制，给unaddressable的值设置值：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> setUnaddressableValue() {</div><div class="line">	<span class="keyword">var</span> x =<span class="number"> 47</span></div><div class="line"></div><div class="line">	v := reflect.ValueOf(x)</div><div class="line">	fmt.Printf(<span class="string">"原始值: %d, CanSet: %v\n"</span>, v.Int(), v.CanSet()) <span class="comment">// 47, false</span></div><div class="line">	<span class="comment">// v.Set(reflect.ValueOf(50))</span></div><div class="line"></div><div class="line">	flagField := reflect.ValueOf(&v).Elem().FieldByName(<span class="string">"flag"</span>)</div><div class="line">	flagPtr := (*<span class="typename">uintptr</span>)(unsafe.Pointer(flagField.UnsafeAddr()))</div><div class="line"></div><div class="line">	<span class="comment">// 修改flag字段的值</span></div><div class="line">	*flagPtr |= <span class="typename">uintptr</span>(flagAddr)          <span class="comment">// 设置可寻址标志位</span></div><div class="line">	fmt.Printf(<span class="string">"CanSet: %v\n"</span>, v.CanSet()) <span class="comment">// true</span></div><div class="line">	v.SetInt<span class="number">(50</span>)</div><div class="line">	fmt.Printf(<span class="string">"修改后的值: %d\n"</span>, v.Int()) <span class="comment">// 50</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>运行这个程序，不会报错，可以看到我们成功的给unaddressable的值设置了新的值。</p>
<h2 id="回顾">回顾</h2>
<p>我们通过修改<code>Value</code>值的flag标志位，可以绕过<code>reflect</code>的检查，实现了访问私有字段、设置私有字段的值、用私有字段设置值，以及给unaddressable的值设置值。</p>
<p>这些都是<code>unsafe</code>的方式，一般情况下不鼓励进行这样的hack操作，但是这种技术也不是完全没有用户，如果你正在写一个debugger，用户在断点出可能想修改某些值，或者你在写深拷贝的库，或者编写某种ORM库，或者你就像突破限制，访问第三方不愿意公开的字段，你有可能会采用这种非常规的技术。</p>
<p>我是鸟窝，一位老程序员，在百度写代码。如果你感觉这篇文章给你带来了帮助，请点击下方点赞按钮或者评论区进行评论。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>在Go语言中,结构体(struct)中的字段如果是私有的,只能在定义该结构体的同一个包内访问。这是为了实现数据的封装和信息隐藏,提高代码的健壮性和安全性。</p>
<p>但是在某些情况下,我们可能需要在外部包中访问或修改结构体的私有字段。这时,我们可以使用 Go 语言提供的反射(reflect)机制来实现这一功能。</p>
<p>即使我们能够实现访问，这些字段你没有办法修改，如果尝试通过反射设置这些私有字段的值，会 panic。</p>
<p>甚至有时，我们通过反射设置一些变量或者字段的值的时候，会 panic, 报错 <code>panic: reflect: reflect.Value.Set using unaddressable value</code>。</p>
<p>在本文中，你将了解到：</p>
<ol>
<li>如何通过 hack 的方式访问外部结构体的私有字段</li>
<li>如何通过 hack 的方式设置外部结构体的私有字段</li>
<li>如何通过 hack 的方式设置 unaddressable 的值</li>
</ol>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[使用eBPF编写系统调用跟踪器]]></title>
    <link href="https://colobu.com/2024/08/04/beetracer/"/>
    <id>https://colobu.com/2024/08/04/beetracer/</id>
    <published>2024-08-04T09:27:25.000Z</published>
    <updated>2024-08-08T15:12:09.510Z</updated>
    <content type="html"><![CDATA[<h2 id="先决条件">先决条件</h2>
<p>系统调用、eBPF、C语言、底层编程基础。</p>
<h2 id="简介">简介</h2>
<p>eBPF(扩展的伯克利数据包过滤器)是一项允许用户在内核中运行自定义程序的技术。BPF或cBPF(经典BPF)是eBPF的前身,它提供了一种简单高效的方法来基于预定义规则过滤数据包。与内核模块相比,eBPF程序提供了更高的安全性、可移植性和可维护性。现有多种高级方法可用于处理eBPF程序,如Cilium的Go语言库、bpftrace、libbpf等。</p>
<ul>
<li><code>注意</code>: 本文要求读者对<code>eBPF</code>有基本了解。如果你不熟悉它,<code>ebpf.io</code>上的这篇文章是很好的参考资料。</li>
</ul>
<a id="more"></a>
<h2 id="目标">目标</h2>
<p>你应该已经熟悉著名的工具 <code>strace</code>。我们将使用eBPF开发类似的工具。例如:</p>
<figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">./beetrace /bin/ls</div></pre></td></tr></table></figure>

<p>以下是该文本的地道中文翻译：</p>
<h2 id="概念">概念</h2>
<p>在开始编写我们的工具之前，我们需要熟悉一些关键概念。</p>
<ol>
<li><code>跟踪点（Tracepoints）</code>：这些是放置在 Linux 内核代码各个部分的检测点。它们提供了一种方法，可以在不修改内核源代码的情况下，钩入内核中的特定事件或代码路径。可用于跟踪的事件可以在 <code>/sys/kernel/debug/tracing/events</code> 中找到。</li>
<li><code>SEC</code> 宏：它在目标 ELF 文件中创建一个新的段，段名与跟踪点的名称相同。例如，<code>SEC(tracepoint/raw_syscalls/sys_enter)</code> 创建了一个具有这个名称的新段。可以使用 readelf 命令查看这些段。</li>
</ol>
<figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">readelf <span class="operator">-s</span> --wide somefile.o</div></pre></td></tr></table></figure>

<ol start="3">
<li><code>映射（Maps）</code>：这些是可以从 eBPF 程序和用户空间运行的应用程序中访问的共享数据结构。</li>
</ol>
<h2 id="编写_eBPF_程序">编写 eBPF 程序</h2>
<p>由于 Linux 内核中存在大量的系统调用，我们不会编写一个全面的工具来跟踪所有系统调用。相反，我们将专注于跟踪几个常见的系统调用。为了实现这一目标，我们将编写两类程序：eBPF 程序和加载器（用于将 BPF 对象加载到内核并将其附加进来）。</p>
<p>让我们首先创建一些数据结构来进行初始设置:</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// controller.h</span></div><div class="line"></div><div class="line"><span class="comment">// SYS_ENTER : for retrieving system call arguments</span></div><div class="line"><span class="comment">// SYS_EXIT : for retrieving the return values of syscalls</span></div><div class="line"></div><div class="line"><span class="keyword">typedef</span> <span class="keyword">enum</span></div><div class="line">{</div><div class="line">    SYS_ENTER,</div><div class="line">    SYS_EXIT</div><div class="line">} event_mode;</div><div class="line"></div><div class="line"><span class="keyword">struct</span> inner_syscall_info</div><div class="line">{</div><div class="line">    <span class="keyword">union</span></div><div class="line">    {</div><div class="line">        <span class="keyword">struct</span></div><div class="line">        {</div><div class="line">            <span class="comment">// For SYS_ENTER mode</span></div><div class="line">            <span class="keyword">char</span> name[<span class="number">32</span>];</div><div class="line">            <span class="keyword">int</span> num_args;</div><div class="line">            <span class="keyword">long</span> syscall_nr;</div><div class="line">            <span class="keyword">void</span> *args[MAX_ARGS];</div><div class="line">        };</div><div class="line">        <span class="keyword">long</span> retval; <span class="comment">// For SYS_EXIT mode</span></div><div class="line">    };</div><div class="line">    event_mode mode;</div><div class="line">};</div><div class="line"></div><div class="line"><span class="keyword">struct</span> default_syscall_info{</div><div class="line">    <span class="keyword">char</span> name[<span class="number">32</span>];</div><div class="line">    <span class="keyword">int</span> num_args;</div><div class="line">};</div><div class="line"></div><div class="line"><span class="comment">// Array for storing the name and argument count of system calls</span></div><div class="line"><span class="keyword">const</span> <span class="keyword">struct</span> default_syscall_info syscalls[MAX_SYSCALL_NR] = {</div><div class="line">    [SYS_fork] = {<span class="string">"fork"</span>, <span class="number">0</span>},</div><div class="line">    [SYS_alarm] = {<span class="string">"alarm"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_brk] = {<span class="string">"brk"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_close] = {<span class="string">"close"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_exit] = {<span class="string">"exit"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_exit_group] = {<span class="string">"exit_group"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_set_tid_address] = {<span class="string">"set_tid_address"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_set_robust_list] = {<span class="string">"set_robust_list"</span>, <span class="number">1</span>},</div><div class="line">    [SYS_access] = {<span class="string">"access"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_arch_prctl] = {<span class="string">"arch_prctl"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_kill] = {<span class="string">"kill"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_listen] = {<span class="string">"listen"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_munmap] = {<span class="string">"sys_munmap"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_open] = {<span class="string">"open"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_stat] = {<span class="string">"stat"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_fstat] = {<span class="string">"fstat"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_lstat] = {<span class="string">"lstat"</span>, <span class="number">2</span>},</div><div class="line">    [SYS_accept] = {<span class="string">"accept"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_connect] = {<span class="string">"connect"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_execve] = {<span class="string">"execve"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_ioctl] = {<span class="string">"ioctl"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_getrandom] = {<span class="string">"getrandom"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_lseek] = {<span class="string">"lseek"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_poll] = {<span class="string">"poll"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_read] = {<span class="string">"read"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_write] = {<span class="string">"write"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_mprotect] = {<span class="string">"mprotect"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_openat] = {<span class="string">"openat"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_socket] = {<span class="string">"socket"</span>, <span class="number">3</span>},</div><div class="line">    [SYS_newfstatat] = {<span class="string">"newfstatat"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_pread64] = {<span class="string">"pread64"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_prlimit64] = {<span class="string">"prlimit64"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_rseq] = {<span class="string">"rseq"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_sendfile] = {<span class="string">"sendfile"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_socketpair] = {<span class="string">"socketpair"</span>, <span class="number">4</span>},</div><div class="line">    [SYS_mmap] = {<span class="string">"mmap"</span>, <span class="number">6</span>},</div><div class="line">    [SYS_recvfrom] = {<span class="string">"recvfrom"</span>, <span class="number">6</span>},</div><div class="line">    [SYS_sendto] = {<span class="string">"sendto"</span>, <span class="number">6</span>},</div><div class="line">};</div></pre></td></tr></table></figure>

<p>加载器将读取用户通过命令行参数提供的待追踪 ELF 文件的路径。然后，加载器会创建一个子进程，并使用 <code>execve</code> 来运行命令行参数中指定的程序。</p>
<p>父进程将处理加载和附加 eBPF 程序所需的所有设置。它还执行一项关键任务：通过 BPF 哈希映射将子进程的 ID 发送给 eBPF 程序。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// loader.c</span></div><div class="line"></div><div class="line"><span class="keyword">int</span> main(<span class="keyword">int</span> argc, <span class="keyword">char</span> **argv)</div><div class="line">{</div><div class="line">  <span class="keyword">if</span> (argc &lt; <span class="number">2</span>)</div><div class="line">  {</div><div class="line">    fatal_error(<span class="string">"Usage: ./beetrace &lt;path_to_program&gt;"</span>);</div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *file_path = argv[<span class="number">1</span>];</div><div class="line"></div><div class="line">  pid_t pid = fork();</div><div class="line">  <span class="keyword">if</span> (pid == <span class="number">0</span>)</div><div class="line">  {</div><div class="line">    <span class="comment">// Child process</span></div><div class="line">    <span class="keyword">int</span> fd = open(<span class="string">"/dev/null"</span>, O_WRONLY);</div><div class="line">    <span class="keyword">if</span>(fd==-<span class="number">1</span>){</div><div class="line">        <span class="comment">// error</span></div><div class="line">    }</div><div class="line">    dup2(fd, <span class="number">1</span>); <span class="comment">// disable stdout for the child process</span></div><div class="line">    sleep(<span class="number">2</span>); <span class="comment">// wait for the parent process to do the required setup for tracing</span></div><div class="line">    execve(file_path, NULL, NULL);</div><div class="line">  }</div><div class="line">  <span class="keyword">else</span>{</div><div class="line">    <span class="comment">// Parent process</span></div><div class="line">  }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>要追踪系统调用，我们需要编写由 <code>tracepoint/raw_syscalls/sys_enter</code> 和 <code>tracepoint/raw_syscalls/sys_exit</code> 跟踪点触发的 eBPF 程序。这些跟踪点提供了对系统调用号和参数的访问。对于给定的系统调用，<code>tracepoint/raw_syscalls/sys_enter</code> 跟踪点总是在 <code>tracepoint/raw_syscalls/sys_exit</code> 跟踪点之前触发。我们可以使用前者获取系统调用参数，使用后者获取返回值。</p>
<p>此外，我们将使用 eBPF 映射在用户空间程序和我们的 eBPF 程序之间共享信息。具体来说，我们将使用两种类型的 eBPF 映射：哈希映射和环形缓冲区。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// controller.c</span></div><div class="line"></div><div class="line"><span class="comment">// Hashmap</span></div><div class="line"><span class="keyword">struct</span></div><div class="line">{</div><div class="line">  __uint(type, BPF_MAP_TYPE_HASH);</div><div class="line">  __uint(key_size, <span class="number">10</span>);</div><div class="line">  __uint(value_size, <span class="number">4</span>);</div><div class="line">  __uint(max_entries, <span class="number">256</span> * <span class="number">1024</span>);</div><div class="line">} pid_hashmap SEC(<span class="string">".maps"</span>);</div><div class="line"></div><div class="line"><span class="comment">// Ring buffer</span></div><div class="line"><span class="keyword">struct</span></div><div class="line">{</div><div class="line">  __uint(type, BPF_MAP_TYPE_RINGBUF);</div><div class="line">  __uint(max_entries, <span class="number">256</span> * <span class="number">1024</span>);</div><div class="line">} syscall_info_buffer SEC(<span class="string">".maps"</span>);</div></pre></td></tr></table></figure>

<p>确定了映射关系之后，我们就可以动手写代码了。首先，让我们来编写针对追踪点 tracepoint/raw_syscalls/sys_enter 的程序代码。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// loader.c</span></div><div class="line"></div><div class="line">SEC(<span class="string">"tracepoint/raw_syscalls/sys_enter"</span>)</div><div class="line"><span class="keyword">int</span> detect_syscall_enter(<span class="keyword">struct</span> trace_event_raw_sys_enter *ctx)</div><div class="line">{</div><div class="line">  <span class="comment">// Retrieve the system call number</span></div><div class="line">  <span class="keyword">long</span> syscall_nr = ctx-&gt;id;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *key = <span class="string">"child_pid"</span>;</div><div class="line">  <span class="keyword">int</span> target_pid;</div><div class="line"></div><div class="line">  <span class="comment">// Reading the process id of the child process in userland</span></div><div class="line">  <span class="keyword">void</span> *value = bpf_map_lookup_elem(&pid_hashmap, key);</div><div class="line">  <span class="keyword">void</span> *args[MAX_ARGS];</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (value)</div><div class="line">  {</div><div class="line">    target_pid = *(<span class="keyword">int</span> *)value;</div><div class="line"></div><div class="line">    <span class="comment">// PID of the process that executed the current system call</span></div><div class="line">    pid_t pid = bpf_get_current_pid_tgid() & <span class="number">0xffffffff</span>;</div><div class="line">    <span class="keyword">if</span> (pid == target_pid && syscall_nr &gt;= <span class="number">0</span> && syscall_nr &lt; MAX_SYSCALL_NR)</div><div class="line">    {</div><div class="line"></div><div class="line">      <span class="keyword">int</span> idx = syscall_nr;</div><div class="line">      <span class="comment">// Reserve space in the ring buffer</span></div><div class="line">      <span class="keyword">struct</span> inner_syscall_info *info = bpf_ringbuf_reserve(&syscall_info_buffer, <span class="keyword">sizeof</span>(<span class="keyword">struct</span> inner_syscall_info), <span class="number">0</span>);</div><div class="line">      <span class="keyword">if</span> (!info)</div><div class="line">      {</div><div class="line">        bpf_printk(<span class="string">"bpf_ringbuf_reserve failed"</span>);</div><div class="line">        <span class="keyword">return</span> <span class="number">1</span>;</div><div class="line">      }</div><div class="line"></div><div class="line">      <span class="comment">// Copy the syscall name into info-&gt;name</span></div><div class="line">      bpf_probe_read_kernel_str(info-&gt;name, <span class="keyword">sizeof</span>(syscalls[syscall_nr].name), syscalls[syscall_nr].name);</div><div class="line">      <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; MAX_ARGS; i++)</div><div class="line">      {</div><div class="line">        info-&gt;args[i] = (<span class="keyword">void</span> *)BPF_CORE_READ(ctx, args[i]);</div><div class="line">      }</div><div class="line">      info-&gt;num_args = syscalls[syscall_nr].num_args;</div><div class="line">      info-&gt;syscall_nr = syscall_nr;</div><div class="line">      info-&gt;mode = SYS_ENTER;</div><div class="line">      <span class="comment">// Insert into ring buffer</span></div><div class="line">      bpf_ringbuf_submit(info, <span class="number">0</span>);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div></pre></td></tr></table></figure>

<p>同理，我们也能编写用于读取返回值并将其传递给用户态空间的程序代码。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// controller.c</span></div><div class="line"></div><div class="line">SEC(<span class="string">"tracepoint/raw_syscalls/sys_exit"</span>)</div><div class="line"><span class="keyword">int</span> detect_syscall_exit(<span class="keyword">struct</span> trace_event_raw_sys_exit *ctx)</div><div class="line">{</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *key = <span class="string">"child_pid"</span>;</div><div class="line">  <span class="keyword">void</span> *value = bpf_map_lookup_elem(&pid_hashmap, key);</div><div class="line">  pid_t pid, target_pid;</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (value)</div><div class="line">  {</div><div class="line">    pid = bpf_get_current_pid_tgid() & <span class="number">0xffffffff</span>;</div><div class="line">    target_pid = *(pid_t *)value;</div><div class="line">    <span class="keyword">if</span> (pid == target_pid)</div><div class="line">    {</div><div class="line">      <span class="keyword">struct</span> inner_syscall_info *info = bpf_ringbuf_reserve(&syscall_info_buffer, <span class="keyword">sizeof</span>(<span class="keyword">struct</span> inner_syscall_info), <span class="number">0</span>);</div><div class="line">      <span class="keyword">if</span> (!info)</div><div class="line">      {</div><div class="line">        bpf_printk(<span class="string">"bpf_ringbuf_reserve failed"</span>);</div><div class="line">        <span class="keyword">return</span> <span class="number">1</span>;</div><div class="line">      }</div><div class="line">      info-&gt;mode = SYS_EXIT;</div><div class="line">      info-&gt;retval = ctx-&gt;ret;</div><div class="line">      bpf_ringbuf_submit(info, <span class="number">0</span>);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div></pre></td></tr></table></figure>

<p>现在，让我们来完善加载器程序中父进程的功能部分。但在进行之前，我们需要理解几个关键函数的工作原理。<br>1、<code>bpf_object__open</code>: 通过打开由传递路径指向的 BPF ELF 对象文件并在内存中加载它，创建一个 <code>bpf_object</code> 结构体实例。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">struct</span> bpf_object *bpf_object__open(<span class="keyword">const</span> <span class="keyword">char</span> *path);</div></pre></td></tr></table></figure>

<p>2、<code>bpf_object__load</code>: 将 BPF 对象加载到内核中。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">int</span> bpf_object__load(<span class="keyword">struct</span> bpf_object *obj);</div></pre></td></tr></table></figure>

<p>3、<code>bpf_object__find_program_by_name</code>: 返回指向有效 BPF 程序的指针。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">struct</span> bpf_program *bpf_object__find_program_by_name(<span class="keyword">const</span> <span class="keyword">struct</span> bpf_object *obj, <span class="keyword">const</span> <span class="keyword">char</span> *name);</div></pre></td></tr></table></figure>

<p>4、<code>bpf_program__attach</code>: 根据自动检测的程序类型、附加类型和适用的额外参数，将 BPF 程序附加到内核。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">struct</span> bpf_link *bpf_program__attach(<span class="keyword">const</span> <span class="keyword">struct</span> bpf_program *prog);</div></pre></td></tr></table></figure>

<p>5、<code>bpf_map__update_elem</code>: 允许在与提供的键对应的 BPF 映射中插入或更新值。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">int</span> bpf_map__update_elem(<span class="keyword">const</span> <span class="keyword">struct</span> bpf_map *<span class="built_in">map</span>, <span class="keyword">const</span> <span class="keyword">void</span> *key, size_t key_sz, <span class="keyword">const</span> <span class="keyword">void</span> *value, size_t value_sz, __u64 flags);</div></pre></td></tr></table></figure>

<p>6、<code>bpf_object__find_map_fd_by_name</code>: 给定一个 BPF 映射名称，返回该映射的文件描述符。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">int</span> bpf_object__find_map_fd_by_name(<span class="keyword">const</span> <span class="keyword">struct</span> bpf_object *obj, <span class="keyword">const</span> <span class="keyword">char</span> *name);</div></pre></td></tr></table></figure>

<p>7、<code>ring_buffer__new</code>: 返回指向环形缓冲区的指针。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">struct</span> ring_buffer *ring_buffer__new(<span class="keyword">int</span> map_fd, ring_buffer_sample_fn sample_cb, <span class="keyword">void</span> *ctx, <span class="keyword">const</span> <span class="keyword">struct</span> ring_buffer_opts *opts);</div></pre></td></tr></table></figure>

<p>第二个参数必须是一个可用于处理从环形缓冲区接收的数据的回调函数。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">bool</span> initialized = <span class="keyword">false</span>;</div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">int</span> syscall_logger(<span class="keyword">void</span> *ctx, <span class="keyword">void</span> *data, size_t len)</div><div class="line">{</div><div class="line">  <span class="keyword">struct</span> inner_syscall_info *info = (<span class="keyword">struct</span> inner_syscall_info *)data;</div><div class="line">  <span class="keyword">if</span> (!info)</div><div class="line">  {</div><div class="line">    <span class="keyword">return</span> -<span class="number">1</span>;</div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (info-&gt;mode == SYS_ENTER)</div><div class="line">  {</div><div class="line">    initialized = <span class="keyword">true</span>;</div><div class="line">    <span class="built_in">printf</span>(<span class="string">"%s("</span>, info-&gt;name);</div><div class="line">    <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; info-&gt;num_args; i++)</div><div class="line">    {</div><div class="line">      <span class="built_in">printf</span>(<span class="string">"%p,"</span>, info-&gt;args[i]);</div><div class="line">    }</div><div class="line">    <span class="built_in">printf</span>(<span class="string">"\b) = "</span>);</div><div class="line">  }</div><div class="line">  <span class="keyword">else</span> <span class="keyword">if</span> (info-&gt;mode == SYS_EXIT)</div><div class="line">  {</div><div class="line">    <span class="keyword">if</span> (initialized)</div><div class="line">    {</div><div class="line">      <span class="built_in">printf</span>(<span class="string">"0x%lx\n"</span>, info-&gt;retval);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div></pre></td></tr></table></figure>

<p>它会打印系统调用的名称和参数。</p>
<p>8、<code>ring_buffer__consume</code>: 此函数处理环形缓冲区中可用的事件。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div></pre></td><td class="code"><pre><div class="line">LIBBPF_API <span class="keyword">int</span> ring_buffer__consume(<span class="keyword">struct</span> ring_buffer *rb);</div></pre></td></tr></table></figure>

<p>现在我们有了编写加载器所需的一切要素。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div><div class="line">98</div><div class="line">99</div><div class="line">100</div><div class="line">101</div><div class="line">102</div><div class="line">103</div><div class="line">104</div><div class="line">105</div><div class="line">106</div><div class="line">107</div><div class="line">108</div><div class="line">109</div><div class="line">110</div><div class="line">111</div><div class="line">112</div><div class="line">113</div><div class="line">114</div><div class="line">115</div><div class="line">116</div><div class="line">117</div><div class="line">118</div><div class="line">119</div><div class="line">120</div><div class="line">121</div><div class="line">122</div><div class="line">123</div><div class="line">124</div><div class="line">125</div><div class="line">126</div><div class="line">127</div><div class="line">128</div><div class="line">129</div><div class="line">130</div><div class="line">131</div><div class="line">132</div><div class="line">133</div><div class="line">134</div><div class="line">135</div><div class="line">136</div><div class="line">137</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// loader.c</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;bpf/libbpf.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> "controller.h"</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;fcntl.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;stdio.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;stdlib.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;sys/wait.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;unistd.h&gt;</span></div><div class="line"></div><div class="line"><span class="keyword">void</span> fatal_error(<span class="keyword">const</span> <span class="keyword">char</span> *message)</div><div class="line">{</div><div class="line">  <span class="built_in">puts</span>(message);</div><div class="line">  <span class="built_in">exit</span>(<span class="number">1</span>);</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">bool</span> initialized = <span class="keyword">false</span>;</div><div class="line"></div><div class="line"><span class="keyword">static</span> <span class="keyword">int</span> syscall_logger(<span class="keyword">void</span> *ctx, <span class="keyword">void</span> *data, size_t len)</div><div class="line">{</div><div class="line">  <span class="keyword">struct</span> inner_syscall_info *info = (<span class="keyword">struct</span> inner_syscall_info *)data;</div><div class="line">  <span class="keyword">if</span> (!info)</div><div class="line">  {</div><div class="line">    <span class="keyword">return</span> -<span class="number">1</span>;</div><div class="line">  }</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (info-&gt;mode == SYS_ENTER)</div><div class="line">  {</div><div class="line">    initialized = <span class="keyword">true</span>;</div><div class="line">    <span class="built_in">printf</span>(<span class="string">"%s("</span>, info-&gt;name);</div><div class="line">    <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; info-&gt;num_args; i++)</div><div class="line">    {</div><div class="line">      <span class="built_in">printf</span>(<span class="string">"%p,"</span>, info-&gt;args[i]);</div><div class="line">    }</div><div class="line">    <span class="built_in">printf</span>(<span class="string">"\b) = "</span>);</div><div class="line">  }</div><div class="line">  <span class="keyword">else</span> <span class="keyword">if</span> (info-&gt;mode == SYS_EXIT)</div><div class="line">  {</div><div class="line">    <span class="keyword">if</span> (initialized)</div><div class="line">    {</div><div class="line">      <span class="built_in">printf</span>(<span class="string">"0x%lx\n"</span>, info-&gt;retval);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">int</span> main(<span class="keyword">int</span> argc, <span class="keyword">char</span> **argv)</div><div class="line">{</div><div class="line">  <span class="keyword">int</span> status;</div><div class="line">  <span class="keyword">struct</span> bpf_object *obj;</div><div class="line">  <span class="keyword">struct</span> bpf_program *enter_prog, *exit_prog;</div><div class="line">  <span class="keyword">struct</span> bpf_map *syscall_map;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *obj_name = <span class="string">"controller.o"</span>;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *map_name = <span class="string">"pid_hashmap"</span>;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *enter_prog_name = <span class="string">"detect_syscall_enter"</span>; </div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *exit_prog_name = <span class="string">"detect_syscall_exit"</span>;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *syscall_info_bufname = <span class="string">"syscall_info_buffer"</span>;</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (argc &lt; <span class="number">2</span>)</div><div class="line">  {</div><div class="line">    fatal_error(<span class="string">"Usage: ./beetrace &lt;path_to_program&gt;"</span>);</div><div class="line">  }</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *file_path = argv[<span class="number">1</span>];</div><div class="line"></div><div class="line">  pid_t pid = fork();</div><div class="line">  <span class="keyword">if</span> (pid == <span class="number">0</span>)</div><div class="line">  {</div><div class="line">    <span class="keyword">int</span> fd = open(<span class="string">"/dev/null"</span>, O_WRONLY);</div><div class="line">    <span class="keyword">if</span>(fd==-<span class="number">1</span>){</div><div class="line">      fatal_error(<span class="string">"failed to open /dev/null"</span>);</div><div class="line">    }</div><div class="line">    dup2(fd, <span class="number">1</span>);</div><div class="line">    sleep(<span class="number">2</span>);</div><div class="line">    execve(file_path, NULL, NULL);</div><div class="line">  }</div><div class="line">  <span class="keyword">else</span></div><div class="line">  {</div><div class="line">    <span class="built_in">printf</span>(<span class="string">"Spawned child process with a PID of %d\n"</span>, pid);</div><div class="line">    obj = bpf_object__open(obj_name);</div><div class="line">    <span class="keyword">if</span> (!obj)</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to open the BPF object"</span>);</div><div class="line">    }</div><div class="line">    <span class="keyword">if</span> (bpf_object__load(obj))</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to load the BPF object into kernel"</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    enter_prog = bpf_object__find_program_by_name(obj, enter_prog_name);</div><div class="line">    exit_prog = bpf_object__find_program_by_name(obj, exit_prog_name);</div><div class="line"></div><div class="line">    <span class="keyword">if</span> (!enter_prog || !exit_prog)</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to find the BPF program"</span>);</div><div class="line">    }</div><div class="line">    <span class="keyword">if</span> (!bpf_program__attach(enter_prog) || !bpf_program__attach(exit_prog))</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to attach the BPF program"</span>);</div><div class="line">    }</div><div class="line">    syscall_map = bpf_object__find_map_by_name(obj, map_name);</div><div class="line">    <span class="keyword">if</span> (!syscall_map)</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to find the BPF map"</span>);</div><div class="line">    }</div><div class="line">    <span class="keyword">const</span> <span class="keyword">char</span> *key = <span class="string">"child_pid"</span>;</div><div class="line">    <span class="keyword">int</span> err = bpf_map__update_elem(syscall_map, key, <span class="number">10</span>, (<span class="keyword">void</span> *)&pid, <span class="keyword">sizeof</span>(pid_t), <span class="number">0</span>);</div><div class="line">    <span class="keyword">if</span> (err)</div><div class="line">    {</div><div class="line">      <span class="built_in">printf</span>(<span class="string">"%d"</span>, err);</div><div class="line">      fatal_error(<span class="string">"failed to insert child pid into the ring buffer"</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">int</span> rbFd = bpf_object__find_map_fd_by_name(obj, syscall_info_bufname);</div><div class="line"></div><div class="line">    <span class="keyword">struct</span> ring_buffer *rbuffer = ring_buffer__new(rbFd, syscall_logger, NULL, NULL);</div><div class="line"></div><div class="line">    <span class="keyword">if</span> (!rbuffer)</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to allocate ring buffer"</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">if</span> (wait(&status) == -<span class="number">1</span>)</div><div class="line">    {</div><div class="line">      fatal_error(<span class="string">"failed to wait for the child process"</span>);</div><div class="line">    }</div><div class="line"></div><div class="line">    <span class="keyword">while</span> (<span class="number">1</span>)</div><div class="line">    {</div><div class="line">      <span class="keyword">int</span> e = ring_buffer__consume(rbuffer);</div><div class="line">      <span class="keyword">if</span> (!e)</div><div class="line">      {</div><div class="line">        <span class="keyword">break</span>;</div><div class="line">      }</div><div class="line">      sleep(<span class="number">1</span>);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div></pre></td></tr></table></figure>

<p>以下便是 eBPF 程序的部分。所有的 C 语言源码最终会被编译整合成单一的对象文件。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div><div class="line">19</div><div class="line">20</div><div class="line">21</div><div class="line">22</div><div class="line">23</div><div class="line">24</div><div class="line">25</div><div class="line">26</div><div class="line">27</div><div class="line">28</div><div class="line">29</div><div class="line">30</div><div class="line">31</div><div class="line">32</div><div class="line">33</div><div class="line">34</div><div class="line">35</div><div class="line">36</div><div class="line">37</div><div class="line">38</div><div class="line">39</div><div class="line">40</div><div class="line">41</div><div class="line">42</div><div class="line">43</div><div class="line">44</div><div class="line">45</div><div class="line">46</div><div class="line">47</div><div class="line">48</div><div class="line">49</div><div class="line">50</div><div class="line">51</div><div class="line">52</div><div class="line">53</div><div class="line">54</div><div class="line">55</div><div class="line">56</div><div class="line">57</div><div class="line">58</div><div class="line">59</div><div class="line">60</div><div class="line">61</div><div class="line">62</div><div class="line">63</div><div class="line">64</div><div class="line">65</div><div class="line">66</div><div class="line">67</div><div class="line">68</div><div class="line">69</div><div class="line">70</div><div class="line">71</div><div class="line">72</div><div class="line">73</div><div class="line">74</div><div class="line">75</div><div class="line">76</div><div class="line">77</div><div class="line">78</div><div class="line">79</div><div class="line">80</div><div class="line">81</div><div class="line">82</div><div class="line">83</div><div class="line">84</div><div class="line">85</div><div class="line">86</div><div class="line">87</div><div class="line">88</div><div class="line">89</div><div class="line">90</div><div class="line">91</div><div class="line">92</div><div class="line">93</div><div class="line">94</div><div class="line">95</div><div class="line">96</div><div class="line">97</div></pre></td><td class="code"><pre><div class="line"><span class="comment">// controller.c</span></div><div class="line"></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> "vmlinux.h"</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;bpf/bpf_helpers.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;bpf/bpf_core_read.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> &lt;sys/syscall.h&gt;</span></div><div class="line"><span class="preprocessor">#<span class="keyword">include</span> "controller.h"</span></div><div class="line"></div><div class="line"><span class="keyword">struct</span></div><div class="line">{</div><div class="line">  __uint(type, BPF_MAP_TYPE_HASH);</div><div class="line">  __uint(key_size, <span class="number">10</span>);</div><div class="line">  __uint(value_size, <span class="number">4</span>);</div><div class="line">  __uint(max_entries, <span class="number">256</span> * <span class="number">1024</span>);</div><div class="line">} pid_hashmap SEC(<span class="string">".maps"</span>);</div><div class="line"></div><div class="line"><span class="keyword">struct</span></div><div class="line">{</div><div class="line">  __uint(type, BPF_MAP_TYPE_RINGBUF);</div><div class="line">  __uint(max_entries, <span class="number">256</span> * <span class="number">1024</span>);</div><div class="line">} syscall_info_buffer SEC(<span class="string">".maps"</span>);</div><div class="line"></div><div class="line"></div><div class="line">SEC(<span class="string">"tracepoint/raw_syscalls/sys_enter"</span>)</div><div class="line"><span class="keyword">int</span> detect_syscall_enter(<span class="keyword">struct</span> trace_event_raw_sys_enter *ctx)</div><div class="line">{</div><div class="line">  <span class="comment">// Retrieve the system call number</span></div><div class="line">  <span class="keyword">long</span> syscall_nr = ctx-&gt;id;</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *key = <span class="string">"child_pid"</span>;</div><div class="line">  <span class="keyword">int</span> target_pid;</div><div class="line"></div><div class="line">  <span class="comment">// Reading the process id of the child process in userland</span></div><div class="line">  <span class="keyword">void</span> *value = bpf_map_lookup_elem(&pid_hashmap, key);</div><div class="line">  <span class="keyword">void</span> *args[MAX_ARGS];</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (value)</div><div class="line">  {</div><div class="line">    target_pid = *(<span class="keyword">int</span> *)value;</div><div class="line"></div><div class="line">    <span class="comment">// PID of the process that executed the current system call</span></div><div class="line">    pid_t pid = bpf_get_current_pid_tgid() & <span class="number">0xffffffff</span>;</div><div class="line">    <span class="keyword">if</span> (pid == target_pid && syscall_nr &gt;= <span class="number">0</span> && syscall_nr &lt; MAX_SYSCALL_NR)</div><div class="line">    {</div><div class="line"></div><div class="line">      <span class="keyword">int</span> idx = syscall_nr;</div><div class="line">      <span class="comment">// Reserve space in the ring buffer</span></div><div class="line">      <span class="keyword">struct</span> inner_syscall_info *info = bpf_ringbuf_reserve(&syscall_info_buffer, <span class="keyword">sizeof</span>(<span class="keyword">struct</span> inner_syscall_info), <span class="number">0</span>);</div><div class="line">      <span class="keyword">if</span> (!info)</div><div class="line">      {</div><div class="line">        bpf_printk(<span class="string">"bpf_ringbuf_reserve failed"</span>);</div><div class="line">        <span class="keyword">return</span> <span class="number">1</span>;</div><div class="line">      }</div><div class="line"></div><div class="line">      <span class="comment">// Copy the syscall name into info-&gt;name</span></div><div class="line">      bpf_probe_read_kernel_str(info-&gt;name, <span class="keyword">sizeof</span>(syscalls[syscall_nr].name), syscalls[syscall_nr].name);</div><div class="line">      <span class="keyword">for</span> (<span class="keyword">int</span> i = <span class="number">0</span>; i &lt; MAX_ARGS; i++)</div><div class="line">      {</div><div class="line">        info-&gt;args[i] = (<span class="keyword">void</span> *)BPF_CORE_READ(ctx, args[i]);</div><div class="line">      }</div><div class="line">      info-&gt;num_args = syscalls[syscall_nr].num_args;</div><div class="line">      info-&gt;syscall_nr = syscall_nr;</div><div class="line">      info-&gt;mode = SYS_ENTER;</div><div class="line">      <span class="comment">// Insert into ring buffer</span></div><div class="line">      bpf_ringbuf_submit(info, <span class="number">0</span>);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div><div class="line"></div><div class="line">SEC(<span class="string">"tracepoint/raw_syscalls/sys_exit"</span>)</div><div class="line"><span class="keyword">int</span> detect_syscall_exit(<span class="keyword">struct</span> trace_event_raw_sys_exit *ctx)</div><div class="line">{</div><div class="line">  <span class="keyword">const</span> <span class="keyword">char</span> *key = <span class="string">"child_pid"</span>;</div><div class="line">  <span class="keyword">void</span> *value = bpf_map_lookup_elem(&pid_hashmap, key);</div><div class="line">  pid_t pid, target_pid;</div><div class="line"></div><div class="line">  <span class="keyword">if</span> (value)</div><div class="line">  {</div><div class="line">    pid = bpf_get_current_pid_tgid() & <span class="number">0xffffffff</span>;</div><div class="line">    target_pid = *(pid_t *)value;</div><div class="line">    <span class="keyword">if</span> (pid == target_pid)</div><div class="line">    {</div><div class="line">      <span class="keyword">struct</span> inner_syscall_info *info = bpf_ringbuf_reserve(&syscall_info_buffer, <span class="keyword">sizeof</span>(<span class="keyword">struct</span> inner_syscall_info), <span class="number">0</span>);</div><div class="line">      <span class="keyword">if</span> (!info)</div><div class="line">      {</div><div class="line">        bpf_printk(<span class="string">"bpf_ringbuf_reserve failed"</span>);</div><div class="line">        <span class="keyword">return</span> <span class="number">1</span>;</div><div class="line">      }</div><div class="line">      info-&gt;mode = SYS_EXIT;</div><div class="line">      info-&gt;retval = ctx-&gt;ret;</div><div class="line">      bpf_ringbuf_submit(info, <span class="number">0</span>);</div><div class="line">    }</div><div class="line">  }</div><div class="line">  <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div><div class="line"></div><div class="line"><span class="keyword">char</span> LICENSE[] SEC(<span class="string">"license"</span>) = <span class="string">"GPL"</span>;</div></pre></td></tr></table></figure>

<p>编译之前，我们不妨先构建一个测试程序，以便后续使用我们的工具对其进行追踪分析。</p>
<figure class="highlight c"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="preprocessor">#<span class="keyword">include</span>&lt;stdio.h&gt;</span></div><div class="line"><span class="keyword">int</span> main(){</div><div class="line">    <span class="built_in">puts</span>(<span class="string">"tracer in action"</span>);</div><div class="line">    <span class="keyword">return</span> <span class="number">0</span>;</div><div class="line">}</div></pre></td></tr></table></figure>

<p>可以利用下面提供的 Makefile 来完成所有相关组件的编译工作。</p>
<figure class="highlight sh"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">compile:</div><div class="line">	clang -O2 -g -Wall -I/usr/include -I/usr/include/bpf -o beetrace loader.c -lbpf</div><div class="line">	clang -O2 -g -target bpf -c controller.c -o controller.o</div></pre></td></tr></table></figure>

<p><img src="beetracer.png" alt=""></p>
<p>整个代码可以在以下的GitHub仓库中找到：<br><a href="https://github.com/0xSh4dy/bee_tracer" target="_blank" rel="external">https://github.com/0xSh4dy/bee_tracer</a></p>
<p>参考链接：</p>
<ul>
<li><a href="https://ebpf.io/" target="_blank" rel="external">https://ebpf.io/</a></li>
<li><a href="https://github.com/libbpf/libbpf" target="_blank" rel="external">https://github.com/libbpf/libbpf</a></li>
</ul>
]]></content>
    <summary type="html">
    <![CDATA[<h2 id="先决条件">先决条件</h2>
<p>系统调用、eBPF、C语言、底层编程基础。</p>
<h2 id="简介">简介</h2>
<p>eBPF(扩展的伯克利数据包过滤器)是一项允许用户在内核中运行自定义程序的技术。BPF或cBPF(经典BPF)是eBPF的前身,它提供了一种简单高效的方法来基于预定义规则过滤数据包。与内核模块相比,eBPF程序提供了更高的安全性、可移植性和可维护性。现有多种高级方法可用于处理eBPF程序,如Cilium的Go语言库、bpftrace、libbpf等。</p>
<ul>
<li><code>注意</code>: 本文要求读者对<code>eBPF</code>有基本了解。如果你不熟悉它,<code>ebpf.io</code>上的这篇文章是很好的参考资料。</li>
</ul>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Russ Cox 引退以及他的新项目 Oscar]]></title>
    <link href="https://colobu.com/2024/08/02/Russ-Cox-and-his-Oscar/"/>
    <id>https://colobu.com/2024/08/02/Russ-Cox-and-his-Oscar/</id>
    <published>2024-08-02T06:51:25.000Z</published>
    <updated>2024-08-08T15:12:09.509Z</updated>
    <content type="html"><![CDATA[<p>Go 第一代技术领导人 Rob Pike, 近两年已经隐居澳大利亚。<br>Go 第二代技术领导人 Russ Cox 2024 年 8 月 2 日宣布卸任，转战 AI 项目，聚焦 Oscar 项目。<br>Go 第三代技术领导人 Austin Clements, 同样和 Russ Cox 一样毕业于美国的一个计算机技术比较出名的一个学院，算是 Russ Cox 的师弟，Austin是Go语言运行时系统和垃圾收集器的主要贡献者之一，在运行时和内存管理等底层系统方面有深入的专长。</p>
<a id="more"></a>
<p>Go 第一代技术领导人 Rob Pike, 近两年已经隐居澳大利亚。<br>Go 第二代技术领导人 Russ Cox 2024 年 8 月 2 日宣布卸任，转战 AI 项目，聚焦 Oscar 项目。<br>Go 第三代技术领导人 Austin Clements, 同样和 Russ Cox 一样毕业于美国的一个计算机技术比较出名的一个学院，算是 Russ Cox 的师弟，Austin是Go语言运行时系统和垃圾收集器的主要贡献者之一，在运行时和内存管理等底层系统方面有深入的专长。</p>
<p>Russ Cox 他在网上的 ID 是 rsc，他是麻省理工学院 MIT 2008 届的博士毕业生，他本科和研究生都是在哈佛大学就读的，Go team 里的又一个学神。他所在的项目组是隶属于 MIT 计算机科学与人工智能实验室的并行与分布式操作系统组，据网上的资料 Austin Clements 也是在这个实验室这个组。</p>
<p>Russ 在哈佛大学就读期间就在 <a href="https://en.wikipedia.org/wiki/Bell_Labs" target="_blank" rel="external">Bell Labs</a> 贝尔实验室里实习（Russ 出生成长的家就在贝尔实验室附近，所以他从高中时期就一直在贝尔实验室的计算机科学部门泡着，所以大佬们是不是关注一下学区房，自己做不到也要为下一代考虑下 ：）），那时候他和 Rob Pike 一起开发贝尔实验室的分布式操作系统 <a href="https://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs" target="_blank" rel="external">Plan 9</a> (上世纪 80 年代末由贝尔实验室的 Ken Thompson 和 Rob Pike 等人发起并领导的项目)，后来他去 MIT 攻读博士学位期间顺便去了 Google 实习，就在他博士快毕业的时候，Rob Pike 和 Ken Thompson 一起和他介绍了他们正在设计的一门新语言 Go，并大概是这么对他说的：“嘿，我们正试图把我们以前在 Plan 9 开发软件时非常喜欢的所有东西用在那些我们想在 Google 里写的软件里，你想过来帮忙一起搞吗？”，然后 Russ 就这样被这两位传奇程序员拉拢进来，事实上 Russ 一直都认为他能在博士毕业以后直接加入 Go 团队是发生在他人生中最幸运的事之一，他说仿佛自己过去十年所学的一切东西就是为了这一刻而准备的。</p>
<p>他加入团队之后就接手了编译器和 Runtime 这两大核心模块，并协助一起开发了标准库，之后依靠这些先前的经验，他和其他人一起完成了标准库后续的所有重构和优化，这就是为什么当你去看 Go 语言的源码的时候会发现 Russ 的名字几乎无处不在，到处都是他的 commits。</p>
<p>在整个 Go 代码仓库中，Russ Cox 提交的代码量是最多的。（请注意，下图中的人物都是大佬，三代领导人都在里面，包括两巨头和 Ian）</p>
<p><img src="contributors.png" alt=""></p>
<p>考虑到Russ Cox在Go项目中的资历和地位以及大学的经历,他可能在某些方面扮演了Austin Clements的导师角色,但这只是推测。我同样推测 Russ Cox 和 Rob Pike 之间也有类似的师承关系。只不过国外可能没有咱们中国这种拜师的礼仪，没有磕头敬过酒。</p>
<p>从 Russ Cox 的信中，可以看到实际他领导 Go 项目已经 12 年，从曾经充满理想、热情澎湃的有志青年，已经进入到经历沧桑的中年，他也在思考自己的职业规划。</p>
<p>尤其最近几位知名 Gopher 大佬对他的批评，不知道是否导致他引退的导火索呢？或许大佬内心受伤了，心灰意冷了。</p>
<p>作为一个绝顶聪明，学历和资历都是金字塔的技术大牛，正处于正当打的年纪，你猜 Ross Cox 下一步会做些什么？</p>
<p>当然是结合当前炙热的 AI 技术，再结合 Go 的经验，做一点有意义的事情，说不定又会发展为一个明星的项目。</p>
<p>这个项目叫 Oscar （奥斯卡），一个开源的贡献者 Agent架构。事实上前几天 Russ Cox 已经透露了它的第一个原型：<a href="https://github.com/gabyhelp" target="_blank" rel="external">gabyhelp</a></p>
<p>Oscar 旨在通过创建用于开源维护的 <code>自动化帮助</code> 或 <code>Agent</code> 来改进开源软件开发。我们相信有很多机会可以减少维护大型和小型开源项目所涉及的辛苦。</p>
<p>这句话已经完全说明这个项目的远景了。</p>
<p>大型语言模型(LLMs)能够对自然语言(如问题报告或维护者指令)进行语义分析,并在自然语言指令和程序代码之间进行转换,这为代理与人更顺畅地交互创造了新的机会。LLMs可能最终只是整个图景中小小的(但关键的!)一部分; Agent的大部分行为将是执行标准的、确定性的代码。</p>
<p>Oscar与许多以开发为中心的LLMs使用方式不同,它完全不试图增强或取代编码过程。毕竟,编写代码是开发软件中最有趣的部分。相反,这个想法是专注于那些不那么有趣的部分,比如处理新提交的问题、将问题与现有文档匹配等。</p>
<p>奥斯卡在很大程度上是一个实验。其实 Russ Cox 目前也还不知道它最终会去哪里。即便如此，他们的第一个原型，<a href="https://github.com/gabyhelp" target="_blank" rel="external">即 @gabyhelp</a> 机器人，已经在 <a href="https://github.com/golang/go/issues?q=label%3Agabywins" target="_blank" rel="external">Go 问题跟踪器中进行了许多成功的交互</a>。这也许是让 Russ Cox 兴奋准备大干一场的动力吧。</p>
<p>目前，Oscar 是在 Go 项目的主持下开发的。在未来的某个时候，它可能会（也可能不会）被分拆成一个单独的项目。</p>
<p>Oscar项目的具体目标是：</p>
<ul>
<li>减少维护人员解决问题的工作量 [请注意，解决并不总是意味着修复]</li>
<li>减少维护人员解决更改列表 （CL） 或拉取请求 （PR） 的工作量 [请注意，解决并不总是意味着提交/合并]</li>
<li>减少维护者解决论坛问题的工作量</li>
<li>让更多人成为高效的维护者</li>
</ul>
<p>和 Copilot 等工具不同，自动化编码不是 Oscar目标。相反，我们专注于<strong>自动化维护人员的工作</strong>。</p>
<p>维护者的辛苦并不是 Go 项目所独有的，因此 Oscar的目标是构建一个任何软件项目都可以重用和扩展的架构，构建他们自己的 Agent，根据项目的需求进行定制。因此 Oscar 是：<em>开源贡献者代理架构</em>。等他们完成额差不多，国内的头部互联网也就会推出类似的产品了。</p>
<p>到目前为止， Russ Cox 已经确定了三项能力，它们将成为奥斯卡的重要组成部分：</p>
<ol>
<li>在贡献者交互期间<strong>索引和显示相关的项目上下文</strong>。</li>
<li>使用<strong>自然语言来控制确定性工具</strong>。</li>
<li><strong>分析问题报告和 CL/PR</strong>，以帮助在提交期间或提交后不久实时改进它们，并适当地标记和路由它们</li>
</ol>
<p>具体的方法 Russ Cox 也在下面这篇文档中描述清楚了，大家可以进一步的了解。<br><a href="https://go.googlesource.com/oscar/+/refs/heads/master/README.md" target="_blank" rel="external">Oscar，一个开源的贡献者代理架构 </a></p>
<p>参考资料：</p>
<ul>
<li><a href="https://mp.weixin.qq.com/s/YF6WGHpY3LYOamG6KmasFg" target="_blank" rel="external">https://mp.weixin.qq.com/s/YF6WGHpY3LYOamG6KmasFg</a></li>
<li><a href="https://golang.design/history/cn.html" target="_blank" rel="external">https://golang.design/history/cn.html</a></li>
<li><a href="https://strikefreedom.top/archives/my-trip-to-san-diego-for-go-contributor-summit-2023" target="_blank" rel="external">https://strikefreedom.top/archives/my-trip-to-san-diego-for-go-contributor-summit-2023</a></li>
<li><a href="https://groups.google.com/g/golang-dev/c/0OqBkS2RzWw" target="_blank" rel="external">https://groups.google.com/g/golang-dev/c/0OqBkS2RzWw</a></li>
<li><a href="https://go.googlesource.com/oscar/+/refs/heads/master/README.md" target="_blank" rel="external">https://go.googlesource.com/oscar/+/refs/heads/master/README.md</a></li>
</ul>
]]></content>
    <summary type="html">
    <![CDATA[<p>Go 第一代技术领导人 Rob Pike, 近两年已经隐居澳大利亚。<br>Go 第二代技术领导人 Russ Cox 2024 年 8 月 2 日宣布卸任，转战 AI 项目，聚焦 Oscar 项目。<br>Go 第三代技术领导人 Austin Clements, 同样和 Russ Cox 一样毕业于美国的一个计算机技术比较出名的一个学院，算是 Russ Cox 的师弟，Austin是Go语言运行时系统和垃圾收集器的主要贡献者之一，在运行时和内存管理等底层系统方面有深入的专长。</p>
]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[128位整数的原子操作]]></title>
    <link href="https://colobu.com/2024/06/16/atomic128/"/>
    <id>https://colobu.com/2024/06/16/atomic128/</id>
    <published>2024-06-16T03:42:29.000Z</published>
    <updated>2024-06-16T03:45:28.407Z</updated>
    <content type="html"><![CDATA[<p>我们已经知道，标准库中的 atomic 针对 int32/uint32、int64/uint64 提供了原子操作的方法和函数，但是如果针对 128 bit 的整数呢？<br><a id="more"></a></p>
<p>当然使用128 bit 整数的原子操作的场景可能比较少，也不会有太多人有这个需求，但是如果我们需要对几个 32 bit、64 bit 变量进行原子操作吗， atomic128 可能就很有用。</p>
<p><a href="https://github.com/tmthrgd/atomic128" target="_blank" rel="external">tmthrgd/atomic128</a> 在几年前提供了 atomic 128 的实验性功能，最后放弃了，但是他提供了一个思路，可以使用 <a href="http://www.felixcloutier.com/x86/CMPXCHG8B:CMPXCHG16B.html" target="_blank" rel="external">CMPXCHG16B</a> 指令为 AMD 64 架构的CPU 提供 atomic 128 功能。</p>
<p><a href="https://github.com/CAFxX/atomic128" target="_blank" rel="external">CAFxX/atomic128</a> fork 了上面的项目，继续维护，还是使用 <code>CMPXCHG16B</code> 指令，只为 AMD 64 架构提供原子操作。</p>
<p>首先我们看看它的功能然后再看一看它的实现，最后我们思路发散一下，看看使用 AVX 为 128 bit 甚至更多 bit 的整数提供原子操作是否可行。</p>
<h2 id="atomic128_的方法">atomic128 的方法</h2>
<p><code>Package atomic128</code> 实现了对 128 bit值的原子操作。在可能的情况下（例如，在支持 <code>CMPXCHG16B</code> 的 amd64 处理器上），它会自动使用 CPU 的原生特性来实现这些操作；否则，它会回退到基于互斥锁（mutexes）的方法。</p>
<p>Go 的基本整数中不包含 int128/uint128，所以这个库先定义了一个 Int128 的类型：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">type</span> Uint128 <span class="keyword">struct</span> {</div><div class="line">	d <span class="number">[3</span>]<span class="typename">uint64</span></div><div class="line">	m sync.Mutex</div><div class="line">}</div></pre></td></tr></table></figure>

<p>然后类似标准库 atomic 中对各种整数的操作，它也提供了类似的方法：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> AddUint128(ptr *Uint128, incr <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="number">[2</span>]<span class="typename">uint64</span></div><div class="line"><span class="keyword">func</span> CompareAndSwapUint128(ptr *Uint128, old, <span class="built_in">new</span> <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="typename">bool</span></div><div class="line"><span class="keyword">func</span> LoadUint128(ptr *Uint128) <span class="number">[2</span>]<span class="typename">uint64</span></div><div class="line"><span class="keyword">func</span> StoreUint128(ptr *Uint128, <span class="built_in">new</span> <span class="number">[2</span>]<span class="typename">uint64</span>)</div><div class="line"><span class="keyword">func</span> SwapUint128(ptr *Uint128, <span class="built_in">new</span> <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="number">[2</span>]<span class="typename">uint64</span></div><div class="line"></div><div class="line"><span class="keyword">func</span> OrUint128(ptr *Uint128, op <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="number">[2</span>]<span class="typename">uint64</span></div><div class="line"><span class="keyword">func</span> AndUint128(ptr *Uint128, op <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="number">[2</span>]<span class="typename">uint64</span></div><div class="line"><span class="keyword">func</span> XorUint128(ptr *Uint128, op <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="number">[2</span>]<span class="typename">uint64</span></div></pre></td></tr></table></figure>

<p>可以看到，除了正常的 Add、CAS、Load、Store、Swap 函数，还贴心的提供了 <code>Or</code>、<code>And</code>、<code>Xor</code> 三个位操作的函数。</p>
<p>下面是一个简单的例子:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div><div class="line">17</div><div class="line">18</div></pre></td><td class="code"><pre><div class="line">n := &atomic128.Uint128{}</div><div class="line">v := atomic128.LoadUint128(n) <span class="comment">// [2]uint64{0, 0}</span></div><div class="line">		</div><div class="line"></div><div class="line">atomic128.StoreUint128(n, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{1</span>, ^<span class="typename">uint64</span><span class="number">(0</span>)})</div><div class="line">v = atomic128.LoadUint128(n) <span class="comment">// [2]uint64{1, ^uint64(0)}</span></div><div class="line"></div><div class="line">v = AddUint128(n, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{2</span>,<span class="number"> 40</span>})</div><div class="line">v = atomic128.LoadUint128(n) <span class="comment">// [2]uint64{3, 40}</span></div><div class="line"></div><div class="line">v = atomic128.SwapUint128(n, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{4</span>,<span class="number"> 50</span>})</div><div class="line">v = atomic128.LoadUint128(n) <span class="comment">// [2]uint64{4, 50}</span></div><div class="line"></div><div class="line">v = atomic128.CompareAndSwapUint128(n, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{4</span>,<span class="number"> 50</span>}, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{5</span>,<span class="number"> 60</span>})</div><div class="line">v = atomic128.LoadUint128(n) <span class="comment">// [2]uint64{5, 60}</span></div><div class="line"></div><div class="line">v = atomic128.OrUint128(n, <span class="number">[2</span>]<span class="typename">uint64</span><span class="number">{0</span>,<span class="number"> 0</span>})</div><div class="line">v = atomic128.LoadUint128(n) <span class="comment">// [2]uint64{5, 60}</span></div></pre></td></tr></table></figure>

<h2 id="atomic128_的实现">atomic128 的实现</h2>
<p>聪明的你也许看到<code>Uint128</code>的定义的时候就会感觉有一点不对劲，为啥128bit的整数要用3个64bit的整数来表示呢？ 2个Uint64不就够了吗？</p>
<p>这是为了保证128位对齐，类似的技术在Go 1.20之前的WaitGroup中也有使用。进一步了解可以查看：</p>
<ul>
<li><a href="https://go101.org/article/memory-layout.html" target="_blank" rel="external">https://go101.org/article/memory-layout.html</a></li>
<li><a href="https://pkg.go.dev/sync/atomic#pkg-note-BUG" target="_blank" rel="external">https://pkg.go.dev/sync/atomic#pkg-note-BUG</a></li>
</ul>
<p>通过包含三个Uint64元素的数组，我们总能通过下面的方法得到128位对齐的地址：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> addr(ptr *Uint128) *<span class="number">[2</span>]<span class="typename">uint64</span> {</div><div class="line">	<span class="keyword">if</span> (<span class="typename">uintptr</span>)((unsafe.Pointer)(&ptr.d<span class="number">[0</span>]))<span class="number">%16</span> ==<span class="number"> 0</span> { <span class="comment">// 指针已经128位对齐</span></div><div class="line">		<span class="keyword">return</span> (*<span class="number">[2</span>]<span class="typename">uint64</span>)((unsafe.Pointer)(&ptr.d<span class="number">[0</span>]))</div><div class="line">	}</div><div class="line">	<span class="keyword">return</span> (*<span class="number">[2</span>]<span class="typename">uint64</span>)((unsafe.Pointer)(&ptr.d<span class="number">[1</span>])) <span class="comment">// 必然ptr.d[1]是128位对齐的 (AMD64架构)</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>通过变量<code>useNativeAmd64</code>判断CPU是否支持<code>CMPXCHG16B</code>指令：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> init() {</div><div class="line">	useNativeAmd64 = cpuid.CPU.Supports(cpuid.CX16)</div><div class="line">}</div></pre></td></tr></table></figure>

<p>如果不支持，回退到使用Mutex实现一个低效的atomic 128bit原子操作:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div><div class="line">13</div><div class="line">14</div><div class="line">15</div><div class="line">16</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">func</span> CompareAndSwapUint128(ptr *Uint128, old, <span class="built_in">new</span> <span class="number">[2</span>]<span class="typename">uint64</span>) <span class="typename">bool</span> {</div><div class="line">	<span class="keyword">if</span> runtime.GOARCH == <span class="string">"amd64"</span> && useNativeAmd64 {</div><div class="line">		<span class="keyword">return</span> compareAndSwapUint128amd64(addr(ptr), old, <span class="built_in">new</span>)</div><div class="line">	}</div><div class="line"></div><div class="line">    <span class="comment">// 不支持CMPXCHG16B指令，使用Mutex</span></div><div class="line">	ptr.m.Lock()</div><div class="line">	v := load(ptr)</div><div class="line">	<span class="keyword">if</span> v != old {</div><div class="line">		ptr.m.Unlock()</div><div class="line">		<span class="keyword">return</span> <span class="constant">false</span></div><div class="line">	}</div><div class="line">	store(ptr, <span class="built_in">new</span>)</div><div class="line">	ptr.m.Unlock()</div><div class="line">	<span class="keyword">return</span> <span class="constant">true</span></div><div class="line">}</div></pre></td></tr></table></figure>

<p>如果支持<code>CMPXCHG16B</code>指令，直接调用<code>compareAndSwapUint128amd64</code>函数：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div></pre></td><td class="code"><pre><div class="line">TEXT ·compareAndSwapUint128amd64(SB),NOSPLIT,<span class="number">$0</span></div><div class="line">	MOVQ addr<span class="number">+0</span>(FP), BP</div><div class="line">	MOVQ old<span class="number">+8</span>(FP), AX</div><div class="line">	MOVQ old<span class="number">+16</span>(FP), DX</div><div class="line">	MOVQ <span class="built_in">new</span><span class="number">+24</span>(FP), BX</div><div class="line">	MOVQ <span class="built_in">new</span><span class="number">+32</span>(FP), CX</div><div class="line">	LOCK</div><div class="line">	CMPXCHG16B (BP)</div><div class="line">	SETEQ swapped<span class="number">+40</span>(FP)</div><div class="line">	RET</div></pre></td></tr></table></figure>

<p>主要依赖<code>CMPXCHG16B</code>实现。</p>
<p><code>CMPXCHG16B</code>是一条X86体系结构中的指令,全称为&quot;Compare and Exchange 16 Bytes&quot;。它用于原子地比较和交换16个字节(128位)的内存区域。<br>这条指令的作用是:</p>
<ul>
<li>将要比较的16个字节的内存值加载到一个寄存器中。</li>
<li>将要写入的16个字节的值加载到另一个寄存器中。</li>
<li>比较内存中的值和第一个寄存器中的值是否相等。</li>
<li>如果相等,则用第二个寄存器中的值覆盖内存中的值。</li>
<li>根据比较结果,设置相应的标志位。</li>
</ul>
<h2 id="思路发散">思路发散</h2>
<p>当前很多号称性能优化的库，可能会使用SIMD指令集来提高性能，比如AVX、SSE等。那么，我们是否可以使用AVX指令集来实现对128位整数甚至256、512位整数的原子操作呢？</p>
<p>有一篇很好的文章介绍了这方面的探索:<a href="https://rigtorp.se/isatomic/" target="_blank" rel="external">Aligned AVX loads and stores are atomic</a>。</p>
<p>各家处理器手册中并没有为AVX指令集提供原子性的担保。<a href="https://www.amd.com/system/files/TechDocs/40332.pdf" target="_blank" rel="external">The AMD64 Architecture Programmer’s Manual</a>只是保证了内存操作最大8个字节，<code>CMPXCHG16B</code>是原子的。<a href="https://software.intel.com/content/www/us/en/develop/articles/intel-sdm.html" target="_blank" rel="external">The Intel® 64 and IA-32 Architectures Software Developer’s Manual</a>也做了类似的保证。此外，Intel手册明确指出AVX指令没有任何原子性保证。</p>
<p>这篇文章的作者做了实验，得出下面的结论:<br><img src="avx.png" alt=""></p>
<p>尽管看起来对齐的 128 位操作室原子的，但是 CPU 提供商没有提供担保，我们还是使用 <code>CMPXCHG16B</code> 指令保险。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>我们已经知道，标准库中的 atomic 针对 int32/uint32、int64/uint64 提供了原子操作的方法和函数，但是如果针对 128 bit 的整数呢？<br>]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Go 朝着错误的方向发展]]></title>
    <link href="https://colobu.com/2024/06/11/go-evolves-in-the-wrong-direction/"/>
    <id>https://colobu.com/2024/06/11/go-evolves-in-the-wrong-direction/</id>
    <published>2024-06-11T14:02:06.000Z</published>
    <updated>2024-06-11T14:40:15.021Z</updated>
    <content type="html"><![CDATA[<p>这是 Aliaksandr Valialkin 昨天刚写的一篇文章, 心有戚戚焉，所以特意翻译成中文，个人感觉，自从Rob Pike退休后，Go在大方向迷失了，正如老貘(Go101)所说，目前Go的开发就像完成KPI一样，也许, 大师不会再回来了。</p>
<p>Aliaksandr Valialkin是fasthttp的作者，也是VictoriaMetrics开发者，一位资深的Go程序员。</p>
<p>以下是译文。<br><a id="more"></a></p>
<p>以下是对原文的地道中文翻译:</p>
<p>Go编程语言以易于使用而闻名。得益于经过深思熟虑的语法、特性和工具,Go允许编写任意复杂度的易读易维护的程序(参见GitHub上的这个<a href="https://github.com/search?q=language%3Ago+stars%3A%3E%3D10000&amp;type=repositories" target="_blank" rel="external">列表</a>)。</p>
<p>有些软件工程师称Go为&quot;无聊&quot;和&quot;过时&quot;,因为它缺乏其他编程语言的高级特性,如单子、Option类型、LINQ、借用检查器、零开销抽象、面向方面编程、继承、函数和运算符重载等。虽然这些特性在特定领域可能可以简化编码,但它们除了好处之外还有非零的成本。这些特性通常对锻炼大脑有好处。但是在处理生产代码时,我们不需要额外的精神负担,因为我们已经很忙于解决业务任务了。所有这些特性的主要成本是<strong>增加了结果代码的复杂性</strong>:</p>
<ul>
<li>仅仅通过阅读代码就变得更难理解正在发生的事情;</li>
<li>调试此类代码变得更加困难,因为您需要跳过数十个非平凡的抽象才能到达业务逻辑;</li>
<li>由于这些特性施加的限制,为此类代码添加新功能变得更加困难。</li>
</ul>
<p>这可能会显著减慢甚至阻碍代码开发的进度。这就是Go一开始就没有这些特性的主要原因。</p>
<p>不幸的是,一些这样的特性开始出现在最新的Go版本中:</p>
<ul>
<li>泛型已在Go1.18中添加。许多软件工程师希望Go有泛型,因为他们认为这将大大提高Go的生产力。Go1.18发布已经两年了,但没有迹象表明生产力有所提高。Go中泛型的整体采用率仍然很低。为什么?因为<strong>大多数实际的Go代码根本不需要泛型</strong>。另一方面,泛型显著增加了Go语言本身的复杂性。例如,尝试理解泛型添加后Go类型推断的所有细节。它的复杂性看起来已经非常接近于C++的类型推断了:)另一个问题是Go中的泛型缺乏C++模板中存在的基本特性。例如,Go泛型不支持泛型类型的泛型方法。它们也不支持模板特化和模板模板参数,以及许多其他需要充分利用泛型编程的特性。让我们将这些缺失的特性添加到Go中吧!等等,那我们就得到另一个过于复杂的C++克隆了。那么,为什么要一开始就将半生不熟的泛型添加到Go中呢?🤦</li>
<li>根据这个提交,Range over functions 又名迭代器、生成器或协程将在 Go 1.23 中添加。让我们仔细看看这个&quot;特性&quot;。</li>
</ul>
<h2 id="Go1-23_中的迭代器">Go1.23 中的迭代器</h2>
<p>如果你不太熟悉Go中的迭代器,请阅读这篇出色的介绍文章。本质上,这是一种语法糖,允许在具有特殊签名的函数上使用<code>for...range</code>循环。这使得可以编写遍历自定义集合和类型的自定义迭代器。听起来像是一个很棒的功能,不是吗?让我们试着弄清楚这一功能解决了哪些实际问题。这在<a href="https://github.com/golang/go/discussions/56413" target="_blank" rel="external">这里</a>有概述:</p>
<blockquote>
<p>Go语言没有标准的方式来遍历一系列值。由于缺乏约定,我们最终使用了各种各样的方法。每种实现都是根据当时的上下文做出最合理的决定,但是孤立地做出的决策导致了用户的困惑。</p>
<p>仅在标准库中,我们就有archive/tar.Reader.Next、bufio.Reader.ReadByte、bufio.Scanner.Scan、container/ring.Ring.Do、database/sql.Rows、expvar.Do、flag.Visit、go/token.FileSet.Iterate、path/filepath.Walk、go/token.FileSet.Iterate、runtime.Frames.Next和sync.Map.Range,几乎没有任何一个在迭代的确切细节上达成一致。即使函数签名相同,语义也不总是一致。例如,大多数返回(T, bool)的迭代函数都遵循Go的惯例,即bool表示T是否有效。相反,runtime.Frames.Next返回的bool则表示下一次调用是否会返回有效的内容。</p>
<p>当你想要遍历某些内容时,你首先必须了解你调用的特定代码是如何处理迭代的。这种不统一阻碍了Go追求的在大型代码库中方便移动的目标。人们常常将Go代码看起来都大致相同作为一个优势,但对于包含自定义迭代的代码而言,这显然是不真实的。</p>
</blockquote>
<p>再说一次,拥有在Go中遍历各种类型的统一方式听起来是合理的。但是对于作为Go主要优势之一的向后兼容性又如何呢?根据Go的兼容性规则,上面提到的标准库中所有现有的自定义迭代器<strong>将永远保留在标准库中</strong>。因此,<strong>所有新的Go版本在标准库中都将至少提供两种不同的方式来遍历各种类型 —— 旧的方式和新的方式</strong>。这增加了Go编程的复杂性,因为:</p>
<ul>
<li>您需要了解遍历各种类型的两种方式,而不是单一方式。</li>
<li>您需要能够阅读和维护使用旧迭代器的旧代码,以及可能使用旧迭代器、新迭代器或同时使用两种迭代器类型的新代码。</li>
<li>在编写新代码时,您需要选择适当的迭代器类型。</li>
</ul>
<h2 id="Go1-23_中迭代器的其他问题">Go1.23 中迭代器的其他问题</h2>
<p>以下是对原文的地道中文翻译:</p>
<p>在Go 1.23之前,<code>for...range</code>循环只能应用于内置类型:整数(从Go1.22开始)、字符串、切片、映射和通道。这些循环的语义很清晰,易于理解(遍历通道的循环语义更加复杂,但如果你处理并发编程,那你应该很容易理解)。</p>
<p>从Go 1.23开始,<code>for...range</code>循环可以应用于具有特殊签名的函数(又称拉取和推送函数)。这使得单凭阅读代码就无法理解给定的看似无辜的<code>for...range</code>循环到底会在底层做什么。它可以做任何事情,就像任何函数调用一样。不同之处在于,Go中的函数调用一直都是<strong>显式的</strong>,比如<code>f(args)</code>，而<code>for...range</code>循环<strong>隐藏了实际的函数调用</strong>。另外,它还对循环体<strong>应用了一些不太明显的转换</strong>:</p>
<ul>
<li>它<strong>隐式地</strong>将循环体包裹在一个匿名函数中,并<strong>隐式地</strong>将这个函数传递给<strong>推送</strong>迭代器函数。</li>
<li>它<strong>隐式地</strong>调用匿名的<strong>拉取</strong>函数,并将返回的结果传递给循环体。</li>
<li>它<strong>隐式地</strong>将<strong>return</strong>、<strong>continue</strong>、<strong>break</strong>、<strong>goto</strong>和<strong>defer</strong>语句转换为另一个不太明显的语句,存在于传递给<strong>推送</strong>迭代器函数的匿名函数中。</li>
</ul>
<p>另外,<strong>在一般情况下,在循环迭代之后使用迭代器函数返回的参数是不安全的</strong>,因为迭代器函数可能会在下一次循环迭代时重用它们。</p>
<p>Go 曾因易于阅读和理解的<strong>显式代码执行路径</strong>而闻名。这一特性在 Go1.23 中<strong>不可逆转地</strong>被破坏了:(我们用什么来交换?<strong>另一种</strong>遍历类型的方式,它具有一些<strong>隐式</strong>的语义,而且在某些情况下行为与广告描述的不同。当遍历可能在迭代过程中返回错误的类型时(例如database/sql.Rows、path/filepath.Walk 或任何其他在迭代过程中进行 IO 操作的类型),这种新方式就无法按预期工作,因为你需要<strong>手动检查迭代错误</strong>,无论是在循环内部还是在循环之后,这与使用旧方法的做法是一样的。</p>
<p>即使你使用不会返回错误的迭代器,生成的 <code>for ... range</code> 循环也看起来比使用显式回调的旧方法更加不清晰。哪种代码更容易理解和调试?</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line">tree.walk(<span class="keyword">func</span>(k, v <span class="typename">string</span>) {</div><div class="line">  <span class="built_in">println</span>(k, v)</div><div class="line">})</div></pre></td></tr></table></figure>



<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">for</span> k, v := <span class="keyword">range</span> tree.walk {</div><div class="line">  <span class="built_in">println</span>(k, v)</div><div class="line">}</div></pre></td></tr></table></figure>

<p>请记住,后一个循环会被隐式地转换为前一个带有显式回调调用的代码。现在让我们从循环中返回一些东西:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">for</span> k, v := <span class="keyword">range</span> tree.walk {</div><div class="line">  <span class="keyword">if</span> k == <span class="string">"foo"</span> {</div><div class="line">    <span class="keyword">return</span> v</div><div class="line">  }</div><div class="line">}</div></pre></td></tr></table></figure>

<p>它被隐式转换为难以跟踪的代码，类似于以下代码：</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> vOuter <span class="typename">string</span></div><div class="line">needOuterReturn := <span class="constant">false</span></div><div class="line">tree.walk(<span class="keyword">func</span>(k, v <span class="typename">string</span>) <span class="typename">bool</span> {</div><div class="line">  <span class="keyword">if</span> k == <span class="string">"foo"</span> {</div><div class="line">    needOuterReturn = <span class="constant">true</span></div><div class="line">    vOuter = v</div><div class="line">    <span class="keyword">return</span> <span class="constant">false</span></div><div class="line">  }</div><div class="line">})</div><div class="line"><span class="keyword">if</span> needOuterReturn {</div><div class="line">  <span class="keyword">return</span> vOuter</div><div class="line">}</div></pre></td></tr></table></figure>

<p>看起来很容易调试:)</p>
<p>如果<code>tree.walk</code>通过从字节切片进行不安全转换将<code>v</code>传递给回调函数,那么这段代码可能会崩溃,因为<code>v</code>的内容在下一次循环迭代时可能会发生变化。因此,<strong>隐式</strong>生成的防弹代码必须使用<code>strings.Clone()</code>函数,这可能导致不必要的内存分配和复制:</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><div class="line">1</div><div class="line">2</div><div class="line">3</div><div class="line">4</div><div class="line">5</div><div class="line">6</div><div class="line">7</div><div class="line">8</div><div class="line">9</div><div class="line">10</div><div class="line">11</div><div class="line">12</div></pre></td><td class="code"><pre><div class="line"><span class="keyword">var</span> vOuter <span class="typename">string</span></div><div class="line">needOuterReturn := <span class="constant">false</span></div><div class="line">tree.walk(<span class="keyword">func</span>(k, v <span class="typename">string</span>) <span class="typename">bool</span> {</div><div class="line">  <span class="keyword">if</span> k == <span class="string">"foo"</span> {</div><div class="line">    needOuterReturn = <span class="constant">true</span></div><div class="line">    vOuter = strings.Clone(v)</div><div class="line">    <span class="keyword">return</span> <span class="constant">false</span></div><div class="line">  }</div><div class="line">})</div><div class="line"><span class="keyword">if</span> needOuterReturn {</div><div class="line">  <span class="keyword">return</span> vOuter</div><div class="line">}</div></pre></td></tr></table></figure>

<p><code>range over func</code>这一特性对函数签名施加了限制。这些限制不适用于所有需要遍历集合元素的场景。这迫使软件工程师在使用<code>for...range</code>循环时进行丑陋的hack,以及编写理想情况下适合给定任务的显式代码之间做出艰难选择。</p>
<h2 id="结论">结论</h2>
<p>令人遗憾的是,Go开始朝着增加复杂性和隐式代码执行的方向发展。也许我们需要停止添加增加Go复杂性的新功能,而是专注于Go的核心特性 - 简单性、高效性和性能。例如,最近Rust开始在对性能要求苛刻的领域取代Go的份额。我相信如果Go核心团队专注于优化热循环,比如循环展开和SIMD使用,这种趋势是可以扭转的。这不应该太过影响编译和链接速度,因为只有少量编译后的Go代码需要优化。没有必要试图优化所有简单代码的变体 - 这些代码即使优化了热循环也仍然会很慢。只需针对那些由注重代码性能的软件工程师故意编写的特定模式进行优化就足够了。<br>Go比Rust容易使用得多。为什么要在性能竞赛中输给Rust呢?<br>Go可以获得的另一个有用特性的例子是,在不增加语言本身和使用这些特性的Go代码复杂性的情况下,进行类似于小的改善代码质量的改进。</p>
<h2 id="我是谁?">我是谁?</h2>
<p>我是一名专门编写简单、面向性能的Go代码的软件工程师,如VictoriaMetrics、quicktemplate、fastjson、fasthttp、fastcache、easyproto等。多亏了Go,我一直试图遵循KISS(Keep It Simple,Stupid)设计原则。</p>
]]></content>
    <summary type="html">
    <![CDATA[<p>这是 Aliaksandr Valialkin 昨天刚写的一篇文章, 心有戚戚焉，所以特意翻译成中文，个人感觉，自从Rob Pike退休后，Go在大方向迷失了，正如老貘(Go101)所说，目前Go的开发就像完成KPI一样，也许, 大师不会再回来了。</p>
<p>Aliaksandr Valialkin是fasthttp的作者，也是VictoriaMetrics开发者，一位资深的Go程序员。</p>
<p>以下是译文。<br>]]>
    
    </summary>
    
      <category term="Go" scheme="https://colobu.com/categories/Go/"/>
    
  </entry>
  
  <entry>
    <title><![CDATA[Rob Pike 语录]]></title>
    <link href="https://colobu.com/2024/06/10/The-Analects-of-Rob-Pike/"/>
    <id>https://colobu.com/2024/06/10/The-Analects-of-Rob-Pike/</id>
    <published>2024-06-10T04:39:59.000Z</published>
    <updated>2024-06-10T09:20:01.990Z</updated>
    <content type="html"><![CDATA[<h2 id="1-_计算机领域里，没有什么问题是加一层间接寻址解决不了的。">1. 计算机领域里，没有什么问题是加一层间接寻址解决不了的。</h2>
<blockquote>
<p>There&#39;s nothing in computing that can&#39;t be broken by another level of indirection.</p>
</blockquote>
<p>这是 Rob Pike 的修改版。</p>
<p>经常 <code>level of insriection</code> 误引用为 <code>abstraction layer</code>。</p>
<blockquote>
<p>原始版本出自 <strong>Butler Lampson</strong><br>All problems in computer science can be solved by another level of indirection</p>
<p>但是 <strong>David Wheeler</strong> 完成了下半句：<br>All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection.</p>
<p>还有 Kevlin Henney 的下半句：<br>ll problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection.&quot;</p>
<p>From <a href="https://www2.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html" target="_blank" rel="external">Beautiful Code: Another Level of Indirection </a></p>
</blockquote>
<p>这句话幽默地指出，在计算机编程中，通过引入额外的抽象层或中间层，几乎可以解决任何复杂的问题。这种思路在软件设计和架构中很常见。</p>
<a id="more"></a>
<h2 id="2-_数据为王。如果你选择了正确的数据结构，并且组织得当，算法几乎总是不言自明的。编程的核心在于数据结构，而不是算法。">2. 数据为王。如果你选择了正确的数据结构，并且组织得当，算法几乎总是不言自明的。编程的核心在于数据结构，而不是算法。</h2>
<blockquote>
<p>Data dominates. If you&#39;ve chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.</p>
</blockquote>
<p>&quot;数据为王&quot;意味着在软件开发中，数据的组织和表示方式比实现算法的具体细节更加重要。<br>这个观点强调了良好的数据结构设计对于编程效率和代码质量的重要性。它鼓励程序员把更多精力放在思考如何组织和表示数据上，而不是过分关注算法的技巧性。</p>
<h2 id="3-_面向对象设计就是计算机界的罗马数字。">3. 面向对象设计就是计算机界的罗马数字。</h2>
<blockquote>
<p>Object-oriented design is the roman numerals of computing.</p>
</blockquote>
<p>这是一句颇具争议性和挑衅性的话。罗马数字在数学史上曾经很重要，但现代计算中已被阿拉伯数字体系取代，因为后者更简单、更高效。这句话暗示面向对象设计（OOD）也是如此。</p>
<p>Pike认为，就像罗马数字对现代数学来说过于繁琐和低效一样，面向对象编程（OOP）的某些方面（如过度的类层次结构和封装）可能导致代码复杂、难以理解和维护。</p>
<p>这个比喻引发了编程圈内的热议。支持者认为它点出了OOP的一些问题，如过度设计和不必要的复杂性。反对者则认为OOP仍然是一个强大和有用的范式。</p>
<p>值得注意的是，Pike是Go语言的创始人之一，Go语言采用了一种不同于传统OOP的设计哲学，更注重简单性和数据结构。</p>
<h2 id="4-_最后，我意识到光想是没什么出路的，该动手实践了。">4. 最后，我意识到光想是没什么出路的，该动手实践了。</h2>
<blockquote>
<p>Eventually, I decided that thinking was not getting me very far and it was time to try building.</p>
</blockquote>
<p>这句话体现了一种在技术和创新领域常见的实用主义态度。</p>
<p>&quot;光想没出路&quot;是一种常见的中文表达，意思是仅靠思考而不付诸行动是难以取得进展的。</p>
<p>&quot;动手实践&quot;强调了在实践中学习和创新的重要性。在编程、创业等领域，这种&quot;边做边学&quot;的方法被广泛推崇。</p>
<p>但这并不是完全否定思考的价值。恰当的思考和规划仍然重要，只是强调不要陷入&quot;分析瘫痪&quot;（analysis paralysis），即过度分析导致迟迟不能行动的状态。</p>
<h2 id="5-_见识短浅，想象力就会受限。">5. 见识短浅，想象力就会受限。</h2>
<blockquote>
<p>Narrowness of experience leads to narrowness of imagination.</p>
</blockquote>
<p>更为甚者， 见识短浅， 反而将大师的作品视为平庸。一个讽刺性的文章： <a href="https://github.com/SuperPaintman/the-evolution-of-a-go-programmer" target="_blank" rel="external">The Evolution of a Go Programmer</a></p>
<h2 id="6-_这就是现代计算：简单的东西变得过于复杂，因为随意捣鼓太容易；复杂的东西依旧复杂，因为修复太难。">6. 这就是现代计算：简单的东西变得过于复杂，因为随意捣鼓太容易；复杂的东西依旧复杂，因为修复太难。</h2>
<blockquote>
<p>Such is modern computing: everything simple is made too complicated because it&#39;s easy to fiddle with; everything complicated stays complicated because it&#39;s hard to fix.</p>
</blockquote>
<p>Rob Pike 作为一位推崇简洁设计的大师，他对当前计算机行业的复杂性提出了犀利的批评。</p>
<p>第一部分讽刺了一些程序员或设计师的倾向：明明有简单的解决方案，却偏爱使用复杂的技术，导致本来简单的问题变得难以理解和维护。</p>
<p>第二部分点出了一个现实问题：一旦系统变得复杂，就很难简化。原因可能包括兼容性问题、团队惯性、或者是简单地因为理解和重构复杂系统需要大量时间和资源。</p>
<p>它呼应了UNIX哲学中的一个核心原则：做好一件事。也就是说，程序应该简单、模块化，只专注于完成一个任务。</p>
<h2 id="7-_过程名应该反映它做什么；函数名应该反映它返回什么。">7. 过程名应该反映它做什么；函数名应该反映它返回什么。</h2>
<blockquote>
<p>Procedure names should reflect what they do; function names should reflect what they return</p>
</blockquote>
<p>这是编程领域的一条重要命名规范，有助于提高代码的可读性和可维护性。</p>
<p>在中文编程圈，&quot;过程&quot;（procedure）和&quot;函数&quot;（function）的区别经常被讨论。简单来说：</p>
<ul>
<li>&quot;过程&quot;执行一系列操作，通常不返回值，重点在于&quot;做&quot;。</li>
<li>&quot;函数&quot;计算并返回一个值，重点在于&quot;得到&quot;什么。</li>
</ul>
<h2 id="8-_花哨的算法在_N_小的时候很慢，而_N_通常都很小。">8. 花哨的算法在 N 小的时候很慢，而 N 通常都很小。</h2>
<blockquote>
<p>Fancy algorithms are slow when N is small, and N is usually small.</p>
</blockquote>
<p>&quot;N&quot; 在算法分析中代表输入规模，如要排序的元素个数、要搜索的数据量等。&quot;大 O 表示法&quot;（如 O(n)、O(n²)）就是用 N 来描述算法在最坏情况下的时间复杂度。</p>
<p>这句话的启示：</p>
<ul>
<li>不要过早优化。在数据量小的情况下，简单直白的算法可能更快、更易理解。</li>
<li>理解实际问题的规模。过度设计（用复杂算法解决小问题）可能适得其反。<br>= 在选择算法时，要考虑具体场景，不能只看理论复杂度。</li>
</ul>
<p>俗语&quot;大炮打蚊子&quot;，就是类似的道理。有时候，简单的方法反而更有效。</p>
<h2 id="9-_UNIX不仅已经死了，臭的都快熏死人了。">9. UNIX不仅已经死了，臭的都快熏死人了。</h2>
<blockquote>
<p>Not only is UNIX dead, it&#39;s starting to smell really bad.</p>
</blockquote>
<p>在1990年代，微软的Windows和IBM的OS/2等图形用户界面（GUI）操作系统开始流行，而基于命令行的UNIX看起来过时了。一些人认为，用户友好的GUI是未来，UNIX这样的系统已经落伍，注定会消亡。<br>讽刺的是，这个预言并没有完全实现：</p>
<ul>
<li>Linux（一个UNIX类操作系统）在服务器领域占据主导地位。</li>
<li>macOS基于BSD（另一个UNIX变种）。</li>
<li>甚至Windows 10也加入了Linux子系统。</li>
</ul>
<p>这句话现在常被用来嘲笑那些过早宣布某项技术&quot;死亡&quot;的人。</p>
<h2 id="10-_想要杜绝傻瓜行为的编程语言，往往自己也变得傻不拉几。">10. 想要杜绝傻瓜行为的编程语言，往往自己也变得傻不拉几。</h2>
<blockquote>
<p>Languages that try to disallow idiocy become themselves idiotic.</p>
</blockquote>
<p>过度限制程序员可能弊大于利：</p>
<ul>
<li>好的程序员应该被信任和赋能，而不是被当成&quot;傻瓜&quot;对待。</li>
<li>有时，所谓的&quot;傻瓜行为&quot;其实是创新和效率的源泉。</li>
<li>语言应该提供工具和指导，而不是强制规定唯一的&quot;正确&quot;方式。</li>
</ul>
<p>这句话也反映了一个更广的设计哲学：过度设计来防止错误，可能带来更多问题。无论是编程语言、产品设计还是管理，给予用户或团队合理的自由和信任，往往比试图规避一切风险更有效。</p>
<h2 id="11、缓存不是架构，只是个优化手段而已。">11、缓存不是架构，只是个优化手段而已。</h2>
<blockquote>
<p>Caches aren&#39;t architecture, they&#39;re just optimization.</p>
</blockquote>
<p>Pike的观点是：</p>
<ul>
<li>不要因为缓存效果好，就把它当成架构的一部分。</li>
<li>如果没有缓存系统就崩溃，那可能是架构有问题。</li>
<li>缓存应该是&quot;锦上添花&quot;，而不是&quot;救命稻草&quot;。</li>
</ul>
<p>但这并不意味着缓存不重要。实际上：</p>
<ul>
<li>合理使用缓存可以极大提升性能。</li>
<li>在某些场景（如高并发网站），缓存几乎是必需的。</li>
</ul>
<p>关键是平衡：</p>
<ul>
<li>先有好的架构和算法。</li>
<li>在合适的地方加缓存，但不要让系统对缓存产生依赖。</li>
<li>缓存失效或穿透时，系统应该能够&quot;优雅降级&quot;。</li>
</ul>
<p>用中文的一句话概括就是：&quot;先治本，再治标&quot;。缓存是&quot;标&quot;（优化性能），好的架构才是&quot;本&quot;。</p>
<h2 id="12、没有类型层次，就不用费劲去管理类型层次了。">12、没有类型层次，就不用费劲去管理类型层次了。</h2>
<blockquote>
<p>When there is no type hierarchy you don&#39;t have to manage the type hierarchy.</p>
</blockquote>
<p>它直指面向对象编程（OOP）中一个常见的复杂性来源。&quot;类型层次&quot;主要指面向对象语言中的类继承结构：</p>
<p>&quot;没有类型层次&quot;并非完全否定OOP，而是指一种不同的设计风格：</p>
<ul>
<li>组合优于继承：用组合（has-a）而不是继承（is-a）来复用代码。</li>
<li>接口而非基类：定义行为协议，而不是强制继承关系。</li>
<li>简单类型：类的职责单一，减少复杂的层次结构。</li>
</ul>
<p>这种思想在Go语言中很明显：</p>
<ul>
<li>Go没有类和继承，但有结构体和接口。</li>
<li>结构体可以嵌入其他结构体来复用字段和方法，但不是继承。</li>
<li>接口是隐式的：只要一个类型实现了接口的所有方法，它就&quot;是&quot;那个接口。</li>
</ul>
<h2 id="13、按工程管理的规则，生产力最重要；可在工程师眼里，乐趣才是第一位。生产力源于乐趣。">13、按工程管理的规则，生产力最重要；可在工程师眼里，乐趣才是第一位。生产力源于乐趣。</h2>
<blockquote>
<p>Productivity is most important by engineering management rules, but enjoyment is most important for engineers. One stems from the other.</p>
</blockquote>
<ul>
<li>这里的&quot;乐趣&quot;不只是表面的快乐，更指：<ul>
<li>解决有趣问题的满足感。</li>
<li>创造优雅代码的成就感。</li>
<li>与团队协作的归属感。</li>
<li>学习新技术的好奇心。</li>
</ul>
</li>
<li>优秀的工程师往往被这些内在动机驱动，而不仅仅是外部压力。</li>
</ul>
<p>Pike指出，高生产力实际上源于工程师的乐趣。<br>当工程师享受工作时，他们会：</p>
<ul>
<li>自发加班，因为问题太有趣了。</li>
<li>主动优化代码，因为看到丑陋的代码会不舒服。</li>
<li>积极学习，因为新技术太酷了。<br>这些行为自然而然地提高了生产力。</li>
</ul>
<p>这一观点在软件行业有广泛共识：</p>
<ul>
<li>Google的20%时间：员工可以花20%工作时间做自己感兴趣的项目。许多重要产品（如Gmail）就是这样诞生的。虽然说现在Google已经去掉了20%工作时间的政策。</li>
<li>开源社区：大多数贡献者是因为热情而不是报酬。这种模式创造了Linux、Python等。</li>
<li>创业文化：舒适的办公环境、弹性工作制，都是为了让员工更快乐，从而更有创造力。</li>
</ul>
<p>对管理者的启示：</p>
<ul>
<li>不要只盯着KPI。创造让工程师愉悦的环境，生产力自然会提高。</li>
<li>理解并尊重工程师的动机。有时，让他们&quot;玩&quot;反而能得到更好的结果。</li>
</ul>
<p>对工程师的启示：</p>
<ul>
<li>追求技术乐趣并不自私。它能让你更高效，也让产品更优秀。</li>
<li>但也要有度。纯粹追求个人兴趣而忽视团队目标，同样问题。</li>
</ul>
<h2 id="14、第一法则：你猜不准程序会在哪里耗时。性能瓶颈总在意想不到的地方冒出来，所以别想当然去优化，除非你证实了那里就是瓶颈所在。">14、第一法则：你猜不准程序会在哪里耗时。性能瓶颈总在意想不到的地方冒出来，所以别想当然去优化，除非你证实了那里就是瓶颈所在。</h2>
<blockquote>
<p>Rule 1. You can&#39;t tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don&#39;t try to second guess and put in a speed hack until you&#39;ve proven that&#39;s where the bottleneck is</p>
</blockquote>
<ul>
<li><p>&quot;猜不准程序会在哪里耗时&quot;：</p>
<ul>
<li>程序性能不是直观的。即使是经验丰富的程序员，也常常错误预测哪部分代码最慢。</li>
<li>现代系统复杂（多线程、缓存、编译器优化等），让性能特征更难预测。</li>
</ul>
</li>
<li><p>&quot;性能瓶颈总在意想不到的地方冒出来&quot;：</p>
<ul>
<li>&quot;性能瓶颈&quot;是限制整体性能的最慢部分。就像木桶，最短的那块板决定了水位。</li>
<li>&quot;意想不到&quot;暗示即使是看似简单的代码，也可能因为被频繁调用或数据量大而成为瓶颈。</li>
</ul>
</li>
<li><p>&quot;别想当然去优化&quot;：</p>
</li>
<li>&quot;想当然&quot;在中文里就是不经证实就认定。这在性能优化中很危险。</li>
<li><p>过早优化（premature optimization）是编程界臭名昭著的反模式。它可能导致：</p>
<ul>
<li>代码复杂化，难以理解和维护。</li>
<li>浪费时间在实际上不慢的部分。</li>
<li>引入新的bug或性能问题。</li>
</ul>
</li>
<li><p>&quot;除非你证实了那里就是瓶颈所在&quot;：</p>
<ul>
<li>&quot;证实&quot;是关键。不是猜测，而是通过性能分析工具（profiler）确定。</li>
<li>现代profiler可以精确定位耗时函数、内存分配等，让优化有的放矢。</li>
</ul>
</li>
</ul>
<p>这一法则的实践：</p>
<ul>
<li>先让它跑起来（Make it work）</li>
<li>然后让它对（Make it right）</li>
<li>最后才是让它快（Make it fast）</li>
<li>每一步都用数据（profile）来指导，不猜测。</li>
</ul>
<h2 id="14、扩展">14、扩展</h2>
<p>第14条事实上来源自Robe Pike的编程五原则, 包括上面的多条名言:</p>
<ul>
<li>第一法则： 你猜不准程序会在哪儿磨蹭。性能瓶颈总冒出在意想不到的地方，所以别瞎猜着去优化，除非你摸清楚了瓶颈的准确位置。</li>
<li>第二法则： 先量化。没测量之前别瞎调速度，就算测出来了，也得是哪块代码严重拖后腿了才优化。</li>
<li>第三法则： 花里胡哨的算法在数据小时龟速，而且数据往往就不大。花哨算法有大常数。除非你明摆着要处理大数据，别整那些花活。（就算真遇到大数据，也先用第二法则。）</li>
<li>第四法则： 花哨的算法比简单的更容易藏bug，而且实现起来费劲得很。算法要简单，数据结构也要简单。</li>
<li>第五法则： 数据为王。如果你选对了数据结构，组织得当，算法几乎都是呼之即来。编程的核心是数据结构，不是算法。</li>
</ul>
<ol>
<li>Pike的第一、二法则重申了Tony Hoare的名言：&quot;过早优化是万恶之源&quot;。 在中文里，&quot;万恶之源&quot;是个很重的词，用在这儿既有警示，也带点调侃。</li>
<li>Ken Thompson把Pike 的第三、四法则总结为：&quot;疑难杂症，暴力破解&quot;。 &quot;暴力破解&quot;在中文程序员圈很常见，指不优雅但直接有效的解法。</li>
<li>第三、四法则体现了KISS（Keep It Simple, Stupid）设计哲学。中文常说&quot;简单就是美&quot;，或者IT圈的&quot;能用短裤的地方，别穿西装&quot;。</li>
<li>第五法则早先出现在Fred Brooks的《人月神话》中。程序员们常把它简化为&quot;写傻瓜代码，用聪明对象&quot;。 这句话在中文圈也很流行，强调代码逻辑要直白，复杂性应该封装在良好设计的数据结构里。</li>
</ol>
<h2 id="15、如果POSIX线程算好东西，我都不敢想它比什么还好。">15、如果POSIX线程算好东西，我都不敢想它比什么还好。</h2>
<blockquote>
<p>If POSIX threads are a good thing, perhaps I don&#39;t want to know what they&#39;re better than. </p>
</blockquote>
<p>这是一句典型的程序员式嘲讽，直指他认为设计糟糕的一个技术标准。相当于说&quot;它已经够糟了，居然还有比它更糟的？&quot;</p>
<p>Pike为什么这么说？</p>
<ul>
<li>复杂性：pthreads API被认为过于复杂和底层，容易出错。</li>
<li>死锁风险：错误使用pthreads很容易导致死锁、竞态条件等并发问题。</li>
<li>可移植性问题：尽管POSIX旨在提高可移植性，但不同系统的pthreads实现仍有差异。</li>
</ul>
<h2 id="16、缓存的bug，哪个不是妖魔鬼怪。">16、缓存的bug，哪个不是妖魔鬼怪。</h2>
<blockquote>
<p>There&#39;s no such thing as a simple cache bug.</p>
</blockquote>
<p>&quot;缓存&quot;在计算机中无处不在：CPU缓存（L1, L2, L3）、内存缓存（如Redis）、浏览器缓存、数据库查询缓存、CDN（内容分发网络）、DNS缓存等等。</p>
<p>Pike之所以这么说，是因为缓存bug的特点：</p>
<ul>
<li>诡异性：缓存问题常常表现得不一致、间歇性，像&quot;鬼打墙&quot;。</li>
<li>隐蔽性：问题可能潜伏很久才暴露，像&quot;潜伏的妖怪&quot;。</li>
<li>牵连性：一个小小的缓存问题可能导致系统范围的故障，像&quot;妖风四起&quot;。</li>
<li>难调试：因为缓存常常是分布式的，跟踪问题如同&quot;捉鬼&quot;。</li>
<li>死灰复燃：以为修好了，问题却在高负载时死灰复燃，像&quot;怪物再生&quot;。</li>
</ul>
<h2 id="17、用_Unix_就跟只听大卫·卡西迪的歌似的，纯属乐坛井底之蛙。">17、用 Unix 就跟只听大卫·卡西迪的歌似的，纯属乐坛井底之蛙。</h2>
<blockquote>
<p>Using Unix is the computing equivalent of listening only to music by David Cassidy.</p>
</blockquote>
<p>Rob Pike在一次采访中说了这句话<a href="https://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds" target="_blank" rel="external">rob pike responds</a>。</p>
<p>Rob Pike可是Unix的大佬级人物，和Ken Thompson、Dennis Ritchie一起在贝尔实验室创造了Unix。他怎么可能真的觉得Unix就像肤浅的流行乐？</p>
<p>现在的macOS（Mach内核）、Linux，甚至Windows（WSL）都有Unix的影子。<br>Pike仿佛预言般地讽刺：&quot;看看，大家最后都来听&#39;大卫·卡西迪&#39;了。&quot;</p>
<p>Rob Pike这句话是典型的技术人的自嘲式炫耀。表面上自贬，实际上是在用幽默的方式表达：&quot;对，我们就是主流，因为我们简单好用。学院派笑话我们Low，但最后还不是得用我们的东西？&quot;这种自黑中带着骄傲的调调。</p>
<h2 id="18、&quot;智能&quot;终端可不是&quot;自作聪明&quot;的终端，而是你能调教的好帮手。">18、&quot;智能&quot;终端可不是&quot;自作聪明&quot;的终端，而是你能调教的好帮手。</h2>
<blockquote>
<p>A smart terminal is not a smartass terminal, but rather a terminal you can educate.</p>
</blockquote>
<p>Rob Pike这句话用带点烟火气的方式，点出了技术设计的一个关键：真正的&quot;智能&quot;不是自作聪明，而是在交互中学习和成长。就像中国老话说的，&quot;学而不舍，才能化茧成蝶&quot;。无论是终端还是人，都是这个道理。</p>
<h2 id="19、Socket_是_IO_接口的_X_Window。">19、Socket 是 IO 接口的 X Window。</h2>
<blockquote>
<p>Sockets are the X windows of IO interfaces.</p>
</blockquote>
<p>表面上是在夸 Socket，就像当年吹捧 X Window 一样，实际上是 Pike 式的反话，意思是：&quot;Socket 复杂、难用，就跟 X Window 一样令人头疼。&quot;</p>
<p>为什么 Pike 这么黑 Socket？</p>
<ul>
<li>复杂性：Socket API 有很多参数、选项和状态，容易用错。</li>
<li>底层性：直接操作网络协议，程序员要处理字节流、缓冲区等底层细节。</li>
<li>错误处理难：网络环境复杂，Socket 编程中的错误情况多，很难全面处理。</li>
<li>跨平台坑多：不同操作系统的 Socket 实现有微妙差异，写出跨平台代码很烦。</li>
</ul>
<h2 id="20、搞个理论上不那么刺激的编程语言干嘛？因为好用啊，能用才是王道。">20、搞个理论上不那么刺激的编程语言干嘛？因为好用啊，能用才是王道。</h2>
<blockquote>
<p>Why would you have a language that is not theoretically exciting? Because it&#39;s very useful.</p>
</blockquote>
<p>这话当然还是出自Rob Pike。作为Unix和Go语言之父，他一贯秉持&quot;实用主义者&quot;的态度。</p>
<p>学院派追求&quot;道&quot;（理论和原则）。Pike强调&quot;术&quot;（实用技巧和方法）。</p>
<h2 id="21、并发不是并行">21、并发不是并行</h2>
<blockquote>
<p>Concurrency is not parallelism</p>
</blockquote>
<h2 id="22、_Go语言_箴言">22、 Go语言 箴言</h2>
<blockquote>
<p>Go Proverbs</p>
</blockquote>
]]></content>
    <summary type="html">
    <![CDATA[<h2 id="1-_计算机领域里，没有什么问题是加一层间接寻址解决不了的。">1. 计算机领域里，没有什么问题是加一层间接寻址解决不了的。</h2>
<blockquote>
<p>There&#39;s nothing in computing that can&#39;t be broken by another level of indirection.</p>
</blockquote>
<p>这是 Rob Pike 的修改版。</p>
<p>经常 <code>level of insriection</code> 误引用为 <code>abstraction layer</code>。</p>
<blockquote>
<p>原始版本出自 <strong>Butler Lampson</strong><br>All problems in computer science can be solved by another level of indirection</p>
<p>但是 <strong>David Wheeler</strong> 完成了下半句：<br>All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection.</p>
<p>还有 Kevlin Henney 的下半句：<br>ll problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection.&quot;</p>
<p>From <a href="https://www2.dmst.aueb.gr/dds/pubs/inbook/beautiful_code/html/Spi07g.html" target="_blank" rel="external">Beautiful Code: Another Level of Indirection </a></p>
</blockquote>
<p>这句话幽默地指出，在计算机编程中，通过引入额外的抽象层或中间层，几乎可以解决任何复杂的问题。这种思路在软件设计和架构中很常见。</p>
]]>
    
    </summary>
    
  </entry>
  
  <entry>
    <title><![CDATA[Rust tips #81 ~ #90]]></title>
    <link href="https://colobu.com/2024/06/09/rust-tips-81-90/"/>
    <id>https://colobu.com/2024/06/09/rust-tips-81-90/</id>
    <published>2024-06-09T03:36:28.000Z</published>
    <updated>2024-06-09T03:37:26.369Z</updated>
    <content type="html"><![CDATA[<a id="more"></a>
]]></content>
    <summary type="html">
    <![CDATA[<a id="more"></a>
]]>
    </summary>
    
      <category term="rust" scheme="https://colobu.com/categories/rust/"/>
    
  </entry>
  
</feed>