CloudNative 架构
CloudNative|云原生应用架构|云原生架构|容器化架构|微服务架构|平台架构|基础架构
2021-06-01T02:12:17.000Z
http://team.jiunile.com/
icyboy
Hexo
Go 代码安全指南
http://team.jiunile.com//blog/2021/06/go-security.html
2021-06-01T14:00:00.000Z
2021-06-01T02:12:17.000Z
<h1 id="通用类"><a href="#通用类" class="headerlink" title="通用类"></a>通用类</h1><h2 id="1-代码实现类"><a href="#1-代码实现类" class="headerlink" title="1. 代码实现类"></a>1. 代码实现类</h2><h3 id="1-1-内存管理"><a href="#1-1-内存管理" class="headerlink" title="1.1 内存管理"></a>1.1 内存管理</h3><h4 id="1-1-1【必须】切片长度校验"><a href="#1-1-1【必须】切片长度校验" class="headerlink" title="1.1.1【必须】切片长度校验"></a>1.1.1【必须】切片长度校验</h4><ul>
<li>在对slice进行操作时,必须判断长度是否合法,防止程序panic</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 未判断data的长度,可导致 index out of range</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">decode</span><span class="params">(data []<span class="keyword">byte</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">if</span> data[<span class="number">0</span>] == <span class="string">'F'</span> && data[<span class="number">1</span>] == <span class="string">'U'</span> && data[<span class="number">2</span>] == <span class="string">'Z'</span> && data[<span class="number">3</span>] == <span class="string">'Z'</span> && data[<span class="number">4</span>] == <span class="string">'E'</span> && data[<span class="number">5</span>] == <span class="string">'R'</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Bad"</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// bad: slice bounds out of range</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> slice = []<span class="keyword">int</span>{<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>}</span><br><span class="line"> fmt.Println(slice[:<span class="number">10</span>])</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good: 使用data前应判断长度是否合法</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">decode</span><span class="params">(data []<span class="keyword">byte</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(data) == <span class="number">6</span> {</span><br><span class="line"> <span class="keyword">if</span> data[<span class="number">0</span>] == <span class="string">'F'</span> && data[<span class="number">1</span>] == <span class="string">'U'</span> && data[<span class="number">2</span>] == <span class="string">'Z'</span> && data[<span class="number">3</span>] == <span class="string">'Z'</span> && data[<span class="number">4</span>] == <span class="string">'E'</span> && data[<span class="number">5</span>] == <span class="string">'R'</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Good"</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<a id="more"></a>
<h4 id="1-1-2【必须】nil指针判断"><a href="#1-1-2【必须】nil指针判断" class="headerlink" title="1.1.2【必须】nil指针判断"></a>1.1.2【必须】nil指针判断</h4><ul>
<li>进行指针操作时,必须判断该指针是否为nil,防止程序panic,尤其在进行结构体Unmarshal时</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Packet <span class="keyword">struct</span> {</span><br><span class="line"> PackeyType <span class="keyword">uint8</span></span><br><span class="line"> PackeyVersion <span class="keyword">uint8</span></span><br><span class="line"> Data *Data</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Data <span class="keyword">struct</span> {</span><br><span class="line"> Stat <span class="keyword">uint8</span></span><br><span class="line"> Len <span class="keyword">uint8</span></span><br><span class="line"> Buf [<span class="number">8</span>]<span class="keyword">byte</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(p *Packet)</span> <span class="title">UnmarshalBinary</span><span class="params">(b []<span class="keyword">byte</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(b) < <span class="number">2</span> {</span><br><span class="line"> <span class="keyword">return</span> io.EOF</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> p.PackeyType = b[<span class="number">0</span>]</span><br><span class="line"> p.PackeyVersion = b[<span class="number">1</span>]</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 若长度等于2,那么不会new Data</span></span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(b) > <span class="number">2</span> {</span><br><span class="line"> p.Data = <span class="built_in">new</span>(Data)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// bad: 未判断指针是否为nil</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> packet := <span class="built_in">new</span>(Packet)</span><br><span class="line"> data := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="number">2</span>)</span><br><span class="line"> <span class="keyword">if</span> err := packet.UnmarshalBinary(data); err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Failed to unmarshal packet"</span>)</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> fmt.Printf(<span class="string">"Stat: %v\n"</span>, packet.Data.Stat)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good: 判断Data指针是否为nil</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> packet := <span class="built_in">new</span>(Packet)</span><br><span class="line"> data := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="number">2</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> err := packet.UnmarshalBinary(data); err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Failed to unmarshal packet"</span>)</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> packet.Data == <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> fmt.Printf(<span class="string">"Stat: %v\n"</span>, packet.Data.Stat)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-3【必须】整数安全"><a href="#1-1-3【必须】整数安全" class="headerlink" title="1.1.3【必须】整数安全"></a>1.1.3【必须】整数安全</h4><ul>
<li><p>在进行数字运算操作时,需要做好长度限制,防止外部输入运算导致异常:</p>
<ul>
<li>确保无符号整数运算时不会反转</li>
<li>确保有符号整数运算时不会出现溢出</li>
<li>确保整型转换时不会出现截断错误</li>
<li>确保整型转换时不会出现符号错误</li>
</ul>
</li>
<li><p>以下场景必须严格进行长度限制:</p>
<ul>
<li>作为数组索引</li>
<li>作为对象的长度或者大小</li>
<li>作为数组的边界(如作为循环计数器)</li>
</ul>
</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 未限制长度,导致整数溢出</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">overflow</span><span class="params">(numControlByUser <span class="keyword">int32</span>)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> numInt <span class="keyword">int32</span> = <span class="number">0</span></span><br><span class="line"> numInt = numControlByUser + <span class="number">1</span></span><br><span class="line"> <span class="comment">// 对长度限制不当,导致整数溢出</span></span><br><span class="line"> fmt.Printf(<span class="string">"%d\n"</span>, numInt)</span><br><span class="line"> <span class="comment">// 使用numInt,可能导致其他错误</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> overflow(<span class="number">2147483647</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">overflow</span><span class="params">(numControlByUser <span class="keyword">int32</span>)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> numInt <span class="keyword">int32</span> = <span class="number">0</span></span><br><span class="line"> numInt = numControlByUser + <span class="number">1</span></span><br><span class="line"> <span class="keyword">if</span> numInt < <span class="number">0</span> {</span><br><span class="line"> fmt.Println(<span class="string">"integer overflow"</span>)</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> fmt.Println(<span class="string">"integer ok"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> overflow(<span class="number">2147483647</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-4【必须】make分配长度验证"><a href="#1-1-4【必须】make分配长度验证" class="headerlink" title="1.1.4【必须】make分配长度验证"></a>1.1.4【必须】make分配长度验证</h4><ul>
<li>在进行make分配内存时,需要对外部可控的长度进行校验,防止程序panic。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">parse</span><span class="params">(lenControlByUser <span class="keyword">int</span>, data []<span class="keyword">byte</span>)</span></span> {</span><br><span class="line"> size := lenControlByUser</span><br><span class="line"> <span class="comment">// 对外部传入的size,进行长度判断以免导致panic</span></span><br><span class="line"> buffer := <span class="built_in">make</span>([]<span class="keyword">byte</span>, size)</span><br><span class="line"> <span class="built_in">copy</span>(buffer, data)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">parse</span><span class="params">(lenControlByUser <span class="keyword">int</span>, data []<span class="keyword">byte</span>)</span> <span class="params">([]<span class="keyword">byte</span>, error)</span></span> {</span><br><span class="line"> size := lenControlByUser</span><br><span class="line"> <span class="comment">// 限制外部可控的长度大小范围</span></span><br><span class="line"> <span class="keyword">if</span> size > <span class="number">64</span>*<span class="number">1024</span>*<span class="number">1024</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, errors.New(<span class="string">"value too large"</span>)</span><br><span class="line"> }</span><br><span class="line"> buffer := <span class="built_in">make</span>([]<span class="keyword">byte</span>, size)</span><br><span class="line"> <span class="built_in">copy</span>(buffer, data)</span><br><span class="line"> <span class="keyword">return</span> buffer, <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-5【必须】禁止SetFinalizer和指针循环引用同时使用"><a href="#1-1-5【必须】禁止SetFinalizer和指针循环引用同时使用" class="headerlink" title="1.1.5【必须】禁止SetFinalizer和指针循环引用同时使用"></a>1.1.5【必须】禁止SetFinalizer和指针循环引用同时使用</h4><ul>
<li>当一个对象从被GC选中到移除内存之前,runtime.SetFinalizer()都不会执行,即使程序正常结束或者发生错误。由指针构成的“循环引用”虽然能被GC正确处理,但由于无法确定Finalizer依赖顺序,从而无法调用runtime.SetFinalizer(),导致目标对象无法变成可达状态,从而造成内存无法被回收。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> a, b Data</span><br><span class="line"> a.o = &b</span><br><span class="line"> b.o = &a</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 指针循环引用,SetFinalizer()无法正常调用</span></span><br><span class="line"> runtime.SetFinalizer(&a, <span class="function"><span class="keyword">func</span><span class="params">(d *Data)</span></span> {</span><br><span class="line"> fmt.Printf(<span class="string">"a %p final.\n"</span>, d)</span><br><span class="line"> })</span><br><span class="line"> runtime.SetFinalizer(&b, <span class="function"><span class="keyword">func</span><span class="params">(d *Data)</span></span> {</span><br><span class="line"> fmt.Printf(<span class="string">"b %p final.\n"</span>, d)</span><br><span class="line"> })</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> foo()</span><br><span class="line"> time.Sleep(time.Millisecond)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-6【必须】禁止重复释放channel"><a href="#1-1-6【必须】禁止重复释放channel" class="headerlink" title="1.1.6【必须】禁止重复释放channel"></a>1.1.6【必须】禁止重复释放channel</h4><ul>
<li>重复释放一般存在于异常流程判断中,如果恶意攻击者构造出异常条件使程序重复释放channel,则会触发运行时恐慌,从而造成DoS攻击。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">(c <span class="keyword">chan</span> <span class="keyword">int</span>)</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> <span class="built_in">close</span>(c)</span><br><span class="line"> err := processBusiness()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> c <- <span class="number">0</span></span><br><span class="line"> <span class="built_in">close</span>(c) <span class="comment">// 重复释放channel</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> c <- <span class="number">1</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">(c <span class="keyword">chan</span> <span class="keyword">int</span>)</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> <span class="built_in">close</span>(c) <span class="comment">// 使用defer延迟关闭channel</span></span><br><span class="line"> err := processBusiness()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> c <- <span class="number">0</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> }</span><br><span class="line"> c <- <span class="number">1</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-7【必须】确保每个协程都能退出"><a href="#1-1-7【必须】确保每个协程都能退出" class="headerlink" title="1.1.7【必须】确保每个协程都能退出"></a>1.1.7【必须】确保每个协程都能退出</h4><ul>
<li>启动一个协程就会做一个入栈操作,在系统不退出的情况下,协程也没有设置退出条件,则相当于协程失去了控制,它占用的资源无法回收,可能会导致内存泄露。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 协程没有设置退出条件</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">doWaiter</span><span class="params">(name <span class="keyword">string</span>, second <span class="keyword">int</span>)</span></span> {</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> time.Sleep(time.Duration(second) * time.Second)</span><br><span class="line"> fmt.Println(name, <span class="string">" is ready!"</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-1-8【推荐】不使用unsafe包"><a href="#1-1-8【推荐】不使用unsafe包" class="headerlink" title="1.1.8【推荐】不使用unsafe包"></a>1.1.8【推荐】不使用unsafe包</h4><ul>
<li>由于unsafe包绕过了 Golang 的内存安全原则,一般来说使用该库是不安全的,可导致内存破坏,尽量避免使用该包。若必须要使用unsafe操作指针,必须做好安全校验。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 通过unsafe操作原始指针</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">unsafePointer</span><span class="params">()</span></span> {</span><br><span class="line"> b := <span class="built_in">make</span>([]<span class="keyword">byte</span>, <span class="number">1</span>)</span><br><span class="line"> foo := (*<span class="keyword">int</span>)(unsafe.Pointer(<span class="keyword">uintptr</span>(unsafe.Pointer(&b[<span class="number">0</span>])) + <span class="keyword">uintptr</span>(<span class="number">0xfffffff</span>e)))</span><br><span class="line"> fmt.Print(*foo + <span class="number">1</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// [signal SIGSEGV: segmentation violation code=0x1 addr=0xc100068f55 pc=0x49142b]</span></span><br></pre></td></tr></table></figure>
<h4 id="1-1-9【推荐】不使用slice作为函数入参"><a href="#1-1-9【推荐】不使用slice作为函数入参" class="headerlink" title="1.1.9【推荐】不使用slice作为函数入参"></a>1.1.9【推荐】不使用slice作为函数入参</h4><ul>
<li>slice是引用类型,在作为函数入参时采用的是地址传递,对slice的修改也会影响原始数据</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: slice作为函数入参时是地址传递</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">modify</span><span class="params">(array []<span class="keyword">int</span>)</span></span> {</span><br><span class="line"> array[<span class="number">0</span>] = <span class="number">10</span> <span class="comment">// 对入参slice的元素修改会影响原始数据</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> array := []<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>}</span><br><span class="line"></span><br><span class="line"> modify(array)</span><br><span class="line"> fmt.Println(array) <span class="comment">// output:[10 2 3 4 5]</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good: 函数使用数组作为入参,而不是slice</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">modify</span><span class="params">(array [5]<span class="keyword">int</span>)</span></span> {</span><br><span class="line"> array[<span class="number">0</span>] = <span class="number">10</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// 传入数组,注意数组与slice的区别</span></span><br><span class="line"> array := [<span class="number">5</span>]<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>}</span><br><span class="line"></span><br><span class="line"> modify(array)</span><br><span class="line"> fmt.Println(array)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-2-文件操作"><a href="#1-2-文件操作" class="headerlink" title="1.2 文件操作"></a>1.2 文件操作</h3><h4 id="1-2-1【必须】-路径穿越检查"><a href="#1-2-1【必须】-路径穿越检查" class="headerlink" title="1.2.1【必须】 路径穿越检查"></a>1.2.1【必须】 路径穿越检查</h4><ul>
<li>在进行文件操作时,如果对外部传入的文件名未做限制,可能导致任意文件读取或者任意文件写入,严重可能导致代码执行。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 任意文件读取</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handler</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> path := r.URL.Query()[<span class="string">"path"</span>][<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 未过滤文件路径,可能导致任意文件读取</span></span><br><span class="line"> data, _ := ioutil.ReadFile(path)</span><br><span class="line"> w.Write(data)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 对外部传入的文件名变量,还需要验证是否存在../等路径穿越的文件名</span></span><br><span class="line"> data, _ = ioutil.ReadFile(filepath.Join(<span class="string">"/home/user/"</span>, path))</span><br><span class="line"> w.Write(data)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// bad: 任意文件写入</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">unzip</span><span class="params">(f <span class="keyword">string</span>)</span></span> {</span><br><span class="line"> r, _ := zip.OpenReader(f)</span><br><span class="line"> <span class="keyword">for</span> _, f := <span class="keyword">range</span> r.File {</span><br><span class="line"> p, _ := filepath.Abs(f.Name)</span><br><span class="line"> <span class="comment">// 未验证压缩文件名,可能导致../等路径穿越,任意文件路径写入</span></span><br><span class="line"> ioutil.WriteFile(p, []<span class="keyword">byte</span>(<span class="string">"present"</span>), <span class="number">0640</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good: 检查压缩的文件名是否包含..路径穿越特征字符,防止任意写入</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">unzipGood</span><span class="params">(f <span class="keyword">string</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> r, err := zip.OpenReader(f)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(<span class="string">"read zip file fail"</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">for</span> _, f := <span class="keyword">range</span> r.File {</span><br><span class="line"> <span class="keyword">if</span> !strings.Contains(f.Name, <span class="string">".."</span>) {</span><br><span class="line"> p, _ := filepath.Abs(f.Name)</span><br><span class="line"> ioutil.WriteFile(p, []<span class="keyword">byte</span>(<span class="string">"present"</span>), <span class="number">0640</span>)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-2-2【必须】-文件访问权限"><a href="#1-2-2【必须】-文件访问权限" class="headerlink" title="1.2.2【必须】 文件访问权限"></a>1.2.2【必须】 文件访问权限</h4><ul>
<li>根据创建文件的敏感性设置不同级别的访问权限,以防止敏感数据被任意权限用户读取。例如,设置文件权限为:<code>-rw-r-----</code></li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">ioutil.WriteFile(p, []<span class="keyword">byte</span>(<span class="string">"present"</span>), <span class="number">0640</span>)</span><br></pre></td></tr></table></figure>
<h3 id="1-3-系统接口"><a href="#1-3-系统接口" class="headerlink" title="1.3 系统接口"></a>1.3 系统接口</h3><p><strong>1.3.1【必须】命令执行检查</strong></p>
<ul>
<li>使用<code>exec.Command</code>、<code>exec.CommandContext</code>、<code>syscall.StartProcess</code>、<code>os.StartProcess</code>等函数时,第一个参数(path)直接取外部输入值时,应使用白名单限定可执行的命令范围,不允许传入<code>bash</code>、<code>cmd</code>、<code>sh</code>等命令;</li>
<li>使用<code>exec.Command</code>、<code>exec.CommandContext</code>等函数时,通过<code>bash</code>、<code>cmd</code>、<code>sh</code>等创建shell,-c后的参数(arg)拼接外部输入,应过滤\n $ & ; | ‘ “ ( ) `等潜在恶意字符;</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">()</span></span> {</span><br><span class="line"> userInputedVal := <span class="string">"&& echo 'hello'"</span> <span class="comment">// 假设外部传入该变量值</span></span><br><span class="line"> cmdName := <span class="string">"ping "</span> + userInputedVal</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 未判断外部输入是否存在命令注入字符,结合sh可造成命令注入</span></span><br><span class="line"> cmd := exec.Command(<span class="string">"sh"</span>, <span class="string">"-c"</span>, cmdName)</span><br><span class="line"> output, _ := cmd.CombinedOutput()</span><br><span class="line"> fmt.Println(<span class="keyword">string</span>(output))</span><br><span class="line"></span><br><span class="line"> cmdName := <span class="string">"ls"</span></span><br><span class="line"> <span class="comment">// 未判断外部输入是否是预期命令</span></span><br><span class="line"> cmd := exec.Command(cmdName)</span><br><span class="line"> output, _ := cmd.CombinedOutput()</span><br><span class="line"> fmt.Println(<span class="keyword">string</span>(output))</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">checkIllegal</span><span class="params">(cmdName <span class="keyword">string</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">if</span> strings.Contains(cmdName, <span class="string">"&"</span>) || strings.Contains(cmdName, <span class="string">"|"</span>) || strings.Contains(cmdName, <span class="string">";"</span>) ||</span><br><span class="line"> strings.Contains(cmdName, <span class="string">"$"</span>) || strings.Contains(cmdName, <span class="string">"'"</span>) || strings.Contains(cmdName, <span class="string">"`"</span>) ||</span><br><span class="line"> strings.Contains(cmdName, <span class="string">"("</span>) || strings.Contains(cmdName, <span class="string">")"</span>) || strings.Contains(cmdName, <span class="string">"\""</span>) {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> userInputedVal := <span class="string">"&& echo 'hello'"</span></span><br><span class="line"> cmdName := <span class="string">"ping "</span> + userInputedVal</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> checkIllegal(cmdName) { <span class="comment">// 检查传给sh的命令是否有特殊字符</span></span><br><span class="line"> <span class="keyword">return</span> <span class="comment">// 存在特殊字符直接return</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> cmd := exec.Command(<span class="string">"sh"</span>, <span class="string">"-c"</span>, cmdName)</span><br><span class="line"> output, _ := cmd.CombinedOutput()</span><br><span class="line"> fmt.Println(<span class="keyword">string</span>(output))</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-4-通信安全"><a href="#1-4-通信安全" class="headerlink" title="1.4 通信安全"></a>1.4 通信安全</h3><h4 id="1-4-1【必须】网络通信采用TLS方式"><a href="#1-4-1【必须】网络通信采用TLS方式" class="headerlink" title="1.4.1【必须】网络通信采用TLS方式"></a>1.4.1【必须】网络通信采用TLS方式</h4><ul>
<li>明文传输的通信协议目前已被验证存在较大安全风险,被中间人劫持后可能导致许多安全风险,因此必须采用至少TLS的安全通信方式保证通信安全,例如gRPC/Websocket都使用TLS1.3。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> http.HandleFunc(<span class="string">"/"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> w.Header().Add(<span class="string">"Strict-Transport-Security"</span>, <span class="string">"max-age=63072000; includeSubDomains"</span>)</span><br><span class="line"> w.Write([]<span class="keyword">byte</span>(<span class="string">"This is an example server.\n"</span>))</span><br><span class="line"> })</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 服务器配置证书与私钥</span></span><br><span class="line"> log.Fatal(http.ListenAndServeTLS(<span class="string">":443"</span>, <span class="string">"yourCert.pem"</span>, <span class="string">"yourKey.pem"</span>, <span class="literal">nil</span>))</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-4-2【推荐】TLS启用证书验证"><a href="#1-4-2【推荐】TLS启用证书验证" class="headerlink" title="1.4.2【推荐】TLS启用证书验证"></a>1.4.2【推荐】TLS启用证书验证</h4><ul>
<li>TLS证书应当是有效的、未过期的,且配置正确的域名,生产环境的服务端应启用证书验证。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"crypto/tls"</span></span><br><span class="line"> <span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">doAuthReq</span><span class="params">(authReq *http.Request)</span> *<span class="title">http</span>.<span class="title">Response</span></span> {</span><br><span class="line"> tr := &http.Transport{</span><br><span class="line"> TLSClientConfig: &tls.Config{InsecureSkipVerify: <span class="literal">true</span>},</span><br><span class="line"> }</span><br><span class="line"> client := &http.Client{Transport: tr}</span><br><span class="line"> res, _ := client.Do(authReq)</span><br><span class="line"> <span class="keyword">return</span> res</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"crypto/tls"</span></span><br><span class="line"> <span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">doAuthReq</span><span class="params">(authReq *http.Request)</span> *<span class="title">http</span>.<span class="title">Response</span></span> {</span><br><span class="line"> tr := &http.Transport{</span><br><span class="line"> TLSClientConfig: &tls.Config{InsecureSkipVerify: <span class="literal">false</span>},</span><br><span class="line"> }</span><br><span class="line"> client := &http.Client{Transport: tr}</span><br><span class="line"> res, _ := client.Do(authReq)</span><br><span class="line"> <span class="keyword">return</span> res</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-5-敏感数据保护"><a href="#1-5-敏感数据保护" class="headerlink" title="1.5 敏感数据保护"></a>1.5 敏感数据保护</h3><h4 id="1-5-1【必须】敏感信息访问"><a href="#1-5-1【必须】敏感信息访问" class="headerlink" title="1.5.1【必须】敏感信息访问"></a>1.5.1【必须】敏感信息访问</h4><ul>
<li>禁止将敏感信息硬编码在程序中,既可能会将敏感信息暴露给攻击者,也会增加代码管理和维护的难度</li>
<li>使用配置中心系统统一托管密钥等敏感信息</li>
</ul>
<h4 id="1-5-2【必须】敏感数据输出"><a href="#1-5-2【必须】敏感数据输出" class="headerlink" title="1.5.2【必须】敏感数据输出"></a>1.5.2【必须】敏感数据输出</h4><ul>
<li>只输出必要的最小数据集,避免多余字段暴露引起敏感信息泄露</li>
<li>不能在日志保存密码(包括明文密码和密文密码)、密钥和其它敏感信息</li>
<li>对于必须输出的敏感信息,必须进行合理脱敏展示</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">serve</span><span class="params">()</span></span> {</span><br><span class="line"> http.HandleFunc(<span class="string">"/register"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> r.ParseForm()</span><br><span class="line"> user := r.Form.Get(<span class="string">"user"</span>)</span><br><span class="line"> pw := r.Form.Get(<span class="string">"password"</span>)</span><br><span class="line"></span><br><span class="line"> log.Printf(<span class="string">"Registering new user %s with password %s.\n"</span>, user, pw)</span><br><span class="line"> })</span><br><span class="line"> http.ListenAndServe(<span class="string">":80"</span>, <span class="literal">nil</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">serve1</span><span class="params">()</span></span> {</span><br><span class="line"> http.HandleFunc(<span class="string">"/register"</span>, <span class="function"><span class="keyword">func</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> r.ParseForm()</span><br><span class="line"> user := r.Form.Get(<span class="string">"user"</span>)</span><br><span class="line"> pw := r.Form.Get(<span class="string">"password"</span>)</span><br><span class="line"></span><br><span class="line"> log.Printf(<span class="string">"Registering new user %s.\n"</span>, user)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> use(pw)</span><br><span class="line"> })</span><br><span class="line"> http.ListenAndServe(<span class="string">":80"</span>, <span class="literal">nil</span>)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<ul>
<li>避免通过GET方法、代码注释、自动填充、缓存等方式泄露敏感信息</li>
</ul>
<h4 id="1-5-3【必须】敏感数据存储"><a href="#1-5-3【必须】敏感数据存储" class="headerlink" title="1.5.3【必须】敏感数据存储"></a>1.5.3【必须】敏感数据存储</h4><ul>
<li>敏感数据应使用SHA2、RSA等算法进行加密存储</li>
<li>敏感数据应使用独立的存储层,并在访问层开启访问控制</li>
<li>包含敏感信息的临时文件或缓存一旦不再需要应立刻删除</li>
</ul>
<h4 id="1-5-4【必须】异常处理和日志记录"><a href="#1-5-4【必须】异常处理和日志记录" class="headerlink" title="1.5.4【必须】异常处理和日志记录"></a>1.5.4【必须】异常处理和日志记录</h4><ul>
<li>应合理使用panic、recover、defer处理系统异常,避免出错信息输出到前端</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">defer</span> <span class="function"><span class="keyword">func</span> <span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">if</span> r := <span class="built_in">recover</span>(); r != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Recovered in start()"</span>)</span><br><span class="line"> }</span><br><span class="line">}()</span><br></pre></td></tr></table></figure>
<ul>
<li>对外环境禁止开启debug模式,或将程序运行日志输出到前端</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">// bad</span><br><span class="line">dlv --listen=:2345 --headless=<span class="literal">true</span> --api-version=2 debug test.go</span><br><span class="line">// good</span><br><span class="line">dlv debug test.go</span><br></pre></td></tr></table></figure>
<h3 id="1-6-加密解密"><a href="#1-6-加密解密" class="headerlink" title="1.6 加密解密"></a>1.6 加密解密</h3><h4 id="1-6-1【必须】不得硬编码密码-密钥"><a href="#1-6-1【必须】不得硬编码密码-密钥" class="headerlink" title="1.6.1【必须】不得硬编码密码/密钥"></a>1.6.1【必须】不得硬编码密码/密钥</h4><ul>
<li>在进行用户登陆,加解密算法等操作时,不得在代码里硬编码密钥或密码,可通过变换算法或者配置等方式设置密码或者密钥。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="keyword">const</span> (</span><br><span class="line"> user = <span class="string">"dbuser"</span></span><br><span class="line"> password = <span class="string">"s3cretp4ssword"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">connect</span><span class="params">()</span> *<span class="title">sql</span>.<span class="title">DB</span></span> {</span><br><span class="line"> connStr := fmt.Sprintf(<span class="string">"postgres://%s:%s@localhost/pqgotest"</span>, user, password)</span><br><span class="line"> db, err := sql.Open(<span class="string">"postgres"</span>, connStr)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> db</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="keyword">var</span> (</span><br><span class="line"> commonkey = []<span class="keyword">byte</span>(<span class="string">"0123456789abcdef"</span>)</span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">AesEncrypt</span><span class="params">(plaintext <span class="keyword">string</span>)</span> <span class="params">(<span class="keyword">string</span>, error)</span></span> {</span><br><span class="line"> block, err := aes.NewCipher(commonkey)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="string">""</span>, err</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-6-2【必须】密钥存储安全"><a href="#1-6-2【必须】密钥存储安全" class="headerlink" title="1.6.2【必须】密钥存储安全"></a>1.6.2【必须】密钥存储安全</h4><ul>
<li>在使用对称密码算法时,需要保护好加密密钥。当算法涉及敏感、业务数据时,可通过非对称算法协商加密密钥。其他较为不敏感的数据加密,可以通过变换算法等方式保护密钥。</li>
</ul>
<h4 id="1-6-3【推荐】不使用弱密码算法"><a href="#1-6-3【推荐】不使用弱密码算法" class="headerlink" title="1.6.3【推荐】不使用弱密码算法"></a>1.6.3【推荐】不使用弱密码算法</h4><ul>
<li>在使用加密算法时,不建议使用加密强度较弱的算法。</li>
</ul>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">// bad</span><br><span class="line">crypto/des,crypto/md5,crypto/sha1,crypto/rc4等。</span><br><span class="line"></span><br><span class="line">// good</span><br><span class="line">crypto/rsa,crypto/aes等。</span><br></pre></td></tr></table></figure>
<h3 id="1-7-正则表达式"><a href="#1-7-正则表达式" class="headerlink" title="1.7 正则表达式"></a>1.7 正则表达式</h3><h4 id="1-7-1【推荐】使用regexp进行正则表达式匹配"><a href="#1-7-1【推荐】使用regexp进行正则表达式匹配" class="headerlink" title="1.7.1【推荐】使用regexp进行正则表达式匹配"></a>1.7.1【推荐】使用regexp进行正则表达式匹配</h4><ul>
<li>正则表达式编写不恰当可被用于DoS攻击,造成服务不可用,推荐使用regexp包进行正则表达式匹配。regexp保证了线性时间性能和优雅的失败:对解析器、编译器和执行引擎都进行了内存限制。但regexp不支持以下正则表达式特性,如业务依赖这些特性,则regexp不适合使用。<ul>
<li>回溯引用<a href="https://www.regular-expressions.info/backref.html" target="_blank" rel="external">Backreferences</a></li>
<li>查看<a href="https://www.regular-expressions.info/lookaround.html" target="_blank" rel="external">Lookaround</a></li>
</ul>
</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line">matched, err := regexp.MatchString(<span class="string">`a.b`</span>, <span class="string">"aaxbb"</span>)</span><br><span class="line">fmt.Println(matched) <span class="comment">// true</span></span><br><span class="line">fmt.Println(err) <span class="comment">// nil</span></span><br></pre></td></tr></table></figure>
<h1 id="后台类"><a href="#后台类" class="headerlink" title="后台类"></a>后台类</h1><h2 id="1-代码实现类-1"><a href="#1-代码实现类-1" class="headerlink" title="1 代码实现类"></a>1 代码实现类</h2><h3 id="1-1-输入校验"><a href="#1-1-输入校验" class="headerlink" title="1.1 输入校验"></a>1.1 输入校验</h3><h4 id="1-1-1【必须】按类型进行数据校验"><a href="#1-1-1【必须】按类型进行数据校验" class="headerlink" title="1.1.1【必须】按类型进行数据校验"></a>1.1.1【必须】按类型进行数据校验</h4><ul>
<li>所有外部输入的参数,应使用<code>validator</code>进行白名单校验,校验内容包括但不限于数据长度、数据范围、数据类型与格式,校验不通过的应当拒绝</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"github.com/go-playground/validator/v10"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> validate *validator.Validate</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">validateVariable</span><span class="params">()</span></span> {</span><br><span class="line"> myEmail := <span class="string">"abc@tencent.com"</span></span><br><span class="line"> errs := validate.Var(myEmail, <span class="string">"required,email"</span>)</span><br><span class="line"> <span class="keyword">if</span> errs != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(errs)</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> <span class="comment">//停止执行</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// 验证通过,继续执行</span></span><br><span class="line"> ...</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> validate = validator.New()</span><br><span class="line"> validateVariable()</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<ul>
<li>无法通过白名单校验的应使用<code>html.EscapeString</code>、<code>text/template</code>或<code>bluemonday</code>对<code><, >, &, ',"</code>等字符进行过滤或编码</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"text/template"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// TestHTMLEscapeString HTML特殊字符转义</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">(inputValue <span class="keyword">string</span>)</span> <span class="title">string</span></span> {</span><br><span class="line"> escapedResult := template.HTMLEscapeString(inputValue)</span><br><span class="line"> <span class="keyword">return</span> escapedResult</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-2-SQL操作"><a href="#1-2-SQL操作" class="headerlink" title="1.2 SQL操作"></a>1.2 SQL操作</h3><h4 id="1-2-1【必须】SQL语句默认使用预编译并绑定变量"><a href="#1-2-1【必须】SQL语句默认使用预编译并绑定变量" class="headerlink" title="1.2.1【必须】SQL语句默认使用预编译并绑定变量"></a>1.2.1【必须】SQL语句默认使用预编译并绑定变量</h4><ul>
<li>使用<code>database/sql</code>的prepare、Query或使用GORM等ORM执行SQL操作</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"github.com/jinzhu/gorm"</span></span><br><span class="line"> _ <span class="string">"github.com/jinzhu/gorm/dialects/sqlite"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> Product <span class="keyword">struct</span> {</span><br><span class="line"> gorm.Model</span><br><span class="line"> Code <span class="keyword">string</span></span><br><span class="line"> Price <span class="keyword">uint</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">...</span><br><span class="line"><span class="keyword">var</span> product Product</span><br><span class="line">...</span><br><span class="line">db.First(&product, <span class="number">1</span>)</span><br></pre></td></tr></table></figure>
<ul>
<li>使用参数化查询,禁止拼接SQL语句,另外对于传入参数用于order by或表名的需要通过校验</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"database/sql"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handler</span><span class="params">(db *sql.DB, req *http.Request)</span></span> {</span><br><span class="line"> q := fmt.Sprintf(<span class="string">"SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='%s' ORDER BY PRICE"</span>,</span><br><span class="line"> req.URL.Query()[<span class="string">"category"</span>])</span><br><span class="line"> db.Query(q)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handlerGood</span><span class="params">(db *sql.DB, req *http.Request)</span></span> {</span><br><span class="line"> <span class="comment">// 使用?占位符</span></span><br><span class="line"> q := <span class="string">"SELECT ITEM,PRICE FROM PRODUCT WHERE ITEM_CATEGORY='?' ORDER BY PRICE"</span></span><br><span class="line"> db.Query(q, req.URL.Query()[<span class="string">"category"</span>])</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-3-网络请求"><a href="#1-3-网络请求" class="headerlink" title="1.3 网络请求"></a>1.3 网络请求</h3><h4 id="1-3-1【必须】资源请求过滤验证"><a href="#1-3-1【必须】资源请求过滤验证" class="headerlink" title="1.3.1【必须】资源请求过滤验证"></a>1.3.1【必须】资源请求过滤验证</h4><ul>
<li><p>使用<code>"net/http"</code>下的方法<code>http.Get(url)</code>、<code>http.Post(url, contentType, body)</code>、<code>http.Head(url)</code>、<code>http.PostForm(url, data)</code>、<code>http.Do(req)</code>时,如变量值外部可控(指从参数中动态获取),应对请求目标进行严格的安全校验。</p>
</li>
<li><p>如请求资源域名归属固定的范围,如只允许<code>a.qq.com</code>和<code>b.qq.com</code>,应做白名单限制。如不适用白名单,则推荐的校验逻辑步骤是:</p>
<ul>
<li><p>第 1 步、只允许HTTP或HTTPS协议</p>
</li>
<li><p>第 2 步、解析目标URL,获取其HOST</p>
</li>
<li><p>第 3 步、解析HOST,获取HOST指向的IP地址转换成Long型</p>
</li>
<li><p>第 4 步、检查IP地址是否为内网IP,网段有:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">// 以RFC定义的专有网络为例,如有自定义私有网段亦应加入禁止访问列表。</span><br><span class="line">10.0.0.0/8</span><br><span class="line">172.16.0.0/12</span><br><span class="line">192.168.0.0/16</span><br><span class="line">127.0.0.0/8</span><br></pre></td></tr></table></figure>
</li>
<li><p>第 5 步、请求URL</p>
</li>
<li><p>第 6 步、如有跳转,跳转后执行1,否则绑定经校验的ip和域名,对URL发起请求</p>
</li>
</ul>
</li>
<li><p>官方库<code>encoding/xml</code>不支持外部实体引用,使用该库可避免xxe漏洞</p>
</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"encoding/xml"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"os"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">type</span> Person <span class="keyword">struct</span> {</span><br><span class="line"> XMLName xml.Name <span class="string">`xml:"person"`</span></span><br><span class="line"> Id <span class="keyword">int</span> <span class="string">`xml:"id,attr"`</span></span><br><span class="line"> UserName <span class="keyword">string</span> <span class="string">`xml:"name>first"`</span></span><br><span class="line"> Comment <span class="keyword">string</span> <span class="string">`xml:",comment"`</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> v := &Person{Id: <span class="number">13</span>, UserName: <span class="string">"John"</span>}</span><br><span class="line"> v.Comment = <span class="string">" Need more details. "</span></span><br><span class="line"></span><br><span class="line"> enc := xml.NewEncoder(os.Stdout)</span><br><span class="line"> enc.Indent(<span class="string">" "</span>, <span class="string">" "</span>)</span><br><span class="line"> <span class="keyword">if</span> err := enc.Encode(v); err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Printf(<span class="string">"error: %v\n"</span>, err)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-4-服务器端渲染"><a href="#1-4-服务器端渲染" class="headerlink" title="1.4 服务器端渲染"></a>1.4 服务器端渲染</h3><h4 id="1-4-1【必须】模板渲染过滤验证"><a href="#1-4-1【必须】模板渲染过滤验证" class="headerlink" title="1.4.1【必须】模板渲染过滤验证"></a>1.4.1【必须】模板渲染过滤验证</h4><ul>
<li>使用<code>text/template</code>或者<code>html/template</code>渲染模板时禁止将外部输入参数引入模板,或仅允许引入白名单内字符。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handler</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> r.ParseForm()</span><br><span class="line"> x := r.Form.Get(<span class="string">"name"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> tmpl = <span class="string">`<!DOCTYPE html><html><body></span><br><span class="line"> <form action="/" method="post"></span><br><span class="line"> First name:<br></span><br><span class="line"> <input type="text" name="name" value=""></span><br><span class="line"> <input type="submit" value="Submit"></span><br><span class="line"> </form><p>`</span> + x + <span class="string">` </p></body></html>`</span></span><br><span class="line"></span><br><span class="line"> t := template.New(<span class="string">"main"</span>)</span><br><span class="line"> t, _ = t.Parse(tmpl)</span><br><span class="line"> t.Execute(w, <span class="string">"Hello"</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"github.com/go-playground/validator/v10"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> validate *validator.Validate</span><br><span class="line">validate = validator.New()</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">validateVariable</span><span class="params">(val)</span></span> {</span><br><span class="line"> errs := validate.Var(val, <span class="string">"gte=1,lte=100"</span>) <span class="comment">// 限制必须是1-100的正整数</span></span><br><span class="line"> <span class="keyword">if</span> errs != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(errs)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">handler</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> r.ParseForm()</span><br><span class="line"> x := r.Form.Get(<span class="string">"name"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> validateVariable(x) {</span><br><span class="line"> <span class="keyword">var</span> tmpl = <span class="string">`<!DOCTYPE html><html><body></span><br><span class="line"> <form action="/" method="post"></span><br><span class="line"> First name:<br></span><br><span class="line"> <input type="text" name="name" value=""></span><br><span class="line"> <input type="submit" value="Submit"></span><br><span class="line"> </form><p>`</span> + x + <span class="string">` </p></body></html>`</span></span><br><span class="line"> t := template.New(<span class="string">"main"</span>)</span><br><span class="line"> t, _ = t.Parse(tmpl)</span><br><span class="line"> t.Execute(w, <span class="string">"Hello"</span>)</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-5-Web跨域"><a href="#1-5-Web跨域" class="headerlink" title="1.5 Web跨域"></a>1.5 Web跨域</h3><h4 id="1-5-1【必须】跨域资源共享CORS限制请求来源"><a href="#1-5-1【必须】跨域资源共享CORS限制请求来源" class="headerlink" title="1.5.1【必须】跨域资源共享CORS限制请求来源"></a>1.5.1【必须】跨域资源共享CORS限制请求来源</h4><ul>
<li>CORS请求保护不当可导致敏感信息泄漏,因此应当严格设置Access-Control-Allow-Origin使用同源策略进行保护。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line">c := cors.New(cors.Options{</span><br><span class="line"> AllowedOrigins: []<span class="keyword">string</span>{<span class="string">"http://qq.com"</span>, <span class="string">"https://qq.com"</span>},</span><br><span class="line"> AllowCredentials: <span class="literal">true</span>,</span><br><span class="line"> Debug: <span class="literal">false</span>,</span><br><span class="line">})</span><br><span class="line"></span><br><span class="line"><span class="comment">// 引入中间件</span></span><br><span class="line">handler = c.Handler(handler)</span><br></pre></td></tr></table></figure>
<h3 id="1-6-响应输出"><a href="#1-6-响应输出" class="headerlink" title="1.6 响应输出"></a>1.6 响应输出</h3><h4 id="1-6-1-【必须】设置正确的HTTP响应包类型"><a href="#1-6-1-【必须】设置正确的HTTP响应包类型" class="headerlink" title="1.6.1 【必须】设置正确的HTTP响应包类型"></a>1.6.1 【必须】设置正确的HTTP响应包类型</h4><ul>
<li>响应头Content-Type与实际响应内容,应保持一致。如:API响应数据类型是json,则响应头使用<code>application/json</code>;若为xml,则设置为<code>text/xml</code>。</li>
</ul>
<h4 id="1-6-2-【必须】添加安全响应头"><a href="#1-6-2-【必须】添加安全响应头" class="headerlink" title="1.6.2 【必须】添加安全响应头"></a>1.6.2 【必须】添加安全响应头</h4><ul>
<li>所有接口、页面,添加响应头 <code>X-Content-Type-Options: nosniff</code>。</li>
<li>所有接口、页面,添加响应头<code>X-Frame-Options</code>。按需合理设置其允许范围,包括:<code>DENY</code>、<code>SAMEORIGIN</code>、<code>ALLOW-FROM origin</code>。用法参考:<a href="https://developer.mozilla.org/zh-CN/docs/Web/HTTP/X-Frame-Options" target="_blank" rel="external">MDN文档</a></li>
</ul>
<h4 id="1-6-3【必须】外部输入拼接到HTTP响应头中需进行过滤"><a href="#1-6-3【必须】外部输入拼接到HTTP响应头中需进行过滤" class="headerlink" title="1.6.3【必须】外部输入拼接到HTTP响应头中需进行过滤"></a>1.6.3【必须】外部输入拼接到HTTP响应头中需进行过滤</h4><ul>
<li>应尽量避免外部可控参数拼接到HTTP响应头中,如业务需要则需要过滤掉<code>\r</code>、<code>\n</code>等换行符,或者拒绝携带换行符号的外部输入。</li>
</ul>
<h4 id="1-6-4【必须】外部输入拼接到response页面前进行编码处理"><a href="#1-6-4【必须】外部输入拼接到response页面前进行编码处理" class="headerlink" title="1.6.4【必须】外部输入拼接到response页面前进行编码处理"></a>1.6.4【必须】外部输入拼接到response页面前进行编码处理</h4><ul>
<li>直出html页面或使用模板生成html页面的,推荐使用<code>text/template</code>自动编码,或者使用<code>html.EscapeString</code>或<code>text/template</code>对<code><, >, &, ',"</code>等字符进行编码。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"html/template"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">outtemplate</span><span class="params">(w http.ResponseWriter, r *http.Request)</span></span> {</span><br><span class="line"> param1 := r.URL.Query().Get(<span class="string">"param1"</span>)</span><br><span class="line"> tmpl := template.New(<span class="string">"hello"</span>)</span><br><span class="line"> tmpl, _ = tmpl.Parse(<span class="string">`{{define "T"}}{{.}}{{end}}`</span>)</span><br><span class="line"> tmpl.ExecuteTemplate(w, <span class="string">"T"</span>, param1)</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-7-会话管理"><a href="#1-7-会话管理" class="headerlink" title="1.7 会话管理"></a>1.7 会话管理</h3><h4 id="1-7-1【必须】安全维护session信息"><a href="#1-7-1【必须】安全维护session信息" class="headerlink" title="1.7.1【必须】安全维护session信息"></a>1.7.1【必须】安全维护session信息</h4><ul>
<li>用户登录时应重新生成session,退出登录后应清理session。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"github.com/gorilla/handlers"</span></span><br><span class="line"> <span class="string">"github.com/gorilla/mux"</span></span><br><span class="line"> <span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="comment">// 创建cookie</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">setToken</span><span class="params">(res http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> expireToken := time.Now().Add(time.Minute * <span class="number">30</span>).Unix()</span><br><span class="line"> expireCookie := time.Now().Add(time.Minute * <span class="number">30</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment">//...</span></span><br><span class="line"></span><br><span class="line"> cookie := http.Cookie{</span><br><span class="line"> Name: <span class="string">"Auth"</span>,</span><br><span class="line"> Value: signedToken,</span><br><span class="line"> Expires: expireCookie, <span class="comment">// 过期失效</span></span><br><span class="line"> HttpOnly: <span class="literal">true</span>,</span><br><span class="line"> Path: <span class="string">"/"</span>,</span><br><span class="line"> Domain: <span class="string">"127.0.0.1"</span>,</span><br><span class="line"> Secure: <span class="literal">true</span>,</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> http.SetCookie(res, &cookie)</span><br><span class="line"> http.Redirect(res, req, <span class="string">"/profile"</span>, <span class="number">307</span>)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// 删除cookie</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">logout</span><span class="params">(res http.ResponseWriter, req *http.Request)</span></span> {</span><br><span class="line"> deleteCookie := http.Cookie{</span><br><span class="line"> Name: <span class="string">"Auth"</span>,</span><br><span class="line"> Value: <span class="string">"none"</span>,</span><br><span class="line"> Expires: time.Now(),</span><br><span class="line"> }</span><br><span class="line"> http.SetCookie(res, &deleteCookie)</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-7-2【必须】CSRF防护"><a href="#1-7-2【必须】CSRF防护" class="headerlink" title="1.7.2【必须】CSRF防护"></a>1.7.2【必须】CSRF防护</h4><ul>
<li>涉及系统敏感操作或可读取敏感信息的接口应校验<code>Referer</code>或添加<code>csrf_token</code>。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"github.com/gorilla/csrf"</span></span><br><span class="line"> <span class="string">"github.com/gorilla/mux"</span></span><br><span class="line"> <span class="string">"net/http"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> r := mux.NewRouter()</span><br><span class="line"> r.HandleFunc(<span class="string">"/signup"</span>, ShowSignupForm)</span><br><span class="line"> r.HandleFunc(<span class="string">"/signup/post"</span>, SubmitSignupForm)</span><br><span class="line"> <span class="comment">// 使用csrf_token验证</span></span><br><span class="line"> http.ListenAndServe(<span class="string">":8000"</span>,</span><br><span class="line"> csrf.Protect([]<span class="keyword">byte</span>(<span class="string">"32-byte-long-auth-key"</span>))(r))</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h3 id="1-8-访问控制"><a href="#1-8-访问控制" class="headerlink" title="1.8 访问控制"></a>1.8 访问控制</h3><h4 id="1-8-1【必须】默认鉴权"><a href="#1-8-1【必须】默认鉴权" class="headerlink" title="1.8.1【必须】默认鉴权"></a>1.8.1【必须】默认鉴权</h4><ul>
<li><p>除非资源完全可对外开放,否则系统默认进行身份认证,使用白名单的方式放开不需要认证的接口或页面。</p>
</li>
<li><p>根据资源的机密程度和用户角色,以最小权限原则,设置不同级别的权限,如完全公开、登录可读、登录可写、特定用户可读、特定用户可写等</p>
</li>
<li><p>涉及用户自身相关的数据的读写必须验证登录态用户身份及其权限,避免越权操作</p>
<figure class="highlight sql"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">-- 伪代码</span></span><br><span class="line"><span class="keyword">select</span> <span class="keyword">id</span> <span class="keyword">from</span> <span class="keyword">table</span> <span class="keyword">where</span> <span class="keyword">id</span>=:<span class="keyword">id</span> <span class="keyword">and</span> userid=session.userid</span><br></pre></td></tr></table></figure>
</li>
<li><p>没有独立账号体系的外网服务使用<code>QQ</code>或<code>微信</code>登录,内网服务使用<code>统一登录服务</code>登录,其他使用账号密码登录的服务需要增加验证码等二次验证</p>
</li>
</ul>
<h3 id="1-9-并发保护"><a href="#1-9-并发保护" class="headerlink" title="1.9 并发保护"></a>1.9 并发保护</h3><h4 id="1-9-1【必须】禁止在闭包中直接调用循环变量"><a href="#1-9-1【必须】禁止在闭包中直接调用循环变量" class="headerlink" title="1.9.1【必须】禁止在闭包中直接调用循环变量"></a>1.9.1【必须】禁止在闭包中直接调用循环变量</h4><ul>
<li>在循环中启动协程,当协程中使用到了循环的索引值,由于多个协程同时使用同一个变量会产生数据竞争,造成执行结果异常。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> runtime.GOMAXPROCS(runtime.NumCPU())</span><br><span class="line"> <span class="keyword">var</span> group sync.WaitGroup</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">5</span>; i++ {</span><br><span class="line"> group.Add(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> group.Done()</span><br><span class="line"> fmt.Printf(<span class="string">"%-2d"</span>, i) <span class="comment">// 这里打印的i不是所期望的</span></span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> group.Wait()</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> runtime.GOMAXPROCS(runtime.NumCPU())</span><br><span class="line"> <span class="keyword">var</span> group sync.WaitGroup</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">5</span>; i++ {</span><br><span class="line"> group.Add(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">(j <span class="keyword">int</span>)</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">if</span> r := <span class="built_in">recover</span>(); r != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Recovered in start()"</span>)</span><br><span class="line"> }</span><br><span class="line"> group.Done()</span><br><span class="line"> }()</span><br><span class="line"> fmt.Printf(<span class="string">"%-2d"</span>, j) <span class="comment">// 闭包内部使用局部变量</span></span><br><span class="line"> }(i) <span class="comment">// 把循环变量显式地传给协程</span></span><br><span class="line"> }</span><br><span class="line"> group.Wait()</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-9-2【必须】禁止并发写map"><a href="#1-9-2【必须】禁止并发写map" class="headerlink" title="1.9.2【必须】禁止并发写map"></a>1.9.2【必须】禁止并发写map</h4><ul>
<li>并发写map容易造成程序崩溃并异常退出,建议加锁保护</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> m := <span class="built_in">make</span>(<span class="keyword">map</span>[<span class="keyword">int</span>]<span class="keyword">int</span>)</span><br><span class="line"> <span class="comment">// 并发读写</span></span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> _ = m[<span class="number">1</span>]</span><br><span class="line"> }</span><br><span class="line"> }()</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> m[<span class="number">2</span>] = <span class="number">1</span></span><br><span class="line"> }</span><br><span class="line"> }()</span><br><span class="line"> <span class="keyword">select</span> {}</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<h4 id="1-9-3【必须】确保并发安全"><a href="#1-9-3【必须】确保并发安全" class="headerlink" title="1.9.3【必须】确保并发安全"></a>1.9.3【必须】确保并发安全</h4><p>敏感操作如果未作并发安全限制,可导致数据读写异常,造成业务逻辑限制被绕过。可通过同步锁或者原子操作进行防护。</p>
<p>通过同步锁共享内存</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">var</span> count <span class="keyword">int</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">Count</span><span class="params">(lock *sync.Mutex)</span></span> {</span><br><span class="line"> lock.Lock() <span class="comment">// 加写锁</span></span><br><span class="line"> count++</span><br><span class="line"> fmt.Println(count)</span><br><span class="line"> lock.Unlock() <span class="comment">// 解写锁,任何一个Lock()或RLock()均需要保证对应有Unlock()或RUnlock()</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> lock := &sync.Mutex{}</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> <span class="keyword">go</span> Count(lock) <span class="comment">// 传递指针是为了防止函数内的锁和调用锁不一致</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> lock.Lock()</span><br><span class="line"> c := count</span><br><span class="line"> lock.Unlock()</span><br><span class="line"> runtime.Gosched() <span class="comment">// 交出时间片给协程</span></span><br><span class="line"> <span class="keyword">if</span> c > <span class="number">10</span> {</span><br><span class="line"> <span class="keyword">break</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<ul>
<li>使用<code>sync/atomic</code>执行原子操作</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// good</span></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"sync"</span></span><br><span class="line"> <span class="string">"sync/atomic"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">type</span> Map <span class="keyword">map</span>[<span class="keyword">string</span>]<span class="keyword">string</span></span><br><span class="line"> <span class="keyword">var</span> m atomic.Value</span><br><span class="line"> m.Store(<span class="built_in">make</span>(Map))</span><br><span class="line"> <span class="keyword">var</span> mu sync.Mutex <span class="comment">// used only by writers</span></span><br><span class="line"> read := <span class="function"><span class="keyword">func</span><span class="params">(key <span class="keyword">string</span>)</span> <span class="params">(val <span class="keyword">string</span>)</span></span> {</span><br><span class="line"> m1 := m.Load().(Map)</span><br><span class="line"> <span class="keyword">return</span> m1[key]</span><br><span class="line"> }</span><br><span class="line"> insert := <span class="function"><span class="keyword">func</span><span class="params">(key, val <span class="keyword">string</span>)</span></span> {</span><br><span class="line"> mu.Lock() <span class="comment">// 与潜在写入同步</span></span><br><span class="line"> <span class="keyword">defer</span> mu.Unlock()</span><br><span class="line"> m1 := m.Load().(Map) <span class="comment">// 导入struct当前数据</span></span><br><span class="line"> m2 := <span class="built_in">make</span>(Map) <span class="comment">// 创建新值</span></span><br><span class="line"> <span class="keyword">for</span> k, v := <span class="keyword">range</span> m1 {</span><br><span class="line"> m2[k] = v</span><br><span class="line"> }</span><br><span class="line"> m2[key] = val</span><br><span class="line"> m.Store(m2) <span class="comment">// 用新的替代当前对象</span></span><br><span class="line"> }</span><br><span class="line"> _, _ = read, insert</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<blockquote>
<p>来源:<a href="https://github.com/Tencent/secguide" target="_blank" rel="external">https://github.com/Tencent/secguide</a></p>
</blockquote>
<h1 id="通用类"><a href="#通用类" class="headerlink" title="通用类"></a>通用类</h1><h2 id="1-代码实现类"><a href="#1-代码实现类" class="headerlink" title="1. 代码实现类"></a>1. 代码实现类</h2><h3 id="1-1-内存管理"><a href="#1-1-内存管理" class="headerlink" title="1.1 内存管理"></a>1.1 内存管理</h3><h4 id="1-1-1【必须】切片长度校验"><a href="#1-1-1【必须】切片长度校验" class="headerlink" title="1.1.1【必须】切片长度校验"></a>1.1.1【必须】切片长度校验</h4><ul>
<li>在对slice进行操作时,必须判断长度是否合法,防止程序panic</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// bad: 未判断data的长度,可导致 index out of range</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">decode</span><span class="params">(data []<span class="keyword">byte</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">if</span> data[<span class="number">0</span>] == <span class="string">'F'</span> && data[<span class="number">1</span>] == <span class="string">'U'</span> && data[<span class="number">2</span>] == <span class="string">'Z'</span> && data[<span class="number">3</span>] == <span class="string">'Z'</span> && data[<span class="number">4</span>] == <span class="string">'E'</span> && data[<span class="number">5</span>] == <span class="string">'R'</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Bad"</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// bad: slice bounds out of range</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">foo</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> slice = []<span class="keyword">int</span>{<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>, <span class="number">5</span>, <span class="number">6</span>}</span><br><span class="line"> fmt.Println(slice[:<span class="number">10</span>])</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// good: 使用data前应判断长度是否合法</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">decode</span><span class="params">(data []<span class="keyword">byte</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">len</span>(data) == <span class="number">6</span> {</span><br><span class="line"> <span class="keyword">if</span> data[<span class="number">0</span>] == <span class="string">'F'</span> && data[<span class="number">1</span>] == <span class="string">'U'</span> && data[<span class="number">2</span>] == <span class="string">'Z'</span> && data[<span class="number">3</span>] == <span class="string">'Z'</span> && data[<span class="number">4</span>] == <span class="string">'E'</span> && data[<span class="number">5</span>] == <span class="string">'R'</span> {</span><br><span class="line"> fmt.Println(<span class="string">"Good"</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
50 张图,掌握 Kubernetes 中优雅且零停机部署的实现
http://team.jiunile.com//blog/2021/02/k8s-graceful-shutdown.html
2021-02-02T14:00:00.000Z
2021-02-02T06:24:19.000Z
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>在本文中,您将了解如何在Pod启动或关闭时防止连接异常,并将学习如何以优雅的方式关闭长时间运行的任务。<br><img src="/images/k8s/g_shutdown_1.png" alt="graceful shutdown"></p>
<a id="more"></a>
<p>在 Kubernetes 中,创建和删除 Pod 是最常见的任务之一。</p>
<p>当您执行滚动更新,扩展部署,每个新发行版,每个作业和 cron 作业等时,都会创建 Pod。</p>
<p>但是在节点被驱逐之后,Pods 也会被删除并重新创建—例如,当您将节点标记为不可调度时。</p>
<p>这些 Pod 的生命是如此短暂,那么当 Pod 在响应请求的过程中却被告知关闭时会发生什么?</p>
<p>请求在关闭之前是否已完成?</p>
<p>接下来的请求又如何呢?</p>
<p>在讨论删除 Pod 时会发生什么之前,有必要讨论一下创建 Pod 时会发生什么。</p>
<p>假设您要在集群中创建以下 Pod:<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> v1</span><br><span class="line"><span class="attr">kind:</span> Pod</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> my-pod</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> containers:</span></span><br><span class="line"><span class="attr"> - name:</span> web</span><br><span class="line"><span class="attr"> image:</span> nginx</span><br><span class="line"><span class="attr"> ports:</span></span><br><span class="line"><span class="attr"> - name:</span> web</span><br><span class="line"><span class="attr"> containerPort:</span> <span class="number">80</span></span><br></pre></td></tr></table></figure></p>
<p>您可以使用以下方式将 YAML 定义提交给集群:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl apply <span class="_">-f</span> pod.yaml</span><br></pre></td></tr></table></figure></p>
<p>输入命令后,kubectl 便将 Pod 定义提交给 Kubernetes API。</p>
<h2 id="在数据库中保存集群的状态"><a href="#在数据库中保存集群的状态" class="headerlink" title="在数据库中保存集群的状态"></a>在数据库中保存集群的状态</h2><p>API 接收和检查 Pod 定义,然后将其存储在数据库 etcd 中。</p>
<p>Pod 也将添加到<a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#scheduling-cycle-binding-cycle" target="_blank" rel="external">调度程序的队列</a>中。</p>
<p>调度程序:</p>
<ol>
<li>检查定义</li>
<li>收集有关工作负载的详细信息,例如 CPU 和内存请求,然后</li>
<li>确定哪个节点最适合运行它。(<a href="https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#extension-points" target="_blank" rel="external">通过Filters 和 Predicates</a>)。</li>
</ol>
<p>在过程结束时:</p>
<ul>
<li>在 etcd 中将 Pod 标记为 Scheduled。</li>
<li>为 Pod 分配了一个节点。</li>
<li>Pod 的状态存储在 etcd 中。</li>
</ul>
<p><strong>但是Pod仍然不存在。</strong></p>
<ol>
<li>当您使用 <code>kubectl apply -f</code> 提交一个 Pod 时,YAML 被发送到 kubernetes api。<br><img src="/images/k8s/g_sd_2.png" alt="graceful shutdown"></li>
<li>API 将 Pod 保存在数据库 etcd g中。<br><img src="/images/k8s/g_sd_3.png" alt="graceful shutdown"> </li>
<li>调度程序为这个 Pod 分配最佳节点,并且 Pod 的状态更改为 Pending。pod 只存在于etcd中。<br><img src="/images/k8s/g_sd_4.png" alt="graceful shutdown"></li>
</ol>
<p>先前的任务发生在控制平面中,并且状态存储在数据库中。</p>
<p>那么谁在您的节点中创建 Pod?</p>
<h2 id="Kubelet-—-Kubernetes-代理"><a href="#Kubelet-—-Kubernetes-代理" class="headerlink" title="Kubelet — Kubernetes 代理"></a>Kubelet — Kubernetes 代理</h2><p><strong>kubelet 的工作是轮询控制平面以获取更新。</strong></p>
<p>您可以想象 kubelet 不断地向主节点询问:“我管理工作节点1,是否对我有任何新的 Pod?”。</p>
<p>当有 Pod 时,kubelet 会创建它。</p>
<p>有一点需要注意。</p>
<p>kubelet 不会自行创建 Pod。而是将工作委托给其他三个组件:</p>
<ol>
<li><strong>容器运行时接口(CRI)</strong> — 为 Pod 创建容器的组件。</li>
<li><strong>容器网络接口(CNI)</strong> — 将容器连接到群集网络并分配IP地址的组件。</li>
<li><strong>容器存储接口(CSI)</strong> — 在容器中装载卷的组件。</li>
</ol>
<p>在大多数情况下,容器运行时接口(CRI)的工作类似于:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">docker run <span class="_">-d</span> <my-container-image></span><br></pre></td></tr></table></figure></p>
<p>容器网络接口(CNI)有点有趣,因为它负责:</p>
<ol>
<li>为 Pod 生成有效的 IP 地址。</li>
<li>将容器连接到网络的其余部分。</li>
</ol>
<p>可以想象,有几种方法可以将容器连接到网络并分配有效的 IP 地址(您可以在 IPv4 或 IPv6 之间进行选择,也可以分配多个 I P地址)。</p>
<p>例如,<a href="https://archive.shivam.dev/docker-networking-explained/" target="_blank" rel="external">Docker 创建虚拟以太网对并将其连接到网桥</a>,而 <a href="https://itnext.io/kubernetes-is-hard-why-eks-makes-it-easier-for-network-and-security-architects-ea6d8b2ca965" target="_blank" rel="external">AWS—CNI 将 Pods 直接连接到虚拟私有云(VPC)</a>。</p>
<p>当容器网络接口完成其工作时,Pod已连接到网络,并分配了有效的IP地址。</p>
<p>还有一个问题。</p>
<p><strong>Kubelet 知道 IP 地址(因为它调用了容器网络接口),但是控制平面却不知道。</strong></p>
<p>没有人告诉主节点,该Pod已分配了IP地址,并准备接收流量。</p>
<p>就控制平面而言,仍在创建 Pod。</p>
<p><strong>Kubelet 的工作是收集 Pod 的所有详细信息(例如 I P地址)并将其报告回控制平面。</strong></p>
<p>您可以想象检查 etcd 不仅可以显示 Pod 的运行位置,还可以显示其 IP 地址。</p>
<ol>
<li>Kubelet 轮询控制平面以获取更新。<br><img src="/images/k8s/g_sd_5.png" alt="graceful shutdown"></li>
<li>当一个新的 Pod 分配给它的节点时,kubelet 会检索详细信息<br><img src="/images/k8s/g_sd_6.png" alt="graceful shutdown"></li>
<li>Kubernetns 不会自己创建 pod。它依赖于三个组件:容器运行时接口、容器网络接口和容器存储接口。<br><img src="/images/k8s/g_sd_7.png" alt="graceful shutdown"></li>
<li>一旦所有三个组件都成功完成,Pod 就在您的节点中运行并分配了一个 IP 地址。<br><img src="/images/k8s/g_sd_8.png" alt="graceful shutdown"></li>
<li>kubelet 向控制平面报告 IP 地址。<br><img src="/images/k8s/g_sd_9.png" alt="graceful shutdown"></li>
</ol>
<p>如果 Pod 不是任何服务的一部分,那么任务将结束。</p>
<p>Pod 已创建并可以使用。</p>
<p>如果 Pod 是服务的一部分,则还需要执行几个步骤。</p>
<h2 id="Pods-和-Services"><a href="#Pods-和-Services" class="headerlink" title="Pods 和 Services"></a>Pods 和 Services</h2><p>创建服务时,通常需要注意以下两条信息:</p>
<ol>
<li><code>selector</code> — 用于指定将接收流量的 Pod。</li>
<li><code>targetPort</code> — 通过 pod 使用的端口接收的流量。</li>
</ol>
<p>服务的典型 YAML 定义如下所示:<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> v1</span><br><span class="line"><span class="attr">kind:</span> Service</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> my-service</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> ports:</span></span><br><span class="line"><span class="attr"> - port:</span> <span class="number">80</span></span><br><span class="line"><span class="attr"> targetPort:</span> <span class="number">3000</span></span><br><span class="line"><span class="attr"> selector:</span></span><br><span class="line"><span class="attr"> name:</span> app</span><br></pre></td></tr></table></figure></p>
<p>将 Service 提交给集群时 <code>kubectl apply</code>,Kubernetes 会找到所有具有与selector(<code>name: app</code>)相同标签的 Pod,并收集其 IP 地址 — 但前提是它们已通过<a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-tcp-liveness-probe" target="_blank" rel="external">Readiness 探针</a>。</p>
<p>然后,对于每个 IP 地址,它将 IP 地址和端口连接在一起。</p>
<p>如果 IP 地址是 <code>10.0.0.3</code> 和,<code>targetPort</code> 是 <code>3000</code>,Kubernetes 将两个结果连接起来并称为 endpoint。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">IP address + port = endpoint</span><br><span class="line">---------------------------------</span><br><span class="line">10.0.0.3 + 3000 = 10.0.0.3:3000</span><br></pre></td></tr></table></figure></p>
<p>endpoint 存储在 etcd 的另一个名为 Endpoint 的对象中。</p>
<p><em>是否有点疑惑?</em></p>
<p>Kubernetes 中定义:</p>
<ul>
<li>endpoint 是 IP 地址 + 端口对(<code>10.0.0.3:3000</code>)。</li>
<li>Endpoint 是 endpoint 的集合。</li>
</ul>
<p>Endpoint 对象是 Kubernetes 中的真实对象,对于每个服务 Kubernetes 都会自动创建一个 endpoint 对象。</p>
<p>您可以使用以下方法进行验证:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ kubectl get services,endpoints</span><br><span class="line">NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)</span><br><span class="line">service/my-service-1 ClusterIP 10.105.17.65 <none> 80/TCP</span><br><span class="line">service/my-service-2 ClusterIP 10.96.0.1 <none> 443/TCP</span><br><span class="line"></span><br><span class="line">NAME ENDPOINTS</span><br><span class="line">endpoints/my-service-1 172.17.0.6:80,172.17.0.7:80</span><br><span class="line">endpoints/my-service-2 192.168.99.100:8443</span><br></pre></td></tr></table></figure></p>
<p>Endpoint 从 Pod 收集所有 IP 地址和端口。</p>
<p>但并不是一次性的。</p>
<p>在以下情况下,将使用新的 endpoint 列表刷新 Endpoint 对象:</p>
<ol>
<li>创建一个 Pod。</li>
<li>Pod 已删除。</li>
<li>在 Pod 上修改了标签。</li>
</ol>
<p>因此,您可以想象,每次创建 Pod 并在 kubelet 将其 IP 地址发布到主节点后,Kubernetes 都会更新所有 endpoint 以反映更改:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ kubectl get services,endpoints</span><br><span class="line">NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)</span><br><span class="line">service/my-service-1 ClusterIP 10.105.17.65 <none> 80/TCP</span><br><span class="line">service/my-service-2 ClusterIP 10.96.0.1 <none> 443/TCP</span><br><span class="line"></span><br><span class="line">NAME ENDPOINTS</span><br><span class="line">endpoints/my-service-1 172.17.0.6:80,172.17.0.7:80,172.17.0.8:80</span><br><span class="line">endpoints/my-service-2 192.168.99.100:8443</span><br></pre></td></tr></table></figure></p>
<p>很好,endpoint 存储在控制平面中,并且 endpoint 对象已更新。</p>
<ol>
<li>在此图中,集群中部署了一个 Pod。Pod 属于服务。如果您要检查 etcd,则可以找到 Pod 的详细信息以及服务。<br><img src="/images/k8s/g_sd_10.png" alt="graceful shutdown"></li>
<li>当部署新 pod 后会发生什么?<br><img src="/images/k8s/g_sd_11.png" alt="graceful shutdown"></li>
<li>Kubernetes 必须跟踪 Pod 及其 IP 地址。服务应该将流量路由到新的 endpoint,因此应该传播 IP 地址和端口。<br><img src="/images/k8s/g_sd_12.png" alt="graceful shutdown"></li>
<li>当部署另一个 Pod 时会发生什么?<br><img src="/images/k8s/g_sd_13.png" alt="graceful shutdown"></li>
<li>完全相同的过程。在数据库中为 Pod 创建一个新的“记录”,并传递给 endpoint。<br><img src="/images/k8s/g_sd_14.png" alt="graceful shutdown"></li>
<li>但是,当一个 Pod 被删除时会发生什么呢?<br><img src="/images/k8s/g_sd_15.png" alt="graceful shutdown"></li>
<li>服务会立即删除 endpoint,最后,Pod 也会从数据库中删除。<br><img src="/images/k8s/g_sd_16.png" alt="graceful shutdown"></li>
<li>Kubernetes 会对集群中的每一个小变化做出反应。<br><img src="/images/k8s/g_sd_17.png" alt="graceful shutdown"></li>
</ol>
<p><em>您准备好开始使用 Pod 了吗?</em></p>
<h2 id="在-Kubernetes-中使用-Endpoint"><a href="#在-Kubernetes-中使用-Endpoint" class="headerlink" title="在 Kubernetes 中使用 Endpoint"></a>在 Kubernetes 中使用 Endpoint</h2><p><strong>endpoint 由 Kubernetes 中的多个组件使用。</strong></p>
<p>Kube-proxy 使用 endpoint 在节点上设置 iptables 规则。</p>
<p>因此,每当 endpoint(对象)发生变化时,kube-proxy 就会检索新的 IP 地址和端口列表,并编写新的 iptables 规则。</p>
<ol>
<li>让我们考虑具有两个 Pod 且不包含 Service 的三节点群集。Pod 的状态存储在 etcd 中。<br><img src="/images/k8s/g_sd_18.png" alt="graceful shutdown"></li>
<li>创建服务时会发生什么?<br><img src="/images/k8s/g_sd_19.png" alt="graceful shutdown"></li>
<li>Kubernetes 创建了一个 endpoint 对象,并从 pod 收集所有 endpoint(IP 地址和端口对)。<br><img src="/images/k8s/g_sd_20.png" alt="graceful shutdown"></li>
<li>Kube-proxy 守护进程监听 endpoint 的更改。<br><img src="/images/k8s/g_sd_21.png" alt="graceful shutdown"></li>
<li>当添加、删除或更新 endpoint 时,kube-proxy 检索 endpoint 的新列表。<br><img src="/images/k8s/g_sd_22.png" alt="graceful shutdown"></li>
<li>Kube-proxy 使用 endpoint 在集群的每个节点上创建 iptables 规则。<br><img src="/images/k8s/g_sd_23.png" alt="graceful shutdown"></li>
</ol>
<p>Ingress controller 使用相同的 endpoint 列表。</p>
<p>Ingress controller 是群集中将外部流量路由到群集中的那个组件。</p>
<p>设置 Ingress 清单时,通常将 Service 指定为目标:<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> networking.k8s.io/v1beta1</span><br><span class="line"><span class="attr">kind:</span> Ingress</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> my-ingress</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> rules:</span></span><br><span class="line"><span class="attr"> - http:</span></span><br><span class="line"><span class="attr"> paths:</span></span><br><span class="line"><span class="attr"> - backend:</span></span><br><span class="line"><span class="attr"> serviceName:</span> my-service</span><br><span class="line"><span class="attr"> servicePort:</span> <span class="number">80</span></span><br><span class="line"><span class="attr"> path:</span> /</span><br></pre></td></tr></table></figure></p>
<p><em>实际上,流量不会路由到服务。</em></p>
<p>取而代之的是,Ingress controller 设置了一个订阅,每次该服务的 endpoint 更改时都将收到通知。</p>
<p><strong>Ingress 会将流量直接路由到 Pod,从而跳过服务</strong>。</p>
<p>可以想象,每次更改 endpoint(对象)时,Ingress 都会检索 IP 地址和端口的新列表,并将控制器重新配置为包括新的 Pod。</p>
<ol>
<li>在这张图片中,有一个 Ingress 控制器,它带有两个副本和一个 Service 的 Deployment。<br><img src="/images/k8s/g_sd_24.png" alt="graceful shutdown"></li>
<li>如果您想通过入口将外部流量路由到 Pods,您应该创建一个入口清单(一个 YAML 文件)。<br><img src="/images/k8s/g_sd_25.png" alt="graceful shutdown"></li>
<li>一旦你运行了 <code>kubectl apply -f ingress.yaml</code>,入口控制器从控制平面检索文件。<br><img src="/images/k8s/g_sd_26.png" alt="graceful shutdown"></li>
<li>Ingress YAML 有一个 <code>serviceName</code> 属性,该属性描述它应该使用哪个服务。<br><img src="/images/k8s/g_sd_27.png" alt="graceful shutdown"></li>
<li>入口控制器从服务检索 Endpoint 列表并跳过它。流量直接流向 endpoint(pod)。<br><img src="/images/k8s/g_sd_28.png" alt="graceful shutdown"></li>
<li>当一个新的 Pod 被创建时会发生什么?<br><img src="/images/k8s/g_sd_29.png" alt="graceful shutdown"></li>
<li>您已经知道 Kubernetes 如何创建 Pod 并通告 endpoint。<br><img src="/images/k8s/g_sd_30.png" alt="graceful shutdown"></li>
<li>入口控制器正在订阅对 endpoint 的更改。因为有一个变更的通知,它检索新的 Endpoint 列表。<br><img src="/images/k8s/g_sd_31.png" alt="graceful shutdown"></li>
<li>入口控制器将流量路由到新的 Pod。<br><img src="/images/k8s/g_sd_32.png" alt="graceful shutdown"></li>
</ol>
<p>有更多的 Kubernetes 组件示例订阅了对 endpoint 的更改。</p>
<p>集群中的 DNS 组件 CoreDNS 是另一个示例。</p>
<p>如果您使用 <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" target="_blank" rel="external">Headless 类型的服务</a>,则每次添加或删除 endpoint 时,CoreDNS 都必须订阅对e ndpoint 的更改并重新配置自身。</p>
<p>相同的 endpoint 被 istio 或 Linkerd 之类的服务网格所使用,<a href="https://thebsdbox.co.uk/2020/03/18/Creating-a-Kubernetes-cloud-doesn-t-required-boiling-the-ocean/" target="_blank" rel="external">云提供商也创建了</a> <code>type:LoadBalancer</code>。</p>
<p>您必须记住,有几个组件订阅了对endpoint的更改,它们可能会在不同时间收到有关 endpoint 更新的通知。</p>
<p><em>够了吗,还是在创建 Pod 之后有什么事发生?</em></p>
<p><strong>这次您完成了!</strong></p>
<p>快速回顾一下创建Pod时发生的情况:</p>
<ol>
<li>Pod 存储在 etcd 中。</li>
<li>调度程序分配一个节点。它将节点写入 etcd。</li>
<li>向 kubelet 通知新的和预定的 Pod。</li>
<li>kubelet 将创建容器的委托委派给容器运行时接口(CRI)。</li>
<li>kubelet 代表将容器附加到容器网络接口(CNI)。</li>
<li>kubelet 将容器中的安装卷委托给容器存储接口(CSI)。</li>
<li>容器网络接口分配 IP 地址。</li>
<li>Kubelet 将 IP 地址报告给控制平面。</li>
<li>IP 地址存储在 etcd 中。</li>
</ol>
<p>如果您的 Pod 属于服务:</p>
<ol>
<li>Kubelet 等待成功的 Readiness 探针。</li>
<li>通知所有相关的 endpoint(对象)更改。</li>
<li>Endpoint 将新 endpoint(IP 地址 + 端口对)添加到其列表中。</li>
<li>Endpoint 更改将通知 Kube-proxy。Kube-proxy 更新每个节点上的 iptables 规则。</li>
<li>通知 Endpoint 变化的入口控制器。控制器将流量路由到新的 IP 地址。</li>
<li>CoreDNS 通知 Endpoint 更改。如果服务的类型为 Headless,则更新 DNS 条目。</li>
<li>向云提供商通知 Endpoint 更改。如果服务为 <code>type: LoadBalancer</code>,则将新 Endpoint 配置为负载均衡器池的一部分。</li>
<li>Endpoint 更改将通知群集中安装的所有服务网格。</li>
<li>订阅 Endpoint 更改的其他运营商也会收到通知。</li>
</ol>
<p>如此长的列表令人惊讶地只是一项常见任务 — 创建 Pod。</p>
<p>Pod 正在运行。现在是时候讨论删除它时会发生什么。</p>
<h2 id="删除-pod"><a href="#删除-pod" class="headerlink" title="删除 pod"></a>删除 pod</h2><p>您可能已经猜到了,但是删除 Pod 时,必须遵循相同的步骤,但要相反。</p>
<p>首先,应从 endpoint(对象)中删除 endpoint。</p>
<p>这次将忽略 “Readiness” 探针,并立即从控制平面移除 endpoint。</p>
<p>依次触发所有事件到 kube-proxy,Ingress 控制器,DNS,服务网格等。</p>
<p>这些组件将更新其内部状态,并停止将流量路由到IP地址。</p>
<p>由于组件可能正在忙于做其他事情,<strong>因此无法保证从其内部状态中删除IP地址将花费多长时间</strong>。</p>
<p>对于某些人来说,可能不到一秒钟。对于其他人,可能需要更多时间。</p>
<ol>
<li>如果您要使用删除 Pod <code>kubectl delete pod</code>,则该命令将首先到达 Kubernetes API<br><img src="/images/k8s/g_sd_33.png" alt="graceful shutdown"></li>
<li>消息被控制平面中的特定控制器截获:Endpoint 控制器。<br><img src="/images/k8s/g_sd_34.png" alt="graceful shutdown"></li>
<li>Endpoint 控制器向 API 发出命令,从端点对象中删除 IP 地址和端口。<br><img src="/images/k8s/g_sd_35.png" alt="graceful shutdown"></li>
<li>谁侦听 Endpoint 更改? Kube-proxy、入口控制器、CoreDNS 等会收到更改通知。<br><img src="/images/k8s/g_sd_36.png" alt="graceful shutdown"></li>
<li>一些组件(如 kube-proxy )可能需要一些额外的时间来进一步传播更改。<br><img src="/images/k8s/g_sd_37.png" alt="graceful shutdown"></li>
</ol>
<p>同时,etcd 中 Pod 的状态更改为 Termination。</p>
<p>将通知 kubelet 更改并委托:</p>
<ol>
<li>将全部容器卸载到容器存储接口(CSI)。</li>
<li>从网络上分离容器并将IP地址释放到容器网络接口(CNI)。</li>
<li>将容器销毁到容器运行时接口(CRI)。</li>
</ol>
<p>换句话说,Kubernetes遵循与创建Pod完全相同的步骤,但相反。</p>
<ol>
<li>如果您要使用删除 Pod <code>kubectl delete pod</code>,则该命令将首先到达 Kubernetes API。<br><img src="/images/k8s/g_sd_38.png" alt="graceful shutdown"></li>
<li>当 kubelet 轮询控制平面以获取更新时,它注意到 Pod 被删除了。<br><img src="/images/k8s/g_sd_39.png" alt="graceful shutdown"></li>
<li>kubelet 将销毁 Pod 委托给容器运行时接口、容器网络接口和容器存储接口。<br><img src="/images/k8s/g_sd_40.png" alt="graceful shutdown"></li>
</ol>
<p>但是,存在细微但必不可少的差异。</p>
<p><strong>当您终止 Pod 时,将同时删除 endpoint 和发送到 kubelet 的信号。</strong></p>
<p>首次创建 Pod 时,Kubernetes 等待 kubelet 报告 IP 地址,然后开始 endpoint 通告。</p>
<p><strong>但是,当您删除 Pod 时,事件将并行开始。</strong></p>
<p>这可能会导致很多竞争情况。</p>
<p><em>如果在通告 endpoint 之前删除 Pod 怎么办?</em></p>
<ol>
<li>删除 endpoint 和删除 Pod 会同时发生。<br><img src="/images/k8s/g_sd_41.png" alt="graceful shutdown"></li>
<li>因此,您可以在 kube-proxy 更新 iptables 规则之前删除 endpoint。<br><img src="/images/k8s/g_sd_42.png" alt="graceful shutdown"></li>
<li>或者更幸运的是,只有在 endpoint 完全通告之后,Pod 才会被删除。<br><img src="/images/k8s/g_sd_43.png" alt="graceful shutdown"></li>
</ol>
<h2 id="正常关机(Graceful)"><a href="#正常关机(Graceful)" class="headerlink" title="正常关机(Graceful)"></a>正常关机(Graceful)</h2><p>当 Pod 从 kube-proxy 或 Ingress 控制器中删除之前终止时,您可能会遇到停机时间。</p>
<p>而且,如果您考虑一下,这是有道理的。</p>
<p>Kubernetes 仍将流量路由到 IP 地址,但 Pod 不再存在。</p>
<p>Ingress 控制器,kube-proxy,CoreDNS 等没有足够的时间从其内部状态中删除 IP 地址。</p>
<p>理想情况下,在删除 Pod 之前,Kubernetes 应该等待集群中的所有组件具有更新的 endpoint 列表。</p>
<p>但是 Kubernetes 不能那样工作。</p>
<p>Kubernetes 提供了健壮的机制来分布 endpoint(即 Endpoint 对象和更高级的抽象功能,例如 <a href="https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/" target="_blank" rel="external">Endpoint Slices</a>)。</p>
<p>但是,Kubernetes 不会验证订阅 endpoint 更改的组件是否是集群状态的最新信息。</p>
<p>那么,如何避免这种竞争情况并确保在通告 endpoint 之后删除 Pod?</p>
<p><strong>你应该等一下!</strong></p>
<p><strong>当 Pod 即将被删除时,它会收到 SIGTERM 信号。</strong></p>
<p>您的应用程序可以捕获该信号并开始关闭。</p>
<p>由于 endpoint 不太可能立即从 Kubernetes 中的所有组件中删除,因此您可以:</p>
<ol>
<li>请稍等片刻,然后退出。</li>
<li>尽管有 SIGTERM,仍然可以处理传入流量。</li>
<li>最后,关闭现有的长期连接(也许是数据库连接或 WebSocket)。</li>
<li>关闭该过程。</li>
</ol>
<p>你应该等多久?</p>
<p><strong>默认情况下,Kubernetes 将发送 SIGTERM 信号并等待 30 秒,然后强制终止该进程。</strong></p>
<p>因此,您可以在最初的15秒内继续操作,以防万一。</p>
<p>希望该间隔应足以将 endpoint 删除通知到 kube-proxy,Ingress 控制器,CoreDNS 等。</p>
<p>因此,越来越少的流量将到达您的 Pod,直到停止为止。</p>
<p>15 秒后,可以安全地关闭与数据库的连接(或任何持久连接)并终止该过程。</p>
<p>如果您认为需要更多时间,则可以在 20 或 25 秒时停止该过程。</p>
<p>但是,您应该记住,Kubernetes 将在 30 秒后强行终止进程(<a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution" target="_blank" rel="external">除非您更改 <code>terminationGracePeriodSecondsPod</code> 定义中的</a>)。</p>
<p>如果您无法更改代码以等待更长的时间怎么办?</p>
<p>您可以调用脚本以等待固定的时间,然后退出应用程序。</p>
<p>在调用 SIGTERM 之前,Kubernetes <code>preStop</code> 在 Pod 中公开一个钩子。</p>
<p>您可以将 <code>preStop</code> 钩子设置为等待 15 秒。</p>
<p>让我们看一个例子:<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> v1</span><br><span class="line"><span class="attr">kind:</span> Pod</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> my-pod</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> containers:</span></span><br><span class="line"><span class="attr"> - name:</span> web</span><br><span class="line"><span class="attr"> image:</span> nginx</span><br><span class="line"><span class="attr"> ports:</span></span><br><span class="line"><span class="attr"> - name:</span> web</span><br><span class="line"><span class="attr"> containerPort:</span> <span class="number">80</span></span><br><span class="line"><span class="attr"> lifecycle:</span></span><br><span class="line"><span class="attr"> preStop:</span></span><br><span class="line"><span class="attr"> exec:</span></span><br><span class="line"><span class="attr"> command:</span> [<span class="string">"sleep"</span>, <span class="string">"15"</span>]</span><br></pre></td></tr></table></figure></p>
<p>该 <code>preStop</code> 钩子是 <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/" target="_blank" rel="external">Pod LifeCycle 钩子之一</a>。</p>
<p><em>建议延迟 15 秒吗?</em></p>
<p>这要视情况而定,但这可能是开始测试的明智方法。</p>
<p>以下是您可以选择的选项的概述:</p>
<ol>
<li>您已经知道,当删除 Pod 时,会通知 kubelet 更改。<br><img src="/images/k8s/g_sd_44.png" alt="graceful shutdown"></li>
<li>如果 Pod 有一个 <code>preStop</code> 钩子,则首先调用它。<br><img src="/images/k8s/g_sd_45.png" alt="graceful shutdown"></li>
<li>当 <code>preStop</code> 完成时,kubelet 向容器发送 SIGTERM 信号。从那时起,容器应该关闭所有长期存在的连接并准备终止。<br><img src="/images/k8s/g_sd_46.png" alt="graceful shutdown"></li>
<li>默认情况下,进程有 30 秒的时间退出,这包括 <code>preStop</code> 钩子。如果进程还没有退出,kubelet 发送 SIGKILL 信号并强制终止进程。<br><img src="/images/k8s/g_sd_47.png" alt="graceful shutdown"></li>
<li>kubelet 通知控制平面 pod 已成功删除。<br><img src="/images/k8s/g_sd_48.png" alt="graceful shutdown"></li>
</ol>
<h2 id="宽限时间(Grace-periods)和滚动更新"><a href="#宽限时间(Grace-periods)和滚动更新" class="headerlink" title="宽限时间(Grace periods)和滚动更新"></a>宽限时间(Grace periods)和滚动更新</h2><p>正常关机适用于要删除的 Pod。</p>
<p>但是,如果不删除 Pod,该怎么办?</p>
<p>即使您不这样做,Kubernetes 也会始终删除 Pod。</p>
<p>尤其是,每次部署较新版本的应用程序时,Kubernetes 都会创建和删除 Pod。</p>
<p>在部署中更改镜像时,Kubernetes 会逐步推出更改。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> apps/v1</span><br><span class="line"><span class="attr">kind:</span> Deployment</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> app</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> replicas:</span> <span class="number">3</span></span><br><span class="line"><span class="attr"> selector:</span></span><br><span class="line"><span class="attr"> matchLabels:</span></span><br><span class="line"><span class="attr"> name:</span> app</span><br><span class="line"><span class="attr"> template:</span></span><br><span class="line"><span class="attr"> metadata:</span></span><br><span class="line"><span class="attr"> labels:</span></span><br><span class="line"><span class="attr"> name:</span> app</span><br><span class="line"><span class="attr"> spec:</span></span><br><span class="line"><span class="attr"> containers:</span></span><br><span class="line"><span class="attr"> - name:</span> app</span><br><span class="line"> <span class="comment"># image: nginx:1.18 OLD</span></span><br><span class="line"><span class="attr"> image:</span> nginx:<span class="number">1.19</span></span><br><span class="line"><span class="attr"> ports:</span></span><br><span class="line"><span class="attr"> - containerPort:</span> <span class="number">3000</span></span><br></pre></td></tr></table></figure></p>
<p>如果您有三个副本,并且一旦提交新的 YAML 资源 Kubernetes,则:</p>
<ul>
<li>用新的容器镜像创建一个 Pod。</li>
<li>销毁现有的 Pod。</li>
<li>等待 Pod 准备就绪。</li>
</ul>
<p>并重复上述步骤,直到所有 Pod 都迁移到较新的版本。</p>
<p>Kubernetes 仅在新的 Pod 准备好接收流量(换句话说,它通过 Readiness 检查)之后才重复每个周期。</p>
<p>Kubernetes 是否在等待 Pod 被删除之后再移到下一个 Pod?</p>
<p><strong>并不会!!!</strong></p>
<p>如果您有 10 个 Pod,并且 Pod 需要 2 秒钟的准备时间和 20 个关闭的时间,则会发生以下情况:</p>
<p>创建第一个 Pod,并终止前一个 Pod。</p>
<p>Kubernetes 创建一个新的 Pod 之后,需要 2 秒钟的准备时间。</p>
<p>同时,被终止的 Pod 会终止 20 秒</p>
<p>20 秒后,所有新的 Pod 均已启用(10 个 Pod ,在 2 秒后就绪),并且所有之前的 10 个Pod 都将终止(第一个 Terminated Pod 将要退出)。</p>
<p>总共,您在短时间内将 Pod 的数量增加了一倍(运行 10 次,终止 10 次)。<br><img src="/images/k8s/g_sd_49.png" alt="graceful shutdown"></p>
<p>与 “Readiness” 探针相比,宽限时间(graceful period)越长,您同时具有 “Running”(和 Terminating )的 Pod 越多。</p>
<p>不好吗?</p>
<p>不一定,因为您要小心不要断开连接。</p>
<h2 id="终止长时间运行的任务"><a href="#终止长时间运行的任务" class="headerlink" title="终止长时间运行的任务"></a>终止长时间运行的任务</h2><p><em>那长期工作呢?</em></p>
<p><em>如果您要对大型视频进行转码,是否有其他方法可以延迟停止 Pod?</em></p>
<p>假设您有一个包含三个副本的 Deployment。</p>
<p>每个副本都分配了一个视频进行转码,该任务可能需要几个小时才能完成。</p>
<p>当您触发滚动更新时,Pod 会在 30 秒内完成任务,然后将其杀死。</p>
<p>如何避免延迟关闭 Pod?</p>
<p>您可以将其增加 <code>terminationGracePeriodSeconds</code> 几个小时。</p>
<p><strong>但是,此时 Pod 的 endpoint不可达。</strong><br><img src="/images/k8s/g_sd_50.png" alt="graceful shutdown"></p>
<p>如果公开指标以监视 Pod,则检测工具将无法访问 Pod。</p>
<p>为什么?</p>
<p><strong>诸如 Prometheus 之类的工具依赖于 Endpoints 来在群集中探测 Pod</strong>。</p>
<p>但是,一旦删除 Pod,endpoint 删除就会在群集中通告,甚至传播到 Prometheus!</p>
<p><strong>您应该考虑为每个新版本创建一个新的 Deployment,而不是增加宽限时间(grace period)</strong>。</p>
<p>当您创建全新的 deployment 时,现有的 deployment 将保持不变。</p>
<p>长时间运行的作业可以照常继续处理视频。</p>
<p>完成后,您可以手动删除它们。</p>
<p>如果希望自动删除它们,则可能需要设置一个弹性伸缩,当它们用尽任务时,可以将部署扩展到零个副本。</p>
<p>这种 Pod 自动伸缩的一个示例是 Osiris,<a href="https://github.com/deislabs/osiris" target="_blank" rel="external">它是 Kubernetes 的通用,从零缩放的组件</a>。</p>
<p>该技术有时被称为 <strong>Rainbow 部署</strong>,并且在每次您必须使以前的 Pod 运行超过宽限期的时间时很有用。</p>
<p><em>另一个很好的例子是 WebSockets。</em></p>
<p>如果您正在向用户流式传输实时更新,则可能不希望在每次发布时都终止 WebSocket。</p>
<p>如果您每天频繁发布,则可能会导致实时 Feed 多次中断。</p>
<p><strong>为每个版本创建一个新的 Deployment 是一个不太明显但却是更好的选择。</strong></p>
<p>现有用户可以继续流更新,而最新的 Deployment 服务于新用户。</p>
<p>当用户断开与旧 Pod 的连接时,您可以逐渐减少副本并退出旧的 Deployment。</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>您应该注意 Pod 从集群中删除,因为它们的IP地址可能仍用于路由流量。</p>
<p>与其立即关闭 Pods,不如考虑在应用程序中等待更长的时间或设置一个 <code>preStop</code> 钩子。</p>
<p>仅在通告集群中的所有 endpoint 并将其从 kube-proxy,Ingress 控制器,CoreDNS 等中删除后,才应删除 Pod。</p>
<p>如果您的 Pod 运行诸如视频转码或使用 WebSocket 进行实时更新之类的长期任务,则应考虑使用 Rainbow 部署。</p>
<p>在 Rainbow 部署中,您为每个版本创建一个新的 Deployment,并在耗尽连接(或任务)后删除上一个版本。</p>
<p>您可以在长时间运行的任务完成后立即手动删除较旧的 Deployment。</p>
<p>或者,您可以自动将 Deployment 扩展到零副本,从而可以自动化该过程。</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<blockquote>
<p>原文:<a href="https://learnk8s.io/graceful-shutdown" target="_blank" rel="external">https://learnk8s.io/graceful-shutdown</a></p>
</blockquote>
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>在本文中,您将了解如何在Pod启动或关闭时防止连接异常,并将学习如何以优雅的方式关闭长时间运行的任务。<br><img src="/images/k8s/g_shutdown_1.png" alt="graceful shutdown"></p>
图文带你了解 Go 中的分配
http://team.jiunile.com//blog/2020/12/go-allocations.html
2020-12-21T14:00:00.000Z
2020-12-22T03:35:11.000Z
<h2 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h2><p>得益于了 Go 运行时高效的内置内存管理,我们通常能够在程序中优先考虑正确性和可维护性,而不需要过多考虑如何进行分配的细节。不过,有时我们可能会发现代码中的性能瓶颈,并希望进行更深入的研究。</p>
<p>任何使用 <code>-benchmem</code> 标志运行基准测试的人都会在输出中看到 <code>allocs/op</code> 的统计。在这篇文章中,我们将看看什么算作一个 alloc,以及我们可以做什么来影响这个数字。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">BenchmarkFunc-8 67836464 16.0 ns/op 8 B/op 1 allocs/op</span><br></pre></td></tr></table></figure></p>
<a id="more"></a>
<h2 id="我们熟悉和喜爱的栈和堆"><a href="#我们熟悉和喜爱的栈和堆" class="headerlink" title="我们熟悉和喜爱的栈和堆"></a>我们熟悉和喜爱的栈和堆</h2><p>要讨论 Go 中的 <code>allocs/op</code> 统计,我们将对 Go 程序中的两个内存区域感兴趣:栈和堆。</p>
<p>在许多流行的编程环境中,栈通常指的是线程的调用栈。调用栈是一个先进先出(LIFO)栈数据结构,它存储了线程执行函数时跟踪的参数、局部变量和其他数据。每一次函数调用都向栈增加(推)一个新的帧,每一次返回函数都会从栈中删除(弹出)。</p>
<p>我们必须能够在最近的栈帧被弹出时安全地释放它的内存。因此,我们不能在栈上存储任何以后需要在其他地方引用的东西。</p>
<p><img src="/images/go/allocs_1.png" alt="调用 println 后的调用栈视图"></p>
<p>由于线程是由操作系统管理的,所以线程栈的可用内存量通常是固定的,例如在许多 Linux 环境中默认为 8MB。这意味着我们还需要注意栈上最终有多少数据,特别是在嵌套较深的递归函数的情况下。如果上图中的栈指针通过了栈保护,程序就会因栈溢出错误而崩溃。</p>
<p>堆是内存中更复杂的区域,与同名的数据结构没有关系。我们可以按需使用堆来存储程序中需要的数据。在这里分配的内存不能在函数返回时简单地释放,需要仔细管理,以避免泄漏和碎片化。堆通常会比任何线程栈大许多倍,任何优化工作的大部分时间都将花费在研究堆的使用上。</p>
<h2 id="Go-栈和堆"><a href="#Go-栈和堆" class="headerlink" title="Go 栈和堆"></a>Go 栈和堆</h2><p>由操作系统管理的线程被 Go 运行时完全抽象出来,我们使用的是一个新的抽象:goroutines。goroutine 在概念上与线程非常相似,但它们存在于用户空间中。这意味着是运行时而不是操作系统来设置栈的行为规则。</p>
<p><img src="/images/go/allocs_2.png" alt="线程被抽离出来"></p>
<p>goroutine 栈并不是由操作系统设置的硬性限制,而是以少量的内存(目前为 2KB)开始。在执行每个函数调用之前,在函数序言中会执行检查,以验证不会发生栈溢出。在下面的图中,<code>convert()</code> 函数可以在当前栈大小的限制下执行(在 SP 不超额处理 <code>stackguard0</code> 的情况下)。</p>
<p><img src="/images/go/allocs_3.png" alt="goroutine 调用栈特写"></p>
<p>如果不是这样,运行时将在执行 <code>convert()</code> 之前将当前栈复制到一个更大的连续内存空间中。这意味着 Go 中的栈是动态大小的,只要有足够的内存可用,通常就可以保持增长。</p>
<p>Go 堆在概念上同样类似于上面描述的线程模型。所有的 goroutines 共享一个公共堆,任何不能存储在栈上的东西都将在那里结束。当对函数进行基准测试时发生堆分配时,我们将看到<code>allocs/ops</code> 属性增加 1。垃圾回收器的工作是稍后释放不再引用的堆变量。</p>
<p>关于Go中如何处理内存管理的详细解释,请参阅 <a href="https://medium.com/@ankur_anand/a-visual-guide-to-golang-memory-allocator-from-ground-up-e132258453ed" target="_blank" rel="external">从头开始的 Go 内存分配器的可视化指南</a>。</p>
<h2 id="我们如何知道一个变量何时被分配给堆?"><a href="#我们如何知道一个变量何时被分配给堆?" class="headerlink" title="我们如何知道一个变量何时被分配给堆?"></a>我们如何知道一个变量何时被分配给堆?</h2><p>这个问题答案在官方 FAQ 中。</p>
<blockquote>
<p>Go 编译器将为函数的栈帧中分配该函数的局部变量。但如果编译器不能证明该变量在函数返回后没有被引用,那么编译器必须在垃圾回收的堆上分配变量,以避免指针悬空错误。而且,如果局部变量非常大,那么将它存储在堆上而不是栈上可能更有意义。</p>
<p>如果某个变量的地址已被占用,那么该变量将成为堆上分配的候选变量。然而,一个基本的转义分析可以识别出一些情况,即这样的变量不会活过函数的返回,可以驻留在栈中。</p>
</blockquote>
<p>由于编译器的实现会随着时间的推移而改变,<strong>所以仅仅通过阅读 Go 代码,是无法知道哪些变量会被分配到堆中的</strong>。不过,可以在编译器的输出中查看上面提到的 escape 分析结果。这可以通过传递给 <code>go build</code> 的 <code>gcflags</code> 参数来实现。完整的选项列表可以通过 <code>go tool compile -help</code> 来查看。</p>
<p>对于转义分析结果,可以使用 <code>-m</code> 选项(<code>打印优化决策</code>)。让我们用一个简单的程序来测试一下,为函数 <code>main1</code> 和 <code>stackIt</code> 创建两个栈帧。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main1</span><span class="params">()</span></span> {</span><br><span class="line"> _ = stackIt()</span><br><span class="line">}</span><br><span class="line"><span class="comment">//go:noinline</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stackIt</span><span class="params">()</span> <span class="title">int</span></span> {</span><br><span class="line"> y := <span class="number">2</span></span><br><span class="line"> <span class="keyword">return</span> y * <span class="number">2</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>因为如果编译器删除了函数调用,我们就无法讨论栈行为,所以在编译代码时使用 <code>noinline</code> <a href="https://dave.cheney.net/2018/01/08/gos-hidden-pragmas" target="_blank" rel="external">pragma</a> 来防止内联。让我们看一下编译器对其优化决策说些什么。<code>-l</code> 选项用于省略内联决策。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ go build -gcflags <span class="string">'-m -l'</span></span><br><span class="line"><span class="comment"># github.com/Jimeux/go-samples/allocations</span></span><br></pre></td></tr></table></figure></p>
<p>在这里,我们看到,没有做出任何关于逃跑分析的决定。换句话说,变量 <code>y</code> 保留在栈中,并没有触发任何堆分配。我们可以用基准测试来验证这一点。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -bench . -benchmem</span><br><span class="line">BenchmarkStackIt-8 680439016 1.52 ns/op 0 B/op 0 allocs/op</span><br></pre></td></tr></table></figure></p>
<p>正如预期的那样,<code>allocs/op</code> 统计值为 <code>0</code>。从这个结果中我们可以得到的一个重要观察是,<strong>复制变量可以让我们将它们保留在栈中</strong>,避免分配到堆中。让我们通过修改程序来验证这一点,以避免使用指针进行复制。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main2</span><span class="params">()</span></span> {</span><br><span class="line"> _ = stackIt2()</span><br><span class="line">}</span><br><span class="line"><span class="comment">//go:noinline</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stackIt2</span><span class="params">()</span> *<span class="title">int</span></span> {</span><br><span class="line"> y := <span class="number">2</span></span><br><span class="line"> res := y * <span class="number">2</span></span><br><span class="line"> <span class="keyword">return</span> &res</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>让我们看以下编译器的输出。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">go build -gcflags <span class="string">'-m -l'</span></span><br><span class="line"><span class="comment"># github.com/Jimeux/go-samples/allocations</span></span><br><span class="line">./main.go:10:2: moved to heap: res</span><br></pre></td></tr></table></figure></p>
<p>编译器告诉我们,它把指针 <code>res</code> 移到了堆上,从而触发了堆分配,这在下面的基准中得到了验证。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -bench . -benchmem</span><br><span class="line">BenchmarkStackIt2-8 70922517 16.0 ns/op 8 B/op 1 allocs/op</span><br></pre></td></tr></table></figure></p>
<p>那么这是否意味着指针一定会创建分配?让我们再次修改程序,这次将指针传到栈下。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main3</span><span class="params">()</span></span> {</span><br><span class="line"> y := <span class="number">2</span></span><br><span class="line"> _ = stackIt3(&y) <span class="comment">// pass y down the stack as a pointer</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">//go:noinline</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">stackIt3</span><span class="params">(y *<span class="keyword">int</span>)</span> <span class="title">int</span></span> {</span><br><span class="line"> res := *y * <span class="number">2</span></span><br><span class="line"> <span class="keyword">return</span> res</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>然而运行基准测试显示没有任何东西被分配到堆中。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -bench . -benchmem</span><br><span class="line">BenchmarkStackIt3-8 705347884 1.62 ns/op 0 B/op 0 allocs/op</span><br></pre></td></tr></table></figure></p>
<p>编译器的输出明确地告诉我们这一点。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ go build -gcflags <span class="string">'-m -l'</span></span><br><span class="line"><span class="comment"># github.com/Jimeux/go-samples/allocations</span></span><br><span class="line">./main.go:10:14: y does not escape</span><br></pre></td></tr></table></figure></p>
<p>为什么会出现这种看似不一致的情况呢?<code>stackIt2</code> 将 <code>y</code> 的地址从栈上传递到 <code>main</code>,在 <code>main</code> 中,<code>y</code> 将在 <code>stackIt2</code> 的栈帧被释放后被引用。因此,编译器能够判断 <code>y</code> 必须被移到堆上才能保持活力。如果它不这样做,当我们试图引用 <code>y</code> 时,就会在 <code>main</code> 中得到一个 <code>nil</code> 指针。</p>
<p>而 <code>stackIt3</code> 则是将 <code>y</code> 传到栈下,而且 <code>y</code> 在 <code>main3</code> 之外的任何地方都不会被引用。因此,编译器能够判断 <code>y</code> 可以单独存在于栈中,而不需要分配到堆中。在任何情况下,我们都无法通过引用 <code>y</code> 来产生一个 <code>nil</code> 指针。</p>
<p><strong>从这里我们可以推断出一个通用规则,即在栈上共享指针会导致分配,而共享栈下的指针则不会</strong>。但是,这并不能保证,所以您仍然需要使用 <code>gcflags</code> 或基准来验证。我们可以肯定的是,任何试图减少 <code>allocs/op</code> 的尝试都将涉及到寻找任性的指针。</p>
<h2 id="我们为什么要关心堆分配?"><a href="#我们为什么要关心堆分配?" class="headerlink" title="我们为什么要关心堆分配?"></a>我们为什么要关心堆分配?</h2><p>我们已经了解了一些关于 <code>allocs/op</code> 中的 <code>alloc</code> 的含义,以及如何验证是否触发了对堆的分配,但是为什么我们要关心这个统计是否是非零呢?我们已经做的基准测试可以回答这个问题。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">BenchmarkStackIt-8 680439016 1.52 ns/op 0 B/op 0 allocs/op</span><br><span class="line">BenchmarkStackIt2-8 70922517 16.0 ns/op 8 B/op 1 allocs/op</span><br><span class="line">BenchmarkStackIt3-8 705347884 1.62 ns/op 0 B/op 0 allocs/op</span><br></pre></td></tr></table></figure></p>
<p>尽管所涉及的变量对内存的需求几乎相等,但相对而言,<code>BenchmarkStackIt2</code> 对 CPU 的开销还是很明显的。我们可以通过生成 <code>stackIt</code> 和 <code>stackIt2</code> 生成的 CPU 曲线的火焰图来了解更多的情况。<br><img src="/images/go/allocs_4.png" alt="stackIt CPU profile"></p>
<p><img src="/images/go/allocs_5.png" alt="stackIt2 CPU profile"></p>
<p><code>stackIt</code> 有一个不起眼的配置文件,它可以预见地从调用栈运行到 <code>stackIt</code> 函数本身。另一方面,<code>stackIt2</code> 大量使用了大量的运行时函数,这些函数消耗了许多额外的 CPU 周期。这说明了分配到堆所涉及的复杂性,并初步了解了每个操作额外的 10 纳秒左右的去向。</p>
<h2 id="那在现实世界中呢?"><a href="#那在现实世界中呢?" class="headerlink" title="那在现实世界中呢?"></a>那在现实世界中呢?</h2><p>如果没有生产条件,性能的许多方面不会变得明显。你的单个功能可能在微基准测试中高效运行,但当它为成千上万的并发用户服务时,它会对你的应用程序有什么影响呢?</p>
<p>我们不会在这篇文章中重新创建一个完整的应用程序,但我们将使用<a href="https://golang.org/cmd/trace/" target="_blank" rel="external">跟踪工具</a>来看看一些更详细的性能诊断。让我们首先定义一个(有点)大的结构体,它有 9 个字段。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> BigStruct <span class="keyword">struct</span> {</span><br><span class="line"> A, B, C <span class="keyword">int</span></span><br><span class="line"> D, E, F <span class="keyword">string</span></span><br><span class="line"> G, H, I <span class="keyword">bool</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>现在我们来定义两个函数:<code>CreateCopy</code>,它在栈帧之间复制 <code>BigStruct</code> 实例;<code>CreatePointer</code>,它在栈上共享 <code>BigStruct</code> 指针,避免复制,但会产生堆分配。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">//go:noinline</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">CreateCopy</span><span class="params">()</span> <span class="title">BigStruct</span></span> {</span><br><span class="line"> <span class="keyword">return</span> BigStruct{</span><br><span class="line"> A: <span class="number">123</span>, B: <span class="number">456</span>, C: <span class="number">789</span>,</span><br><span class="line"> D: <span class="string">"ABC"</span>, E: <span class="string">"DEF"</span>, F: <span class="string">"HIJ"</span>,</span><br><span class="line"> G: <span class="literal">true</span>, H: <span class="literal">true</span>, I: <span class="literal">true</span>,</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"><span class="comment">//go:noinline</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">CreatePointer</span><span class="params">()</span> *<span class="title">BigStruct</span></span> {</span><br><span class="line"> <span class="keyword">return</span> &BigStruct{</span><br><span class="line"> A: <span class="number">123</span>, B: <span class="number">456</span>, C: <span class="number">789</span>,</span><br><span class="line"> D: <span class="string">"ABC"</span>, E: <span class="string">"DEF"</span>, F: <span class="string">"HIJ"</span>,</span><br><span class="line"> G: <span class="literal">true</span>, H: <span class="literal">true</span>, I: <span class="literal">true</span>,</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>我们可以用目前使用的技术来验证上面的解释。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ go build -gcflags <span class="string">'-m -l'</span></span><br><span class="line">./main.go:67:9: &BigStruct literal escapes to heap</span><br><span class="line"></span><br><span class="line">$ go <span class="built_in">test</span> -bench . -benchmem</span><br><span class="line">BenchmarkCopyIt-8 211907048 5.20 ns/op 0 B/op 0 allocs/op</span><br><span class="line">BenchmarkPointerIt-8 20393278 52.6 ns/op 80 B/op 1 allocs/op</span><br></pre></td></tr></table></figure></p>
<p>以下是我们将用于跟踪工具的测试。它们分别用各自的 <code>Create</code> 函数创建 20,000,000 个 <code>BigStruct</code> 实例。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> creations = <span class="number">20</span>_000_000</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">TestCopyIt</span><span class="params">(t *testing.T)</span></span> {</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < creations; i++ {</span><br><span class="line"> _ = CreateCopy()</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">TestPointerIt</span><span class="params">(t *testing.T)</span></span> {</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < creations; i++ {</span><br><span class="line"> _ = CreatePointer()</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>接下来,我们将把 <code>CreateCopy</code> 的跟踪输出保存到文件 <code>copy_trace.out</code> 中。并在浏览器中用跟踪工具打开它。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -run TestCopyIt -trace=copy_trace.out</span><br><span class="line">PASS</span><br><span class="line">ok github.com/Jimeux/go-samples/allocations 0.281s</span><br><span class="line"></span><br><span class="line">$ go tool trace copy_trace.out</span><br><span class="line">Parsing trace...</span><br><span class="line">Splitting trace...</span><br><span class="line">Opening browser. Trace viewer is listening on http://127.0.0.1:57530</span><br></pre></td></tr></table></figure></p>
<p>从菜单中选择 <code>View trace</code>,我们看到了下面的画面,它几乎和我们的 <code>stackIt</code> 功能的火焰图一样不引人注目。8 个潜在的逻辑核(Procs)中只有 2 个被使用,<code>goroutine G19</code> 几乎花费整个时间运行我们的测试循环–这正是我们想要的。<br><img src="/images/go/allocs_6.png" alt="Trace for 20,000,000 CreateCopy calls"></p>
<p>让我们为 <code>CreatePointer</code> 代码生成跟踪数据。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -run TestPointerIt -trace=pointer_trace.out</span><br><span class="line">PASS</span><br><span class="line">ok github.com/Jimeux/go-samples/allocations 2.224s</span><br><span class="line"></span><br><span class="line">go tool trace pointer_trace.out</span><br><span class="line">Parsing trace...</span><br><span class="line">Splitting trace...</span><br><span class="line">Opening browser. Trace viewer is listening on http://127.0.0.1:57784</span><br></pre></td></tr></table></figure></p>
<p>您可能已经注意到,与 <code>CreateCopy</code> 的 0.281 秒相比,测试花费了 2.224 秒,选择 <code>View trace</code> 这次显示的内容更加丰富多彩,更加繁忙。所有的逻辑内核都被利用了,堆操作、线程和 goroutines 似乎比上次多了很多。<br><img src="/images/go/allocs_7.png" alt="Trace for 20,000,000 CreatePointer calls"></p>
<p>如果我们把跟踪的时间放大到一毫秒左右的跨度,我们会看到很多 goroutine 在执行与垃圾回收相关的操作。前面引用的 FAQ 中使用了“垃圾回收堆”这个词,因为垃圾回收器的工作就是清理堆上任何不再被引用的东西。<br><img src="/images/go/allocs_8.png" alt="在CreatePointer跟踪中的GC活动特写"></p>
<p>尽管 Go 的垃圾回收器效率越来越高,但这个过程并不是免费的。我们可以从上面的跟踪输出中直观地验证,测试代码有时完全停止了。对于 <code>CreateCopy</code> 来说,情况并非如此,因为我们所有的 <code>BigStruct</code> 实例仍然在栈上,GC 几乎没有什么事情可做。</p>
<p>比较两组跟踪数据中的 goroutine 分析可以更深入地了解这一点。<code>CreatePointer</code>(底部)花费了超过 15% 的执行时间来清扫或暂停(GC)和调度 goroutines。<br><img src="/images/go/allocs_9.png" alt="CreateCopy 的顶层 goroutine 分析"></p>
<p><img src="/images/go/allocs_10.png" alt="CreatePointer 的顶层 goroutine 分析"></p>
<p>看看跟踪数据中其他地方的一些统计数据,可以进一步说明堆分配的成本,生成的 goroutine数量有明显的差异,<code>CreatePointer</code> 测试有近 400 个 STW(停止世界)事件。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">+------------+------+---------+</span><br><span class="line">| | Copy | Pointer |</span><br><span class="line">+------------+------+---------+</span><br><span class="line">| Goroutines | 41 | 406965 |</span><br><span class="line">| Heap | 10 | 197549 |</span><br><span class="line">| Threads | 15 | 12943 |</span><br><span class="line">| bgsweep | 0 | 193094 |</span><br><span class="line">| STW | 0 | 397 |</span><br><span class="line">+------------+------+---------+</span><br></pre></td></tr></table></figure></p>
<p>但请记住,尽管本节的标题是这样的,但 CreateCopy 测试的条件在一个典型的程序中是非常不现实的。GC 使用一致数量的 CPU 是很正常的,指针是任何真实程序的一个特征。然而,这和前面的火焰图一起给了我们一些启示,为什么我们可能要跟踪 allocs/op 统计,并尽可能避免不必要的堆分配。</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>希望这篇文章能让大家了解到 Go 程序中栈和堆之间的区别、<code>allocs/op</code> 统计的意义,以及我们可以调研内存使用情况的一些方法。</p>
<p>代码的正确性和可维护性通常比减少指针使用和规避 GC 活动的技巧更重要。到目前为止,每个人都知道关于过早优化的那条线,在 Go 中编写代码也不例外。</p>
<p>然而,如果我们确实有严格的性能要求或在其他方面确定了程序中的瓶颈,这里介绍的概念和工具有望成为进行必要优化的有用起点。</p>
<p>如果你想玩玩这篇文章中的简单代码示例,请查看 <a href="https://github.com/Jimeux/go-samples/tree/master/allocations" target="_blank" rel="external">GitHub</a> 上的源代码和 README。</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<blockquote>
<p>译自:<a href="https://medium.com/eureka-engineering/understanding-allocations-in-go-stack-heap-memory-9a2631b5035d" target="_blank" rel="external">https://medium.com/eureka-engineering/understanding-allocations-in-go-stack-heap-memory-9a2631b5035d</a></p>
</blockquote>
<h2 id="介绍"><a href="#介绍" class="headerlink" title="介绍"></a>介绍</h2><p>得益于了 Go 运行时高效的内置内存管理,我们通常能够在程序中优先考虑正确性和可维护性,而不需要过多考虑如何进行分配的细节。不过,有时我们可能会发现代码中的性能瓶颈,并希望进行更深入的研究。</p>
<p>任何使用 <code>-benchmem</code> 标志运行基准测试的人都会在输出中看到 <code>allocs/op</code> 的统计。在这篇文章中,我们将看看什么算作一个 alloc,以及我们可以做什么来影响这个数字。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">BenchmarkFunc-8 67836464 16.0 ns/op 8 B/op 1 allocs/op</span><br></pre></td></tr></table></figure></p>
如何在 Go 中编写无 Bug 的 Goroutines?
http://team.jiunile.com//blog/2020/12/go-nobug-gorotuine.html
2020-12-15T14:00:00.000Z
2020-12-15T13:27:02.000Z
<h2 id="GO-并发"><a href="#GO-并发" class="headerlink" title="GO 并发"></a>GO 并发</h2><p>Go 以其并发性著称,深受人们喜爱。go 运行时管理轻量级线程,称为 goroutines。goroutine 的编写非常快速简单。</p>
<p>你只需在你想异步执行的函数前输入 <code>go</code>,程序就会在另一个线程中执行。</p>
<p><strong>听起来很简单?</strong></p>
<p>goroutines 是 Go 编写异步代码的方式。</p>
<p>重要的是要了解 goroutine 和并发的工作原理。Go 提供了管理 goroutine 的方法,使它们在复杂的程序中更容易管理和预测。</p>
<blockquote>
<p>因为 goroutine 非常容易使用,所以它们很容易被滥用。</p>
</blockquote>
<a id="more"></a>
<h2 id="1-在异步例程中不要对执行顺序进行假设。"><a href="#1-在异步例程中不要对执行顺序进行假设。" class="headerlink" title="1 在异步例程中不要对执行顺序进行假设。"></a>1 在异步例程中不要对执行顺序进行假设。</h2><p>在 Go 中调度并发任务时,要记住异步任务的不可预知性。</p>
<p>可以将异步与同步计算融合在一起,但只要同步任务不对异步任务做任何假设即可。</p>
<p>对于初学者来说,一个常见的错误是创建一个 goroutine,然后根据该 goroutine 的结果继续执行同步任务。例如,如果该 goroutine 要向其作用域外的变量写入,然后在同步任务中使用该变量。</p>
<h3 id="假设执行顺序"><a href="#假设执行顺序" class="headerlink" title="假设执行顺序"></a>假设执行顺序</h3><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"time"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> numbers []<span class="keyword">int</span> <span class="comment">// nil</span></span><br><span class="line"></span><br><span class="line"> <span class="comment">// start a goroutine to initialise array</span></span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span> <span class="params">()</span></span> {</span><br><span class="line"> numbers = <span class="built_in">make</span>([]<span class="keyword">int</span>, <span class="number">2</span>)</span><br><span class="line"> }()</span><br><span class="line"> </span><br><span class="line"> <span class="comment">// do something synchronous</span></span><br><span class="line"> <span class="keyword">if</span> numbers == <span class="literal">nil</span> {</span><br><span class="line"> time.Sleep(time.Second)</span><br><span class="line"> }</span><br><span class="line"> numbers[<span class="number">0</span>] = <span class="number">1</span> <span class="comment">// will sometimes panic here</span></span><br><span class="line"> fmt.Println(numbers[<span class="number">0</span>])</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p><strong>这种模式会导致不可预知的行为</strong>。它引入的代码导致了我们无法控制的因素;这些因素与 go 运行时有关,更具体地说,就是它如何管理 goroutines。</p>
<blockquote>
<p>编写这样的代码意味着假定 goroutine 将在需要结果之前完成它的任务。</p>
</blockquote>
<p><strong>首先</strong>,在没有某种管理技术(我们将讨论)的情况下,交叉异步和同步代码的成功将取决于 CPU 的可用性。</p>
<p>这意味着如果有 CPU 密集型的进程与 goroutines 同时运行,那么执行的时间将会有所不同。</p>
<p><strong>其次</strong>,不同的编译器将以不同的方式调度 goroutines。因此,安全的做法是不要认为 goroutine 会在同步任务期间完成。</p>
<p><strong>如何确保 goroutine 已经完成?</strong></p>
<blockquote>
<p>使用 channel</p>
</blockquote>
<p><strong>在异步任务完成时使用 channel 来通知</strong></p>
<p>channel 应该用于接收来自异步任务(如 goroutines)的值。</p>
<p>如果你想阻止进一步的执行,直到最终从 channel 读取一个值来释放它,可以使用缓冲通道。</p>
<p>如果你想要 1 进 1 出的行为,那么使用非缓冲通道。</p>
<p>在本例中,使用 channel,我们可以确保主任务等待直到异步任务完成。当 goroutine 完成它的工作时,它将通过 <code>done channel</code> 发送一个值,该值将在对 <code>numbers</code> 数组进行操作之前被读取。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"time"</span></span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> numbers []<span class="keyword">int</span> <span class="comment">// nil</span></span><br><span class="line"> done := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">struct</span>{})</span><br><span class="line"> <span class="comment">// start a goroutine to initialise array</span></span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span> <span class="params">()</span></span> {</span><br><span class="line"> numbers = <span class="built_in">make</span>([]<span class="keyword">int</span>, <span class="number">2</span>)</span><br><span class="line"> done <- <span class="keyword">struct</span>{}{}</span><br><span class="line"> }()</span><br><span class="line"> </span><br><span class="line"> <span class="comment">// do something synchronous</span></span><br><span class="line"> <-done <span class="comment">// read done from channel</span></span><br><span class="line"> numbers[<span class="number">0</span>] = <span class="number">1</span> <span class="comment">// will not panic anymore</span></span><br><span class="line"> fmt.Println(numbers[<span class="number">0</span>]) <span class="comment">// 1</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>尽管这是一个人为的示例,但你可以看到它在什么地方会很有用:当主线程与 goroutine 并行处理复杂工作时。这两个任务可以同时完成,而不可能出现 <code>panic</code>。</p>
<h2 id="2-避免跨并发线程访问可变数据"><a href="#2-避免跨并发线程访问可变数据" class="headerlink" title="2 避免跨并发线程访问可变数据"></a>2 避免跨并发线程访问可变数据</h2><p>跨多个 goroutine 访问可变数据是将数据竞争引入程序的“好方法”。</p>
<p>数据竞争是指两个或多个线程(或这里的goroutine)<strong>并发访问同一内存位置</strong>。</p>
<p>这意味着跨线程访问相同的变量可能会产生不可预测的值。如果两个进程同时访问同一个变量,有两种可能性:</p>
<ul>
<li>两个线程的值是相同的(<strong>不正确</strong>)。</li>
<li>对于较慢/较晚的线程,该值是不同的。(<strong>正确</strong>)</li>
</ul>
<p>如果较慢/较晚的线程读取了一个已被较快/较早的线程修改过的更新值,那么它将对更新后的值进行操作。<strong>这是预期的行为</strong>。</p>
<p>否则,就像在数据竞争中看到的那样,两个线程将产生相同的值,因为它们都将对未更改的值进行操作。</p>
<p><strong> 1000 种可能的数据竞争</strong></p>
<p>在这个例子中,我们使用 <code>sync.WaitGroup</code> 来保持程序运行,直到所有的 goroutine 完成,但我们并没有控制对每个 goroutine 内变量的访问。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"sync"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> a := <span class="number">0</span> <span class="comment">// data race</span></span><br><span class="line"> <span class="keyword">var</span> wg sync.WaitGroup</span><br><span class="line"> wg.Add(<span class="number">1000</span>)</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">1000</span>; i++ {</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> a += <span class="number">1</span></span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> wg.Wait()</span><br><span class="line"> fmt.Println(a) <span class="comment">// could theoretical be any number 0-1000 (most likely above 900)</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这段代码可以打印 0-1000 之间的任何数字,具体取决于发生的数据竞争数量。</p>
<p>这段代码的工作原理是,两个线程将对同一个变量各执行 2 次操作,总共有 2 次读 + 2 次写。</p>
<blockquote>
<p>在两个线程都会产生相同的值的情况下,在对变量进行任何写入之前,两个(2)读都必须发生。</p>
</blockquote>
<p><strong>使用互斥锁在 goroutines 之间共享内存</strong></p>
<p>为了防止 goroutines 中的数据竞争,我们需要同步对共享内存的访问。我们可以使用互斥来实现这一点。互斥锁将确保我们不会在同一时间读取或写入相同的值。</p>
<p>它本质上是<strong>暂时锁定对一个变量的访问</strong>。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"fmt"</span></span><br><span class="line"> <span class="string">"sync"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> a := <span class="number">0</span></span><br><span class="line"> <span class="keyword">var</span> wg sync.WaitGroup</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> mu sync.Mutex <span class="comment">// guards access</span></span><br><span class="line"></span><br><span class="line"> wg.Add(<span class="number">1000</span>)</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">1000</span>; i++ {</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> mu.Lock()</span><br><span class="line"> <span class="keyword">defer</span> mu.Unlock()</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> a += <span class="number">1</span></span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> wg.Wait()</span><br><span class="line"> fmt.Println(a) <span class="comment">// will always be 1000</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>就这么简单。</p>
<p>这段代码总是打印 1000,因为对同一个变量的每个后续操作都会对更新后的值进行操作。</p>
<h2 id="3-不要写应该同步的异步任务"><a href="#3-不要写应该同步的异步任务" class="headerlink" title="3 不要写应该同步的异步任务"></a>3 不要写应该同步的异步任务</h2><p>Goroutines 通常被认为是后台任务。它们被视为可以与主程序同时运行的小任务,通过 goroutine 将其委托给另一个线程。</p>
<p>当学习 Go 时,你往往会想到使用 goroutine 来尽量减少阻塞操作,或者让我们的程序性能更强。</p>
<p>但由于对 goroutine 的看法如此简单,很容易养成 “以防万一” 的习惯,把所有东西都做成 goroutine。</p>
<p><strong>如果某些任务本质上是同步的,但你却异步地使用了它们,这就会造成问题</strong>。</p>
<p><strong>并非所有的任务都应该是一个 goroutine</strong>。</p>
<p>有些任务需要秩序。在许多进程中,下一个任务取决于前一个任务的结果。这些顺序性的任务会让你的程序出错,势必需要让这些区域更加同步。</p>
<p>所以有些情况下,你还不如直接忘掉goroutine,一开始就保持同步。</p>
<p><strong>用无限循环浪费 CPU</strong></p>
<p>在这个精心设计的示例中,我们有一个程序,它将所有内容委托给 goroutines,并使用 for 循环来保持程序运行。</p>
<p>这是一个如何不控制 Go 程序流程的例子。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">go</span> doSomething()</span><br><span class="line"> <span class="keyword">go</span> doSomethingElse()</span><br><span class="line"> </span><br><span class="line"> <span class="comment">// execute everything as a goroutine</span></span><br><span class="line"> </span><br><span class="line"> <span class="keyword">for</span> { <span class="comment">// this keeps the program running</span></span><br><span class="line"> </span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>最好保持简单。你可以通过把你的程序看作是主线程加上附加线程的方式来防止这种类型的不良做法。你可以让主线程以同步的方式运行,但如果需要,可以通过 goroutines 将任务委托给另一个线程。</p>
<p>有更好的方法可以控制程序的流程,<strong>比如通过 <a href="https://gobyexample.com/waitgroups" target="_blank" rel="external">WaitGroups</a> 或 <a href="https://gobyexample.com/channels" target="_blank" rel="external">Channels</a></strong>。</p>
<p><strong>使用 WaitGroup 的控制流程</strong></p>
<p>与其浪费宝贵的 CPU 资源,不如使用 WaitGroup 向运行时表明,在程序退出之前,你正在等待 n 个任务的完成。这样就不会让 CPU 一直在无限循环中旋转。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">doSomething</span><span class="params">(wg *sync.WaitGroup)</span></span> {</span><br><span class="line"> <span class="comment">// do something here</span></span><br><span class="line"> fmt.Println(<span class="string">"Done"</span>)</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> wg sync.WaitGroup</span><br><span class="line"> wg.Add(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">defer</span> wg.Wait()</span><br><span class="line"> <span class="keyword">go</span> doSomething(&wg)</span><br><span class="line"> <span class="keyword">go</span> doSomethingElseSync()</span><br><span class="line"> </span><br><span class="line"> <span class="comment">// program will wait until doSomething & doSomethingElseSync is complete</span></span><br><span class="line"></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>首先,您需要将等待完成的任务数量作为参数提供给 <code>wg.Add()</code> 函数。</p>
<p>放置 <code>wg.Wait()</code> 很重要。这是程序中执行将暂停的地方,等待所有任务完成。</p>
<p>一旦任务完成,您可以使用 <code>wg.Done()</code> 让程序知道。</p>
<h2 id="4-不要让-goroutines-挂起"><a href="#4-不要让-goroutines-挂起" class="headerlink" title="4 不要让 goroutines 挂起"></a>4 不要让 goroutines 挂起</h2><p>确保处理不再使用的 goroutines。持续运行的 Goroutines 将会阻塞并浪费宝贵的 CPU 资源。</p>
<p>如果 goroutine 试图将值发送到没有任何读取并等待接收值的 channel,就会发生这种情况。这就意味着这条 channel 将永远卡在那里。</p>
<p>9 个挂起的 goroutine</p>
<p>在这个例子中,channel 只被读取一次。这意味着 9 个 goroutines 在等待通过 channel 发送一个值。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">sendToChan</span><span class="params">()</span> <span class="title">int</span></span> {</span><br><span class="line"> channel := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>)</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> i := i</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> channel <- i <span class="comment">// 9 hanging goroutines</span></span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <-channel</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>为了避免这种情况,请处理不再需要的 goroutines 来释放 CPU。</p>
<p><strong>使通道缓冲</strong></p>
<p>使用缓冲通道意味着您正在为通道提供空间来存储附加值。</p>
<p>对于当前的示例,这意味着所有的 goroutines 都将成功执行,不会阻塞。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">sendToChan</span><span class="params">()</span> <span class="title">int</span></span> {</span><br><span class="line"> channel := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>, <span class="number">9</span>)</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> i := i</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> channel <- i <span class="comment">// all goroutines executed successfully</span></span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <-channel</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p><strong>不要在不知道什么时候停止的情况下开始一个 goroutine。</strong></p>
<p>在不知道何时停止的情况下启动一个 goroutine 会导致以下行为,即 goroutine 被阻塞或浪费 CPU 资源。</p>
<p>您应该总是知道什么时候 goroutine 将停止,什么时候不再需要它。</p>
<p><strong>您可以通过 <code>select</code> 语句和 <code>channel</code> 来实现这一点</strong><br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">done := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">bool</span>)</span><br><span class="line"><span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> <-done:</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}()</span><br><span class="line">done <- <span class="literal">true</span></span><br></pre></td></tr></table></figure></p>
<p>这本质上是一个带有退出条件的异步 for-loop。</p>
<p>重要的逻辑将在默认条件下编写。</p>
<blockquote>
<p>当值被发送到 <code>done</code> 通道时,循环将停止,正如 <code>done <- true</code> 所示。这意味着 channel 读取 <code><-done</code> 成功并返回。</p>
<p>译自:<a href="https://itnext.io/how-to-write-bug-free-goroutines-in-go-golang-59042b1b63fb" target="_blank" rel="external">https://itnext.io/how-to-write-bug-free-goroutines-in-go-golang-59042b1b63fb</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="GO-并发"><a href="#GO-并发" class="headerlink" title="GO 并发"></a>GO 并发</h2><p>Go 以其并发性著称,深受人们喜爱。go 运行时管理轻量级线程,称为 goroutines。goroutine 的编写非常快速简单。</p>
<p>你只需在你想异步执行的函数前输入 <code>go</code>,程序就会在另一个线程中执行。</p>
<p><strong>听起来很简单?</strong></p>
<p>goroutines 是 Go 编写异步代码的方式。</p>
<p>重要的是要了解 goroutine 和并发的工作原理。Go 提供了管理 goroutine 的方法,使它们在复杂的程序中更容易管理和预测。</p>
<blockquote>
<p>因为 goroutine 非常容易使用,所以它们很容易被滥用。</p>
</blockquote>
利用 eBPF 支撑大规模 K8s Service
http://team.jiunile.com//blog/2020/12/k8s-cilium-service-2019.html
2020-12-02T14:00:00.000Z
2020-12-03T02:31:35.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>本文翻译自 2019 年 Daniel Borkmann 和 Martynas Pumputis 在 Linux Plumbers Conference 的一篇分享: <a href="https://linuxplumbersconf.org/event/4/contributions/458/" target="_blank" rel="external">Making the Kubernetes Service Abstraction Scale using eBPF</a> 。 翻译时对大家耳熟能详或已显陈旧的内容(K8s 介绍、Cilium 1.6 之前的版本对 Service 实现等)略有删减,如有需要请查阅原 PDF。</p>
<p>实际上,一年之后 Daniel 和 Martynas 又在 LPC 做了一次分享,内容是本文的延续:<a href="http://team.jiunile.com/blog/2020/11/k8s-cilium-service.html">Cilium:基于 BPF/XDP 实现 K8s Service 负载均衡</a></p>
<p><strong>K8s 当前重度依赖 iptables 来实现 Service 的抽象</strong>。对于每个 Service 及其 backend pods,在 K8s 里会生成很多 iptables 规则。<strong>例如 5K 个 Service 时,iptables 规则将达到 25K 条</strong>,导致的后果:</p>
<ul>
<li><strong>较高、并且不可预测的转发延迟</strong>(packet latency),因为每个包都要遍历这些规则 ,直到匹配到某条规则;</li>
<li><strong>更新规则的操作非常慢</strong>:无法单独更新某条 iptables 规则,只能将全部规则读出来 ,更新整个集合,再将新的规则集合下发到宿主机。在动态环境中这一问题尤其明显,因为每 小时可能都有几千次的 backend pods 创建和销毁。</li>
<li><strong>可靠性问题</strong>:iptables 依赖 Netfilter 和系统的连接跟踪模块(conntrack),在 大流量场景下会出现一些竞争问题(race conditions);<strong>UDP 场景尤其明显</strong>,会导 致丢包、应用的负载升高等问题。</li>
</ul>
<p>本文将介绍如何基于 Cilium/BPF 来解决这些问题,实现 K8s Service 的大规模扩展。<br><a id="more"></a></p>
<h2 id="1-K8s-Service-类型及默认基于-kube-proxy-的实现"><a href="#1-K8s-Service-类型及默认基于-kube-proxy-的实现" class="headerlink" title="1 K8s Service 类型及默认基于 kube-proxy 的实现"></a>1 K8s Service 类型及默认基于 kube-proxy 的实现</h2><p>K8s 提供了 Service 抽象,可以将多个 backend pods 组织为一个<strong>逻辑单元</strong>(logical unit)。K8s 会为这个逻辑单元分配 <strong>虚拟 IP 地址</strong>(VIP),客户端通过该 VIP 就 能访问到这些 pods 提供的服务。</p>
<p>下图是一个具体的例子,<br><img src="/images/k8s/cim_k8s-service.png" alt="k8s-service"></p>
<ol>
<li>右边的 yaml 定义了一个名为 <code>nginx</code> 的 Service,它在 TCP 80 端口提供服务;<ul>
<li>创建:<code>kubectl -f nginx-svc.yaml</code></li>
</ul>
</li>
<li>K8s 会给每个 Service 分配一个虚拟 IP,这里给 <code>nginx</code> 分的是 <code>3.3.3.3</code>;<ul>
<li>查看:<code>kubectl get service nginx</code></li>
</ul>
</li>
<li>左边是 <code>nginx</code> Service 的两个 backend pods(在 K8s 对应两个 endpoint),这里 位于同一台节点,每个 Pod 有独立的 IP 地址;<ul>
<li>查看:<code>kubectl get endpoints nginx</code></li>
</ul>
</li>
</ol>
<p>上面看到的是所谓的 <code>ClusterIP</code> 类型的 Service。实际上,<strong>在 K8s 里有几种不同类型 的 Service</strong>:</p>
<ul>
<li>ClusterIP</li>
<li>NodePort</li>
<li>LoadBalancer</li>
<li>ExternalName</li>
</ul>
<p>本文将主要关注前两种类型。</p>
<p><strong>K8s 里实现 Service 的组件是 kube-proxy</strong>,实现的主要功能就是<strong>将访问 VIP 的请 求转发(及负载均衡)到相应的后端 pods</strong>。前面提到的那些 iptables 规则就是它创建 和管理的。</p>
<p>另外,kube-proxy 是 K8s 的可选组件,如果不需要 Service 功能,可以不启用它。</p>
<h3 id="ClusterIP-Service"><a href="#ClusterIP-Service" class="headerlink" title="ClusterIP Service"></a>ClusterIP Service</h3><p>这是 <strong>K8s 的默认 Service 类型,使得宿主机或 pod 可以通过 VIP 访问一个 Service</strong>。</p>
<ul>
<li>Virtual IP to any endpoint (pod)</li>
<li>Only in-cluster access</li>
</ul>
<p>kube-proxy 是通过如下的 iptables 规则来实现这个功能的:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line">-t nat -A {PREROUTING, OUTPUT} -m conntrack --ctstate NEW -j KUBE-SERVICES</span><br><span class="line"></span><br><span class="line"><span class="comment"># 宿主机访问 nginx Service 的流量,同时满足 4 个条件:</span></span><br><span class="line"><span class="comment"># 1. src_ip 不是 Pod 网段</span></span><br><span class="line"><span class="comment"># 2. dst_ip=3.3.3.3/32 (ClusterIP)</span></span><br><span class="line"><span class="comment"># 3. proto=TCP</span></span><br><span class="line"><span class="comment"># 4. dport=80</span></span><br><span class="line"><span class="comment"># 如果匹配成功,直接跳转到 KUBE-MARK-MASQ;否则,继续匹配下面一条(iptables 是链式规则,高优先级在前)</span></span><br><span class="line"><span class="comment"># 跳转到 KUBE-MARK-MASQ 是为了保证这些包出宿主机时,src_ip 用的是宿主机 IP。</span></span><br><span class="line">-A KUBE-SERVICES ! <span class="_">-s</span> 1.1.0.0/16 <span class="_">-d</span> 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-MARK-MASQ</span><br><span class="line"><span class="comment"># Pod 访问 nginx Service 的流量:同时满足 4 个条件:</span></span><br><span class="line"><span class="comment"># 1. 没有匹配到前一条的,(说明 src_ip 是 Pod 网段)</span></span><br><span class="line"><span class="comment"># 2. dst_ip=3.3.3.3/32 (ClusterIP)</span></span><br><span class="line"><span class="comment"># 3. proto=TCP</span></span><br><span class="line"><span class="comment"># 4. dport=80</span></span><br><span class="line">-A KUBE-SERVICES <span class="_">-d</span> 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-NGINX</span><br><span class="line"></span><br><span class="line"><span class="comment"># 以 50% 的概率跳转到 KUBE-SEP-NGINX1</span></span><br><span class="line">-A KUBE-SVC-NGINX -m statistic --mode random --probability 0.50 -j KUBE-SEP-NGINX1</span><br><span class="line"><span class="comment"># 如果没有命中上面一条,则以 100% 的概率跳转到 KUBE-SEP-NGINX2</span></span><br><span class="line">-A KUBE-SVC-NGINX -j KUBE-SEP-NGINX2</span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果 src_ip=1.1.1.1/32,说明是 Service->client 流量,则</span></span><br><span class="line"><span class="comment"># 需要做 SNAT(MASQ 是动态版的 SNAT),替换 src_ip -> svc_ip,这样客户端收到包时,</span></span><br><span class="line"><span class="comment"># 看到就是从 svc_ip 回的包,跟它期望的是一致的。</span></span><br><span class="line">-A KUBE-SEP-NGINX1 <span class="_">-s</span> 1.1.1.1/32 -j KUBE-MARK-MASQ</span><br><span class="line"><span class="comment"># 如果没有命令上面一条,说明 src_ip != 1.1.1.1/32,则说明是 client-> Service 流量,</span></span><br><span class="line"><span class="comment"># 需要做 DNAT,将 svc_ip -> pod1_ip,</span></span><br><span class="line">-A KUBE-SEP-NGINX1 -p tcp -m tcp -j DNAT --to-destination 1.1.1.1:80</span><br><span class="line"><span class="comment"># 同理,见上面两条的注释</span></span><br><span class="line">-A KUBE-SEP-NGINX2 <span class="_">-s</span> 1.1.1.2/32 -j KUBE-MARK-MASQ</span><br><span class="line">-A KUBE-SEP-NGINX2 -p tcp -m tcp -j DNAT --to-destination 1.1.1.2:80</span><br></pre></td></tr></table></figure></p>
<ol>
<li>Service 既要能被宿主机访问,又要能被 pod 访问(<strong>二者位于不同的 netns</strong>), 因此需要在 <code>PREROUTING</code> 和 <code>OUTPUT</code> 两个 hook 点拦截请求,然后跳转到自定义的 <code>KUBE-SERVICES</code> chain;</li>
<li><code>KUBE-SERVICES</code> chain <strong>执行真正的 Service 匹配</strong>,依据协议类型、目的 IP 和目的端口号。当匹配到某个 Service 后,就会跳转到专门针对这个 Service 创 建的 chain,命名格式为 <code>KUBE-SVC-<Service></code>。</li>
<li><code>KUBE-SVC-<Service></code> chain <strong>根据概率选择某个后端 pod</strong> 然后将请求转发过去。这其实是一种<strong>穷人的负载均衡器</strong> —— 基于 iptables。选中某个 pod 后,会跳转到这个 pod 相关的一条 iptables chain <code>KUBE-SEP-<POD></code>。</li>
<li><code>KUBE-SEP-<POD></code> chain 会<strong>执行 DNAT</strong>,将 VIP 换成 PodIP。</li>
</ol>
<blockquote>
<p>译注:以上解释并不是非常详细和直观,因为这不是本文重点。想更深入地理解基于 iptables 的实现,可参考网上其他一些文章,例如下面这张图所出自的博客 <a href="https://www.stackrox.com/post/2020/01/kubernetes-networking-demystified/" target="_blank" rel="external">Kubernetes Networking Demystified: A Brief Guide</a>,<br><img src="/images/k8s/cim_k8s-net-demystified-svc-lb.png" alt="k8s-net-demystified-svc-lb"></p>
</blockquote>
<h3 id="1-2-NodePort-Service"><a href="#1-2-NodePort-Service" class="headerlink" title="1.2 NodePort Service"></a>1.2 NodePort Service</h3><p>这种类型的 Service 也能被宿主机和 pod 访问,但与 ClusterIP 不同的是,<strong>它还能被集群外的服务访问</strong>。</p>
<ul>
<li>External node IP + port in NodePort range to any endpoint (pod), e.g. 10.0.0.1:31000</li>
<li>Enables access from outside</li>
</ul>
<p>实现上,kube-apiserver 会<strong>从预留的端口范围内分配一个端口给 Service</strong>,然后<strong>每个宿主机上的 kube-proxy 都会创建以下规则</strong>:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">-t nat -A {PREROUTING, OUTPUT} -m conntrack --ctstate NEW -j KUBE-SERVICES</span><br><span class="line"></span><br><span class="line">-A KUBE-SERVICES ! <span class="_">-s</span> 1.1.0.0/16 <span class="_">-d</span> 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-MARK-MASQ</span><br><span class="line">-A KUBE-SERVICES <span class="_">-d</span> 3.3.3.3/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-NGINX</span><br><span class="line"><span class="comment"># 如果前面两条都没匹配到(说明不是 ClusterIP service 流量),并且 dst 是 LOCAL,跳转到 KUBE-NODEPORTS</span></span><br><span class="line">-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS</span><br><span class="line"></span><br><span class="line">-A KUBE-NODEPORTS -p tcp -m tcp --dport 31000 -j KUBE-MARK-MASQ</span><br><span class="line">-A KUBE-NODEPORTS -p tcp -m tcp --dport 31000 -j KUBE-SVC-NGINX</span><br><span class="line"></span><br><span class="line">-A KUBE-SVC-NGINX -m statistic --mode random --probability 0.50 -j KUBE-SEP-NGINX1</span><br><span class="line">-A KUBE-SVC-NGINX -j KUBE-SEP-NGINX2</span><br></pre></td></tr></table></figure></p>
<ol>
<li>前面几步和 ClusterIP Service 一样;如果没匹配到 ClusterIP 规则,则跳转到 <code>KUBE-NODEPORTS</code> chain。</li>
<li><code>KUBE-NODEPORTS</code> chain 里做 Service 匹配,但<strong>这次只匹配协议类型和目的端口号</strong>。</li>
<li>匹配成功后,转到对应的 <code>KUBE-SVC-<Service></code> chain,后面的过程跟 ClusterIP 是一样的。</li>
</ol>
<h3 id="1-3-小结"><a href="#1-3-小结" class="headerlink" title="1.3 小结"></a>1.3 小结</h3><p>以上可以看到,每个 Service 会对应多条 iptables 规则。</p>
<p>Service 数量不断增长时,<strong>iptables 规则的数量增长会更快</strong>。而且,<strong>每个包都需要遍历这些规则</strong>,直到最终匹配到一条相应的规则。如果不幸匹配到最后一条规则才命中, 那相比其他流量,这些包就会有<strong>很高的延迟</strong>。</p>
<p>有了这些背景知识,我们来看如何用 BPF/Cilium 来替换掉 kube-proxy,也可以说是 重新实现 kube-proxy 的逻辑。</p>
<h2 id="2-用-Cilium-BPF-替换-kube-proxy"><a href="#2-用-Cilium-BPF-替换-kube-proxy" class="headerlink" title="2 用 Cilium/BPF 替换 kube-proxy"></a>2 用 Cilium/BPF 替换 kube-proxy</h2><p>我们从 Cilium 早起版本开始,已经逐步用 BPF 实现 Service 功能,但其中仍然有些 地方需要用到 iptables。在这一时期,每台 node 上会同时运行 cilium-agent 和 kube-proxy。</p>
<p>到了 Cilium 1.6,我们已经能<strong>完全基于 BPF 实现,不再依赖 iptables,也不再需要 kube-proxy</strong>。<br><img src="/images/k8s/cim_cilium-cluster-ip.png" alt="cim_cilium-cluster-ip"></p>
<p>这里有一些实现上的考虑:相比于在 TC ingress 层做 Service 转换,我们优先利用 cgroupv2 hooks,<strong>在 socket BPF 层直接做这种转换</strong>(需要高版本内核支持,如果不支 持则 fallback 回 TC ingress 方式)。</p>
<h3 id="2-1-ClusterIP-Service"><a href="#2-1-ClusterIP-Service" class="headerlink" title="2.1 ClusterIP Service"></a>2.1 ClusterIP Service</h3><p>对于 ClusterIP,我们在 BPF 里<strong>拦截 socket 的 <code>connect</code> 和 <code>send</code> 系统调用</strong>; 这些 BPF 执行时,<strong>协议层还没开始执行</strong>(这些系统调用 handlers)。</p>
<ul>
<li>Attach on the cgroupv2 root mount <code>BPF_PROG_TYPE_CGROUP_SOCK_ADDR</code></li>
<li><code>BPF_CGROUP_INET{4,6}_CONNECT</code> - TCP, connected UDP</li>
</ul>
<h4 id="TCP-amp-connected-UDP"><a href="#TCP-amp-connected-UDP" class="headerlink" title="TCP & connected UDP"></a>TCP & connected UDP</h4><p>对于 TCP 和 connected UDP 场景,执行的是下面一段逻辑,<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">int</span> <span class="title">sock4_xlate</span><span class="params">(<span class="keyword">struct</span> bpf_sock_addr *ctx)</span> </span>{</span><br><span class="line"> <span class="keyword">struct</span> lb4_svc_key key = { .dip = ctx->user_ip4, .dport = ctx->user_port };</span><br><span class="line"> svc = lb4_lookup_svc(&key)</span><br><span class="line"> <span class="keyword">if</span> (svc) {</span><br><span class="line"> ctx->user_ip4 = svc->endpoint_addr;</span><br><span class="line"> ctx->user_port = svc->endpoint_port;</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>所做的事情:在 BPF map 中查找 Service,然后做地址转换。但这里的重点是(相比于 TC ingress BPF 实现):</p>
<ol>
<li><strong>不经过连接跟踪(conntrack)模块,也不需要修改包头</strong>(实际上这时候还没有包 ),也不再 mangle 包。这也意味着,<strong>不需要重新计算包的 checksum</strong>。</li>
<li>对于 TCP 和 connected UDP,<strong>负载均衡的开销是一次性的</strong>,只需要在 socket 建立时做一次转换,后面都不需要了,<strong>不存在包级别的转换</strong>。</li>
<li>这种方式是对宿主机 netns 上的 socket 和 pod netns 内的 socket 都是适用的。</li>
</ol>
<h4 id="某些-UDP-应用:存在的问题及解决方式"><a href="#某些-UDP-应用:存在的问题及解决方式" class="headerlink" title="某些 UDP 应用:存在的问题及解决方式"></a>某些 UDP 应用:存在的问题及解决方式</h4><p>但这种方式<strong>对某些 UDP 应用是不适用的</strong>,因为这些 UDP 应用会检查包的源地址,以及 会调用 <code>recvmsg</code> 系统调用。</p>
<p>针对这个问题,我们引入了新的 BPF attach 类型:</p>
<ul>
<li><code>BPF_CGROUP_UDP4_RECVMSG</code></li>
<li><code>BPF_CGROUP_UDP6_RECVMSG</code></li>
</ul>
<p>另外还引入了用于 NAT 的 UDP map、rev-NAT map:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"> BPF rev NAT map</span><br><span class="line">Cookie EndpointIP Port => ServiceID IP Port</span><br><span class="line">-----------------------------------------------------</span><br><span class="line">42 1.1.1.1 80 => 1 3.3.3.30 80</span><br></pre></td></tr></table></figure></p>
<ul>
<li>通过 <code>bpf_get_socket_cookie()</code> 创建 socket cookie。</li>
<li>除了 Service 访问方式,还会有一些<strong>客户端通过 PodIP 直连的方式建立 UDP 连接, cookie 就是为了防止对这些类型的流量做 rev-NAT</strong>。</li>
<li>在 <code>connect(2)</code> 和 <code>sendmsg(2)</code> 时更新 map。</li>
<li>在 <code>recvmsg(2)</code> 时做 rev-NAT。</li>
</ul>
<h3 id="2-2-NodePort-Service"><a href="#2-2-NodePort-Service" class="headerlink" title="2.2 NodePort Service"></a>2.2 NodePort Service</h3><p>NodePort 会更复杂一些,我们先从最简单的场景看起。</p>
<h4 id="2-2-1-后端-pod-在本节点"><a href="#2-2-1-后端-pod-在本节点" class="headerlink" title="2.2.1 后端 pod 在本节点"></a>2.2.1 后端 pod 在本节点</h4><p><img src="/images/k8s/cim_cilium-node-port.png" alt="cilium-node-port"></p>
<p>后端 pod 在本节点时,只需要<strong>在宿主机的网络设备上 attach 一段 tc ingress bpf 程序</strong>,这段程序做的事情:</p>
<ol>
<li>Service 查找</li>
<li>DNAT</li>
<li>redirect 到容器的 lxc0。</li>
</ol>
<p>对于应答包,lxc0 负责 rev-NAT,FIB 查找(因为我们需要设置 L2 地址,否则会被 drop), 然后将其 redirect 回客户端。</p>
<h3 id="2-2-2-后端-pod-在其他节点"><a href="#2-2-2-后端-pod-在其他节点" class="headerlink" title="2.2.2 后端 pod 在其他节点"></a>2.2.2 后端 pod 在其他节点</h3><p>后端 pod 在其他节点时,会复杂一些,因为要转发到其他节点。这种情况下,<strong>需要在 BPF 做 SNAT</strong>,否则 pod 会直接回包给客户端,而由于不同 node 之间没有做连接跟踪( conntrack)同步,因此直接回给客户端的包出 pod 后就会被 drop 掉。</p>
<p>所以需要<strong>在当前节点做一次 SNAT</strong>(<code>src_ip</code> 从原来的 ClientIP 替换为 NodeIP),让回包也经过 当前节点,然后在这里再做 rev-SNAT(<code>dst_ip</code> 从原来的 NodeIP 替换为 ClientIP)。</p>
<p>具体来说,在 <strong>TC ingress</strong> 插入一段 BPF 代码,然后依次执行:Service 查找、DNAT、 选择合适的 egress interface、SNAT、FIB lookup,最后发送给相应的 node,<br><img src="/images/k8s/cim_cilium-node-port-2.png" alt="cilium-node-port-2"></p>
<p>反向路径是类似的,也是回到这个 node,TC ingress BPF 先执行 rev-SNAT,然后 rev-DNAT,FIB lookup,最后再发送回客户端,<br><img src="/images/k8s/cim_cilium-node-port-3.png" alt="cilium-node-port-3"></p>
<p>现在跨宿主机转发是 SNAT 模式,但将来我们打算支持 <strong>DSR 模式</strong>(译注,Cilium 1.8+ 已经支持了)。DSR 的好处是 <strong>backend pods 直接将包回给客户端</strong>,回包不再经过当前 节点转发。</p>
<p>另外,现在 Service 的处理是在 TC ingress 做的,<strong>这些逻辑其实也能够在 XDP 层实现</strong>, 那将会是另一件激动人心的事情(译注,Cilium 1.8+ 已经支持了,性能大幅提升)。</p>
<h4 id="SNAT"><a href="#SNAT" class="headerlink" title="SNAT"></a>SNAT</h4><p>当前基于 BPF 的 SNAT 实现中,用一个 LRU BPF map 存放 Service 和 backend pods 的映 射信息。</p>
<p>需要说明的是,<strong>SNAT 除了替换 <code>src_ip</code>,还可能会替换 <code>src_port</code></strong>:不同客户端的 <code>src_port</code> 可能是相同的,如果只替换 <code>src_ip</code>,不同客户端的应答包在反向转换时就会失 败。因此这种情况下需要做 <code>src_port</code> 转换。现在的做法是,先进行哈希,如果哈希失败, 就调用 <code>prandom()</code> 随机选择一个端口。</p>
<p>此外,我们还需要跟踪宿主机上的流(local flows)信息,因此在 Cilium 里<strong>基于 BPF 实现了一个连接跟踪器</strong>(connection tracker),它会监听宿主机的主物理网络设备( main physical device);我们也会对宿主机上的应用执行 NAT,pod 流量 NAT 之后使用的 是宿主机的 src_port,而宿主机上的应用使用的也是同一个 src_port 空间,它们可能会 有冲突,因此需要在这里处理。</p>
<p>这就是 NodePort Service 类型的流量到达一台节点后,我们在 BPF 所做的事情。</p>
<h3 id="2-2-3-Client-pods-和-backend-pods-在同一节点"><a href="#2-2-3-Client-pods-和-backend-pods-在同一节点" class="headerlink" title="2.2.3 Client pods 和 backend pods 在同一节点"></a>2.2.3 Client pods 和 backend pods 在同一节点</h3><p>另外一种情况是:本机上的 pod 访问某个 NodePort Service,而且 backend pods 也在本机。</p>
<p>这种情况下,流量会从 loopback 口转发到 backend pods,中间会经历路由和转发过程, 整个过程对应用是透明的 —— 我们可以在<strong>应用无感知的情况下,修改二者之间的通信方式</strong>, 只要流量能被双方正确地接受就行。因此,我们在这里<strong>使用了 ClusterIP,并对其进行了一点扩展</strong>,只要连接的 Service 是 loopback 地址或者其他 local 地址,它都能正 确地转发到本机 pods。</p>
<p>另外,比较好的一点是,这种实现方式是基于 cgroups 的,因此独立于 netns。这意味着 我们不需要进入到每个 pod 的 netns 来做这种转换。<br><img src="/images/k8s/cim_cilium-snat.png" alt="cilium-snat"></p>
<h2 id="2-3-Service-规则的规模及请求延迟对比"><a href="#2-3-Service-规则的规模及请求延迟对比" class="headerlink" title="2.3 Service 规则的规模及请求延迟对比"></a>2.3 Service 规则的规模及请求延迟对比</h2><p>有了以上功能,基本上就可以避免 kube-proxy 那样 per-service 的 iptables 规则了, 每个节点上只留下了少数几条由 Kubernetes 自己创建的 iptables 规则:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ iptables-save | grep ‘\-A KUBE’ | wc <span class="_">-l</span></span><br></pre></td></tr></table></figure></p>
<ul>
<li>With kube-proxy: 25401</li>
<li>With BPF: 4</li>
</ul>
<p>在将来,我们有希望连这几条规则也不需要,完全绕开 Netfilter 框架(译注:新版本已经做到了)。</p>
<p>此外,我们做了一些初步的基准测试,如下图所示,<br><img src="/images/k8s/cim_performance.png" alt="performance"></p>
<p>可以看到,随着 Service 数量从 1 增加到 2000+,<strong>kube-proxy/iptables 的请求延 迟增加了将近一倍</strong>,而 Cilium/eBPF 的延迟几乎没有任何增加。</p>
<h2 id="3-相关的-Cilium-BPF-优化"><a href="#3-相关的-Cilium-BPF-优化" class="headerlink" title="3 相关的 Cilium/BPF 优化"></a>3 相关的 Cilium/BPF 优化</h2><p>接下来介绍一些我们在实现 Service 过程中的优化工作,以及一些未来可能会做的事情。</p>
<h3 id="3-1-BPF-UDP-recvmsg-hook"><a href="#3-1-BPF-UDP-recvmsg-hook" class="headerlink" title="3.1 BPF UDP recvmsg() hook"></a>3.1 BPF UDP <code>recvmsg()</code> hook</h3><p>实现 socket 层 UDP Service 转换时,我们发现如果只对 UDP <code>sendmsg</code> 做 hook ,会导致 <strong>DNS 等应用无法正常工作</strong>,会出现下面这种错误:<br><img src="/images/k8s/cim_udp-recvmsg-before.png" alt="udp-recvmsg-before"></p>
<p>深入分析发现,<code>nslookup</code> 及其他一些工具会检查 <strong><code>connect()</code> 时用的 IP 地址和 <code>recvmsg()</code> 读到的 reply message 里的 IP 地址</strong>是否一致。如果不一致,就会 报上面的错误。</p>
<p>原因清楚之后,解决就比较简单了:我们引入了一个做反向映射的 BPF hook,对 <code>recvmsg()</code> 做额外处理,这个问题就解决了:<br><img src="/images/k8s/cim_udp-recvmsg-after.png" alt="udp-recvmsg-after"></p>
<blockquote>
<p><a href="https://github.com/torvalds/linux/commit/983695fa6765" target="_blank" rel="external">983695fa6765</a> bpf: fix unconnected udp hooks。<br>这个 patch 能在不重写包(without packet rewrite)的前提下,会对 BPF ClusterIP 做反向映射(reverse mapping)。</p>
</blockquote>
<h3 id="3-2-全局唯一-socket-cookie"><a href="#3-2-全局唯一-socket-cookie" class="headerlink" title="3.2 全局唯一 socket cookie"></a>3.2 全局唯一 socket cookie</h3><p>BPF ClusterIP Service 为 UDP 维护了一个 LRU 反向映射表(reverse mapping table)。</p>
<p><strong>Socket cookie 是这个映射表的 key 的一部分,但这个 cookie 只在每个 netns 内唯一</strong>,其背后的实现比较简单:每次调用 BPF cookie helper,它都会增加计数器,然后将 cookie 存储到 socket。因此不同 netns 内分配出来的 cookie 值可能会一样,导致冲突。</p>
<p>为解决这个问题,我们将 cookie generator 改成了全局的,见下面的 commit。</p>
<blockquote>
<p><a href="https://github.com/torvalds/linux/commit/cd48bdda4fb8" target="_blank" rel="external">cd48bdda4fb8</a> sock: make cookie generation global instead of per netns。</p>
</blockquote>
<h3 id="3-3-维护邻居表"><a href="#3-3-维护邻居表" class="headerlink" title="3.3 维护邻居表"></a>3.3 维护邻居表</h3><p>Cilium agent 从 K8s apiserver 收到 Service 事件时, 会将 backend entry 更新到 datapath 中的 Service backend 列表。</p>
<p>前面已经看到,当 Service 是 NodePort 类型并且 backend 是 remote 时,需要转发到其 他节点(TC ingress BPF <code>redirect()</code>)。</p>
<p>我们发现<strong>在某些直接路由(direct routing)的场景下,会出现 fib 查找失败的问题</strong> (<code>fib_lookup()</code>),原因是系统中没有对应 backend 的 neighbor entry(IP->MAC 映射 信息),并且接下来<strong>不会主动做 ARP 探测</strong>(ARP probe)。</p>
<blockquote>
<p>Tunneling 模式下这个问题可以忽略,因为本来发送端的 BPF 程 序就会将 src/dst mac 清零,另一台节点对收到的包做处理时, VxLAN 设备上的另一段 BPF 程序会能够正确的转发这个包,因此这种方式更像是 L3 方式。</p>
</blockquote>
<p>我们目前 workaround 了这个问题,解决方式有点丑陋:Cilium 解析 backend,然后直接 将 neighbor entry 永久性地(<code>NUD_PERMANENT</code>)插入邻居表中。</p>
<p>目前这样做是没问题的,因为邻居的数量是固定或者可控的(fixed/controlled number of entries)。但后面我们想尝试的是让内核来做这些事情,因为它能以最好的方式处理这个 问题。实现方式就是引入一些新的 <code>NUD_*</code> 类型,只需要传 L3 地址,然后内核自己将解 析 L2 地址,并负责这个地址的维护。这样 Cilium 就不需要再处理 L2 地址的事情了。 但到今天为止,我并没有看到这种方式的可能性。</p>
<p>对于从集群外来的访问 NodePort Service 的请求,也存在类似的问题, 因为最后将响应流量回给客户端也需要邻居表。由于这些流量都是在 pre-routing,因此我 们现在的处理方式是:自己维护了一个小的 BPF LRU map(L3->L2 mapping in BPF LRU map);由于这是主处理逻辑(转发路径),流量可能很高,因此将这种映射放到 BPF LRU 是更合适的,不会导致邻居表的 overflow。</p>
<h3 id="3-4-LRU-BPF-callback-on-entry-eviction"><a href="#3-4-LRU-BPF-callback-on-entry-eviction" class="headerlink" title="3.4 LRU BPF callback on entry eviction"></a>3.4 LRU BPF callback on entry eviction</h3><p>我们想讨论的另一件事情是:在每个 LRU entry 被 eviction(驱逐)时,能有一个 callback 将会更好。为什么呢?</p>
<p>Cilium 中现在有一个 BPF conntrack table,我们支持到了一些非常老的内核版本 ,例如 4.9。Cilium 在启动时会检查内核版本,优先选择使用 LRU,没有 LRU 再 fallback 到普通的哈希表(Hash Table)。<strong>对于哈希表,就需要一个不断 GC 的过程</strong>。</p>
<p>我们<strong>有意将 NAT map 与 CT map 独立开来</strong>,这是因 为我们要求在 <strong>cilium-agent 升级或降级过程中,现有的连接/流量不能受影响</strong>。 如果二者是耦合在一起的,假如 CT 相关的东西有很大改动,那升级时那要么 是将当前的连接状态全部删掉重新开始;要么就是服务中断,临时不可用,升级完成后再将 老状态迁移到新状态表,但我认为,要轻松、正确地实现这件事情非常困难。 这就是为什么将它们分开的原因。但实际上,GC 在回收 CT entry 的同时, 也会顺便回收 NAT entry。</p>
<p>另外一个问题:<strong>每次从 userspace 操作 conntrack entry 都会破坏 LRU 的正常工作流程</strong>(因为不恰当地更新了所有 entry 的时间戳)。我们通过下面的 commit 解决了这个问题,但要彻底避免这个问题,<strong>最好有一个 GC 以 callback 的方式在第一时 间清理掉这些被 evicted entry</strong>,例如在 CT entry 被 evict 之后,顺便也清理掉 NAT 映射。这是我们正在做的事情(译注,Cilium 1.9+ 已经实现了)。</p>
<blockquote>
<p><a href="https://github.com/torvalds/linux/commit/50b045a8c0cc" target="_blank" rel="external">50b045a8c0cc</a> (“bpf, lru: avoid messing with eviction heuristics upon syscall lookup”) fixed map walking from user space</p>
</blockquote>
<h3 id="3-5-LRU-BPF-eviction-zones"><a href="#3-5-LRU-BPF-eviction-zones" class="headerlink" title="3.5 LRU BPF eviction zones"></a>3.5 LRU BPF eviction zones</h3><p>另一件跟 CT map 相关的比较有意思的探讨:<strong>未来是否能根据流量类型,将 LRU eviction 分割为不同的 zone</strong>?例如,</p>
<ul>
<li>东西向流量分到 zone1:处理 ClusterIP service 流量,都是 pod-{pod,host} 流量, 比较大;</li>
<li>南北向流量分到 zone2:处理 NodePort 和 ExternalName service 流量,相对比较小。</li>
</ul>
<p>这样的好处是:当<strong>对南北向流量 CT 进行操作时,占大头的东西向流量不会受影响</strong>。</p>
<p>理想的情况是这种隔离是有保障的,例如:可以安全地假设,如果正在清理 zone1 内的 entries, 那预期不会对 zone2 内的 entry 有任何影响。不过,虽然分为了多个 zones,但在全局, 只有一个 map。</p>
<h3 id="3-6-BPF-原子操作"><a href="#3-6-BPF-原子操作" class="headerlink" title="3.6 BPF 原子操作"></a>3.6 BPF 原子操作</h3><p>另一个要讨论的内容是原子操作。</p>
<p>使用场景之一是<strong>过期 NAT entry 的快速重复利用</strong>(fast recycling)。 例如,结合前面的 GC 过程,如果一个连接断开时, 不是直接删除对应的 entry,而是更 新一个标记,表明这条 entry 过期了;接下来如果有新的连接刚好命中了这个 entry,就 直接将其标记为正常(非过期),重复利用(循环)这个 entry,而不是像之前一样从新创 建。</p>
<p>现在基于 BPF spinlock 可以实现做这个功能,但并不是最优的方式,因为如果有合适的原 子操作,我们就能节省两次辅助函数调用,然后将 spinlock 移到 map 里。将 spinlock 放到 map 结构体的额外好处是,每个结构体都有自己独立的结构(互相解耦),因此更能 够避免升级/降低导致的问题。</p>
<p>当前内核只有 <code>BPF_XADD</code> 指令,我认为它主要适用于计数(counting),因为它并不像原 子递增(inc)函数一样返回一个值。此外内核中还有的就是针对 maps 的 spinlock。</p>
<p>我觉得如果有 <code>READ_ONCE/WRITE_ONCE</code> 语义将会带来很大便利,现在的 BPF 代码中其实已 经有了一些这样功能的、自己实现的代码。此外,我们还需要 <code>BPF_XCHG</code>, <code>BPF_CMPXCHG</code> 指令,这也将带来很大帮助。</p>
<h3 id="3-7-BPF-getpeername-hook"><a href="#3-7-BPF-getpeername-hook" class="headerlink" title="3.7 BPF getpeername hook"></a>3.7 BPF <code>getpeername</code> hook</h3><p>还有一个 hook —— <code>getpeername()</code> —— 没有讨论到,它<strong>用在 TCP 和 connected UDP 场景</strong>,对应用是透明的。</p>
<p>这里的想法是:永远返回 Service IP 而不是 backend pod IP,这样对应用来说,它看到 就是和 Service IP 建立的连接,而不是和某个具体的 backend pod。</p>
<p>现在返回的是 backend IP 而不是 service IP。从应用的角度看,它连接到的对端并不是 它期望的。</p>
<h3 id="3-8-绕过内核最大-BPF-指令数的限制"><a href="#3-8-绕过内核最大-BPF-指令数的限制" class="headerlink" title="3.8 绕过内核最大 BPF 指令数的限制"></a>3.8 绕过内核最大 BPF 指令数的限制</h3><p>最后再讨论几个非内核的改动(non-kernel changes)。</p>
<p>内核对 <strong>BPF 最大指令数有 4K 条</strong>的限制,现在这个限制已经放大到 <strong>1M</strong>(一百万) 条(但需要 5.1+ 内核,或者稍低版本的内核 + 相应 patch)。</p>
<p>我们的 BPF 程序中包含了 NAT 引擎,因此肯定是超过这个限制的。 但 Cilium 这边,我们目前还并未用到这个新的最大限制,而是通过“外包”的方式将 BPF 切分成了子 BPF 程序,然后通过尾调用(tail call)跳转过去,以此来绕过这个 4K 的限 制。</p>
<p>另外,我们当前使用的是 BPF tail call,而不是 BPF-to-BPF call,因为<strong>二者不能同时使用</strong>。更好的方式是,Cilium agent 在启动时进行检查,如果内核支持 1M BPF insns/complexity limit + bounded loops(我们用于 NAT mappings 查询优化),就用这 些新特性;否则回退到尾调用的方式。</p>
<h2 id="4-Cilium-上手:用-kubeadm-搭建体验环境"><a href="#4-Cilium-上手:用-kubeadm-搭建体验环境" class="headerlink" title="4 Cilium 上手:用 kubeadm 搭建体验环境"></a>4 Cilium 上手:用 kubeadm 搭建体验环境</h2><p>有兴趣尝试 Cilium,可以参考下面的快速安装命令:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ kubeadm init --pod-network-cidr=10.217.0.0/16 --skip-phases=addon/kube-proxy</span><br><span class="line">$ kubeadm join [...]</span><br><span class="line">$ helm template cilium \</span><br><span class="line"> --namespace kube-system --set global.nodePort.enabled=<span class="literal">true</span> \</span><br><span class="line"> --set global.k8sServiceHost=<span class="variable">$API_SERVER_IP</span> \</span><br><span class="line"> --set global.k8sServicePort=<span class="variable">$API_SERVER_PORT</span> \</span><br><span class="line"> --set global.tag=v1.6.1 > cilium.yaml</span><br><span class="line"> kubectl apply <span class="_">-f</span> cilium.yaml</span><br></pre></td></tr></table></figure></p>
<p>附录: <a href="/images/k8s/Making_the_Kubernetes_Service_Abstraction_Scale_using_BPF.pdf">Making_the_Kubernetes_Service_Abstraction_Scale_using_BPF.pdf</a></p>
<blockquote>
<p>译自:ArthurChiao 原文:<a href="https://linuxplumbersconf.org/event/4/contributions/458/" target="_blank" rel="external">https://linuxplumbersconf.org/event/4/contributions/458/</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>本文翻译自 2019 年 Daniel Borkmann 和 Martynas Pumputis 在 Linux Plumbers Conference 的一篇分享: <a href="https://linuxplumbersconf.org/event/4/contributions/458/">Making the Kubernetes Service Abstraction Scale using eBPF</a> 。 翻译时对大家耳熟能详或已显陈旧的内容(K8s 介绍、Cilium 1.6 之前的版本对 Service 实现等)略有删减,如有需要请查阅原 PDF。</p>
<p>实际上,一年之后 Daniel 和 Martynas 又在 LPC 做了一次分享,内容是本文的延续:<a href="http://team.jiunile.com/blog/2020/11/k8s-cilium-service.html">Cilium:基于 BPF/XDP 实现 K8s Service 负载均衡</a></p>
<p><strong>K8s 当前重度依赖 iptables 来实现 Service 的抽象</strong>。对于每个 Service 及其 backend pods,在 K8s 里会生成很多 iptables 规则。<strong>例如 5K 个 Service 时,iptables 规则将达到 25K 条</strong>,导致的后果:</p>
<ul>
<li><strong>较高、并且不可预测的转发延迟</strong>(packet latency),因为每个包都要遍历这些规则 ,直到匹配到某条规则;</li>
<li><strong>更新规则的操作非常慢</strong>:无法单独更新某条 iptables 规则,只能将全部规则读出来 ,更新整个集合,再将新的规则集合下发到宿主机。在动态环境中这一问题尤其明显,因为每 小时可能都有几千次的 backend pods 创建和销毁。</li>
<li><strong>可靠性问题</strong>:iptables 依赖 Netfilter 和系统的连接跟踪模块(conntrack),在 大流量场景下会出现一些竞争问题(race conditions);<strong>UDP 场景尤其明显</strong>,会导 致丢包、应用的负载升高等问题。</li>
</ul>
<p>本文将介绍如何基于 Cilium/BPF 来解决这些问题,实现 K8s Service 的大规模扩展。<br>
使用 Go 实现 Async/Await 模式
http://team.jiunile.com//blog/2020/12/go-async-await.html
2020-12-01T12:00:00.000Z
2020-11-30T15:41:49.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>Golang 是一种并发编程语言。它具有强大的特性,如 <code>Goroutines</code> 和 <code>Channels</code>,可以很好地处理异步任务。另外,<code>goroutines</code> 不是 OS 线程,这就是为什么您可以在不增加开销的情况下根据需要启动任意数量的 <code>goroutine</code> 的原因,它的堆栈大小初始化时仅 <strong>2KB</strong>。那么为什么要 <code>async/await</code> 呢? <code>Async/Await</code> 是一种很好的语言特点,它为异步编程提供了更简单的接口。</p>
<p>项目链接:<a href="https://github.com/icyxp/AsyncGoDemo" target="_blank" rel="external">https://github.com/icyxp/AsyncGoDemo</a></p>
<a id="more"></a>
<h2 id="它是如何工作的?"><a href="#它是如何工作的?" class="headerlink" title="它是如何工作的?"></a>它是如何工作的?</h2><p>从 F# 开始,然后是 C#,到现在 Python 和 Javascript 中,<code>async/await</code> 是一种非常流行的语言特点。它简化了异步方法的执行结构并且读起来像同步代码。对于开发人员来说更容易理解。让我们看看 c# 中的一个简单示例 <code>async/await</code> 是如何工作的。<br><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">static</span> async Task <span class="title">Main</span><span class="params">(<span class="built_in">string</span>[] args)</span></span><br><span class="line"></span>{</span><br><span class="line"> Console.WriteLine(<span class="string">"Let's start ..."</span>);</span><br><span class="line"> var done = DoneAsync();</span><br><span class="line"> Console.WriteLine(<span class="string">"Done is running ..."</span>);</span><br><span class="line"> Console.WriteLine(await done);</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">static</span> async Task<<span class="keyword">int</span>> DoneAsync()</span><br><span class="line">{</span><br><span class="line"> Console.WriteLine(<span class="string">"Warming up ..."</span>);</span><br><span class="line"> await Task.Delay(<span class="number">3000</span>);</span><br><span class="line"> Console.WriteLine(<span class="string">"Done ..."</span>);</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>当程序运行时,我们的 <code>Main</code> 函数将被执行。我们有异步函数 <code>DoneAsync</code>。我们使用 <code>Delay</code> 方法停止执行代码 3 秒钟。Delay 本身是一个异步函数,所以我们用 <code>await</code> 来调用它。</p>
<blockquote>
<p><code>await</code> 只阻塞异步函数内的代码执行</p>
</blockquote>
<p>在 <code>Main</code> 函数中,我们不使用 <code>await</code> 来调用 <code>DoneAsync</code>。但 <code>DoneAsync</code> 开始执行后,只有当我们 <code>await</code> 它的时候,我们才会得到结果。执行流程如下所示:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">Let<span class="string">'s start ...</span><br><span class="line">Warming up ...</span><br><span class="line">Done is running ...</span><br><span class="line">Done ...</span><br><span class="line">1</span></span><br></pre></td></tr></table></figure></p>
<p>对于异步执行,这看起来非常简单。让我们看看如何使用 Golang 的 <code>Goroutines</code> 和 <code>Channels</code> 来做到这一点。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">DoneAsync</span><span class="params">()</span> <span class="title">chan</span> <span class="title">int</span></span> {</span><br><span class="line"> r := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>)</span><br><span class="line"> fmt.Println(<span class="string">"Warming up ..."</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> time.Sleep(<span class="number">3</span> * time.Second)</span><br><span class="line"> r <- <span class="number">1</span></span><br><span class="line"> fmt.Println(<span class="string">"Done ..."</span>)</span><br><span class="line"> }()</span><br><span class="line"> <span class="keyword">return</span> r</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span> <span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(<span class="string">"Let's start ..."</span>)</span><br><span class="line"> val := DoneAsync()</span><br><span class="line"> fmt.Println(<span class="string">"Done is running ..."</span>)</span><br><span class="line"> fmt.Println(<- val)</span><br><span class="line">}</span><br><span class="line"><span class="string">``</span><span class="string">` </span><br><span class="line">在这里,`</span>DoneAsync<span class="string">` 异步运行并返回一个 `</span>channel<span class="string">`。执行完异步任务后,它会将值写入 `</span>channel<span class="string">`。在 `</span>main<span class="string">` 函数中,我们调用 `</span>DoneAsync<span class="string">` 并继续执行后续操作,然后从返回的 `</span>channel<span class="string">` 读取值。它是一个阻塞调用,等待直到将值写入 `</span>channel<span class="string">`,并在获得值后将其输出到终端。</span><br><span class="line">`</span><span class="string">``</span><span class="keyword">go</span></span><br><span class="line">Let<span class="string">'s start ...</span><br><span class="line">Warming up ...</span><br><span class="line">Done is running ...</span><br><span class="line">Done ...</span><br><span class="line">1</span></span><br></pre></td></tr></table></figure></p>
<p>我们看到,我们实现了与 C# 程序相同的结果,但它看起来不像 <code>async/await</code> 那样优雅。尽管这确实不错,但是我们可以使用这种方法轻松地完成很多细粒度的事情,我们还可以用一个简单的结构和接口在 Golang 中实现 <code>async/await</code> 关键字。让我们试试。</p>
<h2 id="实现-Async-Await"><a href="#实现-Async-Await" class="headerlink" title="实现 Async/Await"></a>实现 Async/Await</h2><p>完整代码可在项目链接中找到(在文章开始的地方)。要在 Golang 中实现 <code>async/await</code>,我们将从一个名为 <code>async</code> 的包目录开始。项目结构看起来是这样的。</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">.</span><br><span class="line">├── async</span><br><span class="line">│ └── async.go</span><br><span class="line">├── main.go</span><br><span class="line">└── README.md</span><br></pre></td></tr></table></figure>
<p>在 <code>async</code> 文件中,我们编写了可以处理异步任务最简单的 <code>Future</code> 接口。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> async</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"context"</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// Future interface has the method signature for await</span></span><br><span class="line"><span class="keyword">type</span> Future <span class="keyword">interface</span> {</span><br><span class="line"> Await() <span class="keyword">interface</span>{}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> future <span class="keyword">struct</span> {</span><br><span class="line"> await <span class="function"><span class="keyword">func</span><span class="params">(ctx context.Context)</span> <span class="title">interface</span></span>{}</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(f future)</span> <span class="title">Await</span><span class="params">()</span> <span class="title">interface</span></span>{} {</span><br><span class="line"> <span class="keyword">return</span> f.await(context.Background())</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Exec executes the async function</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">Exec</span><span class="params">(f <span class="keyword">func</span>()</span> <span class="title">interface</span></span>{}) Future {</span><br><span class="line"> <span class="keyword">var</span> result <span class="keyword">interface</span>{}</span><br><span class="line"> c := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">struct</span>{})</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> <span class="built_in">close</span>(c)</span><br><span class="line"> result = f()</span><br><span class="line"> }()</span><br><span class="line"> <span class="keyword">return</span> future{</span><br><span class="line"> await: <span class="function"><span class="keyword">func</span><span class="params">(ctx context.Context)</span> <span class="title">interface</span></span>{} {</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> <-ctx.Done():</span><br><span class="line"> <span class="keyword">return</span> ctx.Err()</span><br><span class="line"> <span class="keyword">case</span> <-c:</span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"> }</span><br><span class="line"> },</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这里发生的事情并不多,我们添加了一个具有 <code>Await</code> 方法标识的 <code>Future</code> 接口。接下来,我们添加一个 <code>future</code> 结构,它包含一个值,即 <code>await</code> 函数的函数标识。现在 <code>futute struct</code> 通过调用自己的 <code>await</code> 函数来实现 <code>Future</code> 接口的 <code>Await</code> 方法。</p>
<p>接下来在 <code>Exec</code> 函数中,我们在 <code>goroutine</code> 中异步执行传递的函数。然后返回 <code>await</code> 函数。它等待 <code>channel</code> 关闭或 <code>context</code> 读取。基于最先发生的情况,它要么返回错误,要么返回作为接口的结果。</p>
<p>现在,有了这个新的 <code>async</code> 包,让我们看看如何更改当前的 go 代码:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">DoneAsync</span><span class="params">()</span> <span class="title">int</span></span> {</span><br><span class="line"> fmt.Println(<span class="string">"Warming up ..."</span>)</span><br><span class="line"> time.Sleep(<span class="number">3</span> * time.Second)</span><br><span class="line"> fmt.Println(<span class="string">"Done ..."</span>)</span><br><span class="line"> <span class="keyword">return</span> <span class="number">1</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Println(<span class="string">"Let's start ..."</span>)</span><br><span class="line"> future := async.Exec(<span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">interface</span></span>{} {</span><br><span class="line"> <span class="keyword">return</span> DoneAsync()</span><br><span class="line"> })</span><br><span class="line"> fmt.Println(<span class="string">"Done is running ..."</span>)</span><br><span class="line"> val := future.Await()</span><br><span class="line"> fmt.Println(val)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>乍一看,它看起来干净得多,这里我们没有显式地使用 <code>goroutine</code> 或 <code>channels</code>。我们的 <code>DoneAsync</code> 函数已更改为完全同步的性质。在 <code>main</code> 函数中,我们使用 <code>async</code> 包的<code>Exec</code> 方法来处理 <code>DoneAsync</code>。在开始执行 <code>DoneAsync</code>。控制流返回到可以执行其他代码的 <code>main</code> 函数中。最后,我们对 <code>Await</code> 进行阻塞调用并回读数据。</p>
<p>现在,代码看起来更加简单易读。我们可以修改我们的 async 包从而能在 Golang 中合并许多其他类型的异步任务,但在本教程中,我们现在只坚持简单的实现。</p>
<h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>我们经历了 <code>async/await</code> 的过程,并在 Golang 中实现了一个简单的版本。我鼓励您进一步研究 <code>async/await</code>,看看它如何更好的让代码库便于易读。</p>
<blockquote>
<p>译自:<a href="https://hackernoon.com/asyncawait-in-golang-an-introductory-guide-ol1e34sg" target="_blank" rel="external">https://hackernoon.com/asyncawait-in-golang-an-introductory-guide-ol1e34sg</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>Golang 是一种并发编程语言。它具有强大的特性,如 <code>Goroutines</code> 和 <code>Channels</code>,可以很好地处理异步任务。另外,<code>goroutines</code> 不是 OS 线程,这就是为什么您可以在不增加开销的情况下根据需要启动任意数量的 <code>goroutine</code> 的原因,它的堆栈大小初始化时仅 <strong>2KB</strong>。那么为什么要 <code>async/await</code> 呢? <code>Async/Await</code> 是一种很好的语言特点,它为异步编程提供了更简单的接口。</p>
<p>项目链接:<a href="https://github.com/icyxp/AsyncGoDemo">https://github.com/icyxp/AsyncGoDemo</a></p>
从 Go 分析 Struct 对齐如何影响内存使用量
http://team.jiunile.com//blog/2020/11/go-struct.html
2020-11-30T14:00:00.000Z
2020-11-30T02:34:25.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>不知道大家在写 Go 时有没有注意过,<strong>一个 struct 所占的空间不见得等于各个 field 加起来的空间</strong>,甚至有时把 field 申明的顺序调换一下,又会得到不同的结果。</p>
<p>今天的文章就是要从 CPU 抓资料的原理开始介绍,然后再讲到 <strong>Data Structure Alignment</strong>(数据结构对齐),希望大家在看完之后能对 CPU 跟记忆体有更多认识~<br><a id="more"></a></p>
<h2 id="直接上例子"><a href="#直接上例子" class="headerlink" title="直接上例子"></a>直接上例子</h2><p>以 T1 为例,整个 <code>struct</code> 共有三个栏位,类型分别是 <code>int8</code>、<code>int64</code> 跟 <code>int32</code>,所以变数 <code>t1</code> 应该需要 <code>1+8+4=13 bytes</code> 的空间。但实际在 <a href="https://goplay.tools/snippet/6kzzmHddQgc" target="_blank" rel="external">Go Playground</a> 上跑,会发现 <code>t1</code> 竟然需要 <code>24 bytes</code>,真奇怪是吧?<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> T1 <span class="keyword">struct</span> {</span><br><span class="line"> f1 <span class="keyword">int8</span> <span class="comment">// 1 byte</span></span><br><span class="line"> f2 <span class="keyword">int64</span> <span class="comment">// 8 bytes</span></span><br><span class="line"> f3 <span class="keyword">int32</span> <span class="comment">// 4 bytes</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> t1 := T1{}</span><br><span class="line"> fmt.Println(unsafe.Sizeof(t1)) <span class="comment">// 24 bytes</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>如果尝试把栏位的顺序调整一下,改成 <code>int8</code>、<code>int32</code>、<code>int64</code> 再跑一次,就只需要 16 bytes,但跟原本预期的 13 bytes 还是有差,那究竟为什么会这样的差异呢?<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> T2 <span class="keyword">struct</span> {</span><br><span class="line"> f1 <span class="keyword">int8</span> <span class="comment">// 1 byte</span></span><br><span class="line"> f3 <span class="keyword">int32</span> <span class="comment">// 4 bytes</span></span><br><span class="line"> f2 <span class="keyword">int64</span> <span class="comment">// 8 bytes</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> t2 := T2{}</span><br><span class="line"> fmt.Println(unsafe.Sizeof(t2)) <span class="comment">// 16 bytes</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<h2 id="从-CPU-如何抓资料开始讲起"><a href="#从-CPU-如何抓资料开始讲起" class="headerlink" title="从 CPU 如何抓资料开始讲起"></a>从 CPU 如何抓资料开始讲起</h2><p>如果买电脑时有在留意 CPU 规格的话(身为工程师一定要的吧XD),应该会发现近几年的 CPU 几乎都是 <code>64 bit</code> 的。而这边的 <code>64 bit</code>,指的就是 CPU 一次可以从记忆体里面抓 <code>64 bits</code> 的资料,换算一下也就是 <code>8 bytes</code>。</p>
<p>虽说是一次抓 <code>8 bytes</code>,但也不是想抓哪就抓哪,因为记忆体也会以 <strong><code>8 bytes</code> 分成一个一个 word</strong>(如下图),而 CPU 只能一次拿某一个 word。所以如果所需的资料刚好横跨两个 word,那就得花两个 <code>CPU cycle</code> 的时间去拿。<br><img src="/images/go/struct_1.png" alt="struct"></p>
<blockquote>
<p>注:在 <code>64 bit</code> 的系统中一个 word 是 <code>8 bytes</code>,<code>32 bit</code> 中则是 <code>4 bytes</code></p>
</blockquote>
<h2 id="所以为什么-struct-会变肥"><a href="#所以为什么-struct-会变肥" class="headerlink" title="所以为什么 struct 会变肥"></a>所以为什么 struct 会变肥</h2><p>了解 CPU 后我们再看一次 T1,他的栏位顺序是 <code>int8</code>、<code>int64</code>、<code>int32</code>,所以把 t1 的资料连续放在记忆体里面就长得像下图:因为第二个栏位 f2(<code>int64</code>) 需要 8 个 bytes,所以<strong>会有一个 byte 会被挤到第二个 word</strong>(第二排)<br><img src="/images/go/struct_2.png" alt="struct"></p>
<p>那这样有什么坏处呢?如果我的程式需要用到 <code>t1.f2</code>,譬如说把他 print 出来,那 CPU 就得花两个 cycle 的时间把 f2 从记忆体抓出来,<strong>因为 f2 分散在两个 word 里面</strong></p>
<p>所以为了让 CPU 可以更快存取到各个栏位,Go 编译器会帮你的 struct 做 <code>Data Structure Align</code>,也就是在 T1 的栏位间加上一些 padding,<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> T1 {</span><br><span class="line"> f1 i8</span><br><span class="line"> _ [<span class="number">7</span>]<span class="keyword">byte</span> <span class="comment">// 7 bytes padding</span></span><br><span class="line"> f2 i64</span><br><span class="line"> f3 i32</span><br><span class="line"> _ [<span class="number">4</span>]<span class="keyword">byte</span> <span class="comment">// 4 bytes padding</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>画成图就长下面这样,有 13 bytes 用来储存 struct 的资料,而深色的 11 个 bytes 则是用来当 padding,<strong>确保每个 field 的所有内容都落在同一个 word 里面</strong>,所以 struct 才会从 13 bytes 肥到 24 bytes<br><img src="/images/go/struct_3.png" alt="struct"></p>
<h2 id="Padding-可以不要那么肥吗?"><a href="#Padding-可以不要那么肥吗?" class="headerlink" title="Padding 可以不要那么肥吗?"></a>Padding 可以不要那么肥吗?</h2><p>虽说 padding 是为了把每个 field 放到更好的位置,但 padding 的空间实际上就是浪费掉了。以 T1 来说,24 bytes 里面就浪费了将近一半,那有什么方法可以兼顾 Alignment 但又不浪费太多空间吗?</p>
<p>再看一次 T1 的记忆体分佈,就会发现最下面 4 bytes 的 f3 其实可以挪到上面的 padding,反正第一排的 padding 空间超大的,不用白不用,<strong>而且挪上去之后每个栏位都还是在同一个 word 里面</strong>。<br><img src="/images/go/struct_4.png" alt="struct"></p>
<p>一旦把 f3 移上去,就可以省掉最下面一整个 word(8 bytes) 的空间,所以 T2 整个 struct 就只需要 16 bytes,是原本 T1 24 bytes 的三分之二<br><img src="/images/go/struct_5.png" alt="struct"></p>
<p>写成程式码的话,因为 Go 会按照栏位的顺序来安排记忆体中的位置,所以要把 f2 跟 f3 的顺序交换,宣告的顺序变成 <code>int8</code>、<code>int32</code>、<code>int64</code>,这样才会顺利排成上面那个图哦~<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> T2 <span class="keyword">struct</span> {</span><br><span class="line"> f1 <span class="keyword">int8</span> <span class="comment">// 1 byte</span></span><br><span class="line"> f3 <span class="keyword">int32</span> <span class="comment">// 4 bytes</span></span><br><span class="line"> f2 <span class="keyword">int64</span> <span class="comment">// 8 bytes</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> t2 := T2{}</span><br><span class="line"> fmt.Println(unsafe.Sizeof(t2)) <span class="comment">// 16 bytes</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<h2 id="编译器没办法自动最佳化吗?"><a href="#编译器没办法自动最佳化吗?" class="headerlink" title="编译器没办法自动最佳化吗?"></a>编译器没办法自动最佳化吗?</h2><p>看到这你一定觉得很麻烦,难不成每次用 Struct 都要自己拼拼凑凑、算算看怎么样的顺序最省空间?这种底层的鸟事应该<a href="https://medium.com/starbugs/see-what-compiler-optimization-do-from-llvm-ir-dfd3774292cb" target="_blank" rel="external">由编译器来最佳化</a>才对啊!</p>
<p>遗憾的是,目前 Go 编译器不会自动做这些最佳化(但 <a href="https://camlorn.net/posts/April%202017/rust-struct-field-reordering/" target="_blank" rel="external">Rust 三年前就支援了</a>,希望 Go 也能赶快跟进XD),所以如果很在意 struct 有没有充分利用记忆体空间,可以自己画图排排看,或是用 <a href="https://github.com/orijtech/structslop" target="_blank" rel="external">structslop</a> 进行分析。</p>
<h3 id="structslop"><a href="#structslop" class="headerlink" title="structslop"></a>structslop</h3><p><code>structslop</code> 是一个用 Go 写成的开源工具,他的功能就是帮你调整 struct 的栏位顺序,<strong>以达到最好的空间利用率</strong>。像下面的例子 Student 里面包含了学号、姓名、班级、成绩等等资讯。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Student <span class="keyword">struct</span> {</span><br><span class="line"> id <span class="keyword">int8</span> <span class="comment">// 1 byte</span></span><br><span class="line"> name <span class="keyword">string</span> <span class="comment">// 16 bytes</span></span><br><span class="line"> classID <span class="keyword">int8</span> <span class="comment">// 1 byte</span></span><br><span class="line"> phone [<span class="number">10</span>]<span class="keyword">byte</span> <span class="comment">// 10 bytes</span></span><br><span class="line"> address <span class="keyword">string</span> <span class="comment">// 16 bytes</span></span><br><span class="line"> grade <span class="keyword">int32</span> <span class="comment">// 4 bytes</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>如果画成图就长这样,可以看到里面还有很多深色的 padding,算一算总共浪费了 16 bytes,感觉不是那么优。<br><img src="/images/go/struct_6.png" alt="struct"></p>
<p>这时就可以用 <code>structslop</code> 帮你分析并且算出一个最佳解,只要把栏位顺序改成他建议的,Student 占用的空间就可以从 64 bytes 最佳化到 48 bytes,共<strong>省下 25% 的空间</strong>。<br><img src="/images/go/struct_7.png" alt="struct"></p>
<p>如果把 <code>structslop</code> 推荐的 field 顺序画成图就长这样,全部排得满满的,没有任何一点 padding,看了心情都好了起来XD<br><img src="/images/go/struct_8.png" alt="struct"></p>
<h2 id="有必要省空间省成这样吗"><a href="#有必要省空间省成这样吗" class="headerlink" title="有必要省空间省成这样吗"></a>有必要省空间省成这样吗</h2><p>讲完怎么省空间后,接著我们来想想,虽然重新排列栏位可以让 struct 更省空间,但真的有必要这样吗?</p>
<p>以 Student 的例子来说,经过重新排列后,一个 struct 可以省下 16 bytes。</p>
<p>如果你要写个程式来排序全校同学的成绩,需要宣告长度十万的 <code>Student array</code>,那省下的记忆体也不过 16 MB,跟现在个人电脑配备的 4GB 到 8GB 比起来根本是零头。</p>
<p>而且笔者我觉得栏位在经过重新排序之后,可读性可能会稍微降低,像 Student 原本的栏位依序是学号、姓名、班级…,满符合直觉的。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Student <span class="keyword">struct</span> {</span><br><span class="line"> id <span class="keyword">int8</span></span><br><span class="line"> name <span class="keyword">string</span></span><br><span class="line"> classID <span class="keyword">int8</span></span><br><span class="line"> phone [<span class="number">10</span>]<span class="keyword">byte</span></span><br><span class="line"> address <span class="keyword">string</span></span><br><span class="line"> grade <span class="keyword">int32</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>但重新排序后顺序就变成姓名、地址、成绩…一直到最后才是学号跟班级,总觉得越重要的栏位应该要放在越前面才是(我自己觉得啦XD)。</p>
<p>所以我的观点是不需要太早进行最佳化,除非你一开始就知道你的程式瓶颈会卡在这(也许程式要跑在嵌入式装置),否则就照平常的方式写 Go 就好,也不用去算这些有的没的,也许 Go 在哪一次更新之后就像 Rust 默默支援 <code>struct field reordering</code> 了</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>最后,我想跟大家分享一个忘记在哪看到的句子:<strong>「Understanding the Hardware Makes You a Better Developer」</strong>,这边的 <strong>Hardware</strong> 我认为不一定是指硬体,而是泛指你所依赖的底层工具。</p>
<p>譬如说我完全不懂浏览器的 Reflow 跟 Repaint 还是可以写前端,但要做动画可能就会遇到效能瓶颈;不懂 Go 的 GC 机制还是可以把 Go 写得不错,但流量大起来时可能就会花太多时间在 GC。</p>
<p>所以虽然这篇文的结论是不需要特别去注意 <code>Data Structure Alignment</code> ,只要知道程式内部是这样运作的,并且顺其自然即可,但如果有一天真的因为这样记忆体不够了,那记得要想到调整一下栏位顺序哦~。</p>
<h2 id="延伸阅读"><a href="#延伸阅读" class="headerlink" title="延伸阅读"></a>延伸阅读</h2><ul>
<li><a href="https://stackoverflow.com/questions/6730664/why-doesnt-c-make-the-structure-tighter" target="_blank" rel="external">Why doesn’t C++ make the structure tighter? — Stack Overflow</a></li>
<li><a href="https://camlorn.net/posts/April%202017/rust-struct-field-reordering/" target="_blank" rel="external">Optimizing Rust Struct Size: A 6-month Compiler Development Project</a></li>
<li><a href="https://techterms.com/help/difference_between_32-bit_and_64-bit_systems" target="_blank" rel="external">What is the difference between a 32-bit and 64-bit system?</a></li>
</ul>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<blockquote>
<p>来源:<a href="https://medium.com/starbugs/illustrate-how-data-alignment-affects-memory-usage-d29bf9d5bf08" target="_blank" rel="external">https://medium.com/starbugs/illustrate-how-data-alignment-affects-memory-usage-d29bf9d5bf08</a></p>
</blockquote>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>不知道大家在写 Go 时有没有注意过,<strong>一个 struct 所占的空间不见得等于各个 field 加起来的空间</strong>,甚至有时把 field 申明的顺序调换一下,又会得到不同的结果。</p>
<p>今天的文章就是要从 CPU 抓资料的原理开始介绍,然后再讲到 <strong>Data Structure Alignment</strong>(数据结构对齐),希望大家在看完之后能对 CPU 跟记忆体有更多认识~<br>
Golang 切片综合指南
http://team.jiunile.com//blog/2020/11/go-slices.html
2020-11-26T14:00:00.000Z
2020-11-27T07:10:03.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在本文中,我们将讨论 “切片” 的概念,它是 Golang 中使用的一种重要数据结构。这一数据结构为你提供了处理与管理数据集合的方法。切片是围绕动态数组的概念构建的,它与动态数组相似,可以根据你的需要而伸缩。</p>
<ul>
<li>切片在增长方面是动态的,因为它们有自己的内置函数 <code>append</code>,可以快速高效地增长切片。</li>
<li>您还可以通过切割底层内存来减小切片的大小。</li>
<li>在底层内存中切片是在连续的块上分配的,因此切片为你提供的便利之处包括:索引、迭代与垃圾回收优化。<a id="more"></a>
<h2 id="切片表示"><a href="#切片表示" class="headerlink" title="切片表示"></a>切片表示</h2></li>
<li>切片不存储任何数据;它只描述底层数组的一部分。</li>
<li>切片使用一个包含三个字段的结构表示:指向底层数组的指针(pointer)、长度(length)与容量(capacity)。</li>
<li>这个数据结构类似于切片的描述符。</li>
</ul>
<p><img src="/images/go/slice_1.png" alt="Slice representation"></p>
<ul>
<li><strong>Pointer</strong>:指针用于指向数组的第一个元素,这个元素可以通过切片进行访问。在这里,指向的元素不必是数组的第一个元素。</li>
<li><strong>Length</strong>:长度代表数组中所有元素的总数。</li>
<li><strong>Capacity</strong>:容量表示切片可扩展的最大大小。</li>
</ul>
<h2 id="使用长度申明切片"><a href="#使用长度申明切片" class="headerlink" title="使用长度申明切片"></a>使用长度申明切片</h2><p>在声明切片过程中,当你仅指定长度(Length)时,容量(Capacity)值与长度(Length)值相同。<br><img src="/images/go/slice_2.png" alt="Declare a slice using the length"><br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Declaring a slice by length. Create a slice of int. </span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 5 elements. </span></span><br><span class="line">slice := <span class="built_in">make</span>([]<span class="keyword">int</span>, <span class="number">5</span>)</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) <span class="comment">// Print 5</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) <span class="comment">// Print 5</span></span><br></pre></td></tr></table></figure></p>
<h2 id="使用长度和容量申明切片"><a href="#使用长度和容量申明切片" class="headerlink" title="使用长度和容量申明切片"></a>使用长度和容量申明切片</h2><p>在声明切片过程中,当你分别指定长度(Length)和容量(Capacity)时,这将初始化一段无法访问的底层数组来创建一个具有可用容量的切片。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/* </span><br><span class="line"> Declaring a slice by length and capacity</span><br><span class="line"> Create a slice of integers. </span><br><span class="line"> Contains a length of 3 and has a capacity of 5 elements.</span><br><span class="line">*/</span></span><br><span class="line">slice := <span class="built_in">make</span>([]<span class="keyword">int</span>, <span class="number">3</span>, <span class="number">5</span>)</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) <span class="comment">// Print 3</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) <span class="comment">// Print 5</span></span><br></pre></td></tr></table></figure></p>
<p><img src="/images/go/slice_3.png" alt="Declare a slice with length and capacity"></p>
<p>但是请注意,尝试创建容量小于长度的切片是不允许的。</p>
<h2 id="使用切片字面量创建切片"><a href="#使用切片字面量创建切片" class="headerlink" title="使用切片字面量创建切片"></a>使用切片字面量创建切片</h2><p>创建切片的惯用方法是使用切片字面量。它与创建数组相似,只是它不需要在 [ ] 操作符中指定值。你初始化切片时所用元素的数量将决定切片的初始长度与容量。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of strings. </span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 5 elements. </span></span><br><span class="line">slice := []<span class="keyword">string</span>{<span class="string">"Red"</span>, <span class="string">"Blue"</span>, <span class="string">"Green"</span>, <span class="string">"Yellow"</span>, <span class="string">"Pink"</span>} </span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) <span class="comment">//Print 5</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) <span class="comment">//Print 5</span></span><br><span class="line"><span class="comment">// Create a slice of integers. </span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 3 elements. </span></span><br><span class="line">intSlice:= []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>}</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(intSlice)) <span class="comment">//Print 3</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(intSlice)) <span class="comment">//Print 3</span></span><br></pre></td></tr></table></figure></p>
<h2 id="声明一个带有索引位置的切片"><a href="#声明一个带有索引位置的切片" class="headerlink" title="声明一个带有索引位置的切片"></a>声明一个带有索引位置的切片</h2><p>当使用切片字面量时,你可以初始化切片的长度与容量。你所需要做的就是初始化表示所需长度和容量的索引。下面的语法将创建一个长度和容量均为 100 的切片。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of strings.</span></span><br><span class="line"><span class="comment">// Initialize the 100th element with an empty string.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">99</span>: <span class="number">88</span>}</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) </span><br><span class="line"><span class="comment">// Print 100</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) </span><br><span class="line"><span class="comment">// Print 100</span></span><br></pre></td></tr></table></figure></p>
<p><img src="/images/go/slice_4.png" alt="Declare a slice with index positions"></p>
<p>声明数组与切片的区别:</p>
<ul>
<li>如果你使用[]操作符中指定一个值,那么你在创建一个数组。</li>
<li>如果你不在[]中指定值,则创建一个切片。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create an array of three integers. </span></span><br><span class="line">array := [<span class="number">3</span>]<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>} </span><br><span class="line"></span><br><span class="line"><span class="comment">//Create a slice of integers with a length and capacity of three.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>}</span><br></pre></td></tr></table></figure>
<h2 id="声明一个-nil-切片"><a href="#声明一个-nil-切片" class="headerlink" title="声明一个 nil 切片"></a>声明一个 nil 切片</h2><ul>
<li>切片用 <code>nil</code> 代表零值。</li>
<li>一个 nil 切片的长度和容量等于 0,且没有底层数组。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a nil slice of integers. </span></span><br><span class="line"><span class="keyword">var</span> slice []<span class="keyword">int32</span></span><br><span class="line">fmt.Println(slice == <span class="literal">nil</span>) </span><br><span class="line"><span class="comment">//This line will print true</span></span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) </span><br><span class="line"><span class="comment">// This line will print 0</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice))</span><br><span class="line"><span class="comment">// This line will print 0</span></span><br></pre></td></tr></table></figure>
<p><img src="/images/go/slice_5.png" alt="Declare a nil slice"></p>
<h2 id="声明一个空切片"><a href="#声明一个空切片" class="headerlink" title="声明一个空切片"></a>声明一个空切片</h2><p>还可以通过初始化声明切片创建一个空切片。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Use make to create an empty slice of integers.</span></span><br><span class="line">sliceOne := <span class="built_in">make</span>([]<span class="keyword">int</span>, <span class="number">0</span>)</span><br><span class="line"><span class="comment">// Use a slice literal to create an empty slice of integers.</span></span><br><span class="line">sliceTwo := []<span class="keyword">int</span>{}</span><br><span class="line">fmt.Println(sliceOne == <span class="literal">nil</span>) <span class="comment">// This will print false</span></span><br><span class="line">fmt.Println(<span class="built_in">len</span>(sliceOne)) <span class="comment">// This will print 0 </span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(sliceOne)) <span class="comment">// This will print 0</span></span><br><span class="line">fmt.Println(sliceTwo == <span class="literal">nil</span>) <span class="comment">// This will print false</span></span><br><span class="line">fmt.Println(<span class="built_in">len</span>(sliceTwo)) <span class="comment">// This will print 0</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(sliceTwo)) <span class="comment">// This will print 0</span></span><br></pre></td></tr></table></figure></p>
<p><img src="/images/go/slice_6.png" alt="Declare an empty slice"></p>
<h2 id="为任何特定索引赋值"><a href="#为任何特定索引赋值" class="headerlink" title="为任何特定索引赋值"></a>为任何特定索引赋值</h2><p>要修改单个元素的值,请使用[]操作符。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 4 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line">fmt.Println(slice) <span class="comment">//This will print [10 20 30 40]</span></span><br><span class="line">slice[<span class="number">1</span>] = <span class="number">25</span> <span class="comment">// Change the value of index 1.</span></span><br><span class="line">fmt.Println(slice) <span class="comment">// This will print [10 25 30 40]</span></span><br></pre></td></tr></table></figure></p>
<p><img src="/images/go/slice_7.png" alt="Assign a value to any specific index"></p>
<h2 id="对切片进行切片"><a href="#对切片进行切片" class="headerlink" title="对切片进行切片"></a>对切片进行切片</h2><p>我们之所以称呼切片为切片,是因为你可以通过对底层数组的一部分进行切片来创建一个新的切片。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/* Create a slice of integers. Contains a </span><br><span class="line">length and capacity of 5 elements.*/</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>, <span class="number">50</span>}</span><br><span class="line">fmt.Println(slice) <span class="comment">// Print [10 20 30 40 50]</span></span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) <span class="comment">// Print 5</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) <span class="comment">// Print 5</span></span><br><span class="line"><span class="comment">/* Create a new slice.Contains a length </span><br><span class="line">of 2 and capacity of 4 elements.*/</span></span><br><span class="line">newSlice := slice[<span class="number">1</span>:<span class="number">3</span>]</span><br><span class="line">fmt.Println(slice) <span class="comment">//Print [10 20 30 40 50]</span></span><br><span class="line">fmt.Println(<span class="built_in">len</span>(newSlice)) <span class="comment">//Print 2</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(newSlice)) <span class="comment">//Print 4</span></span><br></pre></td></tr></table></figure></p>
<p><img src="/images/go/slice_9.png" alt="Take a slice of a slice"></p>
<p>在执行切片操作之后,我们拥有两个共享同一底层数组的切片。然而,这两个切片以不同的方式查看底层数组。原始切片认为底层数组的容量为 5,但 newSlice 与之不同,对 newSlice 而言,底层数组的容量为 4。newSlice 无法访问位于其指针之前的底层数组元素。就 newSlice 而言,这些元素甚至并不存在。使用下面的方式可以为任意切片后的 newSlice 计算长度和容量。</p>
<h3 id="切片的长度与容量如何计算?"><a href="#切片的长度与容量如何计算?" class="headerlink" title="切片的长度与容量如何计算?"></a>切片的长度与容量如何计算?</h3><blockquote>
<p>切片 slice[i:j] 的底层数组容量为 k 长度(Length):j - i 容量(Capacity):k - i</p>
</blockquote>
<h3 id="计算新的长度和容量"><a href="#计算新的长度和容量" class="headerlink" title="计算新的长度和容量"></a>计算新的长度和容量</h3><blockquote>
<p>切片 slice[1:3] 的底层数组容量为 5 长度(Length):3 - 1 = 2 容量(Capacity):5 - 1 = 4</p>
</blockquote>
<h3 id="对一个切片进行更改的结果"><a href="#对一个切片进行更改的结果" class="headerlink" title="对一个切片进行更改的结果"></a>对一个切片进行更改的结果</h3><p>一个切片对底层数组的共享部分所做的更改可以被另一个切片看到。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 5 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>, <span class="number">50</span>}</span><br><span class="line"><span class="comment">// Create a new slice.</span></span><br><span class="line"><span class="comment">// Contains a length of 2 and capacity of 4 elements.</span></span><br><span class="line">newSlice := slice[<span class="number">1</span>:<span class="number">3</span>]</span><br><span class="line"><span class="comment">// Change index 1 of newSlice.</span></span><br><span class="line"><span class="comment">// Change index 2 of the original slice.</span></span><br><span class="line">newSlice[<span class="number">1</span>] = <span class="number">35</span></span><br></pre></td></tr></table></figure></p>
<p>将数值 35 分配给 newSlice 的第二个元素后,该更改也可以在原始切片的元素中被看到。</p>
<h2 id="运行时错误显示索引超出范围"><a href="#运行时错误显示索引超出范围" class="headerlink" title="运行时错误显示索引超出范围"></a>运行时错误显示索引超出范围</h2><p>一个切片只能访问它长度以内的索引位。尝试访问超出长度的索引位元素将引发一个运行时错误。与切片容量相关联的元素只能用于切片增长。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 5 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>, <span class="number">50</span>}</span><br><span class="line"><span class="comment">// Create a new slice.</span></span><br><span class="line"><span class="comment">// Contains a length of 2 and capacity of 4 elements.</span></span><br><span class="line">newSlice := slice[<span class="number">1</span>:<span class="number">3</span>]</span><br><span class="line"><span class="comment">// Change index 3 of newSlice.</span></span><br><span class="line"><span class="comment">// This element does not exist for newSlice.</span></span><br><span class="line">newSlice[<span class="number">3</span>] = <span class="number">45</span></span><br><span class="line"></span><br><span class="line"><span class="comment">/*</span><br><span class="line">Runtime Exception:</span><br><span class="line">panic: runtime error: index out of range</span><br><span class="line">*/</span></span><br></pre></td></tr></table></figure></p>
<h2 id="切片增长"><a href="#切片增长" class="headerlink" title="切片增长"></a>切片增长</h2><p>与使用数组相比,使用切片的优势之一是:你可以根据需要增加切片的容量。当你使用内置函数 「append」 时,Golang 会负责处理所有操作细节。</p>
<ul>
<li>使用 append 前,你需要一个源切片和一个要追加的值。</li>
<li>当你的 append 调用并返回时,它将为你提供一个更改后的新切片。</li>
<li><strong>append</strong> 函数总会增加新切片的长度。</li>
<li>另一方面,容量可能会受到影响,也可能不会受到影响,这取决于源切片的可用容量。</li>
</ul>
<h2 id="使用-append-向切片追加元素"><a href="#使用-append-向切片追加元素" class="headerlink" title="使用 append 向切片追加元素"></a>使用 append 向切片追加元素</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/* Create a slice of integers.</span><br><span class="line"> Contains a length and capacity of 5 elements.*/</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>, <span class="number">50</span>}</span><br><span class="line"></span><br><span class="line"><span class="comment">/* Create a new slice.</span><br><span class="line"> Contains a length of 2 and capacity of 4 elements.*/</span></span><br><span class="line">newSlice := slice[<span class="number">1</span>:<span class="number">3</span>]</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(newSlice)) <span class="comment">// Print 2</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(newSlice)) <span class="comment">// Print 4</span></span><br><span class="line"></span><br><span class="line"><span class="comment">/* Allocate a new element from capacity.</span><br><span class="line"> Assign the value of 60 to the new element.*/</span></span><br><span class="line">newSlice = <span class="built_in">append</span>(newSlice, <span class="number">60</span>)</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(newSlice)) <span class="comment">// Print 3</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(newSlice)) <span class="comment">// Print 4</span></span><br></pre></td></tr></table></figure>
<p>当切片的底层数组没有可用容量时,append 函数将创建一个新的底层数组,拷贝正在引用的现有值,然后再分配新值。</p>
<h2 id="使用-append-增加切片的长度和容量"><a href="#使用-append-增加切片的长度和容量" class="headerlink" title="使用 append 增加切片的长度和容量"></a>使用 append 增加切片的长度和容量</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 4 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(slice)) <span class="comment">// Print 4</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(slice)) <span class="comment">// Print 4</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// Append a new value to the slice.</span></span><br><span class="line"><span class="comment">// Assign the value of 50 to the new element.</span></span><br><span class="line">newSlice= <span class="built_in">append</span>(slice, <span class="number">50</span>)</span><br><span class="line">fmt.Println(<span class="built_in">len</span>(newSlice)) <span class="comment">//Print 5</span></span><br><span class="line">fmt.Println(<span class="built_in">cap</span>(newSlice)) <span class="comment">//Print 8</span></span><br></pre></td></tr></table></figure>
<p><img src="/images/go/slice_9.png" alt="Increase the length and capacity of a slice"></p>
<p>在 append 操作后,newSlice 被给予一个自有的底层数组,该底层数组的容量是原底层数组容量的两倍。在增加底层数组容量时,append 操作十分聪明。举个例子,当切片的容量低于 1,000 个元素时,容量增长总是翻倍的。一旦元素的数量超过 1,000 个,容量就会增长 1.25 倍,即 25%。随着时间的推移,这种增长算法可能会在 Golang 中发生变化。</p>
<p>更改新切片不会对旧切片产生任何影响,因为新切片现在有一个不同的底层数组,它的指针指向一个新分配的数组。</p>
<h2 id="将一个切片追加到另一个切片中"><a href="#将一个切片追加到另一个切片中" class="headerlink" title="将一个切片追加到另一个切片中"></a>将一个切片追加到另一个切片中</h2><p>内置函数 <strong>append</strong> 还是一个<strong>可变参数</strong>函数。这意味着你可以传递多个值来追加到单个切片中。如果你使用 … 运算符,可以将一个切片的所有元素追加到另一个切片中。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create two slices each initialized with two integers.</span></span><br><span class="line">slice1:= []<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>}</span><br><span class="line">slice2 := []<span class="keyword">int</span>{<span class="number">3</span>, <span class="number">4</span>}</span><br><span class="line"><span class="comment">// Append the two slices together and display the results.</span></span><br><span class="line">fmt.Println(<span class="built_in">append</span>(slice1, slice2...))</span><br><span class="line"><span class="comment">//Output: [1 2 3 4]</span></span><br></pre></td></tr></table></figure></p>
<h2 id="对切片执行索引"><a href="#对切片执行索引" class="headerlink" title="对切片执行索引"></a>对切片执行索引</h2><ul>
<li>通过指定一个下限和一个上限来形成切片,例如:<code>a[low:high]</code>。这将选择一个半开范围,其中包含切片的第一个元素,但不包含切片的最后一个元素。</li>
<li>你可以省略上限或下限,这将使用它们的默认值。下限的默认值是 0,上限的默认值是切片的长度。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">a := [...]<span class="keyword">int</span>{<span class="number">0</span>, <span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>} </span><br><span class="line"><span class="comment">// an array</span></span><br><span class="line">s := a[<span class="number">1</span>:<span class="number">3</span>] </span><br><span class="line"><span class="comment">// s == []int{1, 2} </span></span><br><span class="line"><span class="comment">// cap(s) == 3</span></span><br><span class="line">s = a[:<span class="number">2</span>] </span><br><span class="line"><span class="comment">// s == []int{0, 1} </span></span><br><span class="line"><span class="comment">// cap(s) == 4</span></span><br><span class="line">s = a[<span class="number">2</span>:] </span><br><span class="line"><span class="comment">// s == []int{2, 3} </span></span><br><span class="line"><span class="comment">// cap(s) == 2</span></span><br><span class="line">s = a[:] </span><br><span class="line"><span class="comment">// s == []int{0, 1, 2, 3} </span></span><br><span class="line"><span class="comment">// cap(s) == 4</span></span><br></pre></td></tr></table></figure>
<h2 id="遍历切片"><a href="#遍历切片" class="headerlink" title="遍历切片"></a>遍历切片</h2><p>Go 有一个特殊的关键字 <code>range</code>,你可以使用该关键字对切片进行遍历。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 4 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line"><span class="comment">// Iterate over each element and display each value.</span></span><br><span class="line"><span class="keyword">for</span> index, value := <span class="keyword">range</span> slice {</span><br><span class="line"> fmt.Printf(<span class="string">"Index: %d Value: %d\n"</span>, index, value)</span><br><span class="line">}</span><br><span class="line"><span class="comment">/*</span><br><span class="line">Output:</span><br><span class="line">Index: 0 Value: 10</span><br><span class="line">Index: 1 Value: 20</span><br><span class="line">Index: 2 Value: 30</span><br><span class="line">Index: 3 Value: 40</span><br><span class="line">*/</span></span><br></pre></td></tr></table></figure></p>
<ul>
<li>在遍历切片时,关键字 range 将返回两个值。</li>
<li>第一个值是索引下标,第二个值是索引位中值的副本。</li>
<li>一定要知道 range 是在复制值,而不是返回值的引用。</li>
</ul>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/*</span><br><span class="line"> Create a slice of integers.Contains </span><br><span class="line"> a length and capacity of 4 elements.</span><br><span class="line">*/</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line"><span class="comment">/*</span><br><span class="line"> Iterate over each element and display </span><br><span class="line"> the value and addresses.</span><br><span class="line">*/</span></span><br><span class="line"><span class="keyword">for</span> index, value := <span class="keyword">range</span> slice {</span><br><span class="line"> fmt.Printf(<span class="string">"Value: %d Value-Addr: %X ElemAddr: %X\n"</span>,</span><br><span class="line"> value, &value, &slice[index])</span><br><span class="line">}</span><br><span class="line"><span class="comment">/*</span><br><span class="line">Output:</span><br><span class="line">Value: 10 Value-Addr: 10500168 ElemAddr: 1052E100</span><br><span class="line">Value: 20 Value-Addr: 10500168 ElemAddr: 1052E104</span><br><span class="line">Value: 30 Value-Addr: 10500168 ElemAddr: 1052E108</span><br><span class="line">Value: 40 Value-Addr: 10500168 ElemAddr: 1052E10C</span><br><span class="line">*/</span></span><br></pre></td></tr></table></figure>
<p><strong>range</strong> 关键字提供元素的拷贝。</p>
<p>如果你不需要下标值,你可以使用下划线字符丢弃该值。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 4 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line"><span class="comment">// Iterate over each element and display each value.</span></span><br><span class="line"><span class="keyword">for</span> _, value := <span class="keyword">range</span> slice {</span><br><span class="line"> fmt.Printf(<span class="string">"Value: %d\n"</span>, value)</span><br><span class="line">}</span><br><span class="line"><span class="comment">/*</span><br><span class="line">Output:</span><br><span class="line">Value: 10</span><br><span class="line">Value: 20</span><br><span class="line">Value: 30</span><br><span class="line">Value: 40</span><br><span class="line">*/</span></span><br></pre></td></tr></table></figure></p>
<p>关键字 <strong>range</strong> 总是从开始处遍历一个切片。如果你需要对切片的迭代进行更多的控制,你可以使用传统的 <strong>for</strong> 循环。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Create a slice of integers.</span></span><br><span class="line"><span class="comment">// Contains a length and capacity of 4 elements.</span></span><br><span class="line">slice := []<span class="keyword">int</span>{<span class="number">10</span>, <span class="number">20</span>, <span class="number">30</span>, <span class="number">40</span>}</span><br><span class="line"><span class="comment">// Iterate over each element starting at element 3.</span></span><br><span class="line"><span class="keyword">for</span> index := <span class="number">2</span>; index < <span class="built_in">len</span>(slice); index++ {</span><br><span class="line"> fmt.Printf(<span class="string">"Index: %d Value: %d\n"</span>, index, slice[index])</span><br><span class="line">}</span><br><span class="line"><span class="comment">/* </span><br><span class="line">Output:</span><br><span class="line">Index: 2 Value: 30</span><br><span class="line">Index: 3 Value: 40</span><br><span class="line">*/</span></span><br></pre></td></tr></table></figure></p>
<p>##总结<br>在本文中,我们深入探讨了切片的概念。我们了解到,切片并不存储任何数据,而是描述了底层数组的一部分。我们还看到,切片可以在底层数组的范围内增长和收缩,并配合索引可作为数组使用;切片的零值是 nil;函数 <strong>len</strong>、<strong>cap</strong> 和 <strong>append</strong> 都将 nil 看作一个长度和容量都为 0 的<strong>空切片</strong>;你可以通过<strong>切片字面量</strong>或调用 <strong>make</strong> 函数(将长度和容量作为参数)来创建切片。希望这些对你有所帮助!</p>
<h2 id="免责声明"><a href="#免责声明" class="headerlink" title="免责声明"></a>免责声明</h2><p>我参考了各种博客、书籍和媒体故事来撰写这篇文章。如有任何疑问,请在评论中与我联系。</p>
<p>到此为止……开心编码……快乐学习😃</p>
<blockquote>
<p>译自:掘金翻译计划 原文:<a href="https://codeburst.io/a-comprehensive-guide-to-slices-in-golang-bacebfe46669" target="_blank" rel="external">https://codeburst.io/a-comprehensive-guide-to-slices-in-golang-bacebfe46669</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在本文中,我们将讨论 “切片” 的概念,它是 Golang 中使用的一种重要数据结构。这一数据结构为你提供了处理与管理数据集合的方法。切片是围绕动态数组的概念构建的,它与动态数组相似,可以根据你的需要而伸缩。</p>
<ul>
<li>切片在增长方面是动态的,因为它们有自己的内置函数 <code>append</code>,可以快速高效地增长切片。</li>
<li>您还可以通过切割底层内存来减小切片的大小。</li>
<li>在底层内存中切片是在连续的块上分配的,因此切片为你提供的便利之处包括:索引、迭代与垃圾回收优化。
Cilium:基于 BPF/XDP 实现 K8s Service 负载均衡
http://team.jiunile.com//blog/2020/11/k8s-cilium-service.html
2020-11-25T14:00:00.000Z
2020-11-25T06:00:19.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>文章介绍了 K8s 的一些核心网络模型和设计、<code>Cilium</code> 对 <code>K8s Service</code> 的实现、<code>BPF/XDP</code> 性能优化,以及他们从中得到的一些实践经验,全是干货。</p>
<p>去年我们也参加了这个大会(LPC),并做了题为 <a href="https://linuxplumbersconf.org/event/4/contributions/458/" target="_blank" rel="external">Making the Kubernetes Service Abstraction Scale using eBPF</a> 的分享。 今天的内容是去年内容的延续,具体分为三个部分:</p>
<ul>
<li>Kubernetes 网络模型</li>
<li><code>Cilium</code> 对 <code>K8s Service</code> 负载均衡的实现,以及我们的一些实践经验</li>
<li>一些新的 <code>BPF</code> 内核扩展</li>
</ul>
<a id="more"></a>
<h2 id="1-K8s-网络基础:访问集群内服务的几种方式"><a href="#1-K8s-网络基础:访问集群内服务的几种方式" class="headerlink" title="1 K8s 网络基础:访问集群内服务的几种方式"></a>1 K8s 网络基础:访问集群内服务的几种方式</h2><p>Kubernetes 是一个分布式容器调度器,最小调度单位是 Pod。从网络的角度来说,可以认为 一个 pod 就是<strong>网络命名空间的一个实例</strong>(an instance of network namespace)。 一个 pod 内可能会有多个容器,因此,<strong>多个容器可以共存于同一个网络命名空间</strong>。</p>
<p>需要注意的是:<strong>K8s 只定义了网络模型,具体实现则是交给所谓的 CNI 插件</strong>,后者完成 pod 网络的创建和销毁。本文接下来将以 <code>Cilium CNI</code> 插件作为例子。</p>
<p>K8s 规定了<strong>每个 pod 的 IP 在集群内要能访问</strong>,这是通过 CNI 来完成的:CNI 插件负责为 pod 分配 IP 地址,然后为其创建和打通网络。 <strong>除此之外,K8s 没有对 CNI 插件做任何限制</strong>。尤其是,K8s 没有对<strong>从集群外访问 pod 的行为做任何规定</strong>。</p>
<p>接下来我们就来看看如何访问 K8s 集群里的一个<strong>服务</strong>(通常会对应多个 backend pods)。</p>
<h3 id="1-1-PodIP(直连容器-IP)"><a href="#1-1-PodIP(直连容器-IP)" class="headerlink" title="1.1 PodIP(直连容器 IP)"></a>1.1 PodIP(直连容器 IP)</h3><p>第一种方式是<strong>通过 PodIP 直接访问</strong>,这是最简单的方式。<br><img src="/images/k8s/cilium_pod-ip.png" alt="pod-ip"></p>
<p>如上图所示,这个服务的 3 个 backend pods 分别位于两个 node 上。当集群外的客户端 访问这个服务时,它会<strong>直接通过某个具体的 PodIP 来访问</strong>。</p>
<p>假设客户端和 Pod 之间的网络是可达的,那这种访问是没问题的。</p>
<p>但这种方式有几个<strong>缺点</strong>:</p>
<ol>
<li>pod 会因为某些原因重建,而 K8s <strong>无法保证它每次都会分到同一个 IP 地址</strong>。例如,如果 node 重启了,pod 很可能就会分到不同的 IP 地址,这对客户端来说个 大麻烦。</li>
<li><strong>没有内置的负载均衡</strong>。即,客户端选择一个 PodIP 后,所有的请求都会发送到这个 pod,而不是分散到不同的后端 pod。</li>
</ol>
<h3 id="1-2-HostPort(宿主机端口映射)"><a href="#1-2-HostPort(宿主机端口映射)" class="headerlink" title="1.2 HostPort(宿主机端口映射)"></a>1.2 HostPort(宿主机端口映射)</h3><p>第二种方式是使用所谓的 HostPort。<br><img src="/images/k8s/cilium_host-port.png" alt="host-port"></p>
<p>如上图所示,<strong>在宿主机的 netns 分配一个端口</strong>,并将这个端口的所有流量转发到 后端 pod。</p>
<p>这种情况下,</p>
<ol>
<li>客户端通过 Pod 所在的宿主机的 <code>HostIP:HostPort</code> 访问服务,例如上图中访问 <code>10.0.0.1:10000</code>;</li>
<li>宿主机先对<strong>流量进行 DNAT</strong>,然后转发给 Pod。</li>
</ol>
<p>这种方式的<strong>缺点</strong>:</p>
<ol>
<li>宿主机的端口资源是所有 Pod 共享的,任何一个端口只能被一个 pod 使用 ,因此<strong>在每台 node 上,任何一个服务最多只能有一个 pod</strong>(每个 backend 都是一 致的,因此需要使用相同的 HostPort)。对用户非常不友好。</li>
<li>和 PodIP 方式一样,没有内置的负载均衡。</li>
</ol>
<h3 id="1-3-NodePort-Service"><a href="#1-3-NodePort-Service" class="headerlink" title="1.3 NodePort Service"></a>1.3 NodePort Service</h3><p>NodePort 和上面的 HostPort 有点像(可以认为是 HostPort 的增强版),也是将 Pod 暴 露到宿主机 netns 的某个端口,但此时,<strong>集群内的每个 Node 上都会为这个服务的 pods 预留这个端口,并且将流量负载均衡到这些 pods</strong>。</p>
<p>如下图所示,假设这里的 NodePort 是 <code>30001</code>。当客户端请求到达任意一台 node 的 <code>30001</code> 端口时,它可以对请求做 DNAT 然后转发给本节点内的 Pod,如下图所示:<br><img src="/images/k8s/cilium_node-port.png" alt="node-port"></p>
<p>也可以 DNAT 之后将请求转发给其他节点上的 pod,如下图所示:<br><img src="/images/k8s/cilium_node-port-2.png" alt="node-port"></p>
<p>注意在后面跨宿主机转发的情况下,<strong>除了做 DNAT 还需要做 SNAT</strong>。</p>
<p><strong>优点:</strong></p>
<ol>
<li><strong>已经有了服务(service)的概念</strong>,多个 pod 属于同一个 service,挂掉一个时其 他 pod 还能继续提供服务。</li>
<li><strong>客户端不用关心 pod 在哪个 node 上</strong>,因为集群内的所有 node 上都开了这个端 口并监听在那里,它们对全局的 backend 有一致的视图。</li>
<li>已经<strong>有了负载均衡,每个 node 都是 LB</strong>。</li>
<li>在宿主机 netns 内访问这些服务时,通过 <code>localhost:NodePort</code> 就行了,无需 DNS 解析。</li>
</ol>
<p><strong>缺点:</strong></p>
<ol>
<li><strong>大部分实现都是基于 SNAT</strong>,当 pod 不在本节点时,导致 packet 中的<strong>真实客户端 IP 地址</strong>信息丢失,监控、排障等不方便。</li>
<li>Node 做转发使得<strong>转发路径多了一跳,延时变大</strong>。</li>
</ol>
<h3 id="1-4-ExternalIPs-Service"><a href="#1-4-ExternalIPs-Service" class="headerlink" title="1.4 ExternalIPs Service"></a>1.4 ExternalIPs Service</h3><p>第四种从集群外访问 service 的方式是 <code>external IP</code>。</p>
<p>如果有外部可达的 IP ,即<strong>集群外能通过这个 IP 访问到集群内特定的 nodes</strong>,那我 们就可以通过这些 nodes 将流量转发到 service 的后端 pods,并提供负载均衡。</p>
<p>如下图所示,<code>1.1.1.1</code> 是一个 <code>external IP</code>,所有目的 IP 地址是 <code>1.1.1.1</code> 的流量会被底层的网络(K8s 控制之外)转发到 node1。<code>1.1.1.1:8080</code> 在 K8s 里定义了一个 Service,如果它将流量转发到本机内的 backend pod,需要做一次 DNAT:<br><img src="/images/k8s/cilium_external-ip.png" alt="external-ip"></p>
<p>同样,这里的后端 Pod 也可以在其他 node 上,这时除了做 DNAT 还需要做一次 SNAT, 如下图所示:<br><img src="/images/k8s/cilium_external-ip-2.png" alt="external-ip"></p>
<p><strong>优点:可以使用任何外部可达的 IP 地址来定义 Service 入口</strong>,只要用这个 IP 地址能访问集群内的至少一台机器即可。</p>
<p><strong>缺点:</strong></p>
<ol>
<li><strong>External IP 在 k8s 的控制范围之外</strong>,是由底层的网络平台提供的。例如,底层网 络通过 BGP 宣告,使得 IP 能到达某些 nodes。</li>
<li>由于这个 IP 是在 k8s 的控制之外,对 k8s 来说就是黑盒,因此<strong>从集群内访问 external IP 是存在安全隐患的</strong>,例如 <code>external IP</code> 上可能运行了 恶意服务,能够进行中间人攻击。因此,<code>Cilium</code> 目前不支持在集群内通过 <code>external IP</code> 访问 Service。</li>
</ol>
<h3 id="1-5-LoadBalancer-Service"><a href="#1-5-LoadBalancer-Service" class="headerlink" title="1.5 LoadBalancer Service"></a>1.5 LoadBalancer Service</h3><p>第五种访问方式是所谓的 LoadBalancer 模式。针对公有云还是私有云,LoadBalancer 又分为两种。</p>
<h4 id="1-5-1-私有云"><a href="#1-5-1-私有云" class="headerlink" title="1.5.1 私有云"></a>1.5.1 私有云</h4><p>如果是私有云,可以考虑实现一个自己的 <code>cloud provider</code>,或者直接使用 <a href="https://github.com/metallb/metallb" target="_blank" rel="external">MetalLB</a>。</p>
<p>如下图所示,<strong>这种模式和 externalIPs 模式非常相似</strong>,local 转发:<br><img src="/images/k8s/cilium_load-balancer.png" alt="load-balancer"></p>
<p>remote 转发:<br><img src="/images/k8s/cilium_load-balancer-2.png" alt="load-balancer"></p>
<p>但是,二者有重要区别:</p>
<ol>
<li><strong>externalIPs 在 K8s 的控制之外</strong>,使用方式是从某个地方申请一个 external IP, 然后填到 Service 的 Spec 里;这个 <code>external IP</code> 是存在安全隐患的,因为并不是 K8s 分配和控制的;</li>
<li><strong>LoadBalancer 在 K8s 的控制之内</strong>,只需要声明 这是一个 LoadBalancer 类型的 Service,K8s 的 <code>cloud-provider</code> 组件就会自动给这个 Service 分配一个外部可达的 IP,本质上 <code>cloud-provider</code> 做的事情就是从某个 LB 分配一个受信任的 VIP 然后填到 Service 的 Spec 里。</li>
</ol>
<p><strong>优点</strong>:LoadBalancer 分配的 IP 是归 K8s 管的,<strong>用户无法直接配置这些 IP</strong>,因 此也就避免了前面 <code>external IP</code> 的流量欺骗(traffic spoofing)风险。</p>
<p>但<strong>注意这些 IP 不是由 CNI 分配的,而是由 LoadBalancer 实现分配</strong>。</p>
<p><a href="https://github.com/metallb/metallb" target="_blank" rel="external">MetalLB</a> 能完成 LoadBalancer IP 的分配,然后<strong>基于 ARP/NDP 或 BGP 宣告 IP 的可达性</strong>。 此外,<strong>MetalLB 本身并不在 critical fast path</strong> 上(可以认为它只是控制平面,完成 LoadBalancer IP 的生效,接下来的请求和响应流量,即数据平面,都不经过它),因此不 影响 XDP 的使用。</p>
<h4 id="1-5-2-公有云"><a href="#1-5-2-公有云" class="headerlink" title="1.5.2 公有云"></a>1.5.2 公有云</h4><p>主流的云厂商都实现了 LoadBalancer,在它们提供的托管 K8s 内可以直接使用。</p>
<p>特点:</p>
<ol>
<li>有专门的 LB 节点作为统一入口。</li>
<li>LB 节点再将流量转发到 NodePort。</li>
<li>NodePort 再将流量转发到 backend pods。</li>
</ol>
<p>如下图所示,local 转发:<br><img src="/images/k8s/cilium_load-balancer-cloud.png" alt="load-balancer-cloud"></p>
<p>remote 转发:<br><img src="/images/k8s/cilium_load-balancer-cloud-2.png" alt="load-balancer-cloud"></p>
<p><strong>优点:</strong></p>
<ol>
<li>LoadBalancer 由云厂商实现,无需用户安装 BGP 软件、配置 BGP 协议等来宣告 VIP 可达性。</li>
<li>开箱即用,主流云厂商都针对它们的托管 K8s 集群实现了这样的功能。</li>
</ol>
<p>在这种情况下,<strong>Cloud LB 负责检测后端 node(注意不是后端 pod)的健康状态。</strong></p>
<p><strong>缺点:</strong></p>
<ol>
<li>存在两层 LB:LB 节点转发和 node 转发。</li>
<li>使用方式因厂商而已,例如各厂商的 annotations 并没有标准化到 K8s 中,跨云使用会有一些麻烦。</li>
<li><strong>Cloud API 非常慢</strong>,调用厂商的 API 来做拉入拉出非常受影响。</li>
</ol>
<h3 id="1-6-ClusterIP-Service"><a href="#1-6-ClusterIP-Service" class="headerlink" title="1.6 ClusterIP Service"></a>1.6 ClusterIP Service</h3><p>最后一种是<strong>集群内访问 Service 的方式</strong>:ClusterIP 方式。<br><img src="/images/k8s/cilium_cluster-ip.png" alt="cluster-ip"></p>
<p>ClusterIP 也是 Service 的一种 VIP,但这种方式只适用于从集群内访问 Service,例如 从一个 Pod 访问相同集群内的一个 Service。</p>
<p>ClusterIP 的特点:</p>
<ol>
<li>ClusterIP 使用的 IP 地址段是<strong>在创建 K8s 集群之前就预留好的</strong>;</li>
<li>ClusterIP <strong>不可路由</strong>(会在出宿主机之前被拦截,然后 DNAT 成具体的 PodIP);</li>
<li><strong>只能在集群内访问</strong>(For in-cluster access only)。</li>
</ol>
<p>实际上,<strong>当创建一个 LoadBalancer 类型的 Service 时,K8s 会为我们自动创建三种类 型的 Service</strong>:</p>
<ol>
<li>LoadBalancer</li>
<li>NodePort</li>
<li>ClusterIP</li>
</ol>
<p>这三种类型的 Service 对应着同一组 backend pods。</p>
<p>我们此次分享的第一部分,K8s 网络基础至此就要结束了,实际上还有很多与 Service 相 关的 K8s 特性,例如 <code>sessionAffinity</code> 和 <code>externalTrafficPolicy</code>,但这里就不展开了,有兴趣可以参考附录。</p>
<h2 id="2-K8s-Service-负载均衡:Cilium-基于-BPF-XDP-的实现"><a href="#2-K8s-Service-负载均衡:Cilium-基于-BPF-XDP-的实现" class="headerlink" title="2 K8s Service 负载均衡:Cilium 基于 BPF/XDP 的实现"></a>2 K8s Service 负载均衡:Cilium 基于 BPF/XDP 的实现</h2><p><strong>Cilium 基于 eBPF/XDP 实现了前面提到的所有类型的 K8s Service</strong>。实现方式是:</p>
<ol>
<li>在每个 node 上运行一个 <code>cilium-agent</code>;</li>
<li><code>cilium-agent</code> 监听 <code>K8s apiserver</code>,因此能够感知到 K8s 里 Service 的变化;</li>
<li>根据 Service 的变化动态更新 BPF 配置。</li>
</ol>
<p><img src="/images/k8s/cilium_bpf-lb-layers.png" alt="bpf-lb-layers"></p>
<p>如上图所示,Service 的实现由两个主要部分组成:</p>
<ol>
<li>运行在 socket 层的 BPF 程序</li>
<li>运行在 tc/XDP 层的 BPF 程序</li>
</ol>
<p>以上两者共享 <code>service map</code> 等资源,其中存储了 service 及其 backend pods 的映射关系。</p>
<h3 id="2-1-Socket-层负载均衡(东西向流量)"><a href="#2-1-Socket-层负载均衡(东西向流量)" class="headerlink" title="2.1 Socket 层负载均衡(东西向流量)"></a>2.1 Socket 层负载均衡(东西向流量)</h3><p>Socket 层 BPF 负载均衡负责处理<strong>集群内的东西向流量</strong>。</p>
<h4 id="实现"><a href="#实现" class="headerlink" title="实现"></a>实现</h4><p>实现方式是:<strong>将 BPF 程序 attach 到 socket 的系统调用 hooks,使客户端直接和后端 pod 建连和通信</strong>,如下图所示,这里能 hook 的系统调用包括 <code>connect()</code>、<code>sendmsg()</code>、 <code>recvmsg()</code>、<code>getpeername()</code>、<code>bind()</code> 等,<br><img src="/images/k8s/cilium_e-w-lb.png" alt="e-w-lb"></p>
<p>这里的一个问题是,<strong>K8s 使用的还是 cgroup v1,但这个功能需要使用 v2</strong>,而由于 兼容性问题,v2 完全替换 v1 还需要很长时间。所以我们目前所能做的就是 支持 v1 和 v2 的混合模式。这也是为什么 <code>Cilium</code> 会 mount 自己的 <code>cgroup v2 instance</code> 的原因。</p>
<blockquote>
<p>Cilium mounts cgroup v2, attaches BPF to root cgroup. Hybrid use works well for root v2.</p>
</blockquote>
<p>具体到实现上,</p>
<ul>
<li><code>connect + sendmsg</code> 做<strong>正向</strong>变换(translation)</li>
<li><code>recvmsg + getpeername</code> 做<strong>反向</strong>变换,</li>
</ul>
<p>这个变换或转换是<strong>基于 socket structure 的,此时还没有创建 packet</strong>,因此<strong>不存在 packet 级别的 NAT!</strong>目前已经支持 TCP/UDP v4/v6, v4-in-v6。<strong>应用对此是无感知的,它以为自己连接到的还是 Service IP,但其实是 PodIP</strong>。</p>
<h4 id="查找后端-pods"><a href="#查找后端-pods" class="headerlink" title="查找后端 pods"></a>查找后端 pods</h4><p>Service lookup <strong>不一定能选到所有的 backend pods</strong>(scoped lookup),我们将 backend pods 拆成不同的集合。</p>
<p><strong>这样设计的好处</strong>:可以根据<strong>流量类型</strong>,例如是来自集群内还是集群外( internal/external),<strong>来选择不同的 backends</strong>。例如,如果是到达 node 的 external traffic,我们可以限制它只能选择本机上的 backend pods,这样相比于转发到其他 node 上的 backend 就少了一跳。</p>
<p>另外,还支持通配符(wildcard)匹配,这样就能将 Service 暴露到 localhost 或者 loopback 地址,能在宿主机 netns 访问 Service。但这种方式不会将 Service 暴露到宿 主机外面。</p>
<h4 id="好处"><a href="#好处" class="headerlink" title="好处"></a>好处</h4><p>显然,这种 <strong>socket 级别的转换是非常高效和实用的</strong>,它可以直接将客户端 pod 连 接到某个 backend pod,与 kube-proxy 这样的实现相比,转发路径少了好几跳。</p>
<p>此外,<code>bind</code> BPF 程序在 NodePort 冲突时会<strong>直接拒绝应用的请求</strong>,因此相比产生流 量(packet)然后在后面的协议栈中被拒绝,bind 这里要更加高效,<strong>因为此时 流量(packet)都还没有产生</strong>。</p>
<p>对这一功能至关重要的两个函数:</p>
<ul>
<li><p><code>bpf_get_socket_cookie()</code><br>主要用于 UDP sockets,我们希望每个 UDP flow 都能选中相同的 backend pods。</p>
</li>
<li><p><code>bpf_get_netns_cookie()</code><br>用在两个地方:</p>
<ul>
<li>用于区分 <code>host netns</code> 和 <code>pod netns</code>,例如检测到在 <code>host netns</code> 执行 bind 时,直接拒绝(reject);</li>
<li>用于 <code>serviceSessionAffinity</code>,实现在某段时间内永远选择相同的 backend pods。</li>
</ul>
</li>
</ul>
<p>由于 <code>cgroup v2 不感知 netns</code>,因此在这个 context 中我们没用 Pod 源 IP 信 息,通过这个 helper 能让它感知到源 IP,并以此作为它的 <code>source identifier</code>。</p>
<h3 id="2-2-TC-amp-XDP-层负载均衡(南北向流量)"><a href="#2-2-TC-amp-XDP-层负载均衡(南北向流量)" class="headerlink" title="2.2 TC & XDP 层负载均衡(南北向流量)"></a>2.2 TC & XDP 层负载均衡(南北向流量)</h3><p>第二种是进出集群的流量,称为南北向流量,在宿主机 tc 或 XDP hook 里处理。<br><img src="/images/k8s/cilium_n-s-lb.png" alt="n-s-lb"></p>
<p>BPF 做的事情,将入向流量转发到后端 Pod,</p>
<ol>
<li>如果 Pod 在本节点,做 DNAT;</li>
<li>如果在其他节点,还需要做 SNAT 或者 DSR。</li>
</ol>
<p><strong>这些都是 packet 级别的操作</strong>。</p>
<h3 id="2-3-XDP-相关优化"><a href="#2-3-XDP-相关优化" class="headerlink" title="2.3 XDP 相关优化"></a>2.3 XDP 相关优化</h3><p>在引入 XDP 支持时,为了使 context 的抽象更加通用,我们做了很多事情。下面就其中的 一些展开讨论。</p>
<h4 id="BPF-XDP-context-通用化"><a href="#BPF-XDP-context-通用化" class="headerlink" title="BPF/XDP context 通用化"></a>BPF/XDP context 通用化</h4><p>DNAT/SNAT engine, DSR, conntrack 等等都是在 tc BPF 里实现的。 BPF 代码中用 context 结构体传递数据包信息。</p>
<p>支持 XDP 时遇到的一个问题是:到底是将 context 抽象地更通用一些,还是直接实现一个 支持 XDP 的最小子集。我们最后是花大力气重构了以前几乎所有的 BPF 代码,来使得它更 加通用。好处是共用一套代码,这样对代码的优化同时适用于 TC 和 XDP 逻辑。</p>
<p>下面是一个具体例子:</p>
<p><code>ctx</code> 是一个通用抽象,具体是什么类型和 include 的头文件有关,基于 cxt 可以同时处 理 tc BPF 和 XDP BPF 逻辑,<br><img src="/images/k8s/cilium_generic-code.png" alt="generic-code"></p>
<p>例如对于 XDP 场景,编译时这些宏会被相应的 XDP 实现替换掉:<br><img src="/images/k8s/cilium_context-specific-code.png" alt="context-specific-code"></p>
<h4 id="内联汇编:绕过编译器自动优化"><a href="#内联汇编:绕过编译器自动优化" class="headerlink" title="内联汇编:绕过编译器自动优化"></a>内联汇编:绕过编译器自动优化</h4><p>我们遇到的另一个问题是:tc BPF 中已经为 skb 实现了很多的 helper 函数,由于共用一 套抽象,因此现在需要为 XDP 实现对应的一套函数集。这些 helpers 都是 inline 函数, 而 LLVM 会对 inline 函数的自动优化会导致接下来校验器(BPF verifier)失败。</p>
<p>我们的解决方式是用 <strong>inline asm(内联汇编)来绕过这个问题</strong>。</p>
<p>下面是一个具体例子:<code>xdp_load_bytes()</code>,使用下面这段等价的汇编代码,才能让 verifier 认出来:<br><img src="/images/k8s/cilium_inline-asm.png" alt="inline-asm"></p>
<h4 id="避免在用户侧使用-generic-XDP"><a href="#避免在用户侧使用-generic-XDP" class="headerlink" title="避免在用户侧使用 generic XDP"></a>避免在用户侧使用 generic XDP</h4><p>5.6 内核对 XDP 来说是一个里程碑式的版本(但可能不会是一个 LTS 版本),这个版本使得 <strong>XDP 在公有云上大规模可用了</strong>,例如 AWS ENA 和 Azure <code>hv_netvsc</code> 驱动。 但如果想跨平台使用 XDP,那你只应该使用最基本的一些 API,例如 XDP_PASS/DROP/TX 等等。</p>
<p>Cilium 在用户侧只使用 native XDP(only supports native XDP on user side), 我们也用 Generic XDP,但目前只限于 CI 等场景。</p>
<p><strong>为什么我们避免在用户侧使用 generic XDP 呢</strong>?因为这套 LB 逻辑会运行在集群内的 每个 node 上,目前 linearize skb 以及 bypass GRO 会增加太大的 overhead。</p>
<h4 id="自定义内存操作函数"><a href="#自定义内存操作函数" class="headerlink" title="自定义内存操作函数"></a>自定义内存操作函数</h4><p>现在回到加载和存储字节相关的辅助函数(load and store bytes helpers)。</p>
<p>查看 BPF 反汇编代码时,发现内置函数会执行字节级别(byte-wise)的一些操作,因此我们实现了<strong>自己优化过的 <code>mem{cpy,zero,cmp,move}()</code> 函数</strong>。这一点做起来还是比较容 易的,因为 <strong>LLVM 对栈外数据(non-stack data)没有上下文信息</strong>,例如 packet data 、map data,因而它无法准确地知道底层的架构是否支持高效的非对齐访问(unaligned access)。</p>
<p>另外,在基准测试中我们发现,<strong>大流量的场景下,<code>bpf_ktime_get_ns()</code> 在 XDP 中的开 销非常大</strong>,因此我们将 clock source 变成可选的,Cilium 启动时会执行检查,如果内 核支持,就<strong>自动切换到 <code>bpf_jiffies64()</code></strong>(精度更低,但 conntrack 不需要那么高的 精度),这使得转发性能增加了大约 <code>1.1Mpps</code>。</p>
<h4 id="cb-control-buffer"><a href="#cb-control-buffer" class="headerlink" title="cb (control buffer)"></a>cb (control buffer)</h4><p>tc BPF 中大量使用 <code>skb->cb[]</code> 来传递数据,显然,XDP 中也是没有这个东西的。</p>
<p>为了在 XDP 中传递数据,我们最开始使用的是 <code>xdp_adjust_meta()</code>,但有两个缺点:</p>
<ul>
<li>missing driver support</li>
<li>high rate of cache-misses</li>
</ul>
<p><strong>后来换成 per-CPU scratch map</strong>(每个 CPU 独立的、内容可随意修改的 map), 增加了大约 <code>1.2Mpps</code>。</p>
<h4 id="bpf-map-update-elem"><a href="#bpf-map-update-elem" class="headerlink" title="bpf_map_update_elem()"></a>bpf_map_update_elem()</h4><p>在 fast path 中有很多 <code>bpf_map_update_elem()</code> 调用,触发了 bucket spinlock。</p>
<p>如果流量来自多个 CPU,这里可以优化的是:先检查一下是否需要更新(这一步不需要加锁 ),如果原来已经存在,并且需要更新的值并没有变,那就直接返回,<br><img src="/images/k8s/cilium_bpf_map_update_ele.png" alt="bpf_map_update_ele"></p>
<h4 id="bpf-fib-lookup"><a href="#bpf-fib-lookup" class="headerlink" title="bpf_fib_lookup()"></a>bpf_fib_lookup()</h4><p><code>bpf_fib_lookup()</code> 开销非常大,但在 XDP 中,例如 hairpin LB 场景,是不需要这个 函数的,可以在编译时去掉。我们在测试环境的结果显示可以提高 <code>1.5Mpps</code>。</p>
<h4 id="静态-key"><a href="#静态-key" class="headerlink" title="静态 key"></a>静态 key</h4><p>作为这次分享的最后一个例子,不要对不确定的 LLVM 行为做任何假设。</p>
<p>我们在 BPF map 的基础上有大量的尾调用,它们有静态的 keys,能够在编译期间确 定 key 的大小。我们还实现了一个内联汇编来做静态的尾递归调用,保证 LLVM 不会出现 尾调用相关的问题。<br><img src="/images/k8s/cilium_tail_call_static.png" alt="tail_call_static"></p>
<h3 id="2-4-XDP-转发性能"><a href="#2-4-XDP-转发性能" class="headerlink" title="2.4 XDP 转发性能"></a>2.4 XDP 转发性能</h3><p>我们在 K8s 集群测试了 <strong>XDP 对 K8s Service 的转发</strong>。用 pktgen 生成 <code>10Mpps</code> 的入向处理流量,然后让 node 转发到位于其他节点的 backend pods。来看下几种不同的 负载均衡实现分别能处理多少。<br><img src="/images/k8s/cilium_fwd-performance.png" alt="fwd-performance"></p>
<p>由上图可以看出,</p>
<ol>
<li><strong>Cilium XDP 模式</strong>:能够处理全部的 <code>10Mpps</code> 入向流量,将它们转发到其他节点上的 backend pods。</li>
<li><strong>Cilium TC 模式</strong>:可以处理大约 <code>2.8Mpps</code>,虽然它的处理逻辑和 Cilium XDP 是类似的(除了 BPF helpers)。</li>
<li><strong>kube-proxy iptables 模式</strong>:能处理 <code>2.4Mpps</code>,这是 K8s 的默认 Service 负载均衡实现。</li>
<li><strong>kube-proxy IPVS 模式</strong>:性能更差一些,因为它的 <strong>per-packet overhead 更大一些</strong>,这里测试的 Service 只对应一个 backend pod。当 Service 数量更多时, <strong>IPVS 的可扩展性更好</strong>,相比 <code>iptables</code> 模式的 <code>kube-proxy</code> 性能会更好,但仍然没 法跟我们基于 TC BPF 和 XDP 的实现相比(no comparison at all)。</li>
</ol>
<p><strong>softirq 开销</strong>也是类似的,如下图所示,流量从 1Mpps 到 2Mpps 再到 4Mpps 时, XDP 模式下的 softirq 开销都远小于其他几种模式。<br><img src="/images/k8s/cilium_fwd-performance-cpu.png" alt="fwd-performance-cpu"></p>
<p>特别是 pps 到达某个临界点时,TC 和 Netfilter 实现中 <strong>softirq 开销会大到饱和</strong> —— 占用几乎全部 CPU。</p>
<h2 id="3-新的-BPF-内核扩展"><a href="#3-新的-BPF-内核扩展" class="headerlink" title="3 新的 BPF 内核扩展"></a>3 新的 BPF 内核扩展</h2><p>下面介绍几个新的 BPF 内核扩展,主要是 Cilium 相关的场景。</p>
<h3 id="3-1-避免穿越内核协议栈"><a href="#3-1-避免穿越内核协议栈" class="headerlink" title="3.1 避免穿越内核协议栈"></a>3.1 避免穿越内核协议栈</h3><p>主机收到的包,当其 backend 是本机上的 pod 时,或者包是本机产生的,目的端是一个本 机端口,这个包需要跨越不同的 netns,例如从宿主机的 netns 进入到容 器的 netns,<strong>现在 Cilium 的做法是,将包送到内核协议栈</strong>,如下图所示:<br><img src="/images/k8s/cilium_new-bpf-ext.png" alt="new-bpf-ext"></p>
<p>将包送到内核协议栈有两个原因(需要):</p>
<ol>
<li>TPROXY 需要由内核协议栈完成:我们目前的 L7 proxy 功能会用到这个功能,</li>
<li>K8s 默认安装了一些 iptables rule,用来检测<strong>从连接跟踪的角度看是非法的连接</strong>(‘invalid’ connections on asymmetric paths),然后 netfilter 会 drop 这些连接 的包。我们最开始时曾尝试将包从宿主机 tc 层直接 redirect 到 veth,但应答包却要 经过协议栈,因此形成了<strong>非对称路径</strong>,流量被 drop。因此目前进和出都都要经过协议栈。</li>
</ol>
<p>但这样带来两个问题,如下图所示:<br><img src="/images/k8s/cilium_new-bpf-ext-3.png" alt="new-bpf-ext-3"></p>
<ol>
<li>Pod 的出向流量在进入协议栈后,在 socket buffer 层会丢掉 socket 信息(<code>skb->sk</code> gets orphaned at <code>ip_rcv_core()</code>),这导致包从主机设备发出去时, 我们无法在 FQ leaf 获得 TCP 反压(TCP back-pressure)。</li>
<li>转发和处理都是 packet 级别的,因此有 per-packet overhead。</li>
</ol>
<p>不久之前,<strong>BPF TPROXY 已经合并到内核,因此最后一个真正依赖 Netfilter 的东西已经 解决了。因此我们现在可以在 TC 层做全部逻辑处理了,无需进入内核协议栈</strong>,如下图所示:<br><img src="/images/k8s/cilium_new-bpf-ext-2.png" alt="new-bpf-ext"></p>
<h3 id="3-2-Redirection-helpers"><a href="#3-2-Redirection-helpers" class="headerlink" title="3.2 Redirection helpers"></a>3.2 Redirection helpers</h3><p>两个用于 redirection 的 TC BPF helpers:</p>
<ul>
<li><code>bpf_redirect_neigh()</code></li>
<li><code>bpf_redirect_peer()</code></li>
</ul>
<p><strong>从 IPVLAN driver 中借鉴了一些理念,实现到了 veth 驱动中</strong>。</p>
<h4 id="3-2-1-Pod-egress:bpf-redirect-neigh"><a href="#3-2-1-Pod-egress:bpf-redirect-neigh" class="headerlink" title="3.2.1 Pod egress:bpf_redirect_neigh()"></a>3.2.1 Pod egress:<code>bpf_redirect_neigh()</code></h4><p><img src="/images/k8s/cilium_tc-redir-helper.png" alt="tc-redir-helper"></p>
<p>对于 pod egress 流量,我们会填充 src 和 dst mac 地址,这和原来 neighbor subsystem 做的事情相同;此外,我们还可以保留 skb 的 socket。这些都是由 <code>bpf_redirect_neigh()</code> 来完成的:<br><img src="/images/k8s/cilium_tc-redir-helper-2.png" alt="tc-redir-helper"></p>
<p>整个过程大致实现如下,在 veth 主机端的 ingress(对应 pod 的 egress)调用这 个方法的时候:</p>
<ol>
<li>首先会查找路由,<code>ip_route_output_flow()</code></li>
<li>将 skb 和匹配的路由条目(dst entry)关联起来,<code>skb_dst_set()</code></li>
<li>然后调用到 neighbor 子系统,<code>ip_finish_output2()</code><ol>
<li>填充 neighbor 信息,即 src/dst MAC 地址</li>
<li>保留 <code>skb->sk</code> 信息,因此物理网卡上的 qdisc 都能访问到这个字段</li>
</ol>
</li>
</ol>
<p>这就是 pod 出向的处理过程。</p>
<h4 id="3-2-2-Pod-ingress:bpf-redirect-peer"><a href="#3-2-2-Pod-ingress:bpf-redirect-peer" class="headerlink" title="3.2.2 Pod ingress:bpf_redirect_peer()"></a>3.2.2 Pod ingress:<code>bpf_redirect_peer()</code></h4><p>入向流量,<strong>会有快速 netns 切换</strong>,从宿主机 netns 直接进入容器的 netns。<br><img src="/images/k8s/cilium_tc-redir-helper-3.png" alt="tc-redir-helper"></p>
<p>这是由 <code>bpf_redirect_peer()</code> 完成的。<br><img src="/images/k8s/cilium_tc-redir-helper-4.png" alt="tc-redir-helper"></p>
<p>在主机设备的 ingress 执行这个 helper 的时候,</p>
<ol>
<li>首先会获取对应的 veth pair,<code>dev = ops->ndo_get_peer_dev(dev)</code>,然后获取 veth 的对端(在另一个 netns)</li>
<li>然后,<code>skb_scrub_packet()</code></li>
<li>设置包的 dev 为容器内的 dev,<code>skb->dev = dev</code></li>
<li>重新调度一次,<code>sch_handle_ingress()</code>,这不会进入 CPU 的 backlog queue:<ol>
<li>goto another_round</li>
<li>no CPU backlog queue</li>
</ol>
</li>
</ol>
<h4 id="3-2-3-veth-to-veth"><a href="#3-2-3-veth-to-veth" class="headerlink" title="3.2.3 veth to veth"></a>3.2.3 veth to veth</h4><p>同宿主机上的两个 Pod 之间通信时,这两个 helper 也非常有用。 因为我们已经在主机 netns 的 TC ingress 层了,因此能直接将其 redirect 到另一个容 器的 ingress 路径。<br><img src="/images/k8s/cilium_tc-redir-helper-5.png" alt="tc-redir-helper"></p>
<p>这里比较好的一点是,需要针对老版本内核所做的兼容性非常少;因此,我们只需要在启动的 时候检测内核是否有相应的 helper,</p>
<ul>
<li>如果有,就用 redirection 功能;</li>
<li>如果没有,就直接返回 TC_OK,走传统的内核协议栈方式,经过内核邻居子系统。</li>
</ul>
<p>支持这些功能无需对原有的 BPF datapath 进行大规模重构。<br><img src="/images/k8s/cilium_tc-redir-helper-6.png" alt="tc-redir-helper"></p>
<h4 id="3-2-4-BPF-redirection-性能"><a href="#3-2-4-BPF-redirection-性能" class="headerlink" title="3.2.4 BPF redirection 性能"></a>3.2.4 BPF redirection 性能</h4><p>下面看下性能。</p>
<p>TCP stream 场景,相比 Cilium baseline,转发带宽增加了 <code>1.3Gbps</code>,接近线速:<br><img src="/images/k8s/cilium_new-ext-perf.png" alt="new-ext-perf"></p>
<p>更有趣的是 TCP_RR 的场景,以 transactions/second 衡量,提升了 <code>2.9</code> 倍,接近最 大性能:<br><img src="/images/k8s/cilium_new-ext-perf-2.png" alt="new-ext-perf"></p>
<h2 id="4-结束语"><a href="#4-结束语" class="headerlink" title="4 结束语"></a>4 结束语</h2><p><img src="/images/k8s/cilium_try-out.png" alt="try-out"></p>
<p>附录: <a href="/images/k8s/plumbers_2020_cilium_load_balancer.pdf">plumbers_2020_cilium_load_balancer.pdf</a></p>
<blockquote>
<p>译者:ArthurChiao 原文:<a href="https://linuxplumbersconf.org/event/7/contributions/674/" target="_blank" rel="external">https://linuxplumbersconf.org/event/7/contributions/674/</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>文章介绍了 K8s 的一些核心网络模型和设计、<code>Cilium</code> 对 <code>K8s Service</code> 的实现、<code>BPF/XDP</code> 性能优化,以及他们从中得到的一些实践经验,全是干货。</p>
<p>去年我们也参加了这个大会(LPC),并做了题为 <a href="https://linuxplumbersconf.org/event/4/contributions/458/">Making the Kubernetes Service Abstraction Scale using eBPF</a> 的分享。 今天的内容是去年内容的延续,具体分为三个部分:</p>
<ul>
<li>Kubernetes 网络模型</li>
<li><code>Cilium</code> 对 <code>K8s Service</code> 负载均衡的实现,以及我们的一些实践经验</li>
<li>一些新的 <code>BPF</code> 内核扩展</li>
</ul>
Kubernetes 策略引擎工具 - Kyverno
http://team.jiunile.com//blog/2020/11/k8s-kyverno.html
2020-11-23T14:00:00.000Z
2020-11-23T09:10:50.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>Kubernetes 已经能够允许人们大规模地运行分布式应用程序来彻底改变云原生生态系统。虽然 Kubernetes 是一个功能丰富、健壮的容器编排平台,但它也有自己的一套复杂性。与多个团队一起大规模管理 Kubernetes 并不容易,而且要确保人们做正确的事情并且不越界是很难管理的。</p>
<p><a href="https://github.com/kyverno/kyverno" target="_blank" rel="external">Kyverno</a> 正是解决这个问题的合适工具。它是一个开源的 Kubernetes 原生策略引擎,可以帮助您使用简单的 Kubernetes manifests 定义策略。它可以验证、修改和生成 Kubernetes 资源。因此,它允许组织定义和执行策略,以便开发人员和管理员保持一定的标准。</p>
<a id="more"></a>
<h2 id="Kyverno-是如何工作的?"><a href="#Kyverno-是如何工作的?" class="headerlink" title="Kyverno 是如何工作的?"></a>Kyverno 是如何工作的?</h2><p><code>Kyverno</code> 通过使用动态准入控制器来工作,该控制器检查您通过 <code>Kubectl</code> 发送到 <code>Kube API</code> 服务端的每个请求。如果请求与策略匹配,<code>Kyverno</code> 就应用它。否则,它将使用已定义的消息拒绝请求。</p>
<p>所以这使得 <code>Kyverno</code> 能够提供如下特性:</p>
<ul>
<li>检查 CPU 和内存限制。</li>
<li>确保用户不更改默认的网络策略。</li>
<li>检查资源名称是否与特定模式匹配。</li>
<li>确保特定的资源总是包含特定的标签。</li>
<li>拒绝对特定资源的删除和更改。</li>
<li>如果镜像标签是 <code>latest</code> 将自动更改 <code>imagePullPolicy</code> 为 <code>Always</code></li>
<li>为每个新的命名空间生成一个默认的网络策略。</li>
</ul>
<p><code>Kyverno</code> 使用自定义资源定义来定义策略,编写策略就像使用 kubectl 应用它们一样简单。</p>
<p><code>Kyverno</code> 提供了三个主要功能:</p>
<ul>
<li><strong>验证(Validation)</strong></li>
<li><strong>变更(Mutation)</strong></li>
<li><strong>生成(Generation)</strong></li>
</ul>
<p>让我们看一下它们各自的示例清单。</p>
<h2 id="Validation"><a href="#Validation" class="headerlink" title="Validation"></a>Validation</h2><p>一个很好的用例是确保所有的 pods 都设置了资源请求和限制。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> kyverno.io/v1</span><br><span class="line"><span class="attr">kind:</span> ClusterPolicy</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> check-resources</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> validationFailureAction:</span> enforce</span><br><span class="line"><span class="attr"> rules:</span></span><br><span class="line"><span class="attr"> - name:</span> check-pod-resources</span><br><span class="line"><span class="attr"> match:</span></span><br><span class="line"><span class="attr"> resources:</span></span><br><span class="line"><span class="attr"> kinds:</span></span><br><span class="line"><span class="bullet"> -</span> Pod</span><br><span class="line"><span class="attr"> validate:</span></span><br><span class="line"><span class="attr"> message:</span> <span class="string">"CPU and memory resource requests and limits are required"</span></span><br><span class="line"><span class="attr"> pattern:</span></span><br><span class="line"><span class="attr"> spec:</span></span><br><span class="line"><span class="attr"> containers:</span></span><br><span class="line"><span class="attr"> - name:</span> <span class="string">"*"</span></span><br><span class="line"><span class="attr"> resources:</span></span><br><span class="line"><span class="attr"> limits:</span></span><br><span class="line"><span class="attr"> memory:</span> <span class="string">"?*"</span></span><br><span class="line"><span class="attr"> cpu:</span> <span class="string">"?*"</span></span><br><span class="line"><span class="attr"> requests:</span></span><br><span class="line"><span class="attr"> memory:</span> <span class="string">"?*"</span></span><br><span class="line"><span class="attr"> cpu:</span> <span class="string">"?*"</span></span><br></pre></td></tr></table></figure></p>
<p>大多数配置都是比较清楚明白的,<code>validationFailureAction</code> 申明是强制执行(通过使用 <code>enforcement</code> )还是只审计它(通过 <code>audit</code> )并报告违规情况。</p>
<h2 id="Mutation"><a href="#Mutation" class="headerlink" title="Mutation"></a>Mutation</h2><p><strong>Mutation</strong> 意味着如果匹配到满足特定的场景就变更资源属性。一个很好的例子是,如果镜像标签是最新的,那么将 <code>imagePullPolicy</code> 更改为 <code>Always</code>。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> kyverno.io/v1</span><br><span class="line"><span class="attr">kind:</span> ClusterPolicy</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> image-pull-policy-always</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> rules:</span></span><br><span class="line"><span class="attr"> - name:</span> image-pull-policy-latest</span><br><span class="line"><span class="attr"> match:</span></span><br><span class="line"><span class="attr"> resources:</span></span><br><span class="line"><span class="attr"> kinds:</span></span><br><span class="line"><span class="bullet"> -</span> Pod</span><br><span class="line"><span class="attr"> mutate:</span></span><br><span class="line"><span class="attr"> overlay:</span></span><br><span class="line"><span class="attr"> spec:</span></span><br><span class="line"><span class="attr"> containers:</span></span><br><span class="line"><span class="bullet"> -</span> (image): <span class="string">"*:latest"</span></span><br><span class="line"><span class="attr"> imagePullPolicy:</span> <span class="string">"Always"</span></span><br></pre></td></tr></table></figure></p>
<h2 id="Generate"><a href="#Generate" class="headerlink" title="Generate"></a>Generate</h2><p>顾名思义,针对特定事件生成资源。例如,如果有人创建了一个新的名称空间,我们可能希望执行默认的网络策略。<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> kyverno.io/v1</span><br><span class="line"><span class="attr">kind:</span> ClusterPolicy</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> <span class="string">"default"</span></span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> rules:</span></span><br><span class="line"><span class="attr"> - name:</span> <span class="string">"default-deny"</span></span><br><span class="line"><span class="attr"> match:</span></span><br><span class="line"><span class="attr"> resources:</span> </span><br><span class="line"><span class="attr"> kinds:</span></span><br><span class="line"><span class="bullet"> -</span> Namespace</span><br><span class="line"><span class="attr"> name:</span> <span class="string">"*"</span></span><br><span class="line"><span class="attr"> exclude:</span></span><br><span class="line"><span class="attr"> namespaces:</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">"kube-system"</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">"default"</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">"kube-public"</span></span><br><span class="line"><span class="bullet"> -</span> <span class="string">"kyverno"</span></span><br><span class="line"><span class="attr"> generate:</span> </span><br><span class="line"><span class="attr"> kind:</span> NetworkPolicy</span><br><span class="line"><span class="attr"> name:</span> default-deny-all-traffic</span><br><span class="line"><span class="attr"> namespace:</span> <span class="string">"<span class="template-variable">{{request.object.metadata.namespace}}</span>"</span> </span><br><span class="line"><span class="attr"> data:</span> </span><br><span class="line"><span class="attr"> spec:</span></span><br><span class="line"><span class="attr"> podSelector:</span> {}</span><br><span class="line"><span class="attr"> policyTypes:</span> </span><br><span class="line"><span class="bullet"> -</span> Ingress</span><br><span class="line"><span class="bullet"> -</span> Egress</span><br></pre></td></tr></table></figure></p>
<h2 id="体验一把"><a href="#体验一把" class="headerlink" title="体验一把"></a>体验一把</h2><p>现在让我们亲自动手,看看 <code>Kyverno</code> 的行为。我们将安装 <code>Kyverno</code>,然后应用验证策略来检查特定的标签。如果标签不存在,<code>Kyverno</code> 将拒绝请求。否则,它将应用它。</p>
<h3 id="安装-Kyverno"><a href="#安装-Kyverno" class="headerlink" title="安装 Kyverno"></a>安装 Kyverno</h3><p>安装 <code>Kyverno</code> 很简单。你可以应用 GitHub 上的 <code>Kyverno Kubernetes manifest</code>,或者安装最新的 <code>helm chart</code>。</p>
<h4 id="使用-manifest"><a href="#使用-manifest" class="headerlink" title="使用 manifest"></a>使用 manifest</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl create <span class="_">-f</span> https://raw.githubusercontent.com/kyverno/kyverno/master/definitions/release/install.yaml</span><br></pre></td></tr></table></figure>
<h4 id="使用-helm-chart"><a href="#使用-helm-chart" class="headerlink" title="使用 helm chart"></a>使用 helm chart</h4><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">helm repo add kyverno https://kyverno.github.io/kyverno/</span><br><span class="line">kubectl create ns kyverno</span><br><span class="line">helm install kyverno --namespace kyverno kyverno/kyverno</span><br></pre></td></tr></table></figure>
<p>检查我们是否成功安装了 <code>Kyverno</code>,列出 <code>Kyverno</code> 命名空间中的所有资源:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># kubectl get all -n kyverno</span></span><br><span class="line">NAME READY STATUS RESTARTS AGE</span><br><span class="line">pod/kyverno-5f7769d697-x8lkj 0/1 Running 0 21s</span><br><span class="line">NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE</span><br><span class="line">service/kyverno-svc ClusterIP 10.96.167.8 <none> 443/TCP 21s</span><br><span class="line">NAME READY UP-TO-DATE AVAILABLE AGE</span><br><span class="line">deployment.apps/kyverno 0/1 1 0 21s</span><br><span class="line">NAME DESIRED CURRENT READY AGE</span><br><span class="line">replicaset.apps/kyverno-5f7769d697 1 1 0 21s</span><br></pre></td></tr></table></figure></p>
<h3 id="应用策略"><a href="#应用策略" class="headerlink" title="应用策略"></a>应用策略</h3><p>让我们应用一个策略来确保所有 <code>pods</code> 都应该包含一个名为 <code>app</code> 的标签。创建名为<code>require-app-label.yaml</code> 的文件,其内容如下:<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> kyverno.io/v1</span><br><span class="line"><span class="attr">kind:</span> ClusterPolicy</span><br><span class="line"><span class="attr">metadata:</span></span><br><span class="line"><span class="attr"> name:</span> require-app-label</span><br><span class="line"><span class="attr">spec:</span></span><br><span class="line"><span class="attr"> validationFailureAction:</span> enforce</span><br><span class="line"><span class="attr"> rules:</span></span><br><span class="line"><span class="attr"> - name:</span> check-for-app-label</span><br><span class="line"><span class="attr"> match:</span></span><br><span class="line"><span class="attr"> resources:</span></span><br><span class="line"><span class="attr"> kinds:</span></span><br><span class="line"><span class="bullet"> -</span> Pod</span><br><span class="line"><span class="attr"> validate:</span></span><br><span class="line"><span class="attr"> message:</span> <span class="string">"label `app` is required"</span></span><br><span class="line"><span class="attr"> pattern:</span></span><br><span class="line"><span class="attr"> metadata:</span></span><br><span class="line"><span class="attr"> labels:</span></span><br><span class="line"><span class="attr"> app:</span> <span class="string">"?*"</span></span><br></pre></td></tr></table></figure></p>
<p>如果您查看 YAML,会看到有一个匹配部分,其中包含我们应该匹配的资源类型。在这个场景中,我们看到一个 pod。<code>validate</code> 部分定义了验证失败时应该输出的消息,以及定义需要匹配什么内容的模式。</p>
<p>由于这是一个 CRD,我们可以直接应用它,得到想要的结果:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl apply <span class="_">-f</span> require-app-label.yaml</span><br></pre></td></tr></table></figure></p>
<h3 id="测试"><a href="#测试" class="headerlink" title="测试"></a>测试</h3><p>让我们创建一个没有标签的 pod,看看我们会得到什么:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl run nginx --image=nginx</span><br></pre></td></tr></table></figure></p>
<p><img src="/images/k8s/kyverno_1.gif" alt="kyverno"></p>
<p>所以正如我们所看到的,验证失败的原因如下:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">Error from server: admission webhook <span class="string">"nirmata.kyverno.resource.validating-webhook"</span> denied the request:</span><br><span class="line">resource Deployment/default/nginx was blocked due to the following policies</span><br><span class="line">require-app-label:</span><br><span class="line"> autogen-check-for-app-label: <span class="string">'Validation error: label `app` is required; Validation rule autogen-check-for-app-label failed at path /spec/template/metadata/labels/app/'</span></span><br></pre></td></tr></table></figure></p>
<p>这和预期的一样,因为我们还没有提供标签。现在让我们尝试使用标签 <code>name=nginx</code>:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl run nginx --image=nginx --labels=<span class="string">"name=nginx"</span> --generator=run-pod/v1</span><br></pre></td></tr></table></figure></p>
<p><img src="/images/k8s/kyverno_2.gif" alt="kyverno"></p>
<p>这个也失败了,因为 app 标签仍然缺失。让我们用 <code>app=NGINX</code> 标签创建一个 NGINX pod:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl run nginx --image=nginx --labels=<span class="string">"app=nginx"</span> --generator=run-pod/v1</span><br></pre></td></tr></table></figure></p>
<p><img src="/images/k8s/kyverno_3.gif" alt="kyverno"></p>
<p>正如我们所看到的,pod 已经成功创建。现在,让我们使用 kubectl 来获取 pod 和标签:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">kubectl get pod nginx --show-labels</span><br></pre></td></tr></table></figure></p>
<p><img src="/images/k8s/kyverno_4.gif" alt="kyverno"></p>
<p>pod 正在运行,并包含 <code>app=nginx</code> 标签。</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p><code>Kyverno</code> 是一款优秀的 “policy-as-code” 工具,它在组织层执行最佳实践方面非常强大。由于它是 kubernets 原生的,所以编写和操作都很简单,不需要专门的开发人员进行维护。</p>
<p>感谢你的阅读!希望你喜欢这篇文章。</p>
<blockquote>
<p>译自:<a href="https://medium.com/better-programming/policy-as-code-on-kubernetes-with-kyverno-b144749f144" target="_blank" rel="external">https://medium.com/better-programming/policy-as-code-on-kubernetes-with-kyverno-b144749f144</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>Kubernetes 已经能够允许人们大规模地运行分布式应用程序来彻底改变云原生生态系统。虽然 Kubernetes 是一个功能丰富、健壮的容器编排平台,但它也有自己的一套复杂性。与多个团队一起大规模管理 Kubernetes 并不容易,而且要确保人们做正确的事情并且不越界是很难管理的。</p>
<p><a href="https://github.com/kyverno/kyverno">Kyverno</a> 正是解决这个问题的合适工具。它是一个开源的 Kubernetes 原生策略引擎,可以帮助您使用简单的 Kubernetes manifests 定义策略。它可以验证、修改和生成 Kubernetes 资源。因此,它允许组织定义和执行策略,以便开发人员和管理员保持一定的标准。</p>
探索 Go Trace 包
http://team.jiunile.com//blog/2020/11/go-trace.html
2020-11-22T12:00:00.000Z
2020-11-21T16:34:46.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><blockquote>
<p>本文基于 Go 1.13</p>
</blockquote>
<p>Go 为我们提供了一个工具,可以在运行时进行跟踪,并获得程序执行的详细视图。这个工具可以通过在测试中使用标记 <code>-trace</code> 来启用,可以通过 <code>pprof</code> 来进行实时跟踪,也可以通过<code>trace</code> <a href="https://golang.org/pkg/runtime/trace/" target="_blank" rel="external">包</a>在代码中的任何位置启用。这个工具可以更加强大,因为您可以自定义 traces 来增强它。让我们回顾一下它是如何工作的。</p>
<h2 id="流程"><a href="#流程" class="headerlink" title="流程"></a>流程</h2><p>该工具的流程非常简单。每个事件,如内存分配;垃圾回收器的所有阶段;goroutines 在运行、暂停等情况下会被 Go 标准库静态记录,并格式化后显示。然而,在录制开始之前,Go 首先“stops the world”,并对当前的 goroutines 及其状态进行快照。<br><a id="more"></a></p>
<p>这将在之后能让 Go 正确地构建每个 goroutine 的生命周期。流程如下:<br><img src="/images/go/trace_1.png" alt="Initialization phase before tracing"></p>
<p>然后,将收集的事件推送到缓冲区,当达到最大容量时,该缓冲区随后将刷新到完整缓冲区列表。这是此流程的图:<br><img src="/images/go/trace_2.png" alt="Tracing collect events per P"></p>
<p>跟踪器现在需要一种将这些跟踪转储到输出的方法。为此,当追踪开始时,Go 会产生一个专用于此的 goroutine。如果可用,该 goroutine 将转储数据,并将把 goroutine 停放到下一个。这是它的一个表示:<br><img src="/images/go/trace_3.png" alt="A dedicated goroutine reads and dump the traces"></p>
<p>现在流程非常清晰,所以让我们回顾一下记录的跟踪事件。</p>
<h2 id="追踪"><a href="#追踪" class="headerlink" title="追踪"></a>追踪</h2><p>生成跟踪后,就可以通过运行命令 <code>go tool trace my-output.out</code> 来实现可视化。让我们以一些跟踪事件为例:<br><img src="/images/go/trace_4.png" alt="Tracing from go tool"></p>
<p>大多数都很简单。与垃圾回收器相关的跟踪位于蓝色跟踪 <code>GC</code> 下:<br><img src="/images/go/trace_5.png" alt="Traces of the garbage collector"></p>
<p>快速回顾:</p>
<ul>
<li><code>STW</code> 是垃圾回收器中的两个 “Stop the World” 阶段。在这两个阶段,goroutines 被停止。</li>
<li><code>GC (空闲)</code> 是在没有工作要做时标记内存的 goroutine。</li>
<li><code>MARK ASSIST</code> 是在分配期间帮助标记内存的 goroutines。</li>
<li><code>GXX runtime.bgsweep</code> 是垃圾回收器完成后的内存扫描阶段。</li>
<li><code>GXX runtime.gcBgMarkWorker</code> 是帮助标记内存的专用后台 goroutines。</li>
</ul>
<p>然而,有些追踪事件并不容易理解。让我们回顾一下,以便更好地理解:</p>
<ul>
<li><p>当处理器与线程关联时,将调用 <code>proc start</code>。当启动新线程或从 syscall 恢复时,就会发生这种情况。<br><img src="/images/go/trace_6.png" alt="trace"></p>
</li>
<li><p>当线程与当前处理器解除关联时,将调用 <code>proc stop</code>。当线程在 syscall 中被阻塞或线程退出时,就会发生这种情况。<br><img src="/images/go/trace_7.png" alt="trace"></p>
</li>
<li><p>当 goroutine 进行系统调用时,将调用 <code>syscall</code>:<br><img src="/images/go/trace_8.png" alt="trace"></p>
</li>
<li><p>当 goroutine 从 syscall 解除阻止时,将调用 <code>unblock</code> – 在这种情况下,标签 (<code>sysexit</code>) 将从被阻塞的通道显示:<br><img src="/images/go/trace_9.png" alt="trace"></p>
</li>
</ul>
<p>跟踪可以增强,因为 Go 允许您定义和可视化自己的跟踪以及标准库中的跟踪。</p>
<h2 id="用户自定义追踪"><a href="#用户自定义追踪" class="headerlink" title="用户自定义追踪"></a>用户自定义追踪</h2><p>我们可以定义的跟踪有两个级别:</p>
<ul>
<li>在任务的顶层,有开始和结束。</li>
<li>在区域的子级别上。</li>
</ul>
<p>下面是一个简单的例子:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> ctx, task := trace.NewTask(context.Background(), <span class="string">"main start"</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">var</span> wg sync.WaitGroup</span><br><span class="line"> wg.Add(<span class="number">2</span>)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> r := trace.StartRegion(ctx, <span class="string">"reading file"</span>)</span><br><span class="line"> <span class="keyword">defer</span> r.End()</span><br><span class="line"> </span><br><span class="line"> ioutil.ReadFile(<span class="string">`n1.txt`</span>)</span><br><span class="line"> }()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> r := trace.StartRegion(ctx, <span class="string">"writing file"</span>)</span><br><span class="line"> <span class="keyword">defer</span> r.End()</span><br><span class="line"> </span><br><span class="line"> ioutil.WriteFile(<span class="string">`n2.txt`</span>, []<span class="keyword">byte</span>(<span class="string">`42`</span>), <span class="number">0644</span>)</span><br><span class="line"> }()</span><br><span class="line"> </span><br><span class="line"> wg.Wait()</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">defer</span> task.End()</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这些新的跟踪可以通过菜单用户自定义直接从工具中可视化:<br><img src="/images/go/trace_10.png" alt="Custom task and regions"></p>
<p>还可以任意将一些日志记录到任务中:<br><figure class="highlight golang"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">ctx, task := trace.NewTask(context.Background(), <span class="string">"main start"</span>)</span><br><span class="line">trace.Log(ctx, <span class="string">"category"</span>, <span class="string">"I/O file"</span>)</span><br><span class="line">trace.Log(ctx, <span class="string">"goroutine"</span>, <span class="string">"2"</span>)</span><br></pre></td></tr></table></figure></p>
<p>这些日志将在设置任务的 goroutine 下找到:<br><img src="/images/go/trace_11.png" alt="Custom logs in the tracing"></p>
<p>还可以通过派生父任务的上下文等将任务嵌入到其他任务中。</p>
<p>但是,由于 <code>pprof</code> 的存在,在生产中实时跟踪所有这些事件可能会在收集它们时略微降低性能。</p>
<h2 id="性能影响"><a href="#性能影响" class="headerlink" title="性能影响"></a>性能影响</h2><p>一个简单的基准测试可以帮助理解跟踪的影响。其中一个将带标志 <code>-trace</code> 运行,另一个则不带。下面是 <code>ioutil.ReadFile()</code> 函数的基准测试结果,该函数生成了很多事件:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">name time/op</span><br><span class="line">ReadFiles-8 48.1µs ± 0%</span><br><span class="line">name time/op</span><br><span class="line">ReadFiles-8 63.5µs ± 0% // with tracing</span><br></pre></td></tr></table></figure></p>
<p>在这种情况下,影响约为 ~35%,并且可能因应用程序而异。但是有一些工具(如 StackDriver ),允许在生产环境中进行连续的分析,同时又对应用程序保持较小的开销。</p>
<blockquote>
<p>译自:<a href="https://medium.com/a-journey-with-go/go-discovery-of-the-trace-package-e5a821743c3c" target="_blank" rel="external">https://medium.com/a-journey-with-go/go-discovery-of-the-trace-package-e5a821743c3c</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><blockquote>
<p>本文基于 Go 1.13</p>
</blockquote>
<p>Go 为我们提供了一个工具,可以在运行时进行跟踪,并获得程序执行的详细视图。这个工具可以通过在测试中使用标记 <code>-trace</code> 来启用,可以通过 <code>pprof</code> 来进行实时跟踪,也可以通过<code>trace</code> <a href="https://golang.org/pkg/runtime/trace/">包</a>在代码中的任何位置启用。这个工具可以更加强大,因为您可以自定义 traces 来增强它。让我们回顾一下它是如何工作的。</p>
<h2 id="流程"><a href="#流程" class="headerlink" title="流程"></a>流程</h2><p>该工具的流程非常简单。每个事件,如内存分配;垃圾回收器的所有阶段;goroutines 在运行、暂停等情况下会被 Go 标准库静态记录,并格式化后显示。然而,在录制开始之前,Go 首先“stops the world”,并对当前的 goroutines 及其状态进行快照。<br>
在 Go 中我应该使用指针还是拷贝结构体?
http://team.jiunile.com//blog/2020/11/go-copy-struct-or-pointer.html
2020-11-20T14:00:00.000Z
2020-11-19T14:49:22.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>对于许多 Go 开发人员来说,系统地使用指针来共享结构体而不是拷贝本身似乎是性能方面的最佳选择。</p>
<p>为了理解使用指针而不是拷贝结构体的影响,我们将回顾两个用例。<br><a id="more"></a></p>
<h2 id="用例1:数据密集分配"><a href="#用例1:数据密集分配" class="headerlink" title="用例1:数据密集分配"></a>用例1:数据密集分配</h2><p>让我们举一个简单的例子,当你想共享一个结构体的值:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> S <span class="keyword">struct</span> {</span><br><span class="line"> a, b, c <span class="keyword">int64</span></span><br><span class="line"> d, e, f <span class="keyword">string</span></span><br><span class="line"> g, h, i <span class="keyword">float64</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这是一个基本的结构体,可以通过拷贝或指针共享:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">byCopy</span><span class="params">()</span> <span class="title">S</span></span> {</span><br><span class="line"> <span class="keyword">return</span> S{</span><br><span class="line"> a: <span class="number">1</span>, b: <span class="number">1</span>, c: <span class="number">1</span>,</span><br><span class="line"> e: <span class="string">"foo"</span>, f: <span class="string">"foo"</span>,</span><br><span class="line"> g: <span class="number">1.0</span>, h: <span class="number">1.0</span>, i: <span class="number">1.0</span>,</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">byPointer</span><span class="params">()</span> *<span class="title">S</span></span> {</span><br><span class="line"> <span class="keyword">return</span> &S{</span><br><span class="line"> a: <span class="number">1</span>, b: <span class="number">1</span>, c: <span class="number">1</span>,</span><br><span class="line"> e: <span class="string">"foo"</span>, f: <span class="string">"foo"</span>,</span><br><span class="line"> g: <span class="number">1.0</span>, h: <span class="number">1.0</span>, i: <span class="number">1.0</span>,</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>基于这两种方法,我们现在可以编写两个基准测试,其中一个是通过拷贝结构体传递的:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkMemoryStack</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> s S</span><br><span class="line"></span><br><span class="line"> f, err := os.Create(<span class="string">"stack.out"</span>)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">defer</span> f.Close()</span><br><span class="line"></span><br><span class="line"> err = trace.Start(f)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> s = byCopy()</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> trace.Stop()</span><br><span class="line"></span><br><span class="line"> b.StopTimer()</span><br><span class="line"></span><br><span class="line"> _ = fmt.Sprintf(<span class="string">"%v"</span>, s.a)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>另一个,非常相似,通过指针传递:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkMemoryHeap</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> s *S</span><br><span class="line"></span><br><span class="line"> f, err := os.Create(<span class="string">"heap.out"</span>)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">defer</span> f.Close()</span><br><span class="line"></span><br><span class="line"> err = trace.Start(f)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="built_in">panic</span>(err)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> s = byPointer()</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> trace.Stop()</span><br><span class="line"></span><br><span class="line"> b.StopTimer()</span><br><span class="line"></span><br><span class="line"> _ = fmt.Sprintf(<span class="string">"%v"</span>, s.a)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>让我们运行一下基准测试:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">go <span class="built_in">test</span> ./... -bench=BenchmarkMemoryHeap -benchmem -run=^$ -count=10 > head.txt && benchstat head.txt</span><br><span class="line">go <span class="built_in">test</span> ./... -bench=BenchmarkMemoryStack -benchmem -run=^$ -count=10 > stack.txt && benchstat stack.txt</span><br></pre></td></tr></table></figure></p>
<p>以下是统计数据:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">name time/op</span><br><span class="line">MemoryHeap-4 75.0ns ± 5%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryHeap-4 96.0B ± 0%</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryHeap-4 1.00 ± 0%</span><br><span class="line">------------------</span><br><span class="line">name time/op</span><br><span class="line">MemoryStack-4 8.93ns ± 4%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryStack-4 0.00B</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryStack-4 0.00</span><br></pre></td></tr></table></figure></p>
<p>在这里使用结构的拷贝而不是指针要快 <strong>8 倍</strong>。</p>
<p>为了理解其中的原因,让我们来看一下 trace 生成的图表:<br><img src="/images/go/struct_1.png" alt="graph for the struct passed by copy"><br><img src="/images/go/struct_2.png" alt="graph for the struct passed by pointer"></p>
<p>第一个图很简单。由于没有使用堆,因此没有垃圾回收器和额外的 <code>goroutine</code>。</p>
<p>对于第二个图,指针的使用迫使 go 编译器将<a href="https://golang.org/doc/faq#stack_or_heap" target="_blank" rel="external">变量转义到堆中</a>,并对垃圾回收器施加压力。如果我们放大这个图,我们可以看到垃圾回收器在这个过程中扮演了重要的角色:<br><img src="/images/go/struct_3.png" alt="trace struct"></p>
<p>从这个图中我们可以看到,垃圾回收器必须每 4ms 工作一次。</p>
<p>如果我们再次放大,我们可以得到正在发生的事情的详细信息:<br><img src="/images/go/struct_4.png" alt="trace struct"></p>
<p>蓝色、粉色和红色的是垃圾回收器的阶段,而棕色的阶段与堆上的分配有关(在图表上标记为 “<code>runtime.bgsweep</code>”):</p>
<blockquote>
<p>扫描是指回收与堆内存中未标记为正在使用的值相关联的内存。当应用程序 <code>Goroutines</code> 试图在堆内存中分配新值时,就会发生此活动。扫描的延迟会增加在堆内存中执行分配的成本,并且不会与垃圾回收相关的任何延迟相关联。<br><a href="https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html" target="_blank" rel="external">https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html</a></p>
</blockquote>
<p>即使这个示例有点极端,我们也可以看到在堆上分配变量而不是在栈上分配变量的代价有多大。在我们的示例中,代码在栈上分配结构体并拷贝它比在堆上分配结构体并共享其地址要快得多。</p>
<p>如果您不熟悉栈/堆,如果您想了解更多关于每个堆的内部细节,您可以在网上找到许多资源,比如 Paul Gribble 的这篇<a href="https://www.gribblelab.org/CBootCamp/7_Memory_Stack_vs_Heap.html" target="_blank" rel="external">文章</a>。</p>
<p>如果我们将 <code>GOMAXPROCS=1</code> 的处理器限制为 1,情况会更糟:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">name time/op</span><br><span class="line">MemoryHeap 114ns ± 4%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryHeap 96.0B ± 0%</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryHeap 1.00 ± 0%</span><br><span class="line">------------------</span><br><span class="line">name time/op</span><br><span class="line">MemoryStack 8.77ns ± 5%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryStack 0.00B</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryStack 0.00</span><br></pre></td></tr></table></figure></p>
<p>如果在栈上有分配的基准没有改变,那么在堆上的基准已经从 75ns/op 降低到 114ns/op。</p>
<h2 id="用例2:密集的函数调用"><a href="#用例2:密集的函数调用" class="headerlink" title="用例2:密集的函数调用"></a>用例2:密集的函数调用</h2><p>对于第二个用例,我们将在结构体中添加两个空方法,稍微调整一下我们的基准用例:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s S)</span> <span class="title">stack</span><span class="params">(s1 S)</span></span> {}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(s *S)</span> <span class="title">heap</span><span class="params">(s1 *S)</span></span> {}</span><br></pre></td></tr></table></figure></p>
<p>在栈上分配的基准测试将创建一个结构并通过拷贝传递它:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkMemoryStack</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> s S</span><br><span class="line"> <span class="keyword">var</span> s1 S</span><br><span class="line"></span><br><span class="line"> s = byCopy()</span><br><span class="line"> s1 = byCopy()</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">1000000</span>; i++ {</span><br><span class="line"> s.stack(s1)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>而堆的基准测试将通过指针传递结构体:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkMemoryHeap</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> s *S</span><br><span class="line"> <span class="keyword">var</span> s1 *S</span><br><span class="line"></span><br><span class="line"> s = byPointer()</span><br><span class="line"> s1 = byPointer()</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">1000000</span>; i++ {</span><br><span class="line"> s.heap(s1)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>不出所料,结果现在大不相同:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">name time/op</span><br><span class="line">MemoryHeap-4 301µs ± 4%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryHeap-4 0.00B</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryHeap-4 0.00</span><br><span class="line">------------------</span><br><span class="line">name time/op</span><br><span class="line">MemoryStack-4 595µs ± 2%</span><br><span class="line">name alloc/op</span><br><span class="line">MemoryStack-4 0.00B</span><br><span class="line">name allocs/op</span><br><span class="line">MemoryStack-4 0.00</span><br></pre></td></tr></table></figure></p>
<h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>在 go 中,使用指针而不是拷贝结构体并不总是一件好事。</p>
<p>为了你的数据选择好的语义,我强烈建议阅读 <a href="https://twitter.com/goinggodotnet" target="_blank" rel="external">Bill Kennedy</a> 写的<a href="https://www.ardanlabs.com/blog/2017/06/design-philosophy-on-data-and-semantics.html" target="_blank" rel="external">关于值/指针语义</a>的文章。它将让您更好地了解结构和内置类型可以使用的策略。</p>
<p>此外,对内存使用情况的分析肯定会帮助您了解在分配和堆上发生了什么。</p>
<blockquote>
<p>来源:<a href="https://medium.com/a-journey-with-go/go-should-i-use-a-pointer-instead-of-a-copy-of-my-struct-44b43b104963" target="_blank" rel="external">https://medium.com/a-journey-with-go/go-should-i-use-a-pointer-instead-of-a-copy-of-my-struct-44b43b104963</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>对于许多 Go 开发人员来说,系统地使用指针来共享结构体而不是拷贝本身似乎是性能方面的最佳选择。</p>
<p>为了理解使用指针而不是拷贝结构体的影响,我们将回顾两个用例。<br>
Golang 六种错误处理技术,可帮助您编写优雅的代码
http://team.jiunile.com//blog/2020/11/go-handle-error.html
2020-11-20T12:00:00.000Z
2020-11-19T14:47:25.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>当在 GO 中遇到错误时你会怎么做?</p>
<p>处理错误并不简单。在讨论功能性需求时,很少考虑错误处理需求,但是错误处理是软件开发的一个重要部分。</p>
<p>在 GO 中,错误条件以方法返回值的形式返回。在我看来,将错误条件作为主流程的一部分是很有用的 – 它让开发人员在编写功能代码时承担处理错误的责任。这种范例与其他编程语言(如 Java )所提供的非常不同 – 其中异常是完全不同的流程。虽然这种不同的风格使代码更具可读性,但也带来了新的挑战。</p>
<p>本文讨论了六种处理错误、重试和可服务性的技术。虽然很少有想法是琐碎的,但其他想法并不那么受欢迎。</p>
<p>因此,让我们从列表开始!<br><a id="more"></a></p>
<h2 id="1-向左对齐"><a href="#1-向左对齐" class="headerlink" title="1 向左对齐"></a>1 向左对齐</h2><p>处理错误的最佳策略是检查错误并立即从函数返回。在一个函数中有多个错误返回语句是可以的 – 事实上,这是明智的选择。[1]</p>
<p>例如,下面的代码片段展示了如何使用 <code>if err == nil</code> 来处理一个愉快的场景,从而导致嵌套 if 检查。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// Handling Happy case first - leading to nested if checks...</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">example</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> err := somethingThatReturnsError()</span><br><span class="line"> <span class="keyword">if</span> err == <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">//Happy processing</span></span><br><span class="line"> err = somethingElseThatReturnsError()</span><br><span class="line"> <span class="keyword">if</span> err == <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">//More Happy processing</span></span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>上述逻辑可以通过向左对齐逻辑来处理:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">ABetterExample</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> err := somethingThatReturnsError()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> <span class="comment">// Happy processing</span></span><br><span class="line"> err = somethingElseThatReturnsError()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// More Happy processing</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<h2 id="2-重试可恢复错误"><a href="#2-重试可恢复错误" class="headerlink" title="2 重试可恢复错误"></a>2 重试可恢复错误</h2><p>很少有可恢复的错误值得重试 – 网络故障、IO 操作等都可以通过简单的重试恢复。</p>
<p>下面的包可以帮助解决重试带来的麻烦。<br><a href="https://godoc.org/github.com/cenkalti/backoff#example-Retry" target="_blank" rel="external">package backoff</a></p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// An operation that may fail.</span></span><br><span class="line">operation := <span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span> <span class="comment">// or an error</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">err := Retry(operation, NewExponentialBackOff())</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="comment">// Handle error.</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>指数回退意味着重试间隔呈指数增长 – 对于大多数网络 /IO 故障来说,这是一个明智的选择。</p>
<h2 id="3-包装错误"><a href="#3-包装错误" class="headerlink" title="3 包装错误"></a>3 包装错误</h2><p>默认的错误包是有限的 – 错误上下文的详细信息经常会丢失。例如:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">testingError2</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">return</span> errors.New(<span class="string">"New Error"</span>)</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">testingError</span><span class="params">(accountNumber <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> err := testingError2()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> err := testingError(<span class="string">"Acct1"</span>)</span><br><span class="line"> logrus.Error(<span class="string">"Error occurred"</span>, fmt.Sprintf(<span class="string">"%+v"</span>, err))</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>在这种情况下,主函数收到的错误实例没有发生在帐户 <code>Acct1</code> 上的信息。可以在函数 <code>testingErrror</code> 中记录 <code>accountNumber</code>,但是由于当前包错误,无法将该信息传递给主函数。</p>
<p>这就是 <code>github.Com/pkg/errors</code> 的来源。该库与 <code>errors</code> 兼容并带来了一些很酷的功能。</p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">testingError2</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">return</span> errors.New(<span class="string">"New Error"</span>)</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">testingError</span><span class="params">(accountNumber <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> err := testingError2()</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> errors.Wrap(err, <span class="string">"Error occurred while processing Card Number "</span>+accoutNumber)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> err := testingError(<span class="string">"Acct1"</span>)</span><br><span class="line"> logrus.Error(<span class="string">"Error occurred"</span>, fmt.Sprintf(<span class="string">"%+v"</span>, err))</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>在 <code>github.com/pkg/errors</code> 中,您还可以使用一些额外的有用功能 – <code>errors.Unwrap</code> 和 <code>errors.Is</code></p>
<h2 id="4-日志策略"><a href="#4-日志策略" class="headerlink" title="4 日志策略"></a>4 日志策略</h2><p>Golang 的默认包日志不提供使用日志记录级别进行日志记录功能。这里有一些其他的选择:</p>
<ul>
<li><code>Glog</code>: <a href="https://github.com/golang/glog" target="_blank" rel="external">https://github.com/golang/glog</a></li>
<li><code>Logrus</code>: <a href="https://github.com/sirupsen/logrus" target="_blank" rel="external">https://github.com/sirupsen/logrus</a></li>
<li><code>Zap</code>: <a href="https://github.com/uber-go/zap" target="_blank" rel="external">https://github.com/uber-go/zap</a></li>
</ul>
<p><code>Logrus</code> 和 <code>Zap</code> 还提供了<strong>结构化日志输出</strong>的功能 – 这是一个非常方便的功能,因为它为开发人员提供了向错误日志消息添加上下文的能力。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">example</span><span class="params">(accountNumber <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> logrus.SetFormatter(&logrus.JSONFormatter{})</span><br><span class="line">ctxFields := logrus.Fields{</span><br><span class="line"> <span class="string">"accountNumber"</span>: accountNumber,</span><br><span class="line"> <span class="string">"appname"</span>: <span class="string">"my-app"</span>,</span><br><span class="line"> }</span><br><span class="line"><span class="comment">//Happy processing</span></span><br><span class="line"> err := errors.New(<span class="string">"Some test error while doing happy processing"</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> logrus.WithFields(ctxFields).WithError(err).Error(<span class="string">"ErrMsg"</span>)</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>结构化日志输出如下:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">{<span class="string">"accountNumber"</span>:<span class="string">"ABC"</span>,<span class="string">"appname"</span>:<span class="string">"my-app"</span>,<span class="string">"error"</span>:<span class="string">"Some test error while doing happy processing"</span>,<span class="string">"level"</span>:<span class="string">"error"</span>,<span class="string">"msg"</span>:<span class="string">"ErrMsg"</span>,<span class="string">"time"</span>:<span class="string">"2009-11-10T23:00:00Z"</span>}</span><br></pre></td></tr></table></figure></p>
<p>日志的另一个关键方面是获得日志堆栈跟踪的能力。如果你使用 <code>github.com/pkg/errors</code>,你就可以<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">logrus.Error(<span class="string">"Error occurred"</span>, fmt.Sprintf(<span class="string">"%+v"</span>, err))</span><br></pre></td></tr></table></figure></p>
<p>你会得到一个错误堆栈跟踪如下:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">main.testingError2</span><br><span class="line"> /home/nayars/go/src/github.com/nayarsn/temp.go:12</span><br><span class="line">main.testingError</span><br><span class="line"> /home/nayars/go/src/github.com/nayarsn/temp.go:25</span><br><span class="line">main.main</span><br><span class="line"> /home/nayars/go/src/github.com/nayarsn/temp.go:39</span><br><span class="line">runtime.main</span><br><span class="line"> /usr/lib/go-1.15/src/runtime/proc.go:204</span><br><span class="line">runtime.goexit</span><br><span class="line"> /usr/lib/go-1.15/src/runtime/asm_amd64.s:1374</span><br></pre></td></tr></table></figure></p>
<p><code>Zap</code> 为性能进行了缓冲和优化。[2]</p>
<h2 id="5-错误检查"><a href="#5-错误检查" class="headerlink" title="5 错误检查"></a>5 错误检查</h2><p>将错误视为值是好的 – 它是明确的,而明确的有很多意义。但它也可以为开发人员提供跳过的机会。例如:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">testingError</span><span class="params">(accoutNumber <span class="keyword">string</span>)</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">var</span> err error</span><br><span class="line"> _ = errors.New(<span class="string">"errors.New with _"</span></span><br><span class="line"> errors.New(<span class="string">"errors.New not capturing return"</span>)</span><br><span class="line"> <span class="keyword">return</span> err</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>上面的示例显示应用程序程序员是由 <code>errors.New</code> 语句返回的两个错误。这可能是有意或无意发生的。</p>
<p>幸运的是,有一个 linter 实用程序可以帮助您。<br><a href="https://github.com/kisielk/errcheck" target="_blank" rel="external">kisielk/errcheck</a></p>
<p>一旦你安装了 linter,你可以简单地做以下事情:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">errcheck -blank ./...</span><br></pre></td></tr></table></figure></p>
<p>你会得到这样的输出:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">temp.go:16:2: _ = errors.New(<span class="string">"Error capturing return using _"</span>)</span><br><span class="line">temp.go:18:12: errors.New(<span class="string">"Error not capturing return"</span>)</span><br></pre></td></tr></table></figure></p>
<p>这可以作为 CI/CD 流程的一部分,以确保应用程序开发人员不会错过这一部分。</p>
<p><code>errchec</code> 是 Go linters 聚合器实用程序的一部分 – <a href="https://golangci-lint.run/" target="_blank" rel="external">https://golangci-lint.run/</a></p>
<h2 id="6-多个错误"><a href="#6-多个错误" class="headerlink" title="6 多个错误"></a>6 多个错误</h2><p>你有多个错误的场景 – 它们是同一个 <code>go routine</code> 的一部分,你不想停止处理 – 而是继续处理并记录所有错误。这里有一个专门的库:<br><a href="https://github.com/hashicorp/go-multierror" target="_blank" rel="external">hashicorp/go-multierror</a></p>
<p>这里有一个简单的例子:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">step1</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">return</span> errors.New(<span class="string">"Step1"</span>)</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">step2</span><span class="params">()</span> <span class="title">error</span></span> {</span><br><span class="line"> <span class="keyword">return</span> errors.New(<span class="string">"Step2"</span>)</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">var</span> result error</span><br><span class="line"> <span class="keyword">if</span> err := step1(); err != <span class="literal">nil</span> {</span><br><span class="line"> result = multierror.Append(result, err)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := step2(); err != <span class="literal">nil</span> {</span><br><span class="line"> result = multierror.Append(result, err)</span><br><span class="line"> }</span><br><span class="line"> </span><br><span class="line"> fmt.Println(multierror.Flatten(result))</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>同样,对于多个 <code>go routines</code>,可以使用以下库:<br><a href="https://pkg.go.dev/golang.org/x/sync/errgroup" target="_blank" rel="external">errgroup</a></p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>我知道上述列表并非全部。对于你们中的一些人来说,这可能是微不足道的 – 但希望对你们中的一些人来说,这有助于你们掌握错误处理技术。</p>
<h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><ul>
<li>[1] <a href="https://medium.com/@matryer/line-of-sight-in-code-186dd7cdea88" target="_blank" rel="external">https://medium.com/@matryer/line-of-sight-in-code-186dd7cdea88</a></li>
<li>[2] <a href="https://medium.com/a-journey-with-go/go-how-zap-package-is-optimized-dbf72ef48f2d" target="_blank" rel="external">https://medium.com/a-journey-with-go/go-how-zap-package-is-optimized-dbf72ef48f2d</a></li>
</ul>
<blockquote>
<p>译自:<a href="https://medium.com/higher-order-functions/golang-six-error-handling-techniques-to-help-you-write-elegant-code-8e6363e6d2b" target="_blank" rel="external">https://medium.com/higher-order-functions/golang-six-error-handling-techniques-to-help-you-write-elegant-code-8e6363e6d2b</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>当在 GO 中遇到错误时你会怎么做?</p>
<p>处理错误并不简单。在讨论功能性需求时,很少考虑错误处理需求,但是错误处理是软件开发的一个重要部分。</p>
<p>在 GO 中,错误条件以方法返回值的形式返回。在我看来,将错误条件作为主流程的一部分是很有用的 – 它让开发人员在编写功能代码时承担处理错误的责任。这种范例与其他编程语言(如 Java )所提供的非常不同 – 其中异常是完全不同的流程。虽然这种不同的风格使代码更具可读性,但也带来了新的挑战。</p>
<p>本文讨论了六种处理错误、重试和可服务性的技术。虽然很少有想法是琐碎的,但其他想法并不那么受欢迎。</p>
<p>因此,让我们从列表开始!<br>
深入理解 kubernetes iptables proxy 模式
http://team.jiunile.com//blog/2020/11/k8s-proxy-iptables.html
2020-11-19T15:00:00.000Z
2020-11-18T13:56:14.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>最近在面试的时候问了不少 <code>network request</code> 如何到 <code>k8s service backend</code> 的问题,觉得可以整合一下网络上的资料,这篇主要讨论 <code>iptables proxy mode</code>。大部分的情况没有在使用 <code>userspace proxy modes</code>, <code>ipvs proxy mode</code> 可能要等到下一次讨论。</p>
<a id="more"></a>
<h2 id="事先准备"><a href="#事先准备" class="headerlink" title="事先准备"></a>事先准备</h2><p>要先了解 <code>iptable</code> 工作机制,建议可以看这一篇:<a href="https://phoenixnap.com/kb/iptables-tutorial-linux-firewall" target="_blank" rel="external">https://phoenixnap.com/kb/iptables-tutorial-linux-firewall</a>,当然 wikipedia 也是写的不错,我下面的文字也大多数引用:<a href="https://zh.wikipedia.org/wiki/Iptables" target="_blank" rel="external">https://zh.wikipedia.org/wiki/Iptables</a></p>
<h2 id="快速带过-iptable"><a href="#快速带过-iptable" class="headerlink" title="快速带过 iptable"></a>快速带过 <code>iptable</code></h2><p>说到 <code>iptable</code> 要先了解 <strong><code>Tables</code></strong>, <strong><code>Chains</code></strong> 和 <strong><code>Rueles</code></strong>。</p>
<ul>
<li><strong><code>Table</code></strong> 指不同类型的封包处理流程,总共有五种,不同的 <strong><code>Tables</code></strong> 处理不同的行为<ul>
<li><code>raw</code>:处理异常,追踪状态 -> <code>/proc/net/nf_conntrack</code></li>
<li><code>mangle</code>:处理封包,修改 headler 之类的</li>
<li><code>nat</code>:进行位址转换操作</li>
<li><code>filter</code>:进行封包过滤</li>
<li><code>security</code>:SElinux 相关</li>
</ul>
</li>
<li><strong><code>Chains</code></strong> 来对应进行不同的行为。像是 “filter” <strong><code>Tables</code></strong> 进行封包过滤的流程,而 “nat” 针对连接进行位址转换操作。<strong><code>Chains</code></strong> 里面包含许多规则,主要有五种类型的 <strong><code>Chains</code></strong><ul>
<li><code>PREROUTING</code>:处理路由规则前通过此 <code>Chains</code>,通常用于目的位址转换(DNAT)</li>
<li><code>INPUT</code>:发往本机的封包通过此 <strong><code>Chains</code></strong>。</li>
<li><code>FORWARD</code>:本机转发的封包通过此 <strong><code>Chains</code></strong>。</li>
<li><code>OUTPUT</code>:处理本机发出的封包。</li>
<li><code>POSTROUTING</code>:完成路由规则后通过此 <strong><code>Chains</code></strong>,通常用于源位址转换(SNAT)</li>
</ul>
</li>
<li><strong><code>Rules</code></strong> 规则会被逐一进行匹配,如果匹配,可以执行相应的动作 </li>
</ul>
<h2 id="大致的工作流向情况分两种:"><a href="#大致的工作流向情况分两种:" class="headerlink" title="大致的工作流向情况分两种:"></a>大致的工作流向情况分两种:</h2><ol>
<li><p>backend 为本机</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">NIC → PREROUTING → INPUT → Local process </span><br><span class="line">Local process → OUTPUT → POSTROUTING → NIC</span><br></pre></td></tr></table></figure>
</li>
<li><p>backend 目的地非本机</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">NIC→PREROUTING → FORWARD → POSTROUTING→NIC</span><br></pre></td></tr></table></figure>
</li>
</ol>
<p><img src="/images/k8s/proxy_iptable_1.png" alt="Iptables Basics"></p>
<p>下面是比较详细的流程,有包含 <code>EBTABLES</code>,但这个看久头会昏,我这次会主要讨论 Network Layer 这一部分,然后用上面这张比较精简的图<br><img src="/images/k8s/proxy_iptable_2.png" alt="Netfilter pic of wikipedia"></p>
<p><code>Kube-proxy</code> 修改了 filter,nat 两个表,自定义了<br><code>KUBE-SERVICES</code>,<code>KUBE-NODEPORTS</code>,<code>KUBE-POSTROUTING</code>,<code>KUBE-FORWARD</code>,<code>KUBE-MARK-MASQ</code> 和 <code>KUBE-MARK-DROP</code>,所以我这次会 focus on filter ,nat 两个 Table</p>
<h3 id="1-filter-table-有三个-Chain-“INPUT”-“OUTPUT”-“FORWARD”"><a href="#1-filter-table-有三个-Chain-“INPUT”-“OUTPUT”-“FORWARD”" class="headerlink" title="1. filter table 有三个 Chain “INPUT” “OUTPUT” “FORWARD”"></a>1. filter table 有三个 Chain “<strong>INPUT</strong>” “<strong>OUTPUT</strong>” “<strong>FORWARD</strong>”</h3><p><img src="/images/k8s/proxy_iptable_3.png" alt="iptables-routing"></p>
<p><code>kube-proxy</code> 在 filter table 的 “<strong>INPUT</strong>” “<strong>OUTPUT</strong>” chain 增加了 <code>KUBE-FIREWALL</code> 在 “<strong>INPUT</strong>” “<strong>OUTPUT</strong>” “<strong>FORWARD</strong>” chain 增加了 <code>KUBE-SERVICES</code></p>
<p><code>KUBE_FIREWALL</code> 会丢弃所有被 <code>KUBE-MARK-DROP</code> 标记 0x8000 的封包,而标记的动作可以在其他的 table 中(像是第二部分提到的 NAT table 中)<br><img src="/images/k8s/proxy_iptable_4.png" alt="proxy iptable"><br><img src="/images/k8s/proxy_iptable_5.png" alt="proxy iptable"></p>
<p>而 filter table 的 <code>KUBE-SERVICES</code> 可以过滤封包,假如一个 service 没有对应的 endpoint,就会被 reject,这里我先要建立一个 service 和没有正确设定 endpoint。</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">kind:</span> Service </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> test-error-endpoint </span><br><span class="line"><span class="attr"> namespace:</span> default </span><br><span class="line"><span class="attr">spec:</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - protocol:</span> TCP </span><br><span class="line"><span class="attr"> port:</span> <span class="number">7777</span> </span><br><span class="line"><span class="attr"> targetPort:</span> <span class="number">7777</span> </span><br><span class="line"><span class="bullet">-</span>-- </span><br><span class="line"><span class="attr">kind:</span> Endpoints </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> test-error-endpoint </span><br><span class="line"><span class="attr"> namespace:</span> default</span><br></pre></td></tr></table></figure>
<p>service cluster ip 为 10.95.58.92</p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">kind:</span> Service </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> test-error-endpoint </span><br><span class="line"><span class="attr"> namespace:</span> default </span><br><span class="line"><span class="attr"> selfLink:</span> /api/v1/namespaces/default/services/test-error-endpoint </span><br><span class="line"><span class="attr"> uid:</span> <span class="number">5</span>d415d63<span class="bullet">-6</span>fc3<span class="bullet">-444e-8</span>b5a<span class="bullet">-29015</span>b436a83 </span><br><span class="line"><span class="attr"> resourceVersion:</span> <span class="string">' 73026369'</span> </span><br><span class="line"><span class="attr"> creationTimestamp:</span> <span class="string">'2020-11-17T05:48:52Z'</span> </span><br><span class="line"><span class="attr">spec:</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - protocol:</span> TCP </span><br><span class="line"><span class="attr"> port:</span> <span class="number">7777</span> </span><br><span class="line"><span class="attr"> targetPort:</span> <span class="number">7777</span> </span><br><span class="line"><span class="attr"> clusterIP:</span> <span class="number">10.95</span><span class="number">.58</span><span class="number">.92</span> </span><br><span class="line"><span class="attr"> type:</span> ClusterIP </span><br><span class="line"><span class="attr"> sessionAffinity:</span> None </span><br><span class="line"><span class="attr">status:</span> </span><br><span class="line"><span class="attr"> loadBalancer:</span> {}</span><br></pre></td></tr></table></figure>
<p>再次检查 iptable,就可以看到 <code>default/test-error-endpoint: has no endpoints -> tcp dpt:7777 reject-with icmp-port-unreachable</code><br><img src="/images/k8s/proxy_iptable_6.png" alt="proxy iptable"></p>
<h3 id="2-nat-table-有三个-Chain-“PREROUTING”-“OUTPUT”-“POSTROUTING”"><a href="#2-nat-table-有三个-Chain-“PREROUTING”-“OUTPUT”-“POSTROUTING”" class="headerlink" title="2. nat table 有三个 Chain “PREROUTING” “OUTPUT” “POSTROUTING”"></a>2. nat table 有三个 Chain “<strong>PREROUTING</strong>” “<strong>OUTPUT</strong>” “<strong>POSTROUTING</strong>”</h3><p>在前两个封包处理流程是比较相似和复杂的,大体来说是藉由客制化的规则,来处理符合条件封包,帮它们找到正确的 k8s endpoint (后面会细讲),在 <code>POSTROUTING</code> 主要是针对 k8s 处理的封包(标记 0x4000 的封包),在离开 node 的时候做 SNAT</p>
<ul>
<li>(inbound) 在 “<strong>PREROUTING</strong>” 将所有封包转发到 <code>KUBE-SERVICES</code></li>
<li>(outbound) 在 “<strong>OUTPUT</strong>” 将所有封包转发到 <code>KUBE-SERVICES</code></li>
<li>(outbound) 在 “<strong>POSTROUTING</strong>” 将所有封包转发到 <code>KUBE-POSTROUTING</code></li>
</ul>
<p><img src="/images/k8s/proxy_iptable_7.png" alt="iptables-routing"><br>当封包进入 “<strong>PREROUTING</strong>” 和 “<strong>OUTPUT</strong>”,会整个被 <code>KUBE-SERVICES</code> Chain 整个绑架走,开始逐一匹配 <code>KUBE-SERVICES</code> 中的 rule 和打上标签。<br><img src="/images/k8s/proxy_iptable_8.png" alt="nat tables"></p>
<p><code>kube-proxy</code> 的用法是一种 O(n) 算法,其中的 n 随 k8s cluster 的规模同步增加,更简单的说就是 service 和 endpoint 的数量。<br><img src="/images/k8s/proxy_iptable_9.png" alt="kube-services"></p>
<blockquote>
<p>我这里会准备三个最常见的 service type 的 <code>kube-proxy</code> 路由流程</p>
<ul>
<li>cluster IP</li>
<li>nodePort</li>
<li>load balancer</li>
</ul>
</blockquote>
<h4 id="clusterIP-流程"><a href="#clusterIP-流程" class="headerlink" title="clusterIP 流程"></a>clusterIP 流程</h4><p>这里我使用 <code>default/jeff-api(clusterIP: 10.95.57.19)</code> 举例,我下面图过滤掉不必要的资讯<br><img src="/images/k8s/proxy_iptable_10.png" alt="kube-services"></p>
<p>最后会到实际 pod 的位置,<code>podIP: 10.95.35.31,hostIP: 10.20.0.128</code> 是该 pod 所在 node 的 ip<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">kind:</span> Pod </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> jeff-api<span class="bullet">-746</span>f4c9985<span class="bullet">-5</span>qmw6 </span><br><span class="line"><span class="attr"> generateName:</span> jeff-api<span class="bullet">-746</span>f4c9985- </span><br><span class="line"><span class="attr"> namespace:</span> default </span><br><span class="line"><span class="attr">spec:</span> </span><br><span class="line"><span class="attr"> containers:</span> </span><br><span class="line"><span class="attr"> - name:</span> promotion-api </span><br><span class="line"><span class="attr"> image:</span> <span class="string">'gcr.io/jeff-project/jeff /jeff-api:202011161901'</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - name:</span> <span class="number">80</span>tcp02 </span><br><span class="line"><span class="attr"> containerPort:</span> <span class="number">80</span> </span><br><span class="line"><span class="attr"> protocol:</span> TCP </span><br><span class="line"><span class="attr"> nodeName:</span> gke-sit-jeff-k8s-tw<span class="bullet">-01</span>-default-pool<span class="bullet">-7983</span>af35-ug91 </span><br><span class="line"><span class="attr">status:</span> </span><br><span class="line"><span class="attr"> phase:</span> Running </span><br><span class="line"><span class="attr"> hostIP:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.128</span> </span><br><span class="line"><span class="attr"> podIP:</span> <span class="number">10.95</span><span class="number">.35</span><span class="number">.31</span></span><br></pre></td></tr></table></figure></p>
<h4 id="nodePort-流程"><a href="#nodePort-流程" class="headerlink" title="nodePort 流程"></a>nodePort 流程</h4><p>这里有一个关键就是 <code>KUBE-NODEPORTS</code> 一定是在 <code>KUBE-SERVICES</code> 最后一项,iptables 在处理 packet 会先处理 ip 为 cluster ip 的 service,当全部的 <code>KUBE-SVC-XXXXXX</code> 都对应不到的时候就会使用 nodePort 去匹配。<br><img src="/images/k8s/proxy_iptable_11.png" alt="kube-services"></p>
<p>我们看实际 pod 的资讯,<code>podIP: 10.95.32.17,hostIP: 10.20.0.124</code> 是其中一台 node 的 ip<br><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">kind:</span> Service </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> jeff-frontend </span><br><span class="line"><span class="attr"> namespace:</span> jeff-frontend </span><br><span class="line"><span class="attr">spec:</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - protocol:</span> TCP </span><br><span class="line"><span class="attr"> port:</span> <span class="number">80</span> </span><br><span class="line"><span class="attr"> targetPort:</span> <span class="number">80</span> </span><br><span class="line"><span class="attr"> nodePort:</span> <span class="number">31929</span> </span><br><span class="line"><span class="attr"> selector:</span> </span><br><span class="line"><span class="attr"> app:</span> jeff-frontend </span><br><span class="line"><span class="attr"> clusterIP:</span> <span class="number">10.95</span><span class="number">.58</span><span class="number">.51</span> </span><br><span class="line"><span class="attr"> type:</span> NodePort </span><br><span class="line"><span class="attr"> externalTrafficPolicy:</span> Cluster </span><br><span class="line"><span class="bullet">-</span>-- </span><br><span class="line"><span class="attr">kind:</span> Pod </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> jeff-frontend-c94bf68d9-bbmp8 </span><br><span class="line"><span class="attr"> generateName:</span> jeff-frontend-c94bf68d9- </span><br><span class="line"><span class="attr"> namespace:</span> jeff-frontend </span><br><span class="line"><span class="attr">spec:</span> </span><br><span class="line"><span class="attr"> containers:</span> </span><br><span class="line"><span class="attr"> - name:</span> jeff-frontend</span><br><span class="line"><span class="attr"> image:</span> <span class="string">'gcr.io/jeff-project/jeff/jeff-image:jeff-1.0.6.5'</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - name:</span> http </span><br><span class="line"><span class="attr"> containerPort:</span> <span class="number">80</span> </span><br><span class="line"><span class="attr"> protocol:</span> TCP </span><br><span class="line"><span class="attr"> nodeName:</span> gke-sit-jeff-k8s-tw<span class="bullet">-01</span>-default -pool-b5692f8d-enk7 </span><br><span class="line"><span class="attr">status:</span> </span><br><span class="line"><span class="attr"> phase:</span> Running </span><br><span class="line"><span class="attr"> hostIP:</span> <span class="number">10.20</span><span class="number">.0</span><span class="number">.124</span> </span><br><span class="line"><span class="attr"> podIP:</span> <span class="number">10.95</span><span class="number">.32</span><span class="number">.17</span></span><br></pre></td></tr></table></figure></p>
<h4 id="load-balancer流程"><a href="#load-balancer流程" class="headerlink" title="load balancer流程"></a>load balancer流程</h4><p>假如目的地 IP 是 load balancer 就会使用 <code>KUBE-FW-XXXXXX</code>,我建立一个 internal load balancer service 和 endpoint 指到 google postgresql DB(10.28.193.9)<br><img src="/images/k8s/proxy_iptable_12.png" alt="kube-services"></p>
<figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">kind:</span> Service </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> annotations:</span> </span><br><span class="line"> cloud.google.com/load-balancer-type: Internal </span><br><span class="line"> networking.gke.io/internal-load-balancer-allow-global-access: <span class="string">'true'</span> </span><br><span class="line"><span class="attr"> name:</span> external-postgresql </span><br><span class="line">spec : </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - protocol:</span> TCP </span><br><span class="line"><span class="attr"> port:</span> <span class="number">5432</span> </span><br><span class="line"><span class="attr"> targetPort:</span> <span class="number">5432</span> </span><br><span class="line"><span class="attr"> type:</span> LoadBalancer </span><br><span class="line"><span class="bullet">-</span>-- </span><br><span class="line"><span class="attr">apiVersion:</span> v1 </span><br><span class="line"><span class="attr">kind:</span> Endpoints </span><br><span class="line"><span class="attr">metadata:</span> </span><br><span class="line"><span class="attr"> name:</span> external-postgresql </span><br><span class="line"><span class="attr">subsets:</span> </span><br><span class="line"><span class="attr">- addresses:</span> </span><br><span class="line"><span class="attr"> - ip:</span> <span class="number">10.28</span><span class="number">.193</span><span class="number">.9</span> </span><br><span class="line"><span class="attr"> ports:</span> </span><br><span class="line"><span class="attr"> - port:</span> <span class="number">5432</span> </span><br><span class="line"> protocol : TCP</span><br></pre></td></tr></table></figure>
<p><img src="/images/k8s/proxy_iptable_13.png" alt="kube-services"><br>在 NAT table 看到 <code>KUBE-MARK-MASQ</code> 和 <code>KUBE-MARK-DROP</code> 这两个规则主要是经过的封包打上标签,打上标签的封包会做相应的处理。<code>KUBE-MARK-DROP</code> 和 <code>KUBE-MARK-MASQ</code> 本质上就是使用 iptables 的 <a href="https://serverfault.com/questions/514116/how-to-set-mark-on-packet-when-forwarding-it-in-nat-prerouting-table" target="_blank" rel="external">MARK 指令</a><br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000</span><br><span class="line">-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000</span><br></pre></td></tr></table></figure></p>
<p>如果打上了 <code>0x8000</code> 到后面 filter table (上面提到 <code>KUBE_FIREWALL</code> )就会丢弃。</p>
<p>如果打上了 <code>0x4000</code> k8s 将会在 <code>PREROUTING</code> table 的 <code>KUBE-POSTROUTING</code> chain 对它进行 SNAT 转换。<br><img src="/images/k8s/proxy_iptable_14.png" alt="kube-services"><br><img src="/images/k8s/proxy_iptable_15.png" alt="POSTROUTING table"><br><img src="/images/k8s/proxy_iptable_16.png" alt="KUBE-POSTROUTING Chain"><br><img src="/images/k8s/proxy_iptable_17.png" alt="KUBE-SERVICES"></p>
<p>参考:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Netfilter" target="_blank" rel="external">https://en.wikipedia.org/wiki/Netfilter</a></li>
<li><a href="https://zh.wikipedia.org/wiki/Iptables" target="_blank" rel="external">https://zh.wikipedia.org/wiki/Iptables</a></li>
<li><a href="https://phoenixnap.com/kb/iptables-tutorial-linux-firewall" target="_blank" rel="external">https://phoenixnap.com/kb/iptables-tutorial-linux-firewall</a></li>
<li><a href="https://www" target="_blank" rel="external">https://www</a> .cnblogs.com/charlieroro/p/9588019.html</li>
<li><a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go" target="_blank" rel="external">https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go</a></li>
<li><a href="https://www.lijiaocn.com/%E9%" target="_blank" rel="external">https://www.lijiaocn.com/%E9%</a> A1%B9%E7%9B%AE/2017/03/27/Kubernetes-kube-proxy.html</li>
<li><a href="https://juejin.im/post/6844904098605563912" target="_blank" rel="external">https://juejin.im/post/6844904098605563912</a></li>
<li><a href="https://tizeen.github.io/2019/03/19/" target="_blank" rel="external">https://tizeen.github.io/2019/03/19/</a> kubernetes-service-iptables%E5%88%86%E6%9E%90/</li>
<li><a href="https://www.hwchiu.com/kubernetes-service-ii.html" target="_blank" rel="external">https://www.hwchiu.com/kubernetes-service-ii.html</a></li>
<li><a href="https://www.hwchiu.com/kubernetes-service-iii" target="_blank" rel="external">https://www.hwchiu.com/kubernetes-service-iii</a> .html</li>
<li><a href="https://www.itread01.com/content/1542712570.html" target="_blank" rel="external">https://www.itread01.com/content/1542712570.html</a></li>
</ul>
<blockquote>
<p>来源:<a href="https://jeff-yen.medium.com/iptables-proxy-mode-in-kube-proxy-6862bb4b329" target="_blank" rel="external">https://jeff-yen.medium.com/iptables-proxy-mode-in-kube-proxy-6862bb4b329</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>最近在面试的时候问了不少 <code>network request</code> 如何到 <code>k8s service backend</code> 的问题,觉得可以整合一下网络上的资料,这篇主要讨论 <code>iptables proxy mode</code>。大部分的情况没有在使用 <code>userspace proxy modes</code>, <code>ipvs proxy mode</code> 可能要等到下一次讨论。</p>
Go 多重错误管理
http://team.jiunile.com//blog/2020/11/go-multi-errors.html
2020-11-19T14:00:00.000Z
2020-11-18T03:55:12.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在<a href="https://blog.golang.org/survey2019-results" target="_blank" rel="external">年度调查</a>中,关于开发人员在使用 Go 时面临的最大挑战,Go 中的错误管理总是容易引起争论,并且是一个反复出现的话题。然而,当涉及到在并发环境中处理错误或为同一个 <code>goroutine</code> 合并多个错误时,Go 提供了很棒的包,使管理多个错误变得容易。让我们看看如何合并由单个 <code>goroutine</code> 生成的多个错误。<br><a id="more"></a></p>
<h2 id="单个-goroutine-多个错误"><a href="#单个-goroutine-多个错误" class="headerlink" title="单个 goroutine, 多个错误"></a>单个 goroutine, 多个错误</h2><p>例如,当您处理具有重试策略的代码时,将多个错误合并为一个错误会非常有用。这是我们需要收集生成的错误的基本示例:<br><img src="/images/go/manage_muti_error_1.png" alt="multiple errors"></p>
<p>这个程序读取和解析一个 CSV 文本,并显示发现的错误。将错误分组以获得完整的报告可能更方便。要将错误合并为一个,我们可以在两个很棒的包中进行选择:</p>
<ul>
<li>使用 <a href="https://github.com/hashicorp" target="_blank" rel="external">HashiCorp</a> 的 <a href="https://github.com/hashicorp/go-multierror" target="_blank" rel="external">go-multierror</a></li>
</ul>
<p><img src="/images/go/manage_muti_error_2.png" alt="multiple errors"></p>
<p>输出结果如下:<br><img src="/images/go/manage_muti_error_3.png" alt="multiple errors"></p>
<ul>
<li>使用 <a href="https://github.com/uber-go" target="_blank" rel="external">Uber</a> 的 <a href="https://github.com/uber-go/multierr" target="_blank" rel="external">multierr</a></li>
</ul>
<p>这里的实现类似,下面是输出:<br><img src="/images/go/manage_muti_error_4.png" alt="multiple errors"></p>
<p>错误通过分号连接起来,没有任何其他格式。</p>
<p>对于每个包的性能,下面是一个基准测试结论:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">name time/op alloc/op allocs/op</span><br><span class="line">HashiCorpMultiErrors-4 6.01µs ± 1% 6.78kB ± 0% 77.0 ± 0%</span><br><span class="line">UberMultiErrors-4 9.26µs ± 1% 10.3kB ± 0% 126 ± 0%</span><br></pre></td></tr></table></figure></p>
<p>Uber 的实现稍微慢一些,消耗更多的内存。但是,这个包的设计目的是将收集到的错误分组在一起,而不是每次都附加它们。当对错误进行分组时,结果很接近,但是代码不够优雅,因为它需要额外的步骤。以下是最新的测试结果:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">name time/op alloc/op allocs/op</span><br><span class="line">HashiCorpMultiErrors-4 6.01µs ± 1% 6.78kB ± 0% 77.0 ± 0%</span><br><span class="line">UberMultiErrors-4 6.02µs ± 1% 7.06kB ± 0% 77.0 ± 0%</span><br></pre></td></tr></table></figure></p>
<p>这两个包都利用了 Go <code>error</code> 接口,并在其自定义实现中实现了 <code>Error() string</code> 函数。</p>
<h2 id="单个错误-多个-goroutines"><a href="#单个错误-多个-goroutines" class="headerlink" title="单个错误, 多个 goroutines"></a>单个错误, 多个 goroutines</h2><p>当处理多个 <code>goroutines</code> 来执行一个任务时,有必要正确管理结果和将错误聚合,以确保程序的正确性。</p>
<p>让我们从一个使用多个 <code>goroutines</code> 执行一系列操作的程序开始; 每个操作持续一秒钟:<br><img src="/images/go/manage_muti_error_5.png" alt="multiple errors"></p>
<p>为了说明错误传播,第三个 <code>goroutine</code> 的第一个操作将失败。事情是这样的:<br><img src="/images/go/manage_muti_error_6.png" alt="multiple errors"></p>
<p>正如预期的那样,这个程序大约需要 3 秒,因为大多数 <code>goroutines</code> 需要经历三个动作,每个动作需要 1 秒:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go run . 0.30s user 0.19s system 14% cpu 3.274 total</span><br></pre></td></tr></table></figure></p>
<p>但是,我们可能希望使 <code>goroutines</code> 相互依赖,并在其中一个失败时取消它们。避免不必要工作的解决方案是添加上下文,一旦 <code>goroutine</code> 失败,它就会取消它:<br><img src="/images/go/manage_muti_error_7.png" alt="multiple errors"></p>
<p>这正是 <a href="https://pkg.go.dev/golang.org/x/sync/errgroup?tab=doc" target="_blank" rel="external">errgroup</a> 所提供的;处理一组 <code>goroutines</code> 时的错误和上下文传播。下面是使用包 <a href="https://pkg.go.dev/golang.org/x/sync/errgroup?tab=doc" target="_blank" rel="external">errgroup</a> 的新代码:<br><img src="/images/go/manage_muti_error_8.png" alt="multiple errors"></p>
<p>程序现在运行得更快,因为它通过错误传播取消的上下文:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">go run . 0.30s user 0.19s system 38% cpu 1.269 total</span><br></pre></td></tr></table></figure></p>
<p>该包提供的另一个好处是,我们不需要再担心等待组添加和标记 <code>goroutines</code> 完成。包为我们管理这些,我们只需要说我们准备好等待过程的结束。</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<p>译自:<a href="https://medium.com/a-journey-with-go/go-multiple-errors-management-a67477628cf1" target="_blank" rel="external">https://medium.com/a-journey-with-go/go-multiple-errors-management-a67477628cf1</a></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在<a href="https://blog.golang.org/survey2019-results">年度调查</a>中,关于开发人员在使用 Go 时面临的最大挑战,Go 中的错误管理总是容易引起争论,并且是一个反复出现的话题。然而,当涉及到在并发环境中处理错误或为同一个 <code>goroutine</code> 合并多个错误时,Go 提供了很棒的包,使管理多个错误变得容易。让我们看看如何合并由单个 <code>goroutine</code> 生成的多个错误。<br>
Go chan 为啥没有判断 close 的接口
http://team.jiunile.com//blog/2020/11/go-chan-close.html
2020-11-16T14:00:00.000Z
2020-11-16T02:19:55.000Z
<h2 id="大纲"><a href="#大纲" class="headerlink" title="大纲"></a>大纲</h2><ul>
<li>Go 为什么没有判断 close 的接口?</li>
<li>Go 关闭 channel 究竟做了什么?<br> -<code>closechan</code></li>
<li>一个判断 chan 是否 close 的函数<ul>
<li>思考方法一:通过“写”chan 实现</li>
<li>思考方法二:通过“读”chan 实现</li>
<li>chan close 原则</li>
<li>其实并不需要 <code>isChanClose</code> 函数 !!!</li>
</ul>
</li>
<li>怎么优雅关闭 chan ?<ul>
<li>方法一:panic-recover</li>
<li>方法二:sync.Once</li>
<li>方法三:事件同步来解决</li>
</ul>
</li>
<li>总结</li>
</ul>
<a id="more"></a>
<h2 id="Go-为什么没有判断-close-的接口?"><a href="#Go-为什么没有判断-close-的接口?" class="headerlink" title="Go 为什么没有判断 close 的接口?"></a>Go 为什么没有判断 close 的接口?</h2><p><img src="/images/go/chan_close_1.png" alt="go channel close"></p>
<p>相信大家初学 golang chan 的时候应该都遇到过 “<strong>send on closed channel</strong>“ 的 panic 。这个 panic 是当你意图往一个已经 close 的 channel 里面投递元素的时候触发。那么你当你第一次遇到这个问题是否想过 channel 是否能提供一个接口方法来判断是否已经 close 了?我想过这个问题,但是把 chan 的源代码翻了个遍没有找到。为什么?</p>
<p>我先 hold 这个问题,我们捋一下跟 channel close 相关的事情,主要思考到 3 个问题:</p>
<ol>
<li>关闭 channel 究竟做了什么 ?</li>
<li>怎么避免 close channel 导致的 panic 问题 ?</li>
<li>怎么优雅的关闭 channel ?</li>
</ol>
<h2 id="Go-关闭-channel-究竟做了什么?"><a href="#Go-关闭-channel-究竟做了什么?" class="headerlink" title="Go 关闭 channel 究竟做了什么?"></a>Go 关闭 channel 究竟做了什么?</h2><p>首先,用户可以 close channel,如下:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">c := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>)</span><br><span class="line"><span class="comment">// ...</span></span><br><span class="line"><span class="built_in">close</span>(c)</span><br></pre></td></tr></table></figure></p>
<p>用 gdb 或者 delve 调试下就能发现 close 一个 channel,编译器会转换成 <code>closechan</code> 函数,在这个函数里是关闭 channel 的全部实现了,我们可以分析下。</p>
<h3 id="closechan"><a href="#closechan" class="headerlink" title="closechan"></a>closechan</h3><p>对应编译函数为 <code>closechan</code> ,该函数很简单,大概做 3 个事情:</p>
<ol>
<li>标志位置 1 ,也就是 <code>c.closed = 1</code>;</li>
<li>释放资源,唤醒所有等待取元素的协程;</li>
<li>释放资源,唤醒所有等待写元素的协程;</li>
</ol>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">closechan</span><span class="params">(c *hchan)</span></span> {</span><br><span class="line"> <span class="comment">// 以下为锁内操作</span></span><br><span class="line"> lock(&c.lock)</span><br><span class="line"> <span class="comment">// 不能重复 close 一个 channel,否则 panic</span></span><br><span class="line"> <span class="keyword">if</span> c.closed != <span class="number">0</span> {</span><br><span class="line"> unlock(&c.lock)</span><br><span class="line"> <span class="built_in">panic</span>(plainError(<span class="string">"close of closed channel"</span>))</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// closed 标志位置 1</span></span><br><span class="line"> c.closed = <span class="number">1</span></span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> glist gList</span><br><span class="line"> <span class="comment">// 释放所有等待取元素的 waiter 资源</span></span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> <span class="comment">// 等待读的 waiter 出队</span></span><br><span class="line"> sg := c.recvq.dequeue()</span><br><span class="line"> <span class="comment">// 资源一个个销毁</span></span><br><span class="line"> <span class="keyword">if</span> sg.elem != <span class="literal">nil</span> {</span><br><span class="line"> typedmemclr(c.elemtype, sg.elem)</span><br><span class="line"> sg.elem = <span class="literal">nil</span></span><br><span class="line"> }</span><br><span class="line"> gp := sg.g</span><br><span class="line"> gp.param = <span class="literal">nil</span></span><br><span class="line"> <span class="comment">// 相应 goroutine 加到统一队列,下面会统一唤醒</span></span><br><span class="line"></span><br><span class="line"> glist.push(gp)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 释放所有等待写元素的 waiter 资源(他们之后将会 panic)</span></span><br><span class="line"> <span class="keyword">for</span> {</span><br><span class="line"> <span class="comment">// 等待写的 waiter 出队</span></span><br><span class="line"> sg := c.sendq.dequeue()</span><br><span class="line"> <span class="comment">// 资源一个个销毁</span></span><br><span class="line"> sg.elem = <span class="literal">nil</span></span><br><span class="line"> gp := sg.g</span><br><span class="line"> gp.param = <span class="literal">nil</span></span><br><span class="line"> <span class="comment">// 对应 goroutine 加到统一队列,下面会统一唤醒</span></span><br><span class="line"> glist.push(gp)</span><br><span class="line"> }</span><br><span class="line"> unlock(&c.lock)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 唤醒所有的 waiter 对应的 goroutine (这个协程列表是上面 push 进来的)</span></span><br><span class="line"> <span class="keyword">for</span> !glist.empty() {</span><br><span class="line"> gp := glist.pop()</span><br><span class="line"> gp.schedlink = <span class="number">0</span></span><br><span class="line"> goready(gp, <span class="number">3</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>通过上面的代码逻辑,我们窥视到两个重要的信息:</p>
<ol>
<li>close chan 是有标识位的;</li>
<li>close chan 是会唤醒哪些等待的人们的;</li>
</ol>
<p>但是很奇怪的是,我们 golang 官方没有提供一个接口用于判断 chan 是否关闭?那我们能不能实现一个判断 chan 是否 close 的方法呢?</p>
<h2 id="一个判断-chan-是否-close-的函数"><a href="#一个判断-chan-是否-close-的函数" class="headerlink" title="一个判断 chan 是否 close 的函数"></a>一个判断 chan 是否 close 的函数</h2><p>怎么实现?首先 <code>isChanClose</code> 函数有几点要求:</p>
<ol>
<li>能够指明确实是 close 的;</li>
<li>任何时候能够正常运行,且有返回的(非阻塞);</li>
</ol>
<p>想想 <code>send</code>, <code>recv</code> 相关的函数,我们可以知道,当前 channel 给到用户的使用姿势本质上只有两种:读和写,我们实现的 <code>isChanClose</code> 也只能在这个基础上做。</p>
<ul>
<li>写:<code>c <- x</code></li>
<li>读:<code><-c</code> 或 <code>v := <-c</code> 或 <code>v, ok := <-c</code></li>
</ul>
<h3 id="思考方法一:通过“写”chan-实现"><a href="#思考方法一:通过“写”chan-实现" class="headerlink" title="思考方法一:通过“写”chan 实现"></a>思考方法一:通过“写”chan 实现</h3><p>“写”肯定不能作为判断,总不能为了判断 chan 是否 close,我尝试往里面写数据吧?这个会导致 <code>chansend</code> 里面直接 panic 的,如下:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">chansend</span><span class="params">(c *hchan, ep unsafe.Pointer, block <span class="keyword">bool</span>, callerpc <span class="keyword">uintptr</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> <span class="comment">// 当 channel close 之后的处理逻辑</span></span><br><span class="line"> <span class="keyword">if</span> c.closed != <span class="number">0</span> {</span><br><span class="line"> unlock(&c.lock)</span><br><span class="line"> <span class="built_in">panic</span>(plainError(<span class="string">"send on closed channel"</span>))</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>当然了,你路子要是野一点,这样做技术上也能实现,因为 panic 是可以捕捉的,只不过这也太野了吧,不推荐。</p>
<h3 id="思考方法二:通过“读”chan-实现"><a href="#思考方法二:通过“读”chan-实现" class="headerlink" title="思考方法二:通过“读”chan 实现"></a>思考方法二:通过“读”chan 实现</h3><p>“读”来判断。分析函数 <code>chanrecv</code> 可以知道,当尝试从一个已经 close 的 chan 读数据的时候,返回 (selected=true, received=false),我们通过 received = false 即可知道 channel 是否 close 。<code>chanrecv</code> 有如下代码:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">chanrecv</span><span class="params">(c *hchan, ep unsafe.Pointer, block <span class="keyword">bool</span>)</span> <span class="params">(selected, received <span class="keyword">bool</span>)</span></span> {</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> <span class="comment">// 当 channel close 之后的处理逻辑</span></span><br><span class="line"> <span class="keyword">if</span> c.closed != <span class="number">0</span> && c.qcount == <span class="number">0</span> {</span><br><span class="line"> unlock(&c.lock)</span><br><span class="line"> <span class="keyword">if</span> ep != <span class="literal">nil</span> {</span><br><span class="line"> typedmemclr(c.elemtype, ep)</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span>, <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>所以,我们现在知道了,可以通过 “读”的效果来判断,但是我们不能直接写成这样:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// 错误示例</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">isChanClose</span><span class="params">(ch <span class="keyword">chan</span> <span class="keyword">int</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> _, ok := <- c</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>上面是个<strong>错误示例</strong>,因为 <code>_, ok := <-c</code> 编译出来的是 <code>chanrecv2</code> ,这个函数 block 赋值 true 传入的,所以当 c 是正常的时候,这里是阻塞的,所以这个不能用来作为一个正常的函数调用,因为会卡死协程,怎么解决这个问题?用 <code>select</code> 和 <code><-chan</code> 来结合可以解决这个问题,<code>select</code> 和 <code><-chan</code> 结合起来是对应 <code>selectnbrecv</code> 和 <code>selectnbrecv2</code> 这两个函数,这两个函数是非阻塞的( <code>block = false</code> )。</p>
<p>正确示例:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">isChanClose</span><span class="params">(ch <span class="keyword">chan</span> <span class="keyword">int</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> _, received := <- ch:</span><br><span class="line"> <span class="keyword">return</span> !received</span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>网上很多人举了一个 <code>isChanClose</code> 错误的例子,错误示例:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">isChanClose</span><span class="params">(ch <span class="keyword">chan</span> <span class="keyword">int</span>)</span> <span class="title">bool</span></span> {</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> <- ch:</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line"> <span class="keyword">default</span>:</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">false</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>思考下:为什么第一个例子是对的,第二个例子是错的?</p>
<p>因为,第一个例子编译出来对应的函数是 <code>selectnbrecv2</code> ,第二个例子编译出来对应的是 <code>selectnbrecv1</code> ,这两个函数的区别在于 <code>selectnbrecv2</code> 多了一个返回参数 <code>received</code>,只有这个函数才能指明是否元素出队成功,而 <code>selected</code> 只是判断是否要进到 select case 分支。我们通过 <code>received</code> 这个返回值(其实是一个入参,只不过是指针类型,函数内可修改)来反向推断 chan 是否 close 了。</p>
<h4 id="小结:"><a href="#小结:" class="headerlink" title="小结:"></a>小结:</h4><ol>
<li>case 的代码必须是 <code>_, received := <- ch</code> 的形式,如果仅仅是 <code><- ch</code> 来判断,是错的逻辑,因为我们关注的是 <code>received</code> 的值;</li>
<li>select 必须要有 default 分支,否则会阻塞函数,我们这个函数要保证一定能正常返回;</li>
</ol>
<h3 id="chan-close-原则"><a href="#chan-close-原则" class="headerlink" title="chan close 原则"></a>chan close 原则</h3><ol>
<li>永远不要尝试在读取端关闭 channel ,写入端无法知道 channel 是否已经关闭,往已关闭的 channel 写数据会 panic ;</li>
<li>一个写入端,在这个写入端可以放心关闭 channel;</li>
<li>多个写入端时,不要在写入端关闭 channel ,其他写入端无法知道 channel 是否已经关闭,关闭已经关闭的 channel 会发生 panic (你要想个办法保证只有一个人调用 close);</li>
<li>channel 作为函数参数的时候,最好带方向;</li>
</ol>
<p>其实这些原则只有一点:一定要是安全的是否才能去 close channel 。</p>
<h3 id="其实并不需要-isChanClose-函数"><a href="#其实并不需要-isChanClose-函数" class="headerlink" title="其实并不需要 isChanClose 函数 !!!"></a>其实并不需要 isChanClose 函数 !!!</h3><p>上面实现的 <code>isChanClose</code> 是可以判断出 channel 是否 close,但是适用场景优先,因为可能等你 <code>isChanClose</code> 判断的时候返回值 false,你以为 channel 还是正常的,但是下一刻 channel 被关闭了,这个时候往里面“写”数据就又会 panic ,如下:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> isChanClose( c ) {</span><br><span class="line"> <span class="comment">// 关闭的场景,exit </span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line">}</span><br><span class="line"><span class="comment">// 未关闭的场景,继续执行(可能还是会 panic)</span></span><br><span class="line">c <- x</span><br></pre></td></tr></table></figure></p>
<p>因为判断之后还是有时间窗,所以 <code>isChanClose</code> 的适用还是有限,那么是否有更好的办法?</p>
<p>我们换一个思路,你其实并不是一定要判断 channel 是否 close,真正的目的是:<strong>安全的使用 channel,避免使用到已经关闭的 closed channel,从而导致 panic</strong> 。</p>
<p>这个问题的本质上是保证一个事件的时序,官方推荐通过 <code>context</code> 来配合使用,我们可以通过一个 ctx 变量来指明 close 事件,而不是直接去判断 channel 的一个状态。举个栗子:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> {</span><br><span class="line"><span class="keyword">case</span> <-ctx.Done():</span><br><span class="line"> <span class="comment">// ... exit</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"><span class="keyword">case</span> v, ok := <-c:</span><br><span class="line"> <span class="comment">// do something....</span></span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line"> <span class="comment">// do default ....</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p><code>ctx.Done()</code> 事件发生之后,我们就明确不去读 channel 的数据。</p>
<p>或者<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">select</span> {</span><br><span class="line"><span class="keyword">case</span> <-ctx.Done():</span><br><span class="line"> <span class="comment">// ... exit</span></span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line"> <span class="comment">// push </span></span><br><span class="line"> c <- x</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p><code>ctx.Done()</code> 事件发生之后,我们就明确不写数据到 channel ,或者不从 channel 里读数据,那么保证这个时序即可。就一定不会有问题。</p>
<p>我们只需要确保一点:</p>
<ol>
<li>触发时序保证:一定要先触发 ctx.Done() 事件,再去做 close channel 的操作,保证这个时序的才能保证 select 判断的时候没有问题;<br> a. 只有这个时序,才能保证在获悉到 Done 事件的时候,一切还是安全的;</li>
<li>条件判断顺序:select 的 case 先判断 ctx.Done() 事件,这个很重要哦,否则很有可能先执行了 chan 的操作从而导致 panic 问题;</li>
</ol>
<h2 id="怎么优雅关闭-chan-?"><a href="#怎么优雅关闭-chan-?" class="headerlink" title="怎么优雅关闭 chan ?"></a>怎么优雅关闭 chan ?</h2><h3 id="方法一:panic-recover"><a href="#方法一:panic-recover" class="headerlink" title="方法一:panic-recover"></a>方法一:panic-recover</h3><p>关闭一个 channel 直接调用 close 即可,但是关闭一个已经关闭的 channel 会导致 panic,怎么办?panic-recover 配合使用即可。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">SafeClose</span><span class="params">(ch <span class="keyword">chan</span> <span class="keyword">int</span>)</span> <span class="params">(closed <span class="keyword">bool</span>)</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">if</span> <span class="built_in">recover</span>() != <span class="literal">nil</span> {</span><br><span class="line"> closed = <span class="literal">false</span></span><br><span class="line"> }</span><br><span class="line"> }()</span><br><span class="line"> <span class="comment">// 如果 ch 是一个已经关闭的,会 panic 的,然后被 recover 捕捉到;</span></span><br><span class="line"> <span class="built_in">close</span>(ch)</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">true</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这并不优雅。</p>
<h3 id="方法二:sync-Once"><a href="#方法二:sync-Once" class="headerlink" title="方法二:sync.Once"></a>方法二:sync.Once</h3><p>可以使用 <code>sync.Once</code> 来确保 <code>close</code> 只执行一次。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> ChanMgr <span class="keyword">struct</span> {</span><br><span class="line"> C <span class="keyword">chan</span> <span class="keyword">int</span></span><br><span class="line"> once sync.Once</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">NewChanMgr</span><span class="params">()</span> *<span class="title">ChanMgr</span></span> {</span><br><span class="line"> <span class="keyword">return</span> &ChanMgr{C: <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>)}</span><br><span class="line">}</span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(cm *ChanMgr)</span> <span class="title">SafeClose</span><span class="params">()</span></span> {</span><br><span class="line"> cm.once.Do(<span class="function"><span class="keyword">func</span><span class="params">()</span></span> { <span class="built_in">close</span>(cm.C) })</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这看着还可以。</p>
<h3 id="方法三:事件同步来解决"><a href="#方法三:事件同步来解决" class="headerlink" title="方法三:事件同步来解决"></a>方法三:事件同步来解决</h3><p>对于关闭 channel 这个我们有两个简要的原则:</p>
<ol>
<li>永远不要尝试在读端关闭 channel ;</li>
<li>永远只允许一个 goroutine(比如,只用来执行关闭操作的一个 goroutine )执行关闭操作;</li>
</ol>
<p>可以使用 <code>sync.WaitGroup</code> 来同步这个关闭事件,遵守以上的原则,举几个例子:</p>
<h4 id="第一个例子:一个-sender"><a href="#第一个例子:一个-sender" class="headerlink" title="第一个例子:一个 sender"></a>第一个例子:一个 sender</h4><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">"sync"</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// channel 初始化</span></span><br><span class="line"> c := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>, <span class="number">10</span>)</span><br><span class="line"> <span class="comment">// 用来 recevivers 同步事件的</span></span><br><span class="line"> wg := sync.WaitGroup{}</span><br><span class="line"></span><br><span class="line"> <span class="comment">// sender(写端)</span></span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// 入队</span></span><br><span class="line"> c <- <span class="number">1</span></span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> <span class="comment">// 满足某些情况,则 close channel</span></span><br><span class="line"> <span class="built_in">close</span>(c)</span><br><span class="line"> }()</span><br><span class="line"></span><br><span class="line"> <span class="comment">// receivers (读端)</span></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> wg.Add(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> <span class="comment">// ... 处理 channel 里的数据</span></span><br><span class="line"> <span class="keyword">for</span> v := <span class="keyword">range</span> c {</span><br><span class="line"> _ = v</span><br><span class="line"> }</span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// 等待所有的 receivers 完成;</span></span><br><span class="line"> wg.Wait()</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>这里例子里面,我们在 sender 的 goroutine 关闭 channel,因为只有一个 sender,所以关闭自然是安全的。receiver 使用 <code>WaitGroup</code> 来同步事件,receiver 的 for 循环只有在 channel close 之后才会退出,主协程的 <code>wg.Wait()</code> 语句只有所有的 receivers 都完成才会返回。所以,事件的顺序是:</p>
<ol>
<li>写端入队一个整形元素</li>
<li>关闭 channel</li>
<li>所有的读端安全退出</li>
<li>主协程返回</li>
</ol>
<p>一切都是安全的</p>
<h4 id="第二个例子:多个-sender"><a href="#第二个例子:多个-sender" class="headerlink" title="第二个例子:多个 sender"></a>第二个例子:多个 sender</h4><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"> <span class="string">"context"</span></span><br><span class="line"> <span class="string">"sync"</span></span><br><span class="line"> <span class="string">"time"</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// channel 初始化</span></span><br><span class="line"> c := <span class="built_in">make</span>(<span class="keyword">chan</span> <span class="keyword">int</span>, <span class="number">10</span>)</span><br><span class="line"> <span class="comment">// 用来 recevivers 同步事件的</span></span><br><span class="line"> wg := sync.WaitGroup{}</span><br><span class="line"> <span class="comment">// 上下文</span></span><br><span class="line"> ctx, cancel := context.WithCancel(context.TODO())</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 专门关闭的协程</span></span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> time.Sleep(<span class="number">2</span> * time.Second)</span><br><span class="line"> cancel()</span><br><span class="line"> <span class="comment">// ... 某种条件下,关闭 channel</span></span><br><span class="line"> <span class="built_in">close</span>(c)</span><br><span class="line"> }()</span><br><span class="line"></span><br><span class="line"> <span class="comment">// senders(写端)</span></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">(ctx context.Context, id <span class="keyword">int</span>)</span></span> {</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> <-ctx.Done():</span><br><span class="line"> <span class="keyword">return</span></span><br><span class="line"> <span class="keyword">case</span> c <- id: <span class="comment">// 入队</span></span><br><span class="line"> <span class="comment">// ...</span></span><br><span class="line"> }</span><br><span class="line"> }(ctx, i)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// receivers(读端)</span></span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < <span class="number">10</span>; i++ {</span><br><span class="line"> wg.Add(<span class="number">1</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="keyword">defer</span> wg.Done()</span><br><span class="line"> <span class="comment">// ... 处理 channel 里的数据</span></span><br><span class="line"> <span class="keyword">for</span> v := <span class="keyword">range</span> c {</span><br><span class="line"> _ = v</span><br><span class="line"> }</span><br><span class="line"> }()</span><br><span class="line"> }</span><br><span class="line"> <span class="comment">// 等待所有的 receivers 完成;</span></span><br><span class="line"> wg.Wait()</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>这个例子我们看到有多个 sender 和 receiver ,这种情况我们还是要保证一点:close(ch) 操作的只能有一个人,我们单独抽出来一个 goroutine 来做这个事情,并且使用 context 来做事件同步,事件发生顺序是:</p>
<ol>
<li>10 个写端协程(sender)运行,投递元素;</li>
<li>10 个读端协程(receiver)运行,读取元素;</li>
<li>2 分钟超时之后,单独协程执行 <code>close(channel)</code> 操作;</li>
<li>主协程返回;</li>
</ol>
<p>一切都是安全的。</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><ol>
<li>channel 并没有直接提供判断是否 close 的接口,官方推荐使用 context 和 select 语法配合使用,事件通知的方式,达到优雅判断 channel 关闭的效果;</li>
<li>channel 关闭姿势也有讲究,永远不要尝试在读端关闭,永远保持一个关闭入口处,使用 sync.WaitGroup 和 context 实现事件同步,达到优雅关闭效果;</li>
</ol>
<p>作者:奇伢 来源:奇伢云存储</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="大纲"><a href="#大纲" class="headerlink" title="大纲"></a>大纲</h2><ul>
<li>Go 为什么没有判断 close 的接口?</li>
<li>Go 关闭 channel 究竟做了什么?<br> -<code>closechan</code></li>
<li>一个判断 chan 是否 close 的函数<ul>
<li>思考方法一:通过“写”chan 实现</li>
<li>思考方法二:通过“读”chan 实现</li>
<li>chan close 原则</li>
<li>其实并不需要 <code>isChanClose</code> 函数 !!!</li>
</ul>
</li>
<li>怎么优雅关闭 chan ?<ul>
<li>方法一:panic-recover</li>
<li>方法二:sync.Once</li>
<li>方法三:事件同步来解决</li>
</ul>
</li>
<li>总结</li>
</ul>
Go Sync.Pool 背后的想法
http://team.jiunile.com//blog/2020/11/go-sync-pool.html
2020-11-14T14:00:00.000Z
2020-11-13T09:45:15.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>我最近在我的一个项目中遇到了垃圾回收问题。大量对象被重复分配,并导致 GC 的巨大工作量。使用 <code>sync.Pool</code>,我能够减少分配和 GC 工作负载。</p>
<h2 id="什么是-sync-Pool?"><a href="#什么是-sync-Pool?" class="headerlink" title="什么是 sync.Pool?"></a>什么是 sync.Pool?</h2><p>Go 1.3 版本的亮点之一是同步池。它是 <code>sync</code> 包下的一个组件,用于创建自我管理的临时检索对象池。</p>
<h2 id="为什么要使用-sync-Pool?"><a href="#为什么要使用-sync-Pool?" class="headerlink" title="为什么要使用 sync.Pool?"></a>为什么要使用 sync.Pool?</h2><p>我们希望尽可能减少 GC 开销。频繁的内存分配和回收会给 GC 带来沉重的负担。<code>sync.Poll</code> 可以缓存暂时不使用的对象,并在下次需要时直接使用它们(无需重新分配)。这可能会减少 GC 工作负载并提高性能。<br><a id="more"></a></p>
<h2 id="怎么使用-sync-Pool?"><a href="#怎么使用-sync-Pool?" class="headerlink" title="怎么使用 sync.Pool?"></a>怎么使用 sync.Pool?</h2><p>首先,您需要设置新函数。当池中没有缓存对象时将使用此函数。之后,您只需要使用 <code>Get</code> 和 <code>Put</code> 方法来检索和返回对象。另外,池在第一次使用后绝对不能复制。</p>
<p>由于 <code>New</code> 函数类型是 <code>func() interface{}</code>,<code>Get</code> 方法返回一个 <code>interface{}</code>。为了得到具体对象,你需要做一个类型断言。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// A dummy struct</span></span><br><span class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</span><br><span class="line"> Name <span class="keyword">string</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Initializing pool</span></span><br><span class="line"><span class="keyword">var</span> personPool = sync.Pool{</span><br><span class="line"> <span class="comment">// New optionally specifies a function to generate</span></span><br><span class="line"> <span class="comment">// a value when Get would otherwise return nil.</span></span><br><span class="line"> New: <span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">interface</span></span>{} { <span class="keyword">return</span> <span class="built_in">new</span>(Person) },</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="comment">// Main function</span></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// Get hold of an instance</span></span><br><span class="line"> newPerson := personPool.Get().(*Person)</span><br><span class="line"> <span class="comment">// Defer release function</span></span><br><span class="line"> <span class="comment">// After that the same instance is </span></span><br><span class="line"> <span class="comment">// reusable by another routine</span></span><br><span class="line"> <span class="keyword">defer</span> personPool.Put(newPerson)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Using the instance</span></span><br><span class="line"> newPerson.Name = <span class="string">"Jack"</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<h2 id="基准测试"><a href="#基准测试" class="headerlink" title="基准测试"></a>基准测试</h2><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</span><br><span class="line"> Age <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> personPool = sync.Pool{</span><br><span class="line"> New: <span class="function"><span class="keyword">func</span><span class="params">()</span> <span class="title">interface</span></span>{} { <span class="keyword">return</span> <span class="built_in">new</span>(Person) },</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkWithoutPool</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> p *Person</span><br><span class="line"> b.ReportAllocs()</span><br><span class="line"> b.ResetTimer()</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> <span class="keyword">for</span> j := <span class="number">0</span>; j < <span class="number">10000</span>; j++ {</span><br><span class="line"> p = <span class="built_in">new</span>(Person)</span><br><span class="line"> p.Age = <span class="number">23</span></span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkWithPool</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> p *Person</span><br><span class="line"> b.ReportAllocs()</span><br><span class="line"> b.ResetTimer()</span><br><span class="line"> <span class="keyword">for</span> i := <span class="number">0</span>; i < b.N; i++ {</span><br><span class="line"> <span class="keyword">for</span> j := <span class="number">0</span>; j < <span class="number">10000</span>; j++ {</span><br><span class="line"> p = personPool.Get().(*Person)</span><br><span class="line"> p.Age = <span class="number">23</span></span><br><span class="line"> personPool.Put(p)</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>测试结果:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">BenchmarkWithoutPool</span><br><span class="line">BenchmarkWithoutPool-8 160698 ns/op 80001 B/op 10000 allocs/op</span><br><span class="line">BenchmarkWithPool</span><br><span class="line">BenchmarkWithPool-8 191163 ns/op 0 B/op 0 allocs/op</span><br></pre></td></tr></table></figure></p>
<h2 id="权衡"><a href="#权衡" class="headerlink" title="权衡"></a>权衡</h2><p>生活中的一切都是一种权衡。池也有它的性能成本。使用 <code>sync.Pool</code> 比简单的初始化要慢得多。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkPool</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> <span class="keyword">var</span> p sync.Pool</span><br><span class="line"> b.RunParallel(<span class="function"><span class="keyword">func</span><span class="params">(pb *testing.PB)</span></span> {</span><br><span class="line"> <span class="keyword">for</span> pb.Next() {</span><br><span class="line"> p.Put(<span class="number">1</span>)</span><br><span class="line"> p.Get()</span><br><span class="line"> }</span><br><span class="line"> })</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">BenchmarkAllocation</span><span class="params">(b *testing.B)</span></span> {</span><br><span class="line"> b.RunParallel(<span class="function"><span class="keyword">func</span><span class="params">(pb *testing.PB)</span></span> {</span><br><span class="line"> <span class="keyword">for</span> pb.Next() {</span><br><span class="line"> i := <span class="number">0</span></span><br><span class="line"> i = i</span><br><span class="line"> }</span><br><span class="line"> })</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>压测结果:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">BenchmarkPool</span><br><span class="line">BenchmarkPool-8 283395016 4.40 ns/op</span><br><span class="line">BenchmarkAllocation</span><br><span class="line">BenchmarkAllocation-8 1000000000 0.344 ns/op</span><br></pre></td></tr></table></figure></p>
<h2 id="sync-Pool-是如何工作的?"><a href="#sync-Pool-是如何工作的?" class="headerlink" title="sync.Pool 是如何工作的?"></a>sync.Pool 是如何工作的?</h2><p><code>sync.Pool</code> 有两个对象容器: 本地池 (活动) 和受害者缓存 (存档)。</p>
<p>根据 <code>sync/pool.go</code> ,包 <code>init</code> 函数作为清理池的方法<a href="https://golang.org/src/sync/pool.go?s=8003:8060#L271" target="_blank" rel="external">注册到运行时</a>。此方法将由 GC 触发。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">init</span><span class="params">()</span></span> {</span><br><span class="line"> runtime_registerPoolCleanup(poolCleanup)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>当 GC 被触发时,受害者缓存中的对象将被收集,然后本地池中的对象将被移动到受害者缓存中。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">poolCleanup</span><span class="params">()</span></span> {</span><br><span class="line"> <span class="comment">// Drop victim caches from all pools.</span></span><br><span class="line"> <span class="keyword">for</span> _, p := <span class="keyword">range</span> oldPools {</span><br><span class="line"> p.victim = <span class="literal">nil</span></span><br><span class="line"> p.victimSize = <span class="number">0</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Move primary cache to victim cache.</span></span><br><span class="line"> <span class="keyword">for</span> _, p := <span class="keyword">range</span> allPools {</span><br><span class="line"> p.victim = p.local</span><br><span class="line"> p.victimSize = p.localSize</span><br><span class="line"> p.local = <span class="literal">nil</span></span><br><span class="line"> p.localSize = <span class="number">0</span></span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> oldPools, allPools = allPools, <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>新对象被放入本地池中。调用 <code>Put</code> 方法也会将对象放入本地池中。调用 <code>Get</code> 方法将首先从受害者缓存中获取对象,如果受害者缓存为空,则对象将从本地池中获取。<br><img src="/images/go/syncpool_1.gif" alt="sync.Pool localPool and victimCache"></p>
<p>供你参考,Go 1.12 sync.pool 实现使用基于 <code>mutex</code> 的锁,用于来自多个 Goroutines 的线程安全操作。Go 1.13 <a href="https://github.com/golang/go/commit/d5fd2dd6a17a816b7dfd99d4df70a85f1bf0de31#diff-491b0013c82345bf6cfa937bd78b690d" target="_blank" rel="external">引入了一个双链表</a>作为共享池,它删除了 <code>mutex</code> 并改善了共享访问。</p>
<h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>当有一个昂贵的对象需要频繁创建时,使用 <code>sync.Pool</code> 是非常有益的。</p>
<p>译自:<a href="https://medium.com/swlh/go-the-idea-behind-sync-pool-32da5089df72" target="_blank" rel="external">https://medium.com/swlh/go-the-idea-behind-sync-pool-32da5089df72</a></p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>我最近在我的一个项目中遇到了垃圾回收问题。大量对象被重复分配,并导致 GC 的巨大工作量。使用 <code>sync.Pool</code>,我能够减少分配和 GC 工作负载。</p>
<h2 id="什么是-sync-Pool?"><a href="#什么是-sync-Pool?" class="headerlink" title="什么是 sync.Pool?"></a>什么是 sync.Pool?</h2><p>Go 1.3 版本的亮点之一是同步池。它是 <code>sync</code> 包下的一个组件,用于创建自我管理的临时检索对象池。</p>
<h2 id="为什么要使用-sync-Pool?"><a href="#为什么要使用-sync-Pool?" class="headerlink" title="为什么要使用 sync.Pool?"></a>为什么要使用 sync.Pool?</h2><p>我们希望尽可能减少 GC 开销。频繁的内存分配和回收会给 GC 带来沉重的负担。<code>sync.Poll</code> 可以缓存暂时不使用的对象,并在下次需要时直接使用它们(无需重新分配)。这可能会减少 GC 工作负载并提高性能。<br>
我在 Go 中犯了 5 个错误
http://team.jiunile.com//blog/2020/11/go-5-mistakes.html
2020-11-13T14:00:00.000Z
2020-11-13T04:58:39.000Z
<blockquote>
<p>人皆犯错,宽恕是德 — Alexander Pope</p>
</blockquote>
<p>这些都是我在写 Go 中犯的错误。尽管这些可能不会导致任何类型的错误,但它们可能会潜在地影响软件。</p>
<h2 id="1-内循环"><a href="#1-内循环" class="headerlink" title="1 内循环"></a>1 内循环</h2><p>有几种方法可以造成循环内部的混乱,你需要注意。<br><a id="more"></a></p>
<h3 id="1-1-使用引用循环迭代变量"><a href="#1-1-使用引用循环迭代变量" class="headerlink" title="1.1 使用引用循环迭代变量"></a>1.1 使用引用循环迭代变量</h3><p>由于效率的原因,循环迭代变量是单个变量,在每次循环迭代中采用不同的值。这可能会导致不知情的行为。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">in := []<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> out []*<span class="keyword">int</span></span><br><span class="line"><span class="keyword">for</span> _, v := <span class="keyword">range</span> in {</span><br><span class="line"> out = <span class="built_in">append</span>(out, &v)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">fmt.Println(<span class="string">"Values:"</span>, *out[<span class="number">0</span>], *out[<span class="number">1</span>], *out[<span class="number">2</span>])</span><br><span class="line">fmt.Println(<span class="string">"Addresses:"</span>, out[<span class="number">0</span>], out[<span class="number">1</span>], out[<span class="number">2</span>])</span><br></pre></td></tr></table></figure></p>
<p>结果将是:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Values: 3 3 3</span><br><span class="line">Addresses: 0xc000014188 0xc000014188 0xc000014188</span><br></pre></td></tr></table></figure></p>
<p>正如你所看到的,<code>out</code> 切片中的所有元素都是 3。实际上,实际上很容易解释为什么会发生这种情况:在每次迭代中,我们都会将 <code>v</code> 的地址附加到 <code>out</code> 切片中。如前所述,<code>v</code> 是在每次迭代中接受新值的单个变量。因此,正如您在输出的第二行中看到的,地址是相同的,并且所有地址都指向相同的值。</p>
<p>简单的解决方法是将循环迭代器变量复制到新变量中:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">in := []<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> out []*<span class="keyword">int</span></span><br><span class="line"><span class="keyword">for</span> _, v := <span class="keyword">range</span> in {</span><br><span class="line"> v := v</span><br><span class="line"> out = <span class="built_in">append</span>(out, &v)</span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">fmt.Println(<span class="string">"Values:"</span>, *out[<span class="number">0</span>], *out[<span class="number">1</span>], *out[<span class="number">2</span>])</span><br><span class="line">fmt.Println(<span class="string">"Addresses:"</span>, out[<span class="number">0</span>], out[<span class="number">1</span>], out[<span class="number">2</span>])</span><br></pre></td></tr></table></figure></p>
<p>新的输出:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Values: 1 2 3</span><br><span class="line">Addresses: 0xc0000b6010 0xc0000b6018 0xc0000b6020</span><br></pre></td></tr></table></figure></p>
<p>同样的问题可以找到正在 Goroutine 中使用的循环迭代变量。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">list := []<span class="keyword">int</span>{<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>}</span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> _, v := <span class="keyword">range</span> list {</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> fmt.Printf(<span class="string">"%d "</span>, v)</span><br><span class="line"> }()</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>结果将是:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">3 3 3</span><br></pre></td></tr></table></figure></p>
<p>它可以使用上面提到的相同的解决方案来修复。注意,如果不使用 Goroutine 运行该函数,代码将按照预期运行。</p>
<h3 id="1-2-在循环中调用-WaitGroup-Wait"><a href="#1-2-在循环中调用-WaitGroup-Wait" class="headerlink" title="1.2 在循环中调用 WaitGroup.Wait"></a>1.2 在循环中调用 WaitGroup.Wait</h3><p>使用 <code>WaitGroup</code> 类型的共享变量会犯此错误,如下面的代码所示,当第 5 行的 <code>Done()</code> 被调用 <code>len(tasks)</code> 次数时,第 7 行的 <code>Wait()</code> 只能被解除阻塞,因为它被用作参数在第 2 行调用 <code>Add()</code>。但是,<code>Wait()</code> 在循环中被调用,因此在下一个迭代中,它会阻止在第 4 行创建 Goroutine。简单的解决方案是将 <code>Wait()</code> 的调用移出循环。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> wg sync.WaitGroup</span><br><span class="line">wg.Add(<span class="built_in">len</span>(tasks))</span><br><span class="line"><span class="keyword">for</span> _, t := <span class="keyword">range</span> tasks {</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">(t *task)</span></span> { </span><br><span class="line"> <span class="keyword">defer</span> group.Done()</span><br><span class="line"> }(t)</span><br><span class="line"> <span class="comment">// group.Wait()</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line">group.Wait()</span><br></pre></td></tr></table></figure></p>
<h3 id="1-3-在循环中使用-defer"><a href="#1-3-在循环中使用-defer" class="headerlink" title="1.3 在循环中使用 defer"></a>1.3 在循环中使用 defer</h3><p><code>defer</code> 直到函数返回才执行。除非你确定你在做什么,否则你不应该在循环中使用 <code>defer</code>。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> mutex sync.Mutex</span><br><span class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</span><br><span class="line"> Age <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line">persons := <span class="built_in">make</span>([]Person, <span class="number">10</span>)</span><br><span class="line"><span class="keyword">for</span> _, p := <span class="keyword">range</span> persons {</span><br><span class="line"> mutex.Lock()</span><br><span class="line"> <span class="comment">// defer mutex.Unlock()</span></span><br><span class="line"> p.Age = <span class="number">13</span></span><br><span class="line"> mutex.Unlock()</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>在上面的例子中,如果你使用第 8 行而不是第 10 行,下一次迭代就不能持有互斥锁,因为锁已经在使用中,并且循环永远阻塞。</p>
<p>如果你真的需要使用 defer 内循环,你可能想委托另一个函数来做这项工作。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">var</span> mutex sync.Mutex</span><br><span class="line"><span class="keyword">type</span> Person <span class="keyword">struct</span> {</span><br><span class="line"> Age <span class="keyword">int</span></span><br><span class="line">}</span><br><span class="line">persons := <span class="built_in">make</span>([]Person, <span class="number">10</span>)</span><br><span class="line"><span class="keyword">for</span> _, p := <span class="keyword">range</span> persons {</span><br><span class="line"> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> mutex.Lock()</span><br><span class="line"> <span class="keyword">defer</span> mutex.Unlock()</span><br><span class="line"> p.Age = <span class="number">13</span></span><br><span class="line"> }()</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>但是,有时使用 <code>defer</code> 在循环可能会变得很方便。所以你真的需要知道你在做什么。</p>
<blockquote>
<p>Go 不能容忍愚蠢者</p>
</blockquote>
<h2 id="2-发送到一个无保证的-channel"><a href="#2-发送到一个无保证的-channel" class="headerlink" title="2 发送到一个无保证的 channel"></a>2 发送到一个无保证的 channel</h2><p>您可以将值从一个 Goroutine 发送到 channels,并将这些值接收到另一个 Goroutine。默认情况下,发送和接收,直到另一方准备好。这允许 Goroutines 在没有显式锁或条件变量的情况下进行同步。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">doReq</span><span class="params">(timeout time.Duration)</span> <span class="title">obj</span></span> {</span><br><span class="line"> <span class="comment">// ch :=make(chan obj)</span></span><br><span class="line"> ch := <span class="built_in">make</span>(<span class="keyword">chan</span> obj, <span class="number">1</span>)</span><br><span class="line"> <span class="keyword">go</span> <span class="function"><span class="keyword">func</span><span class="params">()</span></span> {</span><br><span class="line"> obj := do()</span><br><span class="line"> ch <- result</span><br><span class="line"> } ()</span><br><span class="line"> <span class="keyword">select</span> {</span><br><span class="line"> <span class="keyword">case</span> result = <- ch :</span><br><span class="line"> <span class="keyword">return</span> result</span><br><span class="line"> <span class="keyword">case</span><- time.After(timeout):</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span> </span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>让我们检查一下上面的代码。<code>doReq</code> 函数在第 4 行创建一个子 Goroutine 来处理请求,这在Go服务程序中是一种常见的做法。子 Goroutine 执行 <code>do</code> 函数并通过第 6 行通道 <code>ch</code> 将结果发送回父节点。子进程会在第 6 行阻塞,直到父进程在第 9 行接收到 <code>ch</code> 的结果。同时,父进程将阻塞 <code>select</code>,直到子进程将结果发送给 <code>ch</code>(第9行)或发生超时(第11行)。如果超时发生在更早的时候,父函数将从第 12 行 <code>doReq</code> 方法返回,并且没有人可以再接收 <code>ch</code> 的结果,这将导致子函数永远被阻塞。解决方案是将 <code>ch</code> 从无缓冲通道更改为缓冲通道,这样即使父及退出,子 Goroutine 也始终可以发送结果。另一个修复方法是在第 6 行使用默认为空的 <code>select</code> 语句,这样如果没有 Goroutine 接收 <code>ch</code>,就会发生默认情况。尽管这种解决方案可能并不总是有效。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">...</span><br><span class="line"><span class="keyword">select</span> { </span><br><span class="line"><span class="keyword">case</span> ch <- result: </span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line">}</span><br><span class="line">...</span><br></pre></td></tr></table></figure></p>
<h2 id="3-不使用接口"><a href="#3-不使用接口" class="headerlink" title="3 不使用接口"></a>3 不使用接口</h2><p>接口可以使代码更加灵活。这是在代码中引入多态的一种方法。接口允许您请求一组行为,而不是特定类型。不使用接口可能不会导致任何错误,但它会导致代码不简单、不灵活和不具有可扩展性。</p>
<p>在众多接口中,<code>io.Reader</code> 和 <code>io.Writer</code> 可能是最受欢迎的。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Reader <span class="keyword">interface</span> {</span><br><span class="line"> Read(p []<span class="keyword">byte</span>) (n <span class="keyword">int</span>, err error)</span><br><span class="line">}</span><br><span class="line"><span class="keyword">type</span> Writer <span class="keyword">interface</span> {</span><br><span class="line"> Write(p []<span class="keyword">byte</span>) (n <span class="keyword">int</span>, err error)</span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>这些接口可以非常强大。假设您要将对象写入文件中,因此您定义了一个 Save 方法:<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(o *obj)</span> <span class="title">Save</span><span class="params">(file os.File)</span> <span class="title">error</span></span></span><br></pre></td></tr></table></figure></p>
<p>如果您第二天需要写入 <code>http.ResponseWriter</code> 该怎么办?您不想定义新方法。是吧?所以使用 <code>io.Writer</code>。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(o *obj)</span> <span class="title">Save</span><span class="params">(w io.Writer)</span> <span class="title">error</span></span></span><br></pre></td></tr></table></figure></p>
<p>还有一个重要的注意事项,你应该知道,总是要求你要使用的行为。在上面的例子中,请求一个<code>io.ReadWriteCloser</code> 也可以工作得很好,但当你要使用的唯一方法是 <code>Write</code> 时,这不是一个最佳实践。接口越大,抽象就越弱。</p>
<p>所以大多数时候你最好专注于行为而不是具体的类型。</p>
<h2 id="4-不好的顺序结构"><a href="#4-不好的顺序结构" class="headerlink" title="4 不好的顺序结构"></a>4 不好的顺序结构</h2><p>这个错误也不会导致任何错误,但是它会导致更多的内存使用。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> BadOrderedPerson <span class="keyword">struct</span> {</span><br><span class="line"> Veteran <span class="keyword">bool</span> <span class="comment">// 1 byte</span></span><br><span class="line"> Name <span class="keyword">string</span> <span class="comment">// 16 byte</span></span><br><span class="line"> Age <span class="keyword">int32</span> <span class="comment">// 4 byte</span></span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> OrderedPerson <span class="keyword">struct</span> {</span><br><span class="line"> Name <span class="keyword">string</span></span><br><span class="line"> Age <span class="keyword">int32</span></span><br><span class="line"> Veteran <span class="keyword">bool</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>似乎两种类型的大小都相同,为 21 个字节,但结果显示出完全不同。使用 <code>GOARCH=amd64</code> 编译代码,<code>BadOrderedPerson</code> 类型分配 32 字节,而 <code>OrderedPerson</code> 类型分配 24 字节。为什么?原因是<a href="https://en.wikipedia.org/wiki/Data_structure_alignment" target="_blank" rel="external">数据结构对齐</a>。在 64 位体系结构中,内存分配 8 字节的连续数据包。需要添加的填充可以通过以下方式计算:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">padding = (align - (offset mod align)) mod align</span><br><span class="line">aligned = offset + padding</span><br><span class="line"> = offset + ((align - (offset mod align)) mod align)</span><br></pre></td></tr></table></figure></p>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> BadOrderedPerson <span class="keyword">struct</span> {</span><br><span class="line"> Veteran <span class="keyword">bool</span> <span class="comment">// 1 byte</span></span><br><span class="line"> _ [<span class="number">7</span>]<span class="keyword">byte</span> <span class="comment">// 7 byte: padding for alignment</span></span><br><span class="line"> Name <span class="keyword">string</span> <span class="comment">// 16 byte</span></span><br><span class="line"> Age <span class="keyword">int32</span> <span class="comment">// 4 byte</span></span><br><span class="line"> _ <span class="keyword">struct</span>{} <span class="comment">// to prevent unkeyed literals</span></span><br><span class="line"> <span class="comment">// zero sized values, like struct{} and [0]byte occurring at </span></span><br><span class="line"> <span class="comment">// the end of a structure are assumed to have a size of one byte.</span></span><br><span class="line"> <span class="comment">// so padding also will be addedd here as well.</span></span><br><span class="line"> </span><br><span class="line">}</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> OrderedPerson <span class="keyword">struct</span> {</span><br><span class="line"> Name <span class="keyword">string</span></span><br><span class="line"> Age <span class="keyword">int32</span></span><br><span class="line"> Veteran <span class="keyword">bool</span></span><br><span class="line"> _ <span class="keyword">struct</span>{} </span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>当您有一个大的常用类型时,它可能会导致性能问题。但是不要担心,您不必手动处理所有的结构。使用 <a href="https://github.com/mdempsky/maligned" target="_blank" rel="external">maligned</a> 你可以轻松检查代码以解决此问题。</p>
<h2 id="5-在测试中没有使用-race-detector"><a href="#5-在测试中没有使用-race-detector" class="headerlink" title="5 在测试中没有使用 race detector"></a>5 在测试中没有使用 race detector</h2><p>数据竞争会导致神秘的故障,通常是在代码部署到生产环境很久之后。正因为如此,它们是并发系统中最常见也是最难调试的 bug 类型。为了帮助区分这些 bug, Go 1.1 引入了一个内置的数据竞争检测器。它可以简单地添加 <code>-race</code> 标志。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">$ go <span class="built_in">test</span> -race pkg // to <span class="built_in">test</span> the package</span><br><span class="line">$ go run -race pkg.go // to run the <span class="built_in">source</span> file</span><br><span class="line">$ go build -race // to build the package</span><br><span class="line">$ go install -race pkg // to install the package</span><br></pre></td></tr></table></figure></p>
<p>启用 race 检测器后,编译器将记录在代码中访问内存的时间和方式,而 <code>runtime</code> 监视对共享变量的不同步访问。</p>
<p>当发现数据竞争时,竞争检测器将打印一份报告,其中包含冲突访问的堆栈跟踪。下面是一个例子:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br></pre></td><td class="code"><pre><span class="line">WARNING: DATA RACE</span><br><span class="line">Read by goroutine 185:</span><br><span class="line"> net.(*pollServer).AddFD()</span><br><span class="line"> src/net/fd_unix.go:89 +0x398</span><br><span class="line"> net.(*pollServer).WaitWrite()</span><br><span class="line"> src/net/fd_unix.go:247 +0x45</span><br><span class="line"> net.(*netFD).Write()</span><br><span class="line"> src/net/fd_unix.go:540 +0x4d4</span><br><span class="line"> net.(*conn).Write()</span><br><span class="line"> src/net/net.go:129 +0x101</span><br><span class="line"> net.func·060()</span><br><span class="line"> src/net/timeout_test.go:603 +0xaf</span><br><span class="line">Previous write by goroutine 184:</span><br><span class="line"> net.setWriteDeadline()</span><br><span class="line"> src/net/sockopt_posix.go:135 +0xdf</span><br><span class="line"> net.setDeadline()</span><br><span class="line"> src/net/sockopt_posix.go:144 +0x9c</span><br><span class="line"> net.(*conn).SetDeadline()</span><br><span class="line"> src/net/net.go:161 +0xe3</span><br><span class="line"> net.func·061()</span><br><span class="line"> src/net/timeout_test.go:616 +0x3ed</span><br><span class="line">Goroutine 185 (running) created at:</span><br><span class="line"> net.func·061()</span><br><span class="line"> src/net/timeout_test.go:609 +0x288</span><br><span class="line">Goroutine 184 (running) created at:</span><br><span class="line"> net.TestProlongTimeout()</span><br><span class="line"> src/net/timeout_test.go:618 +0x298</span><br><span class="line"> testing.tRunner()</span><br><span class="line"> src/testing/testing.go:301 +0xe8</span><br></pre></td></tr></table></figure></p>
<h2 id="6-最后一句"><a href="#6-最后一句" class="headerlink" title="6 最后一句"></a>6 最后一句</h2><p>唯一真正的错误是我们什么也没学到。</p>
<blockquote>
<p>译自:<a href="https://medium.com/swlh/5-mistakes-ive-made-in-go-75fb64b943b8" target="_blank" rel="external">https://medium.com/swlh/5-mistakes-ive-made-in-go-75fb64b943b8</a></p>
</blockquote>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<blockquote>
<p>人皆犯错,宽恕是德 — Alexander Pope</p>
</blockquote>
<p>这些都是我在写 Go 中犯的错误。尽管这些可能不会导致任何类型的错误,但它们可能会潜在地影响软件。</p>
<h2 id="1-内循环"><a href="#1-内循环" class="headerlink" title="1 内循环"></a>1 内循环</h2><p>有几种方法可以造成循环内部的混乱,你需要注意。<br>
用 Go 从头开始构建容器(第1部分:命名空间)
http://team.jiunile.com//blog/2020/11/go-build-container-ns.html
2020-11-12T14:00:00.000Z
2020-11-13T01:53:02.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在过去几年中,容器的使用显著增加。容器的概念已经出现好几年了,但是 Docker 易于使用的命令行才从 2013 年开始在开发人员中普及容器。</p>
<p>在这个系列中,我试图演示容器是如何在下面工作的,以及我是如何开发容器的。</p>
<a id="more"></a>
<h2 id="什么是-vessel?"><a href="#什么是-vessel?" class="headerlink" title="什么是 vessel?"></a>什么是 vessel?</h2><p><a href="https://github.com/0xc0d/vessel" target="_blank" rel="external">vessel</a> 是我的一个教学目的的项目,它实现了一个小版本的 Docker 来管理容器。它既不使用 <a href="https://containerd.io/" target="_blank" rel="external">containerd</a> 也不使用 <a href="https://github.com/opencontainers/runc" target="_blank" rel="external">runc</a>,而是使用一组 Linux 特性来创建容器。</p>
<p>vessel 既不是生产就绪的,也没有经过良好测试的软件。这只是一个简单的项目来了解更多关于容器的知识。</p>
<h2 id="让我们开始:阅读-Docker!"><a href="#让我们开始:阅读-Docker!" class="headerlink" title="让我们开始:阅读 Docker!"></a>让我们开始:阅读 Docker!</h2><p>我发现,在开始编写代码之前,先看一下 <a href="https://docs.docker.com/get-started/overview/" target="_blank" rel="external">Docker 文档</a>,了解一下容器是很有用的。</p>
<p>Docker 就其<a href="https://docs.docker.com/get-started/overview/#the-underlying-technology" target="_blank" rel="external">文档</a>而言,利用了 linux 内核的几个特性,并将它们组合成一个称为容器格式的包装器。这些特性是:</p>
<ul>
<li><strong>Namespaces</strong></li>
<li><strong>Control groups</strong></li>
<li><strong>Union file systems</strong></li>
</ul>
<p>现在让我们浏览一下上面的列表,并简要地了解一下它们是什么。</p>
<h2 id="什么是命名空间(Namespace-)"><a href="#什么是命名空间(Namespace-)" class="headerlink" title="什么是命名空间(Namespace!)?"></a>什么是命名空间(Namespace!)?</h2><p>Linux 命名空间是最现代容器实现背后的基础技术。名称空间是进程对周围运行的其他事物的感知。命名空间允许隔离一组进程中的全局系统资源。例如,网络命名空间隔离网络堆栈,这意味着该网络命名空间中的进程可以拥有自己的独立路由、防火墙规则和网络设备。</p>
<p>因此,如果没有命名空间,容器中的进程可能(例如)卸载文件系统,或在另一个容器中设置网络接口。</p>
<h3 id="哪些资源可以使用命名空间进行隔离?"><a href="#哪些资源可以使用命名空间进行隔离?" class="headerlink" title="哪些资源可以使用命名空间进行隔离?"></a>哪些资源可以使用命名空间进行隔离?</h3><p>在当前的 linux 内核 (5.9) 中,有 8 种类型的不同命名空间。每个命名空间可以隔离某个全局系统资源。</p>
<ul>
<li><strong>Cgroup</strong>: 此命名空间隔离控制组根目录。我将在第 2 部分中解释什么是 cgroups。但简而言之,cgroup 允许系统为一组进程定义资源限制。但要注意的是,“cgroup namespce” 仅控制在命名空间中哪些 cgroup 可见。命名空间无法分配资源限制。我们稍后将对此进行深入解释。</li>
<li><strong>IPC</strong>: 此命名空间隔离进程间通信机制,如 System V 和 POSIX 消息队列。理解IPC 并不难,但这篇文章不会讨论这个主题。</li>
<li><strong>Network</strong>: 此名称空间隔离路由、防火墙规则和名称空间内的一组进程可以看到的网络设备。</li>
<li><strong>Mount</strong>:此名称空间隔离每个名称空间中的挂载点列表。在单独的挂载名称空间中运行的进程可以挂载和卸载,而不会影响其他名称空间。</li>
<li><strong>PID</strong>:这个命名空间隔离进程 ID 号空间。它支持在名称空间内挂起/恢复进程之类的函数。</li>
<li><strong>Time</strong>:这个命名空间隔离了 <code>CLOCK_MONOTONIC</code> 和 <code>CLOCK_BOOTTIME</code> 系统时钟,它们影响了针对这些时钟(如系统正常运行时间)测量的 API。</li>
<li><strong>User</strong>:此名称空间隔离用户 id、组 id、根目录、密钥和功能。这允许进程在名称空间内是根,但不在命名空间外(如在主机中)。</li>
<li><strong>UTS</strong>:这个命名空间隔离主机名和域名</li>
</ul>
<h3 id="关于命名空间的重要注意事项"><a href="#关于命名空间的重要注意事项" class="headerlink" title="关于命名空间的重要注意事项"></a>关于命名空间的重要注意事项</h3><p>命名空间除了隔离之外什么也没做,这意味着,例如,加入一个新的网络名称空间不会给您提供一组隔离的网络设备,您必须自己创建它们。UTS 命名空间也是如此,它不会改变您的主机名。它所做的唯一事情就是隔离与主机名相关的系统调用。我们将在这个系列中一起做这些事情。</p>
<h3 id="命名空间生命周期"><a href="#命名空间生命周期" class="headerlink" title="命名空间生命周期"></a>命名空间生命周期</h3><p>当命名空间中的最后一个进程离开命名空间时,命名空间将自动删除。然而,有许多例外情况使名称空间在没有任何成员进程的情况下保持活动。我们将在为 vessel 创建网络名称空间时解释其中一个例外。</p>
<h3 id="命名空间的系统调用"><a href="#命名空间的系统调用" class="headerlink" title="命名空间的系统调用"></a>命名空间的系统调用</h3><p>现在我们已经简要了解了命名空间是什么,接下来看看如何与命名空间交互。在 Linux 中,有一组允许创建、加入和发现命名空间的系统调用。</p>
<ul>
<li><strong><code>clone</code></strong>:此系统调用实际上创建了一个新进程。但是借助 flags 参数,新进程将创建自己的新命名空间。</li>
<li><strong><code>setns</code></strong>:此系统调用允许正在运行的进程加入现有命名空间。</li>
<li><strong><code>unshare</code></strong>:此系统调用实际上与克隆相同,但不同之处在于此系统调用将创建当前进程并将其移动到新的命名空间,而 <code>clone</code> 将创建具有新的命名空间的新进程。</li>
</ul>
<p>额外提示:<code>fork</code> 和 <code>vfork</code> 内部系统调用只是使用不同的参数调用 <code>clone()</code>。</p>
<h3 id="命名空间-Flags"><a href="#命名空间-Flags" class="headerlink" title="命名空间 Flags"></a>命名空间 Flags</h3><p>上面提到的系统调用需要一个能够指定所需命名空间的 flag。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">CLONE_NEWCGROUP Cgroup namespaces</span><br><span class="line">CLONE_NEWIPC IPC namespaces</span><br><span class="line">CLONE_NEWNET Network namespaces</span><br><span class="line">CLONE_NEWNS Mount namespaces$$ </span><br><span class="line">CLONE_NEWPID PID namespaces</span><br><span class="line">CLONE_NEWTIME Time namespaces</span><br><span class="line">CLONE_NEWUSER User namespaces</span><br><span class="line">CLONE_NEWUTS UTS namespaces</span><br></pre></td></tr></table></figure></p>
<p>例如,如果你想为当前进程创建一个新的网络命名空间,你应该用 <code>CLONE_NEWNET</code> 标记调用<code>unshare</code>,如果您想使用新用户和 UTS 命名空间创建新进程,你应该用<code>CLONE_NEWUSER|CLONE_NEWUTS</code> 调用 clone。竖线表示或按位组合两个标记。</p>
<h3 id="命名空间文件"><a href="#命名空间文件" class="headerlink" title="命名空间文件"></a>命名空间文件</h3><p>在上面我提到过 <code>setns</code> 系统调用将在名称空间之间移动一个正在运行的进程。但是,如何指定要移动到哪个名称空间呢?好的,在创建名称空间之后,成员进程将具有指向命名空间文件的符号链接。</p>
<blockquote>
<p>在 Unix 中,所有内容都是文件。</p>
</blockquote>
<p>例如,在您的 shell 中,通过列出 /proc/[pid]/ns 目录下的文件,您可以看到进程命名空间。在这里你可以看到正在运行的 shell 的当前命名空间(<code>self</code> 代表当前 shell pid):<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">$ ls <span class="_">-l</span> /proc/self/ns | cut <span class="_">-d</span> <span class="string">' '</span> <span class="_">-f</span> 10-12</span><br><span class="line">cgroup -> cgroup:[4026531835]</span><br><span class="line">ipc -> ipc:[4026531839]</span><br><span class="line">mnt -> mnt:[4026531840]</span><br><span class="line">net -> net:[4026532008]</span><br><span class="line">pid -> pid:[4026531836]</span><br><span class="line">pid_<span class="keyword">for</span>_children -> pid:[4026531836]</span><br><span class="line">time -> time:[4026531834]</span><br><span class="line">time_<span class="keyword">for</span>_children -> time:[4026531834]</span><br><span class="line">user -> user:[4026531837]</span><br><span class="line">uts -> uts:[4026531838]</span><br></pre></td></tr></table></figure></p>
<p>同样使用 <code>lsns</code> 命令,您也可以看到进程命名空间列表:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># lsns</span></span><br><span class="line"> NS TYPE NPROCS PID USER COMMAND</span><br><span class="line">4026531834 time 244 1 root /sbin/init</span><br><span class="line">4026531835 cgroup 244 1 root /sbin/init</span><br><span class="line">4026531836 pid 199 1 root /sbin/init</span><br><span class="line">4026531837 user 198 1 root /sbin/init</span><br><span class="line">4026531838 uts 241 1 root /sbin/init</span><br><span class="line">4026531839 ipc 244 1 root /sbin/init</span><br><span class="line">4026531840 mnt 234 1 root /sbin/init</span><br></pre></td></tr></table></figure></p>
<p>实际上 <code>setns</code> syscall 所做的是更改 <code>/proc/[pid]/ns</code> 目录下文件的链接。</p>
<h2 id="废话少说,让我们编码吧!"><a href="#废话少说,让我们编码吧!" class="headerlink" title="废话少说,让我们编码吧!"></a>废话少说,让我们编码吧!</h2><p>现在我们知道我们想要的一切。是时候编写第一个在单独命名空间上运行的代码了。首先让我们看看 <code>unshare</code> 是如何工作的。下面的代码,在第 1 行使用 <code>syscall</code> 包和 <code>Unshare</code> 方法为当前运行的 Go 程序创建一个新的名称空间,然后在第 5 行将主机名设置为“container”,然后在第 9 行,它创建一个新命令并运行它。<code>Run</code> 启动命令并等待其完成。</p>
<blockquote>
<p>除用户命名空间外,创建命名空间需要 <code>CAP_SYS_ADMIN</code> 功能。因此,您需要以 root 用户来运行该程序。</p>
</blockquote>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">err := syscall.Unshare(syscall.CLONE_NEWPID|syscall.CLONE_NEWUTS)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Fprintln(os.Stderr, err)</span><br><span class="line">}</span><br><span class="line">err = syscall.Sethostname([]<span class="keyword">byte</span>(<span class="string">"container"</span>))</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> fmt.Fprintln(os.Stderr, err)</span><br><span class="line">}</span><br><span class="line">cmd := exec.Command(<span class="string">"/bin/sh"</span>)</span><br><span class="line">cmd.Stdin = os.Stdin</span><br><span class="line">cmd.Stdout = os.Stdout</span><br><span class="line">cmd.Stderr = os.Stderr</span><br><span class="line">cmd.Run()</span><br></pre></td></tr></table></figure>
<p>让我们构建程序并进行测试。对于 host 中的第一个命令,我运行 ps 来监视正在运行的进程,然后获取主机名和当前 shell PID(例如 self,$$ 代表当前进程 PID)。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ ps</span><br><span class="line"> PID TTY TIME CMD</span><br><span class="line"> 27973 pts/2 00:00:00 sh</span><br><span class="line"> 27984 pts/2 00:00:00 ps</span><br><span class="line">$ hostname</span><br><span class="line">host</span><br><span class="line">$ <span class="built_in">echo</span> $$</span><br><span class="line">27973</span><br></pre></td></tr></table></figure></p>
<p>现在让我们看看运行程序后会发生什么。获取主机名它返回“container”。似乎有用!<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ hostname</span><br><span class="line">container</span><br></pre></td></tr></table></figure></p>
<p>让我们看看进程 ID 是什么。是的!它是 1,可行。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">echo</span> $$</span><br><span class="line">1</span><br></pre></td></tr></table></figure></p>
<p>让我们运行 ps 来查看在容器内运行的进程。<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">$ ps</span><br><span class="line"> PID TTY TIME CMD</span><br><span class="line"> 27973 pts/2 00:00:00 sh</span><br><span class="line"> 27998 pts/2 00:00:00 unshare</span><br><span class="line"> 28003 pts/2 00:00:00 sh</span><br><span class="line"> 28011 pts/2 00:00:00 ps</span><br></pre></td></tr></table></figure></p>
<p>发生什么事了!?我们可以看到带有大型 pid 的容器内的主机进程没有意义。</p>
<p>我将终止其中一个进程,看看会发生什么:<br><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">kill</span> 27998</span><br><span class="line">sh: <span class="built_in">kill</span>: (27998) - No such process</span><br></pre></td></tr></table></figure></p>
<p>没有这样的进程,它说。精彩吗?让我解释一下。代码实际上是有效的,我们在一个新的 PID 命名空间中,我们可以看到我们的进程 ID 是 1。问题是 ps 命令。下面的 ps 使用 proc 伪文件系统列出所有正在运行的进程。为了能够拥有我们自己的 proc 文件系统,我们需要一个新的挂载名称空间,以及一个新的根路径来将 proc 挂载到其中。我们将在下一部分深入讨论。</p>
<h3 id="Clone-in-Go"><a href="#Clone-in-Go" class="headerlink" title="Clone in Go"></a>Clone in Go</h3><p>在我看来,Go 没有 clone 功能。但是,有一个名为 <a href="https://github.com/liquidgecka/goclone" target="_blank" rel="external">goclone</a> 的包,它包装了 Go 的 clone 系统调用。但是我们将要使用的解决方案是不同的。在 vessel 中,我们使用一个叫做 <code>reexec</code> 的包,它是 Docker 团队开发的。</p>
<h3 id="reexec-是什么?"><a href="#reexec-是什么?" class="headerlink" title="reexec 是什么?"></a>reexec 是什么?</h3><p>Go 允许您使用一组新的名称空间运行命令。<code>reexec</code> 背后的思想是用新的名称空间重新执行正在运行的程序本身。<code>reexec</code> 包,后台的 <code>reexec</code> 包将从调用 <code>/proc/self/exe</code> 的 Go 标准库返回 <code>*exec.Cmd</code>。该文件基本上是指向正在运行的程序可执行文件的链接。</p>
<p>现在您已经了解了 <code>reexec</code> 是如何工作的,让我们从容器中深入研究一些代码。下面的代码,是在 vessel 的早期阶段。它实际上是使用一组新名称空间运行新进程的代码。这个过程就是我们的容器。在第 1 行到第 4 行,函数创建参数和新的 reexec 命令,然后为其设置标准的输入、输出和错误。</p>
<blockquote>
<p>注意: 容器的 <code>fork</code> 子命令(第一行)是容器模式。虽然它被隐藏在使用中。</p>
</blockquote>
<figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">args := []<span class="keyword">string</span>{<span class="string">"fork"</span>}</span><br><span class="line">...</span><br><span class="line"></span><br><span class="line">cmd := reexec.Command(args...)</span><br><span class="line">cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr</span><br><span class="line">cmd.SysProcAttr = &syscall.SysProcAttr{</span><br><span class="line"> Cloneflags: syscall.CLONE_NEWUTS |</span><br><span class="line"> syscall.CLONE_NEWIPC |</span><br><span class="line"> syscall.CLONE_NEWPID |</span><br><span class="line"> syscall.CLONE_NEWNS,</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<p>Go 中的 <code>SysProcAttr</code> 命令包含操作系统特定的属性。这些属性之一是 <code>Cloneflags</code>,通过将 flags 传递到这个值,该命令将使用新的特定名称空间运行。这样,我们的新进程就有了新的 IPC、UTS、PID 和 Mount (NS) 命名空间。但是网络命名空间呢?!</p>
<h3 id="深入研究网络命名空间"><a href="#深入研究网络命名空间" class="headerlink" title="深入研究网络命名空间"></a>深入研究网络命名空间</h3><p>正如我已经提到的,命名空间只能隔离资源和容器感知的边界。因此,使用新的网络命名空间运行容器不会有太大帮助。我们也应该做一些连接容器到外部网络的事情。但这怎么可能?!</p>
<h3 id="什么是虚拟以太网设备?"><a href="#什么是虚拟以太网设备?" class="headerlink" title="什么是虚拟以太网设备?"></a>什么是虚拟以太网设备?</h3><p><code>veth</code> 可以充当网络命名空间之间的隧道。这意味着它可以在另一个命名空间中创建与网络设备的连接。<br><img src="/images/go/docker_ns_1.png" alt="figure 1: Virtual Ethernet Devices"></p>
<blockquote>
<p>虚拟以太网设备总是成对地创建,并相互连接。在一对中的一个设备上传输的所有数据将立即在另一个设备上接收。当任一设备关闭时,这对设备的链路状态也关闭。</p>
</blockquote>
<p>例如,在图 1 中,有两个 veth 对。在每对设备中,一个对等设备位于主机网络命名空间内,另一个位于容器内。主机命名空间中的设备连接到网桥,该网桥被路由到名为 <code>eth0</code> 的物理互联网连接设备。</p>
<p>现在让我们来看看 vessel 是如何创建这样一个网络的。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="params">(c *Container)</span> <span class="title">SetupNetwork</span><span class="params">(bridge <span class="keyword">string</span>)</span> <span class="params">(filesystem.Unmounter, error)</span></span> {</span><br><span class="line"> nsMountTarget := filepath.Join(netnsPath, c.Digest)</span><br><span class="line"> vethName := fmt.Sprintf(<span class="string">"veth%.7s"</span>, c.Digest)</span><br><span class="line"> peerName := fmt.Sprintf(<span class="string">"P%s"</span>, vethName)</span><br><span class="line"> </span><br><span class="line"> <span class="keyword">if</span> err := network.SetupVirtualEthernet(vethName, peerName); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkSetMaster(vethName, bridge); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, err</span><br><span class="line"> }</span><br><span class="line"> unmount, err := network.MountNewNetworkNamespace(nsMountTarget)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkSetNsByFile(nsMountTarget, peerName); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// Change current network namespace to setup the veth</span></span><br><span class="line"> unset, err := network.SetNetNSByFile(nsMountTarget)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, <span class="literal">nil</span></span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">defer</span> unset()</span><br><span class="line"></span><br><span class="line"> ctrEthName := <span class="string">"eth0"</span></span><br><span class="line"> ctrEthIPAddr := c.GetIP()</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkRename(peerName, ctrEthName); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkAddAddr(ctrEthName, ctrEthIPAddr); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkSetup(ctrEthName); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkAddGateway(ctrEthName, <span class="string">"172.30.0.1"</span>); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">if</span> err := network.LinkSetup(<span class="string">"lo"</span>); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> unmount, <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>上面的代码涵盖了容器包的 <code>SetupNetwork</code> 方法。这个方法的职责是创建一个如图 1 所示的网络。</p>
<p>在调用此方法之前,vessel 将创建其名为 <code>vessel0</code> 的桥梁。这是实际传递给 <code>SetupNetwork</code> 网桥值的名称。</p>
<p>从现在开始,事情可能会有点混乱,但别担心。请务必多阅读几次,并遵循代码。</p>
<p>在第 3-4 行,定义了 veth 设备对名称。然后在第 6 行,将使用关联的名称创建 veth。在第 9 行,veth 将指定 <code>vessel0</code> 作为其主服务器,以便进一步通信。<br><img src="/images/go/docker_ns_2.png" alt="docker_ns_2"></p>
<p>现在是时候创建一个新的网络名称空间,并将其中一个 veth 对移动到其中。我们的容器终究会加入这个网络命名空间。然而,问题是命名空间的生命周期!如前所述,当最后一个进程成员离开名称空间时,名称空间将被删除。我也提到了一些例外。其中一个例外是绑定挂载命名空间时。这就是为什么有一个名为 <code>MountNewNetworkNamespace</code> 的函数。这个函数创建一个新的名称空间,并将其绑定到一个文件以保持其活动。下面的代码涵盖了此功能。<br><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">MountNewNetworkNamespace</span><span class="params">(nsTarget <span class="keyword">string</span>)</span> <span class="params">(filesystem.Unmounter, error)</span></span> {</span><br><span class="line"> _, err := os.OpenFile(nsTarget, syscall.O_RDONLY|syscall.O_CREAT|syscall.O_EXCL, <span class="number">0644</span>)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, errors.Wrap(err, <span class="string">"unable to create target file"</span>)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// store current network namespace</span></span><br><span class="line"> file, err = os.OpenFile(<span class="string">"/proc/self/ns/net"</span>, os.O_RDONLY, <span class="number">0</span>)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, err</span><br><span class="line"> }</span><br><span class="line"> <span class="keyword">defer</span> file.Close()</span><br><span class="line"></span><br><span class="line"> <span class="keyword">if</span> err := syscall.Unshare(syscall.CLONE_NEWNET); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> <span class="literal">nil</span>, errors.Wrap(err, <span class="string">"unshare syscall failed"</span>)</span><br><span class="line"> }</span><br><span class="line"> mountPoint := filesystem.MountOption{</span><br><span class="line"> Source: <span class="string">"/proc/self/ns/net"</span>,</span><br><span class="line"> Target: nsTarget,</span><br><span class="line"> Type: <span class="string">"bind"</span>,</span><br><span class="line"> Flag: syscall.MS_BIND,</span><br><span class="line"> }</span><br><span class="line"> unmount, err := filesystem.Mount(mountPoint)</span><br><span class="line"> <span class="keyword">if</span> err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, err</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// reset previous network namespace</span></span><br><span class="line"> <span class="keyword">if</span> err := unix.Setns(<span class="keyword">int</span>(file.Fd()), syscall.CLONE_NEWNET); err != <span class="literal">nil</span> {</span><br><span class="line"> <span class="keyword">return</span> unmount, errors.Wrap(err, <span class="string">"setns syscall failed: "</span>)</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="keyword">return</span> unmount, <span class="literal">nil</span></span><br><span class="line">}</span><br></pre></td></tr></table></figure></p>
<p>在第 2 行,函数创建一个文件。此文件将用于绑定新的网络命名空间。然后在第 9 行,函数存储了当前的命名空间链接,以便能够返回到它。现在是时候创建一个新的网络命名空间,并在第 15 行使用 <code>unshare</code> 系统调用连接它。该函数现在将 <code>/proc/self/ns/net</code> 绑定到第 2 行创建的文件。记住,<code>/proc/self/ns/net</code> 将在 <code>unshare</code> 系统调用后改变。</p>
<p>这一切都很好,我们只需要离开当前的网络命名空间,然后使用第 29 行的 <code>setns</code> 系统调用返回到我们以前的命名空间。这就是为什么函数首先存储了进程网络名称空间(第 9 行)。</p>
<p>回到 <code>SetupNetwork</code> 函数,现在让我们将对等设备移动到我们刚刚在 <code>MountNewNetworkNamespace</code> 函数中创建的命名空间。由于 <code>nsMountTarget</code> 值绑定到网络名称空间,因此它表示命名空间本身。因此,我们可以使用该文件的描述符来指定命名空间。</p>
<p>好吧,毕竟我们有一个虚拟以太网设备对,其中一个设备在主机网络命名空间内,另一个在新的命名空间内。</p>
<p>现在剩下的唯一任务是在新命名空间内配置设备。问题是设备在主机网络命名空间中不再可见,因此,我们需要使用 <code>SetNetNsByFile</code> 函数(第21行)再次加入网络命名空间。此函数仅使用给定文件的描述符调用 <code>setns</code> 系统调用。注意,我们需要 <code>defer</code> <code>unset</code> 函数(第 25 行),以将容器网络命名空间保留在函数的末尾。</p>
<p>现在,代码的其余部分(第 22-43 行)在容器网络命名空间内运行。首先要做的是将容器设备重命名为 eth0(第 29行),然后关联一个新的 IP 地址(第 32 行),设置设备(第 35 行),添加设备的网关(第 38 行),最后设置回环(127.0.0.1)网络接口。现在我们完成了这里的工作,我们的网络命名空间已经完全准备好了。</p>
<p>还要提到 172.30.0.1 是 <code>vessel0</code> 网桥的默认 IP 地址,这并不是最好的方法,因为这个 IP 地址可能已经在使用了。我这样做是为了简单。现在你的任务是让它变得更好,并发送一个 Pull 请求。</p>
<h2 id="结论"><a href="#结论" class="headerlink" title="结论"></a>结论</h2><p>我们了解到命名空间是 Linux 特性之一,它为一组进程隔离全局系统资源,因此它是大多数容器中的基本技术。我们还学习了如何在 Go 中使用 <code>unshare</code>、<code>clone</code> 和 <code>setns</code> 系统调用与命名空间进行交互。</p>
<p>它还没有完成。我们将在下一部分中讨论 union 文件系统,但是现在让我们试着阅读容器代码来理解它。</p>
<p>另外,别忘了用谷歌搜索 “Liz Rice”,看她谈论容器。</p>
<p>感谢阅读!</p>
<p>作者:Ali Josie 来源:medium.com</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p>在过去几年中,容器的使用显著增加。容器的概念已经出现好几年了,但是 Docker 易于使用的命令行才从 2013 年开始在开发人员中普及容器。</p>
<p>在这个系列中,我试图演示容器是如何在下面工作的,以及我是如何开发容器的。</p>
Kubernetes 网络模型来龙去脉
http://team.jiunile.com//blog/2020/11/k8s-network-source.html
2020-11-11T14:00:00.000Z
2020-11-11T03:09:30.000Z
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p><img src="/images/k8s/network_1.jpg" alt="network"><br>容器网络发端于 <code>Docker</code> 的网络。<code>Docker</code> 使用了一个比较简单的网络模型,即内部的网桥加内部的保留 IP。这种设计的好处在于容器的网络和外部世界是解耦的,无需占用宿主机的 IP 或者宿主机的资源,完全是虚拟的。<br><a id="more"></a><br>它的设计初衷是:当需要访问外部世界时,会采用 <code>SNAT</code> 这种方法来借用 Node 的 IP 去访问外面的服务。比如容器需要对外提供服务的时候,所用的是 <code>DNAT</code> 技术,也就是在 Node 上开一个端口,然后通过 <code>iptable</code> 或者别的某些机制,把流导入到容器的进程上以达到目的。</p>
<p>该模型的问题在于,外部网络无法区分哪些是容器的网络与流量、哪些是宿主机的网络与流量。比如,如果要做一个高可用的时候,172.16.1.1 和 172.16.1.2 是拥有同样功能的两个容器,此时我们需要将两者绑成一个 Group 对外提供服务,而这个时候我们发现从外部看来两者没有相同之处,它们的 IP 都是借用宿主机的端口,因此很难将两者归拢到一起。<br><img src="/images/k8s/network_2.jpg" alt="network"></p>
<p>在此基础上,<code>Kubernetes</code> 提出了这样一种机制:即每一个 Pod,也就是一个功能聚集小团伙应有自己的“身份证”,或者说 ID。在 TCP 协议栈上,这个 ID 就是 IP。</p>
<p>这个 IP 是真正属于该 Pod 的,外部世界不管通过什么方法一定要给它。对这个 Pod IP 的访问就是真正对它的服务的访问,中间拒绝任何的变化。比如以 10.1.1.1 的 IP 去访问 10.1.2.1 的 Pod,结果到了 10.1.2.1 上发现,它实际上借用的是宿主机的 IP,而不是源 IP,这样是不被允许的。Pod 内部会要求共享这个 IP,从而解决了一些功能内聚的容器如何变成一个部署的原子的问题。</p>
<p>剩下的问题是我们的部署手段。<code>Kubernetes</code> 对怎么实现这个模型其实是没有什么限制的,用 <code>underlay</code> 网络来控制外部路由器进行导流是可以的;如果希望解耦,用 <code>overlay</code> 网络在底层网络之上再加一层叠加网,这样也是可以的。总之,只要达到模型所要求的目的即可。</p>
<h2 id="Pod-究竟如何上网"><a href="#Pod-究竟如何上网" class="headerlink" title="Pod 究竟如何上网"></a>Pod 究竟如何上网</h2><p>容器网络的网络包究竟是怎么传送的?<br><img src="/images/k8s/network_3.jpg" alt="network"></p>
<p>我们可以从以下两个维度来看:</p>
<ul>
<li>协议层次</li>
<li>网络拓扑</li>
</ul>
<h3 id="2-1-协议层次"><a href="#2-1-协议层次" class="headerlink" title="2.1 协议层次"></a>2.1 协议层次</h3><p>它和 TCP 协议栈的概念是相同的,需要从两层、三层、四层一层层地摞上去,发包的时候从右往左,即先有应用数据,然后发到了 TCP 或者 UDP 的四层协议,继续向下传送,加上 IP 头,再加上 MAC 头就可以送出去了。收包的时候则按照相反的顺序,首先剥离 MAC 的头,再剥离 IP 的头,最后通过协议号在端口找到需要接收的进程。</p>
<h3 id="2-2-网络拓扑"><a href="#2-2-网络拓扑" class="headerlink" title="2.2 网络拓扑"></a>2.2 网络拓扑</h3><p>一个容器的包所要解决的问题分为两步:</p>
<ul>
<li>第一步,如何从容器的空间 (c1) 跳到宿主机的空间 (infra);</li>
<li>第二步,如何从宿主机空间到达远端。</li>
</ul>
<p>我个人的理解是,容器网络的方案可以通过接入、流控、通道这三个层面来考虑。</p>
<ul>
<li>第一个是接入,就是说我们的容器和宿主机之间是使用哪一种机制做连接,比如 <code>Veth + bridge</code>、<code>Veth + pair</code> 这样的经典方式,也有利用高版本内核的新机制等其他方式(如 mac/IPvlan 等),来把包送入到宿主机空间;</li>
<li>第二个是流控,就是说我的这个方案要不要支持 <code>Network Policy</code>,如果支持的话又要用何种方式去实现。这里需要注意的是,我们的实现方式一定需要在数据路径必经的一个关节点上。如果数据路径不通过该 Hook 点,那就不会起作用;</li>
<li>第三个是通道,即两个主机之间通过什么方式完成包的传输。我们有很多种方式,比如以路由的方式,具体又可分为 BGP 路由或者直接路由。还有各种各样的隧道技术等等。最终我们实现的目的就是一个容器内的包通过容器,经过接入层传到宿主机,再穿越宿主机的流控模块(如果有)到达通道送到对端。</li>
</ul>
<h3 id="2-3-一个最简单的路由方案:Flannel-host-gw"><a href="#2-3-一个最简单的路由方案:Flannel-host-gw" class="headerlink" title="2.3 一个最简单的路由方案:Flannel-host-gw"></a>2.3 一个最简单的路由方案:Flannel-host-gw</h3><p>这个方案采用的是每个 Node 独占网段,每个 Subnet 会绑定在一个 Node 上,网关也设置在本地,或者说直接设在 cni0 这个网桥的内部端口上。该方案的好处是管理简单,坏处就是无法跨 Node 迁移 Pod。就是说这个 IP、网段已经是属于这个 Node 之后就无法迁移到别的 Node 上。<br><img src="/images/k8s/network_4.jpg" alt="network"></p>
<p>这个方案的精髓在于 route 表的设置,如上图所示。接下来为大家一一解读一下。</p>
<ul>
<li>第一条很简单,我们在设置网卡的时候都会加上这一行。就是指定我的默认路由是通过哪个 IP 走掉,默认设备又是什么;</li>
<li>第二条是对 Subnet 的一个规则反馈。就是说我的这个网段是 10.244.0.0,掩码是 24 位,它的网关地址就在网桥上,也就是 10.244.0.1。这就是说这个网段的每一个包都发到这个网桥的 IP 上;</li>
<li>第三条是对对端的一个反馈。如果你的网段是 10.244.1.0(上图右边的 Subnet),我们就把它的 Host 的网卡上的 IP (10.168.0.3) 作为网关。也就是说,如果数据包是往 10.244.1.0 这个网段发的,就请以 10.168.0.3 作为网关。</li>
</ul>
<p>再来看一下这个数据包到底是如何跑起来的?</p>
<p>假设容器 (10.244.0.2) 想要发一个包给 10.244.1.3,那么它在本地产生了 TCP 或者 UDP 包之后,再依次填好对端 IP 地址、本地以太网的 MAC 地址作为源 MAC 以及对端 MAC。一般来说本地会设定一条默认路由,默认路由会把 cni0 上的 IP 作为它的默认网关,对端的 MAC 就是这个网关的 MAC 地址。然后这个包就可以发到桥上去了。如果网段在本桥上,那么通过 MAC 层的交换即可解决。</p>
<p>这个例子中我们的 IP 并不属于本网段,因此网桥会将其上送到主机的协议栈去处理。主机协议栈恰好找到了对端的 MAC 地址。使用 10.168.0.3 作为它的网关,通过本地 ARP 探查后,我们得到了 10.168.0.3 的 MAC 地址。即通过协议栈层层组装,我们达到了目的,将 Dst-MAC 填为右图主机网卡的 MAC 地址,从而将包从主机的 eth0 发到对端的 eth0 上去。</p>
<p>所以大家可以发现,这里有一个隐含的限制,上图中的 MAC 地址填好之后一定是能到达对端的,但如果这两个宿主机之间不是二层连接的,中间经过了一些网关、一些复杂的路由,那么这个 MAC 就不能直达,这种方案就是不能用的。当包到达了对端的 MAC 地址之后,发现这个包确实是给它的,但是 IP 又不是它自己的,就开始 Forward 流程,包上送到协议栈,之后再走一遍路由,刚好会发现 10.244.1.0/24 需要发到 10.244.1.1 这个网关上,从而到达了 cni0 网桥,它会找到 10.244.1.3 对应的 MAC 地址,再通过桥接机制,这个包就到达了对端容器。</p>
<p>大家可以看到,整个过程总是二层、三层,发的时候又变成二层,再做路由,就是一个大环套小环。这是一个比较简单的方案,如果中间要走隧道,则可能会有一条 <code>vxlan tunnel</code> 的设备,此时就不填直接的路由,而填成对端的隧道号。</p>
<h2 id="Service-究竟如何工作"><a href="#Service-究竟如何工作" class="headerlink" title="Service 究竟如何工作"></a>Service 究竟如何工作</h2><p>Service 其实是一种负载均衡 (Load Balance) 的机制。</p>
<p>我们认为它是一种用户侧(Client Side) 的负载均衡,也就是说 VIP 到 RIP 的转换在用户侧就已经完成了,并不需要集中式地到达某一个 NGINX 或者是一个 ELB 这样的组件来进行决策。<br><img src="/images/k8s/network_5.jpg" alt="network"></p>
<p>它的实现是这样的:首先是由一群 Pod 组成一组功能后端,再在前端上定义一个虚 IP 作为访问入口。一般来说,由于 IP 不太好记,我们还会附赠一个 DNS 的域名,Client 先访问域名得到虚 IP 之后再转成实 IP。<code>Kube-proxy</code>则是整个机制的实现核心,它隐藏了大量的复杂性。它的工作机制是通过 <code>apiserver</code> 监控 Pod/Service 的变化(比如是不是新增了 Service、Pod)并将其反馈到本地的规则或者是用户态进程。</p>
<h2 id="一个-LVS-版的-Service"><a href="#一个-LVS-版的-Service" class="headerlink" title="一个 LVS 版的 Service"></a>一个 LVS 版的 Service</h2><p>我们来实际做一个 LVS 版的 Service。LVS 是一个专门用于负载均衡的内核机制。它工作在第四层,性能会比用 <code>iptable</code> 实现好一些。</p>
<p>假设我们是一个 <code>Kube-proxy</code>,拿到了一个 Service 的配置,如下图所示:它有一个 <code>Cluster IP</code>,在该 IP 上的端口是 9376,需要反馈到容器上的是 80 端口,还有三个可工作的 Pod,它们的 IP 分别是 10.1.2.3, 10.1.14.5, 10.1.3.8。<br><img src="/images/k8s/network_6.jpg" alt="network"></p>
<p>它要做的事情就是:<br><img src="/images/k8s/network_7.png" alt="network"></p>
<ul>
<li>第 1 步,绑定 VIP 到本地(欺骗内核);</li>
</ul>
<p>首先需要让内核相信它拥有这样的一个虚 IP,这是 LVS 的工作机制所决定的,因为它工作在第四层,并不关心 IP 转发,只有它认为这个 IP 是自己的才会拆到 TCP 或 UDP 这一层。在第一步中,我们将该 IP 设到内核中,告诉内核它确实有这么一个 IP。实现的方法有很多,我们这里用的是 ip route 直接加 local 的方式,用 Dummy 设备上加 IP 的方式也是可以的。</p>
<ul>
<li>第 2 步,为这个虚 IP 创建一个 IPVS 的 <code>virtual server</code>;</li>
</ul>
<p>告诉它我需要为这个 IP 进行负载均衡分发,后面的参数就是一些分发策略等等。<code>virtual server</code> 的 IP 其实就是我们的 <code>Cluster IP</code>。</p>
<ul>
<li>第 3 步,为这个 <code>IPVS service</code> 创建相应的 <code>real server</code>。</li>
</ul>
<p>我们需要为 <code>virtual server</code> 配置相应的 <code>real server</code>,就是真正提供服务的后端是什么。比如说我们刚才看到有三个 Pod,于是就把这三个的 IP 配到 <code>virtual server</code> 上,完全一一对应过来就可以了。<code>Kube-proxy</code> 工作跟这个也是类似的。只是它还需要去监控一些 Pod 的变化,比如 Pod 的数量变成 5 个了,那么规则就应变成 5 条。如果这里面某一个 Pod 死掉了或者被杀死了,那么就要相应地减掉一条。又或者整个 Service 被撤销了,那么这些规则就要全部删掉。所以它其实做的是一些管理层面的工作。</p>
<h2 id="啥?负载均衡还分内部外部"><a href="#啥?负载均衡还分内部外部" class="headerlink" title="啥?负载均衡还分内部外部"></a>啥?负载均衡还分内部外部</h2><p>最后我们介绍一下 Service 的类型,可以分为以下 4 类。</p>
<h3 id="5-1-ClusterIP"><a href="#5-1-ClusterIP" class="headerlink" title="5.1 ClusterIP"></a>5.1 ClusterIP</h3><p>集群内部的一个虚拟 IP,这个 IP 会绑定到一堆服务的 Group Pod 上面,这也是默认的服务方式。它的缺点是这种方式只能在 Node 内部也就是集群内部使用。</p>
<h3 id="5-2-NodePort"><a href="#5-2-NodePort" class="headerlink" title="5.2 NodePort"></a>5.2 NodePort</h3><p>供集群外部调用。将 Service 承载在 Node 的静态端口上,端口号和 Service 一一对应,那么集群外的用户就可以通过 <code><NodeIP>:<NodePort></code> 的方式调用到 Service。</p>
<h3 id="5-3-LoadBalancer"><a href="#5-3-LoadBalancer" class="headerlink" title="5.3 LoadBalancer"></a>5.3 LoadBalancer</h3><p>给云厂商的扩展接口。像阿里云、亚马逊这样的云厂商都是有成熟的 LB 机制的,这些机制可能是由一个很大的集群实现的,为了不浪费这种能力,云厂商可通过这个接口进行扩展。它首先会自动创建 NodePort 和 ClusterIP 这两种机制,云厂商可以选择直接将 LB 挂到这两种机制上,或者两种都不用,直接把 Pod 的 RIP 挂到云厂商的 ELB 的后端也是可以的。</p>
<h3 id="5-4-ExternalName"><a href="#5-4-ExternalName" class="headerlink" title="5.4 ExternalName"></a>5.4 ExternalName</h3><p>摈弃内部机制,依赖外部设施,比如某个用户特别强,他觉得我们提供的都没什么用,就是要自己实现,此时一个 Service 会和一个域名一一对应起来,整个负载均衡的工作都是外部实现的。</p>
<p>下图是一个实例。它灵活地应用了 ClusterIP、NodePort 等多种服务方式,又结合了云厂商的 ELB,变成了一个很灵活、极度伸缩、生产上真正可用的一套系统。<br><img src="/images/k8s/network_8.png" alt="network"></p>
<p>首先我们用 ClusterIP 来做功能 Pod 的服务入口。大家可以看到,如果有三种 Pod 的话,就有三个 <code>Service Cluster IP</code> 作为它们的服务入口。这些方式都是 Client 端的,如何在 Server 端做一些控制呢?</p>
<p>首先会起一些 Ingress 的 Pod(Ingress 是 K8s 后来新增的一种服务,本质上还是一堆同质的 Pod),然后将这些 Pod 组织起来,暴露到一个 NodePort 的 IP,K8s 的工作到此就结束了。</p>
<p>任何一个用户访问 23456 端口的 Pod 就会访问到 Ingress 的服务,它的后面有一个 Controller,会把 Service IP 和 Ingress 的后端进行管理,最后会调到 ClusterIP,再调到我们的功能 Pod。前面提到我们去对接云厂商的 ELB,我们可以让 ELB 去监听所有集群节点上的 23456 端口,只要在 23456 端口上有服务的,就认为有一个 Ingress 的实例在跑。</p>
<p>整个的流量经过外部域名的一个解析跟分流到达了云厂商的 ELB,ELB 经过负载均衡并通过 NodePort 的方式到达 Ingress,Ingress 再通过 ClusterIP 调用到后台真正的 Pod。这种系统看起来比较丰富,健壮性也比较好。任何一个环节都不存在单点的问题,任何一个环节也都有管理与反馈。</p>
<h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>本文的主要内容就到此为止了,这里为大家简单总结一下:</p>
<ul>
<li>大家要从根本上理解 Kubernetes 网络模型的演化来历,理解 PerPodPerIP 的用心在哪里;</li>
<li>网络的事情万变不离其宗,按照模型从 4 层向下就是发包过程,反正层层剥离就是收包过程,容器网络也是如此;</li>
<li>Ingress 等机制是在更高的层次上(服务<->端口)方便大家部署集群对外服务,通过一个真正可用的部署实例,希望大家把 <code>Ingress + Cluster IP + PodIP</code> 等概念联合来看,理解社区出台新机制、新资源对象的思考。</li>
</ul>
<p>作者:叶磊 来源:阿里巴巴云原生</p>
<p><img src="/images/wx_dyh.png" alt="微信订阅号"></p>
<h2 id="概述"><a href="#概述" class="headerlink" title="概述"></a>概述</h2><p><img src="/images/k8s/network_1.jpg" alt="network"><br>容器网络发端于 <code>Docker</code> 的网络。<code>Docker</code> 使用了一个比较简单的网络模型,即内部的网桥加内部的保留 IP。这种设计的好处在于容器的网络和外部世界是解耦的,无需占用宿主机的 IP 或者宿主机的资源,完全是虚拟的。<br>