-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
503 lines (386 loc) · 86.3 KB
/
index.html
File metadata and controls
503 lines (386 loc) · 86.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" name="viewport">
<title>The Cabin in the City</title>
<link rel="stylesheet" href="/css/style.css">
<link rel="stylesheet" href="/css/gitment.css">
<link rel="stylesheet" type="text/css" href="//at.alicdn.com/t/font_859455_eaq7v6w8ktj.css">
<meta name="generator" content="Hexo 4.2.1"></head>
<body>
<header class="header">
<div class="header-inner">
<div class="header-title">
</div>
<nav class="header-nav">
<a href="/" class="header-nav-link" >
Home
</a>
<a href="/archives" class="header-nav-link">
Archives
</a>
<a href="/tags" class="header-nav-link">
Tags
</a>
<a href="/about/" class="header-nav-link">
About
</a>
</nav>
</div>
</header>
<header class="mobile-header">
<div class="mobile-nav">
<div class="mobile-nav-icon">
<span></span>
<span></span>
<span></span>
</div>
<div class="mobile-nav-title">
<a href="/" class="mobile-nav-title-link">'s Blog</a>
</div>
</div>
<nav class="mobile-menu">
<ul class="mobile-menu-list">
<li class="mobile-menu-item">
<i class="iconfont icon-home"></i>
<a href="/" class="mobile-nav-link">Home</a>
</li>
<li class="mobile-menu-item">
<i class="iconfont icon-archive"></i>
<a href="/archives" class="mobile-nav-link">Archives</a>
</li>
<li class="mobile-menu-item">
<i class="iconfont icon-tag"></i>
<a href="/tags" class="mobile-nav-link">Tags</a>
</li>
<li class="mobile-menu-item">
<i class="iconfont icon-about"></i>
<a href="/about/" class="mobile-nav-link">About</a>
</li>
</ul>
</nav>
</header>
<div class="main">
<div class="content-inner">
<div class="posts">
<article class="post-abstract">
<div class="post-title">
<h2><a class="post-title-link" href="/%E5%A4%A7%E6%95%B0%E6%8D%AE/Flink/%E6%97%B6%E9%97%B4%E4%B8%8E%E7%AA%97%E5%8F%A3/">时间与窗口</a></h2>
</div>
<div class="post-meta">
<span class="post-time">2024-04-07</span>
<span class="post-category">
<a class="category" href="/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/">大数据</a>
</span>
</div>
<div class="post-content">
<p>时间与窗口的验证</p>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> flink.demo.time</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.time.<span class="type">Duration</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.api.common.eventtime.{<span class="type">SerializableTimestampAssigner</span>, <span class="type">WatermarkStrategy</span>}</span><br><span class="line"><span class="keyword">import</span> org.apache.flink.api.scala._</span><br><span class="line"><span class="keyword">import</span> org.apache.flink.streaming.api.<span class="type">CheckpointingMode</span></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.streaming.api.environment.<span class="type">CheckpointConfig</span>.<span class="type">ExternalizedCheckpointCleanup</span></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.streaming.api.scala.<span class="type">StreamExecutionEnvironment</span></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.streaming.api.windowing.assigners.<span class="type">TumblingEventTimeWindows</span></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.streaming.api.windowing.time.<span class="type">Time</span></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.util.<span class="type">Collector</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * @author guo</span></span><br><span class="line"><span class="comment"> * @date 2022/11/12</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">FlinkTimeJob</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">main</span></span>(args: <span class="type">Array</span>[<span class="type">String</span>]): <span class="type">Unit</span> = {</span><br><span class="line"></span><br><span class="line"> <span class="keyword">val</span> env = <span class="type">StreamExecutionEnvironment</span>.getExecutionEnvironment</span><br><span class="line"> env.setParallelism(<span class="number">1</span>)</span><br><span class="line"></span><br><span class="line"> env.enableCheckpointing(<span class="number">600</span> * <span class="number">1000</span>, <span class="type">CheckpointingMode</span>.<span class="type">EXACTLY_ONCE</span>)</span><br><span class="line"> env.getCheckpointConfig.setMinPauseBetweenCheckpoints(<span class="number">500</span>)</span><br><span class="line"> env.getCheckpointConfig.setCheckpointTimeout(<span class="number">60000</span>)</span><br><span class="line"> env.getCheckpointConfig.setMaxConcurrentCheckpoints(<span class="number">1</span>)</span><br><span class="line"> env.getCheckpointConfig.setExternalizedCheckpointCleanup(<span class="type">ExternalizedCheckpointCleanup</span>.<span class="type">RETAIN_ON_CANCELLATION</span>)</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 水印周期</span></span><br><span class="line"> env.getConfig.setAutoWatermarkInterval(<span class="number">2000</span>)</span><br><span class="line"></span><br><span class="line"> env.socketTextStream(<span class="string">"localhost"</span>, <span class="number">9192</span>, '\n')</span><br><span class="line"> .map(str => {</span><br><span class="line"> <span class="keyword">val</span> keyAndTimestamp = str.split(<span class="string">","</span>)</span><br><span class="line"> <span class="type">Tuple2</span>(keyAndTimestamp(<span class="number">0</span>), keyAndTimestamp(<span class="number">1</span>).toLong)</span><br><span class="line"> })</span><br><span class="line"> .assignTimestampsAndWatermarks(</span><br><span class="line"> <span class="type">WatermarkStrategy</span>.forGenerator[(<span class="type">String</span>, <span class="type">Long</span>)](_ => <span class="keyword">new</span> <span class="type">BoundedOutOfOrdernessWatermarksWithLog</span>(<span class="type">Duration</span>.ofSeconds(<span class="number">10</span>)))</span><br><span class="line"> .withTimestampAssigner(<span class="keyword">new</span> <span class="type">SerializableTimestampAssigner</span>[(<span class="type">String</span>, <span class="type">Long</span>)] {</span><br><span class="line"> <span class="keyword">override</span> <span class="function"><span class="keyword">def</span> <span class="title">extractTimestamp</span></span>(element: (<span class="type">String</span>, <span class="type">Long</span>), recordTimestamp: <span class="type">Long</span>): <span class="type">Long</span> = element._2</span><br><span class="line"> })</span><br><span class="line"> )</span><br><span class="line"> .keyBy(_._1)</span><br><span class="line"> .window(<span class="type">TumblingEventTimeWindows</span>.of(<span class="type">Time</span>.seconds(<span class="number">3</span>)))</span><br><span class="line"> <span class="comment">// .windowAll(TumblingEventTimeWindows.of(Time.seconds(3)))</span></span><br><span class="line"> <span class="comment">// .allowedLateness(Time.seconds(2)) // 允许数据迟到, 只要当前 water_mark < window_end_time + lateness, 迟到的数据还是可以出发已经执行过窗口的处理</span></span><br><span class="line"> .apply((key, window, iterable, collector: <span class="type">Collector</span>[<span class="type">String</span>]) => {</span><br><span class="line"> <span class="keyword">val</span> start = window.getStart</span><br><span class="line"> <span class="keyword">val</span> end = window.getEnd</span><br><span class="line"> <span class="keyword">val</span> windowElementConcat = iterable.mkString(<span class="string">","</span>)</span><br><span class="line"> <span class="keyword">val</span> res = <span class="string">s"key: <span class="subst">$key</span>, Window: [<span class="subst">$start</span>, <span class="subst">$end</span>), [<span class="subst">${DateTimeUtil.toDateTime(start)}</span>, <span class="subst">${DateTimeUtil.toDateTime(end)}</span>), window elements: <span class="subst">$windowElementConcat</span>"</span></span><br><span class="line"> collector.collect(res)</span><br><span class="line"> })</span><br><span class="line"> .print()</span><br><span class="line"></span><br><span class="line"> env.execute(<span class="string">"flink demo"</span>)</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> flink.demo.time;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.flink.api.common.eventtime.<span class="type">Watermark</span>;</span><br><span class="line"><span class="keyword">import</span> org.apache.flink.api.common.eventtime.<span class="type">WatermarkGenerator</span>;</span><br><span class="line"><span class="keyword">import</span> org.apache.flink.api.common.eventtime.<span class="type">WatermarkOutput</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.time.<span class="type">Duration</span>;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> static org.apache.flink.util.<span class="type">Preconditions</span>.checkArgument;</span><br><span class="line"><span class="keyword">import</span> static org.apache.flink.util.<span class="type">Preconditions</span>.checkNotNull;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * @author guo</span></span><br><span class="line"><span class="comment"> * @date 2024/4/4</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">public <span class="class"><span class="keyword">class</span> <span class="title">BoundedOutOfOrdernessWatermarksWithLog<T></span> <span class="title">implements</span> <span class="title">WatermarkGenerator<T></span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * The maximum timestamp encountered so far.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="keyword">private</span> long maxTimestamp;</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * The maximum out-of-orderness that this watermark generator assumes.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> <span class="keyword">private</span> <span class="keyword">final</span> long outOfOrdernessMillis;</span><br><span class="line"></span><br><span class="line"> <span class="comment">/**</span></span><br><span class="line"><span class="comment"> * Creates a new watermark generator with the given out-of-orderness bound.</span></span><br><span class="line"><span class="comment"> *</span></span><br><span class="line"><span class="comment"> * @param maxOutOfOrderness The bound for the out-of-orderness of the event timestamps.</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"> public <span class="type">BoundedOutOfOrdernessWatermarksWithLog</span>(<span class="type">Duration</span> maxOutOfOrderness) {</span><br><span class="line"> checkNotNull(maxOutOfOrderness, <span class="string">"maxOutOfOrderness"</span>);</span><br><span class="line"> checkArgument(!maxOutOfOrderness.isNegative(), <span class="string">"maxOutOfOrderness cannot be negative"</span>);</span><br><span class="line"></span><br><span class="line"> <span class="keyword">this</span>.outOfOrdernessMillis = maxOutOfOrderness.toMillis();</span><br><span class="line"></span><br><span class="line"> <span class="comment">// start so that our lowest watermark would be Long.MIN_VALUE.</span></span><br><span class="line"> <span class="keyword">this</span>.maxTimestamp = <span class="type">Long</span>.<span class="type">MIN_VALUE</span> + outOfOrdernessMillis + <span class="number">1</span>;</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="comment">// ------------------------------------------------------------------------</span></span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> public void onEvent(<span class="type">T</span> event, long eventTimestamp, <span class="type">WatermarkOutput</span> output) {</span><br><span class="line"> maxTimestamp = <span class="type">Math</span>.max(maxTimestamp, eventTimestamp);</span><br><span class="line"></span><br><span class="line"> <span class="type">System</span>.err.println(<span class="string">"onEvent: eventTimestamp:"</span> + <span class="type">DateTimeUtil</span>.toDateTimeStr(eventTimestamp)</span><br><span class="line"> + <span class="string">", maxTimestamp:"</span> + <span class="type">DateTimeUtil</span>.toDateTimeStr(maxTimestamp));</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="meta">@Override</span></span><br><span class="line"> public void onPeriodicEmit(<span class="type">WatermarkOutput</span> output) {</span><br><span class="line"> output.emitWatermark(<span class="keyword">new</span> <span class="type">Watermark</span>(maxTimestamp - outOfOrdernessMillis - <span class="number">1</span>));</span><br><span class="line"> <span class="type">System</span>.err.println(<span class="string">"emit watermark:"</span> + <span class="type">DateTimeUtil</span>.toDateTimeStr(maxTimestamp - outOfOrdernessMillis - <span class="number">1</span>));</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<figure class="highlight scala"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> flink.demo.time</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.time.format.<span class="type">DateTimeFormatter</span></span><br><span class="line"><span class="keyword">import</span> java.time.{<span class="type">Instant</span>, <span class="type">ZoneId</span>}</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * @author guo</span></span><br><span class="line"><span class="comment"> * @date 2024/4/4</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line"><span class="class"><span class="keyword">object</span> <span class="title">DateTimeUtil</span> </span>{</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">toDateTime</span></span>(ts: <span class="type">Long</span>): <span class="type">String</span> = {</span><br><span class="line"> <span class="type">Instant</span>.ofEpochMilli(ts).atZone(<span class="type">ZoneId</span>.systemDefault()).toLocalDateTime.format(<span class="type">DateTimeFormatter</span>.ofPattern(<span class="string">"yyyy-MM-dd HH:mm:ss.SSS"</span>))</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line"> <span class="function"><span class="keyword">def</span> <span class="title">toDateTimeStr</span></span>(ts: <span class="type">Long</span>): <span class="type">String</span> = {</span><br><span class="line"> ts + <span class="string">" <=> "</span> + <span class="type">Instant</span>.ofEpochMilli(ts).atZone(<span class="type">ZoneId</span>.systemDefault()).toLocalDateTime.format(<span class="type">DateTimeFormatter</span>.ofPattern(<span class="string">"yyyy-MM-dd HH:mm:ss.SSS"</span>))</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
<figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta"><?xml version="1.0" encoding="UTF-8"?></span></span><br><span class="line"><span class="tag"><<span class="name">project</span> <span class="attr">xmlns</span>=<span class="string">"http://maven.apache.org/POM/4.0.0"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xmlns:xsi</span>=<span class="string">"http://www.w3.org/2001/XMLSchema-instance"</span></span></span><br><span class="line"><span class="tag"> <span class="attr">xsi:schemaLocation</span>=<span class="string">"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">modelVersion</span>></span>4.0.0<span class="tag"></<span class="name">modelVersion</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.example<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-test<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>1.0-SNAPSHOT<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">properties</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">project.build.sourceEncoding</span>></span>UTF-8<span class="tag"></<span class="name">project.build.sourceEncoding</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">flink.version</span>></span>1.15.0<span class="tag"></<span class="name">flink.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scala.binary.version</span>></span>2.12<span class="tag"></<span class="name">scala.binary.version</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">target.java.version</span>></span>1.8<span class="tag"></<span class="name">target.java.version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">maven.compiler.source</span>></span>${target.java.version}<span class="tag"></<span class="name">maven.compiler.source</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">maven.compiler.target</span>></span>${target.java.version}<span class="tag"></<span class="name">maven.compiler.target</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">log4j.version</span>></span>2.12.1<span class="tag"></<span class="name">log4j.version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">properties</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">repositories</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">repository</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">id</span>></span>apache.snapshots<span class="tag"></<span class="name">id</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>Apache Development Snapshot Repository<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">url</span>></span>https://repository.apache.org/content/repositories/snapshots/<span class="tag"></<span class="name">url</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">releases</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">enabled</span>></span>false<span class="tag"></<span class="name">enabled</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">releases</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">snapshots</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">enabled</span>></span>true<span class="tag"></<span class="name">enabled</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">snapshots</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">repository</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">repositories</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependencies</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-clients<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${flink.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-streaming-scala_${scala.binary.version}<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${flink.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-connector-kafka<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${flink.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.flink<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>flink-walkthrough-common<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${flink.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- Add logging framework, to produce console output when running in the IDE. --></span></span><br><span class="line"> <span class="comment"><!-- These dependencies are excluded from the application JAR by default. --></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-slf4j-impl<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>runtime<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-api<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>runtime<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.logging.log4j<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>log4j-core<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>${log4j.version}<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">scope</span>></span>runtime<span class="tag"></<span class="name">scope</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependency</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">dependencies</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">build</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">plugins</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="tag"><<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.scala-tools<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>maven-scala-plugin<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>2.15.2<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugin</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- Java Compiler --></span></span><br><span class="line"> <span class="tag"><<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.maven.plugins<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>maven-compiler-plugin<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>3.1<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">source</span>></span>${target.java.version}<span class="tag"></<span class="name">source</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">target</span>></span>${target.java.version}<span class="tag"></<span class="name">target</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugin</span>></span></span><br><span class="line"></span><br><span class="line"> <span class="comment"><!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. --></span></span><br><span class="line"> <span class="comment"><!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. --></span></span><br><span class="line"> <span class="tag"><<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">groupId</span>></span>org.apache.maven.plugins<span class="tag"></<span class="name">groupId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactId</span>></span>maven-shade-plugin<span class="tag"></<span class="name">artifactId</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">version</span>></span>3.0.0<span class="tag"></<span class="name">version</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">executions</span>></span></span><br><span class="line"> <span class="comment"><!-- Run shade goal on package phase --></span></span><br><span class="line"> <span class="tag"><<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">phase</span>></span>package<span class="tag"></<span class="name">phase</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">goal</span>></span>shade<span class="tag"></<span class="name">goal</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">goals</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">excludes</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>org.apache.flink:force-shading<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>com.google.code.findbugs:jsr305<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>org.slf4j:*<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>org.apache.logging.log4j:*<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">excludes</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">artifactSet</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">filters</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">filter</span>></span></span><br><span class="line"> <span class="comment"><!-- Do not copy the signatures in the META-INF folder.</span></span><br><span class="line"><span class="comment"> Otherwise, this might cause SecurityExceptions when using the JAR. --></span></span><br><span class="line"> <span class="tag"><<span class="name">artifact</span>></span>*:*<span class="tag"></<span class="name">artifact</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">excludes</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>META-INF/*.SF<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>META-INF/*.DSA<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">exclude</span>></span>META-INF/*.RSA<span class="tag"></<span class="name">exclude</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">excludes</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">filter</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">filters</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">transformers</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">transformer</span></span></span><br><span class="line"><span class="tag"> <span class="attr">implementation</span>=<span class="string">"org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">mainClass</span>></span>spendreport.FraudDetectionJob<span class="tag"></<span class="name">mainClass</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">transformer</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">transformers</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">configuration</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">execution</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">executions</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugin</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">plugins</span>></span></span><br><span class="line"> <span class="tag"></<span class="name">build</span>></span></span><br><span class="line"><span class="tag"></<span class="name">project</span>></span></span><br></pre></td></tr></table></figure>
</div>
</article>
<article class="post-abstract">
<div class="post-title">
<h2><a class="post-title-link" href="/%E5%A4%A7%E6%95%B0%E6%8D%AE/Hbase/%E4%BF%9D%E6%8C%81%E4%B8%8E%20Redis%20%E7%9B%B8%E5%90%8C%E7%9A%84%20TTL%20%E6%93%8D%E4%BD%9C/">保持与 Redis 相同的 TTL 操作</a></h2>
</div>
<div class="post-meta">
<span class="post-time">2024-04-07</span>
<span class="post-category">
<a class="category" href="/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/">大数据</a>
</span>
</div>
<div class="post-content">
<p>保持与 Redis 相同的 TTL 操作</p>
<p>Redis 中没有 Hbase 中时间版本的概念与机制，也可以认为只有当前版本，并且 TTL 是针对整个 Key 的，而 Hbase 的 TTL 可以基于列族或者单元设定，不是基于整个行键的，如果我们将 Redis 中的数据迁移到 Hbase 中，对应的 TTL 就是一个问题，原来基于 Redis 机制的应用中使用也是一个问题，所以为了保持旧的应用兼容（在不对原有应用做大量重构测试的情况下，只在 API 层面兼容修改），只能手动针对 Hbase 的单元的 TTL 进行设置。</p>
<p>假设现在 API 层面，需要将原有的 Redis 访问替换为 Hbase 的访问，需要做一些 API 层面的映射。首先抛开所使用的底层缓存方案，将缓存抽象：每个缓存数据都属于一张表，这张表中缓存的是某个特定领域，特定应用的数据，访问时，需要指定表名，有两种基本的结构（和 Redis 保持一致） Key-Value, Key-HashKey-Value， 这样就可以为开发人员提供一个抽象层的 API, 开发人员并不需要特别关注底层缓存的方案，如果后面更换缓存方案，可以提供 API 层的兼容。</p>
<p>Hbase 中单元有时间版本的概念（VERSIONS）, 一般设置 1 就够了，还有最小版本（MIN_VERSIONS）, 可以设置为 0， 保证早于 TTL 的时间版本的数据不会返回。如果我们多次对一个单元进行写，即使设置了最大版本为 1， 如果最近的版本因为 TTL 过期，之前的版本没有过期，而且因为没有合并，没有被删除，还是会被查询出来，这显示对于原来使用 Redis 的应用来说，是不可接受的，所以只能在设置 TTL 时，手动删除之前的版本，而且， TTL 一般来说只是影响数据保存的时间长短，我们是为了清理数据设置的 TTL, 所以这个操作可以异步来进行。即使某次操作失败了，也可以等待下一次的操作进行设置。</p>
<p>这是一个同步的测试，使用 checkAndMutate 方法</p>
<figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> test.bigdata</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.time.{Duration, Instant, LocalDateTime, ZoneId}</span><br><span class="line"><span class="keyword">import</span> java.util</span><br><span class="line"><span class="keyword">import</span> java.util.Date</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.hbase.{CellUtil, HBaseConfiguration, TableName}</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.hbase.client.{ConnectionFactory, Delete, Get, Put, RowMutations, Table}</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.hbase.util.Bytes</span><br><span class="line"><span class="keyword">import</span> org.apache.hadoop.hbase.util.Bytes.toBytes</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@author</span> guo</span></span><br><span class="line"><span class="comment"> * <span class="doctag">@date</span> 2022/7/24</span></span><br><span class="line"><span class="comment"> */</span></span><br><span class="line">object HbaseExpireTest {</span><br><span class="line"></span><br><span class="line"> <span class="function">def <span class="title">main</span><span class="params">(args: Array[String])</span>: Unit </span>= {</span><br><span class="line"> val configuration = HBaseConfiguration.create()</span><br><span class="line"></span><br><span class="line"> <span class="comment">// hbase pseudo distributed cluster on nas docker</span></span><br><span class="line"> configuration.set(<span class="string">"hbase.zookeeper.quorum"</span>, <span class="string">"harisekhon-hbase1"</span>)</span><br><span class="line"> configuration.set(<span class="string">"hbase.zookeeper.property.clientPort"</span>, <span class="string">"2181"</span>)</span><br><span class="line"></span><br><span class="line"> <span class="keyword">var</span> connection = ConnectionFactory.createConnection(configuration)</span><br><span class="line"> <span class="keyword">var</span> table: Table = connection.getTable(TableName.valueOf(<span class="string">"Test:user"</span>))</span><br><span class="line"></span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> connection = ConnectionFactory.createConnection(configuration)</span><br><span class="line"> table = connection.getTable(TableName.valueOf(<span class="string">"Test:user"</span>))</span><br><span class="line"> val rowKey = toBytes(<span class="string">"r14"</span>)</span><br><span class="line"> val family = toBytes(<span class="string">"info"</span>)</span><br><span class="line"> val qualifier = toBytes(<span class="string">"name"</span>)</span><br><span class="line"> val value = toBytes(<span class="string">"hello world"</span>)</span><br><span class="line"> val ttl = Duration.ofMinutes(<span class="number">10</span>).toMillis</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 先不设置 TTL, 直接 Put</span></span><br><span class="line"> val put = <span class="keyword">new</span> Put(rowKey).addColumn(family, qualifier, value)</span><br><span class="line"> table.put(put)</span><br><span class="line"></span><br><span class="line"> val get = <span class="keyword">new</span> Get(rowKey).addColumn(family, qualifier)</span><br><span class="line"> println(<span class="string">"after first set:"</span> + Bytes.toString(table.get(get).getValue(family, qualifier)))</span><br><span class="line"></span><br><span class="line"> <span class="comment">// 设置 TTL, 先检查值是否相等，再进行原子操作: 删除，新增并设置 TTL</span></span><br><span class="line"> <span class="comment">// 如果不相等，说明其它实例可能修改了单元的值，所以此次不再设置，等待其它实例设置即可</span></span><br><span class="line"> val time = <span class="keyword">new</span> Date().getTime</span><br><span class="line"> val firstDel = <span class="keyword">new</span> Delete(rowKey, time).addColumns(family, qualifier)</span><br><span class="line"> val secondPut = <span class="keyword">new</span> Put(rowKey, time + <span class="number">100</span>).addColumn(family, qualifier, value).setTTL(ttl)</span><br><span class="line"> val delAndPut = RowMutations.of(util.Arrays.asList(firstDel, secondPut))</span><br><span class="line"></span><br><span class="line"> val bool = table.checkAndMutate(rowKey, family)</span><br><span class="line"> .qualifier(qualifier)</span><br><span class="line"> .ifEquals(value)</span><br><span class="line"> .thenMutate(delAndPut)</span><br><span class="line"> println(<span class="string">"check and execute result:"</span> + bool)</span><br><span class="line"></span><br><span class="line"> val limitMinutes = <span class="number">5</span></span><br><span class="line"> val end = LocalDateTime.now().plusMinutes(limitMinutes)</span><br><span class="line"> <span class="keyword">while</span> (LocalDateTime.now().isBefore(end)) {</span><br><span class="line"> val get = <span class="keyword">new</span> Get(rowKey).addColumn(family, qualifier).readAllVersions</span><br><span class="line"> val cells = table.get(get).rawCells()</span><br><span class="line"> <span class="keyword">if</span> (cells != <span class="keyword">null</span> && cells.nonEmpty) {</span><br><span class="line"> <span class="keyword">for</span> (cell <- cells) {</span><br><span class="line"> val timestamp = cell.getTimestamp</span><br><span class="line"> val family = Bytes.toString(CellUtil.cloneFamily(cell))</span><br><span class="line"> val qualifier = Bytes.toString(CellUtil.cloneQualifier(cell))</span><br><span class="line"> val value = Bytes.toString(CellUtil.cloneValue(cell))</span><br><span class="line"> val row = Bytes.toString(CellUtil.cloneRow(cell))</span><br><span class="line"> println(s<span class="string">"${LocalDateTime.now()} row:$row, family: $family, qualifier:$qualifier, timestamp:$timestamp, value:$value, ${Instant.ofEpochMilli(timestamp).atZone(ZoneId.systemDefault()).toLocalDateTime}"</span>)</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">else</span> {</span><br><span class="line"> println(<span class="string">"cells null or empty"</span>)</span><br><span class="line"> }</span><br><span class="line"> Thread.sleep(<span class="number">1000</span>)</span><br><span class="line"> }</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> table.close()</span><br><span class="line"> connection.close()</span><br><span class="line"> }</span><br><span class="line"> }</span><br><span class="line">}</span><br></pre></td></tr></table></figure>
</div>
</article>
<article class="post-abstract">
<div class="post-title">
<h2><a class="post-title-link" href="/Linux/Vim%20%E4%BB%8B%E7%BB%8D/">Vim 介绍</a></h2>
</div>
<div class="post-meta">
<span class="post-time">2023-12-17</span>
<span class="post-category">
<a class="category" href="/categories/Linux/">Linux</a>
</span>
</div>
<div class="post-content">
<h2 id="Vim-的基本使用"><a href="#Vim-的基本使用" class="headerlink" title="Vim 的基本使用"></a>Vim 的基本使用</h2><p>内容取自 byte-of-vim， 查看原文，请参考文末链接.</p>
<h3 id="Vim"><a href="#Vim" class="headerlink" title="Vim"></a>Vim</h3><p>Vim 是一个用于写作的计算机程序（computer program），提供一系列特性帮助你更好地进行写作。 可以用来编写购物清单，写书，或者是程序代码。</p>
<h3 id="为什么选择-Vim"><a href="#为什么选择-Vim" class="headerlink" title="为什么选择 Vim"></a>为什么选择 Vim</h3><h4 id="简单"><a href="#简单" class="headerlink" title="简单"></a>简单</h4><ul>
<li>极简主义界面，帮助你集中注意力</li>
<li>少许的核心概念</li>
</ul>
<h4 id="高效"><a href="#高效" class="headerlink" title="高效"></a>高效</h4><blockquote>
<p>There is no great writing, only great rewriting.</p>
<p> ——Louis Brandeis</p>
</blockquote>
<p>与纯文本或富文本编辑器对比，Vim 可以使得复杂大量频繁的修改该变得更容易，更快，更好。Minimal effort. Maximal effect.</p>
<h3 id="Vim-能干什么？"><a href="#Vim-能干什么？" class="headerlink" title="Vim 能干什么？"></a>Vim 能干什么？</h3><p><strong>说了这么多漂亮话，Vim 到底强在哪里？</strong></p>
<h4 id="Example"><a href="#Example" class="headerlink" title="Example"></a>Example</h4><table>
<thead>
<tr>
<th>编辑</th>
<th>Vim 中的操作</th>
</tr>
</thead>
<tbody><tr>
<td>如何将光标向下移动 7 行</td>
<td>Press 7j</td>
</tr>
<tr>
<td>删除一个单词</td>
<td>Press dw</td>
</tr>
<tr>
<td>搜索文档中光标所在的单词</td>
<td>Press *</td>
</tr>
<tr>
<td>在 50-100 行进行查找并替换</td>
<td>Run :50,100s/old/new/g (s = substitute, c = confirmation )</td>
</tr>
<tr>
<td>如果光标所在的单词是一个文件，如何打开</td>
<td>gf (g = goto, f = file)</td>
</tr>
<tr>
<td>如果每行只有前N个字符是有用的？</td>
<td>ctrl-v –> y (copy)</td>
</tr>
<tr>
<td>……</td>
<td>……</td>
</tr>
</tbody></table>
<p><strong>所以，你被说服了吗？</strong></p>
<p>我们为什么学习 Vim ?</p>
<h4 id="Vim-图形化界面版本"><a href="#Vim-图形化界面版本" class="headerlink" title="Vim 图形化界面版本"></a>Vim 图形化界面版本</h4><p><strong>Windows</strong></p>
<p><strong>Mac OS X</strong></p>
<p><strong>Linux/BSD</strong></p>
<p><img src="https://raw.githubusercontent.com/weirdWimp/blog-store/main/img/image-20200901204140178.png" alt="image-20200901204140178"></p>
<p> Windos 下的 Vim</p>
<h4 id="终端版本"><a href="#终端版本" class="headerlink" title="终端版本"></a>终端版本</h4><p><strong>Windows</strong></p>
<p><strong>Mac OS X</strong></p>
<p><strong>Linux/BSD</strong></p>
<p><img src="https://raw.githubusercontent.com/weirdWimp/blog-store/main/img/image-20200901204402296.png" alt="image-20200901204402296"></p>
<h3 id="最重要的概念-模式"><a href="#最重要的概念-模式" class="headerlink" title="最重要的概念 - 模式"></a>最重要的概念 - 模式</h3><p>Vim 中的模式也许是最令刚接触 Vim 的人最疑惑的部分，为什么打开不能直接输入文字？该怎么样保存并退出 Vim？据说 Stack Overflow 上关于如何退出 Vim 的问题，其点击量已有上百万次了</p>
<p>Vim 中不同的模式就像是电视机中的正常模式和 DVD 模式，每种模式都有其特定的功能。模式使得功能区得以划分，使得事情变得尽可能简单。Vim 的目标之一就是使用键盘就可以完成所有工作，而不需要借助鼠标。</p>
<h4 id="Normal-Mode-正常模式-命令行模式"><a href="#Normal-Mode-正常模式-命令行模式" class="headerlink" title="Normal Mode (正常模式/命令行模式)"></a>Normal Mode (正常模式/命令行模式)</h4><p><strong>A mode for running commands</strong></p>
<p>Vim 启动的默认模式</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Vim command :echo</span></span><br><span class="line">:<span class="built_in">echo</span> <span class="string">"Hello world"</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># Vim command :help takes us to the table of contents of the reference manual</span></span><br><span class="line">:<span class="built_in">help</span> usr_toc</span><br></pre></td></tr></table></figure>
<h4 id="Insert-Mode-插入模式-编辑模式"><a href="#Insert-Mode-插入模式-编辑模式" class="headerlink" title="Insert Mode (插入模式/编辑模式)"></a>Insert Mode (插入模式/编辑模式)</h4><p><strong>A mode for writing text</strong></p>
<p>打开 Vim，命令行模式下运行 <code>:e temp.txt</code>, 按下 <code>i</code>, 进入编辑模式，输入文本；</p>
<p>按下 <code><ESC></code>, 切换回命令行模式，运行 <code>:w</code> 命令保存</p>
<p><strong>使用命令进行更方便的移动光标</strong></p>
<p>除了 <code>i</code> 之外，你还可以 </p>
<table>
<thead>
<tr>
<th>Command</th>
<th>Action</th>
</tr>
</thead>
<tbody><tr>
<td>i</td>
<td>insert text just before the cursor</td>
</tr>
<tr>
<td>I</td>
<td>insert text at the start of the line</td>
</tr>
<tr>
<td>a</td>
<td>append text just after the cursor</td>
</tr>
<tr>
<td>A</td>
<td>append text at the end of the line</td>
</tr>
</tbody></table>
<p>光标移动，并切换为 Insert Mode</p>
<p><strong>其他常用操作命令</strong></p>
<table>
<thead>
<tr>
<th>Command</th>
<th>Action</th>
</tr>
</thead>
<tbody><tr>
<td>o</td>
<td>open a new line below</td>
</tr>
<tr>
<td>O</td>
<td>open a new line above</td>
</tr>
<tr>
<td>s</td>
<td>substitute character</td>
</tr>
<tr>
<td>S</td>
<td>substitutes the whole line</td>
</tr>
<tr>
<td>r</td>
<td>replace the current character</td>
</tr>
<tr>
<td>R</td>
<td>replace continuous characters</td>
</tr>
</tbody></table>
<p>编辑完文本，切换回 Normal Mode, 是一个良好的习惯，完成文档的初始编写时，最好切换为 Normal Mode。</p>
<p>这两种模式的切换时如此的简单，按下 <code>i</code> 即可进入编辑模式， <code><ESC></code> 即可切换为命令行模式，图形化界面看似提供了很多菜单选项，可是如果上百个命令，以及这些命令的组合，图形化界面是难以做到的。</p>
<p>一旦你理解了 Vim 的模式 （哲学）， 你就不会觉得好难用，好奇怪，是吧？</p>
<h4 id="Visual-mode"><a href="#Visual-mode" class="headerlink" title="Visual mode"></a>Visual mode</h4><p>假如想要选中一连串的单词（words），并将它们完全替换为新的文本， 该如何做呢？你可不想按住删除键全部删除，再重新添加新的文本吧？Visual Mode 就将发挥它的作用。</p>
<p>Normal Mode 下，按下 <code>v</code> 或则 <code>V</code> 进入 Visual Mode</p>
<table>
<thead>
<tr>
<th>Command</th>
<th>desc</th>
</tr>
</thead>
<tbody><tr>
<td>v</td>
<td>Visual Mode, character basis</td>
</tr>
<tr>
<td>V</td>
<td>Visual Mode, line basis</td>
</tr>
</tbody></table>
<p><img src="https://raw.githubusercontent.com/weirdWimp/blog-store/main/img/image-20200901215958686.png" alt="image-20200901215958686"></p>
<p> relation between the difference modes</p>
<h3 id="Graphical-cheat-sheet"><a href="#Graphical-cheat-sheet" class="headerlink" title="Graphical cheat sheet"></a>Graphical cheat sheet</h3><p><img src="https://raw.githubusercontent.com/weirdWimp/blog-store/main/img/image-20200901221532521.png" alt="image-20200901221532521"></p>
<h3 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h3><p>理解模式是如何工作的，以及如何进行模式的切换是成为一个 <strong>Vimmer</strong> 的关键，当然， Vim 也只是众多编辑器中的一种 （如 Vim 于 Emacs 之争），如何选择，取决于你的个人习惯和偏好，工欲善其事必先利其器，Vim 也只是提供了一种选择，这个看似古老的软件依然在庞大的程序员群体中占有一席之地，继续发光发热。</p>
<h3 id="参考文献"><a href="#参考文献" class="headerlink" title="参考文献"></a>参考文献</h3><p>[1] <a href="https://github.com/swaroopch/byte-of-vim" target="_blank" rel="noopener">A Byte of Vim</a></p>
<p>[2] <a href="https://initialcommit.com/blog/7-versatile-vim-commands" target="_blank" rel="noopener">7 versatile Vim commands that are easy to memorize</a></p>
<p>[3] <a href="https://github.com/wsdjeg/vim-galore-zh_cn" target="_blank" rel="noopener">Vim Galore</a></p>
<p>[4] <a href="http://www.viemu.com/a_vi_vim_graphical_cheat_sheet_tutorial.html" target="_blank" rel="noopener">Graphical vi-vim Cheat Sheet and Tutorial</a></p>
</div>
</article>
<article class="post-abstract">
<div class="post-title">
<h2><a class="post-title-link" href="/Linux/cURL%E6%89%B9%E9%87%8F%E8%AF%B7%E6%B1%82/">cURL批量请求</a></h2>
</div>
<div class="post-meta">
<span class="post-time">2023-12-17</span>
<span class="post-category">
<a class="category" href="/categories/Linux/">Linux</a>
</span>
</div>
<div class="post-content">
<h1 id="cURL-批量请求"><a href="#cURL-批量请求" class="headerlink" title="cURL 批量请求"></a>cURL 批量请求</h1><p>req_params 文件中每一行都是一个 JSON 字符串，表示每次请求的请求体</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br><span class="line">{"a":"b", "c":"{\"d\":\"f\"}"}</span><br></pre></td></tr></table></figure>
<p>现在需要针对每个请求体进行调用</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># 重点在于 read 的参数 -r， 表示 raw 模式，不把输入的反斜杠字符解释为转义字符</span></span><br><span class="line"><span class="comment"># 如果不指定该参数，每行中的 \ 会被认为是转义字符，表示对后一个字符的转义，就不是一个合法的 JSON</span></span><br><span class="line"><span class="keyword">while</span> <span class="built_in">read</span> -r line; <span class="keyword">do</span> </span><br><span class="line"> curl -H <span class="string">'Content-Type: application/json'</span> -d <span class="string">"<span class="variable">$line</span>"</span> http://192.168.31.224:8080/curl/<span class="built_in">print</span></span><br><span class="line"><span class="keyword">done</span> < req_params</span><br></pre></td></tr></table></figure>
<p>使用 here document 进行请求参数的填充</p>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">while</span> <span class="built_in">read</span> ph; <span class="keyword">do</span></span><br><span class="line">input=$(cat << EOF</span><br><span class="line">{<span class="string">"phone"</span>:<span class="string">"<span class="variable">${ph}</span>"</span>}</span><br><span class="line">EOF</span><br><span class="line">)</span><br><span class="line">curl -H <span class="string">'Content-Type: application/json'</span> -d <span class="string">"<span class="variable">${input}</span>"</span> http://192.168.31.224:8080/curl/<span class="built_in">print</span></span><br><span class="line"><span class="built_in">echo</span></span><br><span class="line"><span class="keyword">done</span> < phone</span><br></pre></td></tr></table></figure>
<p><a href="https://wangdoc.com/bash/read" target="_blank" rel="noopener">read 命令 - Bash 脚本教程 - 网道 (wangdoc.com)</a></p>
</div>
</article>
<article class="post-abstract">
<div class="post-title">
<h2><a class="post-title-link" href="/Linux/xxd%E5%91%BD%E4%BB%A4/">xxd命令</a></h2>
</div>
<div class="post-meta">
<span class="post-time">2022-11-12</span>
<span class="post-category">
<a class="category" href="/categories/Linux/">Linux</a>
</span>
</div>
<div class="post-content">
<h1 id="xxd-命令"><a href="#xxd-命令" class="headerlink" title="xxd 命令"></a>xxd 命令</h1><p><code>xxd</code> 用来对指定的文件或来自标准输入的数据进行<strong>十六进制转储</strong>，也可以将将十六进制转换为原始的二进制格式，允许以邮件安全的 ASCII 表示传输二进制数据，<code>base64</code> 其实也可以达到相同的目的。也可以用来进行二进制文件 patching</p>
<h2 id="选项"><a href="#选项" class="headerlink" title="选项"></a>选项</h2><p>如果没有指定输入文件或输出文件，或者指定为 <code>-</code> ，使用标准输入或标准输出</p>
<ul>
<li><p><code>-a | -autoskip</code> 开启自动跳过，以 <code>*</code> 代替空行（nul-lines），默认关闭</p>
</li>
<li><p><code>-b | -bits</code> 切换到二进制位模式，而不是十六进制, <code>-r</code>, <code>-p</code>,<code>-i</code> 下不生效 </p>
</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">echo</span> -n <span class="string">"hello"</span> | xxd -b</span><br><span class="line"></span><br><span class="line">00000000: 01101000 01100101 01101100 01101100 01101111 hello</span><br></pre></td></tr></table></figure>
<ul>
<li><code>-c cols | -cols cols</code> 输出的每行包含 cols 个字节，默认是 16 个 (-i: 12, -ps: 30, -b: 6).最大 256</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">echo</span> -n <span class="string">"hello"</span> | xxd -b -c 2</span><br><span class="line"></span><br><span class="line">00000000: 01101000 01100101 he</span><br><span class="line">00000002: 01101100 01101100 ll</span><br><span class="line">00000004: 01101111 o</span><br></pre></td></tr></table></figure>
<ul>
<li><p><code>-E | -EBCDIC</code> ASCII 表示切换为 EBCDIC（国际）编码 meaningless in combinations with -r, -p or -i</p>
</li>
<li><p><code>-g bytes | -groupsize bytes</code> 每 bytes 个字节，添加一个空格进行分隔分组</p>
</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">echo</span> <span class="string">"hello"</span> | xxd -g 1</span><br><span class="line">00000000: 68 65 6c 6c 6f 0a hello.</span><br><span class="line"></span><br><span class="line">$ <span class="built_in">echo</span> <span class="string">"hello"</span> | xxd -g 2</span><br><span class="line">00000000: 6865 6c6c 6f0a hello.</span><br></pre></td></tr></table></figure>
<ul>
<li><code>-p | -ps | -postscript | -plain</code> 以连续连续十六进制样式输出</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ <span class="built_in">echo</span> <span class="string">"hello"</span> | xxd -p</span><br><span class="line">68656c6c6f0a</span><br></pre></td></tr></table></figure>
<ul>
<li><p><code>-r | -revert</code> 将十六进制转为二进制。如果不是写到 stdout, xxd 在不截断的情况下将其写入输出文件。使用 <code>-r -p</code> 读取连续的十六进制输出（不包含行号信息，没有特定的列布局）</p>
</li>
<li><p><code>-seek offset</code></p>
<p>When used after <em>-r</em>: revert with <<em>offset</em>> added to file positions found in hexdump.</p>
</li>
<li><p><code>-s [+][-]seek</code> </p>
<p>start at <<em>seek</em>> bytes abs. (or rel.) infile offset. <em>+</em> indicates that the seek is relative to the current stdin file position (meaningless when not reading from stdin). <em>-</em> indicates that the seek should be that many characters from the end of the input (or if combined with <em>+</em>: before the current stdin file position). Without -s option, xxd starts at the current file position.</p>
<p>跳过多少字节</p>
</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br></pre></td><td class="code"><pre><span class="line">xxd text</span><br><span class="line">00000000: 6865 6c6c 6f0a 4f6e 2054 7565 7364 6179 hello.On Tuesday</span><br><span class="line">00000010: 2c20 7468 6520 4c69 6e75 7820 5379 7374 , the Linux Syst</span><br><span class="line">00000020: 656d 0a41 646d 696e 6973 7472 6174 6f72 em.Administrator</span><br><span class="line">00000030: 2773 2067 726f 7570 206d 6565 7469 6e67 <span class="string">'s group meeting</span></span><br><span class="line"><span class="string">00000040: 2077 696c 6c20 6265 2068 656c 642e 5379 will be held.Sy</span></span><br><span class="line"><span class="string">00000050: 7374 656d 0a41 646d 696e 6973 7472 6174 stem.Administrat</span></span><br><span class="line"><span class="string">00000060: 6f72 2041 6c6c 2053 7973 7465 6d20 4164 or All System Ad</span></span><br><span class="line"><span class="string">00000070: 6d69 6e69 7374 7261 746f 7273 2073 686f ministrators sho</span></span><br><span class="line"><span class="string">00000080: 756c 6420 6174 7465 6e64 2e20 5379 7374 uld attend. Syst</span></span><br><span class="line"><span class="string">00000090: 656d 0a41 646d 696e 6973 7472 6174 6f72 em.Administrator</span></span><br><span class="line"><span class="string">000000a0: 2054 6861 6e6b 2079 6f75 2066 6f72 2079 Thank you for y</span></span><br><span class="line"><span class="string">000000b0: 6f75 7220 6174 7465 6e64 616e 6365 2e0a our attendance..</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string"># xxd 每行默认 16 个字节，0x30 也就是 48 个字节，刚好是前 3 行</span></span><br><span class="line"><span class="string">xxd -s 0x30 text</span></span><br><span class="line"><span class="string">00000030: 2773 2067 726f 7570 206d 6565 7469 6e67 '</span>s group meeting</span><br><span class="line">00000040: 2077 696c 6c20 6265 2068 656c 642e 5379 will be held.Sy</span><br><span class="line">00000050: 7374 656d 0a41 646d 696e 6973 7472 6174 stem.Administrat</span><br><span class="line">00000060: 6f72 2041 6c6c 2053 7973 7465 6d20 4164 or All System Ad</span><br><span class="line">00000070: 6d69 6e69 7374 7261 746f 7273 2073 686f ministrators sho</span><br><span class="line">00000080: 756c 6420 6174 7465 6e64 2e20 5379 7374 uld attend. Syst</span><br><span class="line">00000090: 656d 0a41 646d 696e 6973 7472 6174 6f72 em.Administrator</span><br><span class="line">000000a0: 2054 6861 6e6b 2079 6f75 2066 6f72 2079 Thank you <span class="keyword">for</span> y</span><br><span class="line">000000b0: 6f75 7220 6174 7465 6e64 616e 6365 2e0a our attendance..</span><br></pre></td></tr></table></figure>
<ul>
<li><code>-l len | -len len</code> 只打印出 len 个字节</li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">xxd -l 5 text</span><br><span class="line">00000000: 6865 6c6c 6f hello</span><br></pre></td></tr></table></figure>
<ul>
<li><p><code>-u</code> 使用大写的十六进制字母，默认是小写的</p>
</li>
<li></li>
</ul>
<figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment"># Create a 65537 byte file with all bytes 0x00, except for the last one which is 'A' (hex 0x41).</span></span><br><span class="line">% <span class="built_in">echo</span> <span class="string">"010000: 41"</span> | xxd -r > file</span><br></pre></td></tr></table></figure>
<h2 id="参考"><a href="#参考" class="headerlink" title="参考"></a>参考</h2><p><a href="https://www.commandlinux.com/man-page/man1/xxd.1.html" target="_blank" rel="noopener">XXD</a></p>
</div>
</article>
</div>
<div class="paginator">
<a class="next" href="/page/2/">
Next
<i class="iconfont icon-next"></i>
</a>
</div>
</div>
</div>
<footer class="footer-social">
<a class="social-link" target="_blank" href="https://github.com/weirdwimp">
<i class="iconfont icon-github"></i>
</a>
<div class="footer-copyright">
<p class="time-line">
©
2024
<i class="iconfont icon-heart"></i>
<a target="_blank" href="https://github.com/iJinxin"></a>
</p>
<p class="theme-info">
Powered by <a target="_blank" href="https://hexo.io">Hexo</a> | Theme -
<a target="_blank" href="https://github.com/iJinxin/hexo-theme-sky">Sky</a>
</p>
</div>
</footer>
</body>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="/js/index.js"></script>
<script async src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script>
<script src="https://imsun.github.io/gitment/dist/gitment.browser.js"></script>
<script>
</script>
</html>