实例介绍
【实例简介】Observability Engineering：可观测性工程
【实例截图】
【核心代码】
Table of Contents
Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Part I. The Path to Observability
1. What Is Observability?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
The Mathematical Definition of Observability 4
Applying Observability to Software Systems 4
Mischaracterizations About Observability for Software 7
Why Observability Matters Now 8
Is This Really the Best Way? 9
Why Are Metrics and Monitoring Not Enough? 9
Debugging with Metrics Versus Observability 11
The Role of Cardinality 13
The Role of Dimensionality 14
Debugging with Observability 16
Observability Is for Modern Systems 17
Conclusion 17
2. How Debugging Practices Differ Between Observability and Monitoring. . . . . . . . . . . 19
How Monitoring Data Is Used for Debugging 19
Troubleshooting Behaviors When Using Dashboards 21
The Limitations of Troubleshooting by Intuition 23
Traditional Monitoring Is Fundamentally Reactive 24
How Observability Enables Better Debugging 26
Conclusion 28
iii3. Lessons from Scaling Without Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
An Introduction to Parse 29
Scaling at Parse 31
The Evolution Toward Modern Systems 33
The Evolution Toward Modern Practices 36
Shifting Practices at Parse 38
Conclusion 41
4. How Observability Relates to DevOps, SRE, and Cloud Native. . . . . . . . . . . . . . . . . . . . . 43
Cloud Native, DevOps, and SRE in a Nutshell 43
Observability: Debugging Then Versus Now 45
Observability Empowers DevOps and SRE Practices 46
Conclusion 48
Part II. Fundamentals of Observability
5. Structured Events Are the Building Blocks of Observability. . . . . . . . . . . . . . . . . . . . . . 51
Debugging with Structured Events 52
The Limitations of Metrics as a Building Block 53
The Limitations of Traditional Logs as a Building Block 55
Unstructured Logs 55
Structured Logs 56
Properties of Events That Are Useful in Debugging 57
Conclusion 59
6. Stitching Events into Traces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
Distributed Tracing and Why It Matters Now 61
The Components of Tracing 63
Instrumenting a Trace the Hard Way 65
Adding Custom Fields into Trace Spans 68
Stitching Events into Traces 70
Conclusion 71
7. Instrumentation with OpenTelemetry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
A Brief Introduction to Instrumentation 74
Open Instrumentation Standards 74
Instrumentation Using Code-Based Examples 75
Start with Automatic Instrumentation 76
Add Custom Instrumentation 78
Send Instrumentation Data to a Backend System 80
Conclusion 82
iv | Table of Contents8. Analyzing Events to Achieve Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Debugging from Known Conditions 84
Debugging from First Principles 85
Using the Core Analysis Loop 86
Automating the Brute-Force Portion of the Core Analysis Loop 88
This Misleading Promise of AIOps 91
Conclusion 92
9. How Observability and Monitoring Come Together. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Where Monitoring Fits 96
Where Observability Fits 97
System Versus Software Considerations 97
Assessing Your Organizational Needs 99
Exceptions: Infrastructure Monitoring That Can’t Be Ignored 101
Real-World Examples 101
Conclusion 103
Part III. Observability for Teams
10. Applying Observability Practices in Your Team. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Join a Community Group 107
Start with the Biggest Pain Points 109
Buy Instead of Build 109
Flesh Out Your Instrumentation Iteratively 111
Look for Opportunities to Leverage Existing Efforts 112
Prepare for the Hardest Last Push 114
Conclusion 115
11. Observability-Driven Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Test-Driven Development 117
Observability in the Development Cycle 118
Determining Where to Debug 119
Debugging in the Time of Microservices 120
How Instrumentation Drives Observability 121
Shifting Observability Left 123
Using Observability to Speed Up Software Delivery 123
Conclusion 125
12. Using Service-Level Objectives for Reliability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Traditional Monitoring Approaches Create Dangerous Alert Fatigue 127
Threshold Alerting Is for Known-Unknowns Only 129
Table of Contents | vUser Experience Is a North Star 131
What Is a Service-Level Objective? 132
Reliable Alerting with SLOs 133
Changing Culture Toward SLO-Based Alerts: A Case Study 135
Conclusion 138
13. Acting on and Debugging SLO-Based Alerts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Alerting Before Your Error Budget Is Empty 139
Framing Time as a Sliding Window 141
Forecasting to Create a Predictive Burn Alert 142
The Lookahead Window 144
The Baseline Window 151
Acting on SLO Burn Alerts 152
Using Observability Data for SLOs Versus Time-Series Data 154
Conclusion 156
14. Observability and the Software Supply Chain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Why Slack Needed Observability 159
Instrumentation: Shared Client Libraries and Dimensions 161
Case Studies: Operationalizing the Supply Chain 164
Understanding Context Through Tooling 164
Embedding Actionable Alerting 166
Understanding What Changed 168
Conclusion 170
Part IV. Observability at Scale
15. Build Versus Buy and Return on Investment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
How to Analyze the ROI of Observability 174
The Real Costs of Building Your Own 175
The Hidden Costs of Using “Free” Software 175
The Benefits of Building Your Own 176
The Risks of Building Your Own 177
The Real Costs of Buying Software 179
The Hidden Financial Costs of Commercial Software 179
The Hidden Nonfinancial Costs of Commercial Software 180
The Benefits of Buying Commercial Software 181
The Risks of Buying Commercial Software 182
Buy Versus Build Is Not a Binary Choice 182
Conclusion 183
vi | Table of Contents16. Efficient Data Storage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
The Functional Requirements for Observability 185
Time-Series Databases Are Inadequate for Observability 187
Other Possible Data Stores 189
Data Storage Strategies 190
Case Study: The Implementation of Honeycomb’s Retriever 193
Partitioning Data by Time 194
Storing Data by Column Within Segments 195
Performing Query Workloads 197
Querying for Traces 199
Querying Data in Real Time 200
Making It Affordable with Tiering 200
Making It Fast with Parallelism 201
Dealing with High Cardinality 202
Scaling and Durability Strategies 202
Notes on Building Your Own Efficient Data Store 204
Conclusion 205
17. Cheap and Accurate Enough: Sampling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Sampling to Refine Your Data Collection 207
Using Different Approaches to Sampling 209
Constant-Probability Sampling 209
Sampling on Recent Traffic Volume 210
Sampling Based on Event Content (Keys) 210
Combining per Key and Historical Methods 211
Choosing Dynamic Sampling Options 211
When to Make a Sampling Decision for Traces 211
Translating Sampling Strategies into Code 212
The Base Case 212
Fixed-Rate Sampling 213
Recording the Sample Rate 213
Consistent Sampling 215
Target Rate Sampling 216
Having More Than One Static Sample Rate 218
Sampling by Key and Target Rate 218
Sampling with Dynamic Rates on Arbitrarily Many Keys 220
Putting It All Together: Head and Tail per Key Target Rate Sampling 222
Conclusion 223
18. Telemetry Management with Pipelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Attributes of Telemetry Pipelines 226
Routing 226
Table of Contents | viiSecurity and Compliance 227
Workload Isolation 227
Data Buffering 228
Capacity Management 228
Data Filtering and Augmentation 229
Data Transformation 230
Ensuring Data Quality and Consistency 230
Managing a Telemetry Pipeline: Anatomy 231
Challenges When Managing a Telemetry Pipeline 233
Performance 233
Correctness 233
Availability 233
Reliability 234
Isolation 234
Data Freshness 234
Use Case: Telemetry Management at Slack 235
Metrics Aggregation 235
Logs and Trace Events 236
Open Source Alternatives 238
Managing a Telemetry Pipeline: Build Versus Buy 239
Conclusion 240
Part V. Spreading Observability Culture
19. The Business Case for Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
The Reactive Approach to Introducing Change 243
The Return on Investment of Observability 245
The Proactive Approach to Introducing Change 246
Introducing Observability as a Practice 248
Using the Appropriate Tools 249
Instrumentation 250
Data Storage and Analytics 250
Rolling Out Tools to Your Teams 251
Knowing When You Have Enough Observability 252
Conclusion 253
20. Observability’s Stakeholders and Allies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
Recognizing Nonengineering Observability Needs 255
Creating Observability Allies in Practice 258
Customer Support Teams 258
Customer Success and Product Teams 259
viii | Table of ContentsSales and Executive Teams 260
Using Observability Versus Business Intelligence Tools 261
Query Execution Time 262
Accuracy 262
Recency 262
Structure 263
Time Windows 263
Ephemerality 264
Using Observability and BI Tools Together in Practice 264
Conclusion 265
21. An Observability Maturity Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
A Note About Maturity Models 267
Why Observability Needs a Maturity Model 268
About the Observability Maturity Model 269
Capabilities Referenced in the OMM 270
Respond to System Failure with Resilience 271
Deliver High-Quality Code 273
Manage Complexity and Technical Debt 274
Release on a Predictable Cadence 275
Understand User Behavior 276
Using the OMM for Your Organization 277
Conclusion 277
22. Where to Go from Here. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
Observability, Then Versus Now 279
Additional Resources 281
Predictions for Where Observability Is Going 282
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
好例子网口号：伸出你的我的手 — 分享！
相关软件
小贴士
感谢您为本站写下的评论，您的评论对其它用户来说具有重要的参考价值，所以请认真填写。
- 类似“顶”、“沙发”之类没有营养的文字，对勤劳贡献的楼主来说是令人沮丧的反馈信息。
- 相信您也不想看到一排文字/表情墙，所以请不要反馈意义不大的重复字符，也请尽量不要纯表情的回复。
- 提问之前请再仔细看一遍楼主的说明，或许是您遗漏了。
- 请勿到处挖坑绊人、招贴广告。既占空间让人厌烦，又没人会搭理，于人于己都无利。
关于好例子网
本站旨在为广大IT学习爱好者提供一个非营利性互相学习交流分享平台。本站所有资源都可以被免费获取学习研究。本站资源来自网友分享，对搜索内容的合法性不具有预见性、识别性、控制性，仅供学习研究，请务必在下载后24小时内给予删除，不得用于其他任何用途，否则后果自负。基于互联网的特殊性，平台无法对用户传输的作品、信息、内容的权属或合法性、安全性、合规性、真实性、科学性、完整权、有效性等进行实质审查；无论平台是否已进行审查，用户均应自行承担因其传输的作品、信息、内容而可能或已经产生的侵权或权属纠纷等法律责任。本站所有资源不代表本站的观点或立场，基于网友分享，根据中国法律《信息网络传播权保护条例》第二十二与二十三条之规定，若资源存在侵权或相关问题请联系本站客服人员，点此联系我们。关于更多版权及免责申明参见 版权及免责申明
网友评论
我要评论