Skip to main content
the invisible-layer how abstraction is making software engineers dumber

Measuring Your Progress

11 min read Chapter 55 of 56
Summary

Provides four self-assessment tests for measuring growth in...

Provides four self-assessment tests for measuring growth in systems understanding: the debugging speed test (track time-to-root-cause), the explain-to-a-junior test (can you explain at the layer level?), the incident contribution test (are you the person people call?), and the prediction test (can you predict root causes from symptoms?). Includes quarterly layer map tracking, realistic timelines (6 months for noticeable improvement, 2 years for confident depth), and closes with encouragement grounded in the universal experience of engineers who took this path.

Measuring Your Progress

You’ve started the Curiosity Protocol. You’re spending fifteen minutes a day investigating the layer beneath your daily work. You’re writing TIL entries. But how do you know it’s working? How do you differentiate genuine progress from the comforting illusion of productivity?

You measure. Not with certifications or course completions — those measure exposure, not understanding. Instead, you run yourself through four tests, regularly, and track the results. These are self-assessments, which means they work only if you’re honest. But if you’ve made it this far in this book, you’ve already demonstrated the capacity for uncomfortable honesty about your own knowledge gaps.

Test 1: The Debugging Speed Test

This is the most objective measure you have. Track how long it takes you to find the root cause of production issues. Not “how long until the issue is resolved” — that includes deployment time, approval processes, and other noise. Specifically: how many minutes elapsed between “I started investigating” and “I know what’s causing this”?

Keep a simple log:

Date       | Issue                          | Time to Root Cause | Layer
2026-02-15 | API timeouts to payment svc    | 47 min             | Application (conn pool)
2026-03-02 | Slow dashboard page            | 22 min             | Database (missing index)
2026-03-18 | Pod OOMKilled intermittently    | 85 min             | OS/container (memory limit)
2026-04-01 | Intermittent 502s from LB       | 12 min             | Network (health check path)

The “Layer” column matters. Early on, your debugging will be fastest at the layer you’re most familiar with and painfully slow at layers you’re still learning. Over months, you should see the times at unfamiliar layers decrease. You should also see yourself correctly identifying the layer faster — spending less time investigating at the wrong layer before pivoting to the right one.

A realistic trajectory: in month one, a cross-layer issue (symptom at the application layer, root cause at the network or OS layer) might take you 60–90 minutes. By month six, the same class of issue should take 15–30 minutes. By year one, you should be consistently under 15 minutes for issues in your primary layer and under 30 minutes for issues one layer below.

If you’re not seeing improvement after three months, check your Curiosity Protocol. Are you actually investigating daily, or are you skipping most days? Are you investigating your own systems, or reading abstract tutorials? The protocol works when applied to the systems you operate. It doesn’t work as an intellectual exercise.

Test 2: The Explain-to-a-Junior Test

Pick a production incident you resolved recently. Now explain it to a junior engineer — or, if none is available, explain it to yourself out loud. But here’s the constraint: you must explain both the symptom and the mechanism. Not “the database was slow” but “the query planner chose a sequential scan because the statistics on the orders table were stale after a bulk import, so it estimated 200 rows when there were actually 2 million, and a sequential scan of 2 million rows takes 8 seconds on spinning disk.”

Grade yourself on three criteria:

Layer identification: Can you name the specific layer where the root cause lived? Not “the backend” — that’s a region, not a layer. “The PostgreSQL query planner” is a layer. “The Linux CFS scheduler’s throttling mechanism” is a layer. “The browser’s layout engine” is a layer.

Mechanism description: Can you describe how the failure propagated? The root cause is the spark, but the outage is the fire. Can you trace the path from cause to symptom? “The stale table statistics caused the query planner to underestimate row count, which caused it to choose a sequential scan over an index scan, which increased query time from 3ms to 8 seconds, which exhausted the application’s database connection pool because each connection was held for 8 seconds instead of 3ms, which caused new requests to queue, which triggered the load balancer’s timeout, which returned 504 to the user.”

Counterfactual reasoning: Can you describe what should have happened, at the layer level? “If the autovacuum daemon had been running on schedule, it would have updated the statistics after the bulk import, and the query planner would have estimated the correct row count, and it would have chosen the index scan.” This matters because counterfactual reasoning demonstrates that you understand the system’s intended behavior, not just the failure mode.

When you first start running this test, you’ll notice gaps. You’ll be able to describe the symptom and maybe the root cause, but you won’t be able to trace the propagation path or articulate the counterfactual. That’s normal. Those abilities develop as your mental model gains resolution over time.

Run this test after every significant incident. After six months, compare your earliest explanations with your most recent ones. The difference will be visible.

Test 3: The Incident Contribution Test

This one requires honesty, and it’s binary: when something mysterious breaks in production, are you a person who gets called, or a person who waits for the resolution?

Early in this process, you won’t be called. Your team doesn’t yet know that you’ve been building cross-layer understanding. That’s fine. But you should be volunteering. Jump on the incident call. Offer hypotheses. Run commands. Even if your first three hypotheses are wrong, the act of generating hypotheses at multiple layers and testing them is the practice itself.

Over months, two things happen. First, you start being right more often — not because you’ve memorized failure modes, but because your mental model covers enough layers to generate plausible hypotheses quickly. Second, your team notices. When someone says “pod X is OOMKilled but the application doesn’t seem to be leaking memory,” you’ll be the one who says “check the kernel’s memory accounting — the container’s memory limit includes kernel buffers, not just heap allocation. Check /sys/fs/cgroup/memory/kubepods/pod<uid>/memory.stat for cache and RSS breakdown.”

You’ll know you’ve made real progress when one of these things happens:

  • Someone Slacks you directly during an incident, bypassing the on-call rotation.
  • A post-mortem credits your diagnosis.
  • A junior engineer asks you — specifically you — to explain how something works beneath the application layer.

These aren’t vanity metrics. They’re external validation that your internal mental model has crossed a threshold of practical usefulness.

Test 4: The Prediction Test

This is the most demanding test, and the most rewarding. When someone describes a production symptom, can you predict the root cause — or at least the correct layer — before anyone starts investigating?

The format is simple. Hear the symptom. Generate a hypothesis. Write it down (even if just mentally). Then watch the investigation unfold. Were you right? Were you in the right layer but wrong about the specific mechanism? Were you in the wrong layer entirely?

Some examples of what developing prediction ability looks like:

Month 1: Someone reports “the API is slow.” Your prediction: “Maybe the database?” That’s a layer guess, barely even a hypothesis. It’s a start.

Month 4: Someone reports “the API is slow, but only for authenticated users.” Your prediction: “The auth middleware is making a synchronous call to the identity provider, and that call is slow. Check the network latency between the API pod and the IdP endpoint.” That’s a mechanism hypothesis at a specific layer.

Month 8: Someone reports “the API is slow, but only for authenticated users, and only since yesterday’s deploy.” Your prediction: “Yesterday’s deploy changed the auth middleware to validate JWT signatures synchronously instead of caching the validation result. Each request is now fetching the JWKS endpoint, and the DNS resolution for the JWKS URL is hitting a stale resolver cache on the new nodes, adding 5 seconds per cold DNS lookup.” That’s a multi-layer hypothesis with a specific, testable prediction.

You won’t reach month 8’s level in month 1. But you should be able to track movement along this trajectory. Your hypotheses should grow more specific, span more layers, and name concrete mechanisms over time.

Quarterly Layer Map Review

In Chapter 15, we introduced the concept of a personal layer map — a visual representation of the layers you work on and your current understanding of each. Pull it out every three months and update it.

For each layer, rate yourself on a three-point scale:

  • 1 — Awareness: You know this layer exists and can name its key concepts, but you can’t use them in practice. You know what TCP is, but you can’t read ss output.
  • 2 — Working knowledge: You can use the tools and concepts at this layer for debugging and basic decision-making. You can read ss output, identify problematic connection states, and correlate them with application behavior.
  • 3 — Mental model: You can predict behavior at this layer, reason about failure modes, and explain mechanisms to others. You understand why CLOSE-WAIT accumulates (the local application hasn’t called close() on the socket), what causes it (connection leak, daemon thread not properly shutting down connections), and how to fix it.

Track the ratings over time. The goal is not to reach 3 on every layer — that would take a lifetime, and most of it would be wasted. The goal is to reach 2 on the layer directly below your primary work layer and 1 on the layer below that. If you’re an application developer, reach 2 on OS/runtime/database and 1 on networking/hardware. If you’re a frontend developer, reach 2 on browser internals/networking and 1 on OS/server.

A realistic progression: if you start at 1 on a layer, expect 3–6 months to reach 2 (with consistent fifteen-minute daily practice). Moving from 2 to 3 takes longer — 12–18 months — because it requires not just knowledge but internalized mental models that only develop through repeated application to real problems.

Realistic Timelines

People abandon practices that don’t produce visible results fast enough. So let me set your expectations honestly:

Weeks 1–4: You feel like you’re learning disconnected trivia. DNS TTLs, TCP states, cgroup limits. It’s interesting but doesn’t seem to help with your daily work. This is normal. Stick with it.

Months 2–3: You start noticing connections. A production issue comes up and, for the first time, you check the layer below first. You might not find the answer there, but the fact that you checked changes your debugging approach. Your TIL journal has 40–60 entries and you’ve referenced it during a real investigation at least once.

Months 4–6: The inflection point. Your debugging speed on cross-layer issues decreases noticeably — maybe 30% faster than when you started. You’ve generated a correct root-cause prediction at least once. A colleague has asked you “how did you know to check that?” and you were able to explain the mental model behind it. You’re starting to feel like the investment is paying off.

Months 7–12: You’re no longer consciously applying the Curiosity Protocol — it’s become automatic. When you see a 504, your brain generates three hypotheses at different layers before you’ve opened a terminal. Your layer map has moved from 1s to 2s in critical areas. You’re the person people pull into incidents, even if you’re not on call.

Year 2: You’re confident at the layer below your primary one. Not expert-level — that would mean changing your specialization — but confident enough to diagnose most issues independently and to know when you’ve hit the boundary of your understanding and need to escalate to a specialist. Your debugging speed has plateaued, which is fine — the improvement from “helpless” to “competent” is the big leap. The improvement from “competent” to “expert” is smaller and takes longer.

The Universal Regret

I’ve talked to dozens of engineers who took this path — who decided, at some point in their careers, to look beneath their primary layer and build a working understanding of the systems they depend on. Backend engineers who learned operating systems. Frontend engineers who learned browser internals. DevOps engineers who learned kernel mechanics. Data engineers who learned storage formats and query planners.

Every single one says a version of the same thing: “I can’t believe I waited so long.”

Not because the knowledge was hard to acquire — most of them are surprised by how accessible it was once they started. Not because it required a dramatic career change — they kept doing their jobs the entire time. But because the gap between working on a system and working with a system is so large, and so immediately apparent once you cross it, that the years spent on the other side feel like wasted potential.

They’re not wasted, of course. You built real things during those years. You shipped features, served users, grew your career. But you did it with one hand tied behind your back, solving problems at one layer that had root causes at another, spending hours on issues that take minutes when you can see the full stack.

You have the protocol. You have the tests. You have realistic timelines. The only remaining question is whether you start — and the best time to answer that question was a year ago. The second-best time is today.