15.01 Readings and Lecture Notes

April 26-30 · Reading: 11 pages · About 3 hrs with the worked example

What to do this week, and when it is due, is on the Module 15 Overview.

Readings

Source	Sections	Printed pages	Length	Time
CyBOK v1.1.0	§8.1 Fundamental concepts: workflows and vocabulary	253-256	3 pp	20 min
CyBOK v1.1.0	§8.2 Monitor: data sources (skim)	256-263	7 pp	20 min
CyBOK v1.1.0	§8.3.1-8.3.3 Misuse detection, anomaly detection, blended	264-268	4 pp	30 min
CyBOK v1.1.0	§8.3.6 The base-rate fallacy	270	1 p	15 min
CyBOK v1.1.0	§8.7 Human factors: Incident management	283-286	3 pp	25 min
NIST SP 800-61r3	The incident response life cycle (skim)	none	skim	20 min

NIST SP 800-61r3 (April 2025): https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r3.pdf

§8.3.6 is one page and it is the most important page of the week.

Worked example

Open data/auth.log and data/web_access.log. This section walks the method; Lab 10 asks you to do it properly.

1. Start by counting, not reading

548 lines of authentication log is too many to read and trivial to count. Almost every log investigation starts by grouping and counting:

grep -oE 'from [0-9.]+' auth.log | sort | uniq -c | sort -rn | head

 328 from 203.0.113.47
  28 from 10.12.4.44
  24 from 10.12.9.8
  24 from 10.12.4.31

Three internal addresses with a couple of dozen events each, and one external address with 328. You have found the interesting thing in about fifteen seconds without reading a line.

Now look at what that address was doing:

grep -c "Failed password.*203.0.113.47" auth.log      # 184
grep "Accepted password" auth.log

184 failed authentications over about ten minutes, cycling through root, admin, oracle, test, postgres, ubuntu, git, jenkins, deploy. That username list tells you something: it is a generic list, not an organization-specific one. Nobody researched this target.

And then the line that changes everything:

Apr 21 03:27:27 vault-api-01 sshd[6485]: Accepted password for deploy from 203.0.113.47 port 51884 ssh2

One of the guesses worked. This is no longer a failed brute-force in a log full of internet background noise; it is an intrusion. Everything after that timestamp is the intruder, and the four lines that follow (a session opening, sudo cat /etc/shadow, a service status check, and the session closing twenty minutes later) are what they did with it.

Method to take away: count first, then read only what the counting pointed at, then read everything after the moment it succeeded.

2. What the logs cannot tell you

Objective 7.1 is as much about limits as about findings.

auth.log can tell you which account authenticated, from which address, by which method, and when. It cannot tell you what the intruder did once they had a shell: that needs process auditing or command logging, neither of which is here. It cannot tell you whether the credential was guessed or already known. And it cannot be fully trusted at all, because the intruder used sudo and could have edited it. That is one reason to ship logs off the host as they are written, which Syslog (§8.2.6) makes routine.

web_access.log can tell you which paths were requested, from where, with what result and response size. It cannot tell you what was in a POST body, what the response contained, or whether an authenticated user was doing something they should not: that needs application logging.

3. The two detection models

§8.3 divides detection into two approaches, and the division is the practical one.

Misuse detection encodes what bad looks like. More than 20 failed authentications from one source address in 5 minutes → alert. Precise, explainable, cheap, and it catches this brute-force immediately. It also catches only what you thought to write down: an attacker who tries three passwords an hour from a different address each time sails past it.

Anomaly detection encodes what normal looks like and flags deviation. You would first measure a baseline (this server sees authentication from three internal addresses, around the clock, publickey only) and then flag departures from it. It would have caught the slow attacker the misuse rule misses. It would also have flagged the new contractor, the changed backup schedule, and the day somebody worked from a hotel.

In practice you blend them (§8.3.3): misuse rules for the known and cheap, anomaly detection to surface the unknown, and the two feeding one queue.

Notice, incidentally, that the intruder's success is visible to a third kind of rule that is neither: any successful password authentication on a host meant for publickey only. Legitimate staff in this log all use Accepted publickey. That single rule would have fired on line 350 with a near-zero false positive rate. The best detections usually come from knowing your own environment, not from a better algorithm.

4. The arithmetic

Here is the number that explains the whole course's recurring failure.

Your detector watches authentication on this server. Assume:

• 50,000 authentication events per day
• Genuine intrusion attempts on 1 day in 100: about 500 malicious events per 100 days
• True positive rate 99%: it catches almost everything malicious
• False positive rate 1%: it fires on 1 in 100 benign events

Over 100 days: 5,000,000 events, of which 500 are malicious and 4,999,500 are benign.

True positives  = 0.99  ×     500 =     495
False positives = 0.01  × 4,999,500 =  49,995
Total alerts                        =  50,490

Probability an alert is real = 495 / 50,490 = 0.98%
Alerts per day               = 50,490 / 100 ≈ 505

A 99% accurate detector produces a queue in which fewer than one alert in a hundred is real, and it produces five hundred of them a day.

Nothing is wrong with the detector. The arithmetic is driven by the base rate: malicious events are 0.01% of the total, so even a very small false positive rate applied to an enormous benign population swamps the true positives. This is the base-rate fallacy, and §8.3.6 is one page on it because it takes one page to state and a career to internalize.

Now go back to the NORTHWIND MEADOW report: "The endpoint agent flagged the PowerShell execution as suspicious and raised a medium alert at 09:26. The alert entered a queue that averaged 400 medium alerts per day. It was not reviewed."

The alert was correct. The sensor worked. The analyst did not read it, because reading five hundred alerts a day to find five real ones is not a thing a person can do. The failure was not in detection. It was in the arithmetic of the queue.

5. So what do you actually change?

Improving the detector's accuracy is the obvious answer and the weakest one. Going from 99% to 99.9% true positive rate barely moves the ratio; the false positive rate is what dominates, and driving it from 1% to 0.1% still leaves about 5,000 false positives to 495 true ones.

The moves that work change something other than accuracy:

• Raise the base rate. Do not run the detector against everything. Run it against the population where malicious events are concentrated: administrative accounts, servers, off-hours activity. Same detector, much better ratio.
• Require corroboration. Alert on failed authentications followed by a success from the same source, rather than on failures alone. Combining two weak signals into one strong one is the cheapest large improvement available.
• Automate the response for the cheap cases. If the action on a brute-force is to block the source address, do that automatically and let a human see the summary.
• Tier the queue. Give analysts fifty things that are 20% likely rather than five hundred that are 1% likely.

6. Writing it up

The four phases below come from the previous revision, NIST SP 800-61r2. §8.7 groups the same work into three activities (prepare, handle, follow up). The current revision, r3, files it under the CSF 2.0 Functions (Govern, Identify, Protect, Detect, Respond, Recover) instead. These four headings are no longer its life cycle, but its Table 1 maps each one onto those Functions:

Phase	r3 Functions	The question
Preparation	Govern, Identify, Protect	What was in place before? Logging, backups, a plan, somebody to call.
Detection and analysis	Detect, Identify (Improvement)	What happened, how do we know, and how far does it go?
Containment, eradication, and recovery	Respond, Recover, Identify (Improvement)	Stop it, remove it, come back, and how do we decide the host is clean?
Post-incident activity	Identify (Improvement)	What changes, so this is less likely or gets caught sooner?

Three things that separate a usable memo from a bad one:

• Separate what you know from what you infer. "The account authenticated from 203.0.113.47 at 03:27" is a fact. "The attacker obtained the credential from a previous breach" is a hypothesis. Label it.
• Say what you would not trust. Once an intruder has had root, the host's own logs and its package manifest are no longer evidence. Recovery from a known-good image beats cleaning.
• Say what your recommendation costs. Containment is disruptive by definition. Say what it disrupts, so the person approving it knows what they are approving.

Key terms

Term	Short form
Security operations	The ongoing work of monitoring, detecting, and responding.
Data source	What detection draws on: network traffic, flow records, application logs, system and kernel logs, syslog.
Misuse detection	Encodes what bad looks like. Precise; catches only what was written down.
Anomaly detection	Encodes what normal looks like and flags deviation. Catches the unknown; noisier.
True / false positive rate	Fraction of malicious events detected / benign events flagged.
Base-rate fallacy	Ignoring how rare the thing is, and so overestimating what an alert means.
SIEM	Collects, correlates, and presents events from many sources.
Alert fatigue	What a queue of 500 alerts a day produces in the people who have to read it.
Dwell time	Time between compromise and detection.
Containment / eradication / recovery	Stop the bleeding; remove the attacker; restore service.
Indicator of compromise	An observable artifact of an intrusion.

Looking ahead

Finals week, May 3-7, has one item: D6, a short reflection. There is no final exam.

Thank you for the semester.