This is the “production reality” half of the deployment runbook: how to scale safely, keep the environment governable, and avoid the most common causes of MTTR spikes.

1) Gradual Expansion (Production)

1.1 Telemetry + tuning before scaling

Before you expand scope beyond the pilot rings, validate three things with evidence:

• Stability

  • Windows: crash/BSOD signals

  • macOS: kernel panic signals

• Performance

  • CPU/IO p95 during peak hours

  • boot/login impact (baseline vs post-deployment)

• Noise

  • alert volume by module/blade

  • top noisy endpoints and recurring detections

TAC rule: if you can’t show stability + p95 performance + noise baseline, you’re not ready to scale.

1.2 Exceptions management (governance)

An exception must be:

• Scoped by Virtual Group (never global by default)

• Justified (incident / validated false positive / business requirement)

• Time-bounded with a review date (and owner)

Avoid: “global permanent exceptions” for a single application.
Prefer: function-based scoping (e.g., Dev vs Finance) and the smallest possible exception surface.

2) Continuous Operations (Day-2)

2.1 Recommended operational cadence

A cadence that keeps the environment “boring” (in a good way):

• Weekly

  • Top detections (by severity + volume)

  • Noisiest endpoints (repeat offenders)

• Monthly

  • Exceptions review (keep/expire/refine)

  • Policy deltas (what changed + why + impact)

• Quarterly

  • Drift audit: group mappings, client versions, enabled modules, ring alignment

2.2 Controlled upgrades (no component drift)

Golden rule

• Do not change components during an upgrade.

• Change components before or after — never “during”.

Why (TAC view): upgrade + module change at the same time multiplies variables and makes RCA unreliable when something breaks.

Best practice

• Upgrade by rings (Pilot → Wave 1 → Wave 2 → Full)

• Treat “enable/disable modules” as a separate change request with its own validation gates

3) Policy Best Practices (engineering-grade)

3.1 Enforcement strategy by maturity

• Initial phase: stable coverage + visibility (reduce surprises)

• Evolve: harden (more blocking) based on evidence (alert trends + validation)

Practical note: “start restrictive” only works if you have triage capacity and governed exceptions. In many orgs, the fastest path is:
start stable → harden quickly by waves.

3.2 Group-based policy (AD / Virtual Groups)

Group policies by:

• Risk (high-risk / privileged)

• Function (dev, finance, third-party)

• Technology (VDI, macOS, specialized endpoints)

This prevents an ungovernable monolithic policy.

3.3 User experience and ticket reduction

• Reduce pop-ups and user prompts where possible (keep alerts actionable)

• Standardize messaging + escalation paths:

  • what goes to SOC

  • what goes to Service Desk

  • what is “known benign” and should be exception-handled

•  

3.4 Documentation and change control

Every policy change should capture:

• Reason (incident / false positive / audit requirement)

• Scope (which groups)

• Expected impact (what could break)

• Rollback plan (how to revert safely)

•  

4) TAC-Style Runbooks (must exist before go-live)

4.1 “Installed but not visible / policy not applied”

Checklist:

• Is the endpoint in the correct group?

• Is the Deployment Policy hitting the target?

• Portal connectivity constraints (proxy/DNS/SSL inspection)?

• Is the client version compatible with the tenant/policies?

•  

4.2 “Performance degraded”

Process:

• Identify the active module when the impact started (what changed recently?)

• Correlate with:

  • High IO (scanning)

  • High CPU (emulation/behavioral engines)

  • Timing patterns (logon storm, VDI cycles)

• Action:

  • Tune/reduce scope in the affected group, not globally

  •  

4.3 “False positive on a critical app”

Process:

• Collect evidence (hash, path, signer, behavior)

• Create a granular exception (group + app) with expiration

• Validate in a small ring, then expand

•  

5) High-Value Recommendations (incident prevention)

• Do not change modules during upgrades

• Ring-based upgrades via Deployment Policy

• Air-gapped/offline: plan packages and manual updates (no improvisation)

• FDE: plan keys/recovery/helpdesk workflows before mass encryption

• VPN + Endpoint on the same host: validate interoperability and impact (latency, split tunneling, DNS)

•  

6) Validation metrics (what Security and IT both need)

• Coverage: % endpoints active + blades enabled

• Health: crash/incident rate per 100 endpoints

• Performance: CPU/IO p95 at peak

• Efficacy: unique detections, meaningful blocks, response time

• Operations: endpoint MTTR, ticket volume per wave

• Governance: number of active exceptions + average age (stale exceptions = risk)

•  

7) Official references

• sk154072 — Harmony Endpoint Client Deployment and Upgrade Best Practice

• sk182659 — Harmony Endpoint Onboarding Best Practices

• Infinity Portal Administration Guide