SIEM and Security Operations: Building a SOC That Works

A SIEM collects security logs from your identity provider, endpoints, cloud accounts and applications, runs detection rules over them, and gives analysts one place to investigate. A SOC is the team and process that responds to what it finds. The platform is the easy part. An effective security operations function comes from collecting the right logs first, writing detections as tested code tied to a response runbook, tuning ruthlessly for precision, and having someone able to act at 3 a.m., whether that is your own team or a managed provider.

Key takeaways

• Start from the attacks you need to catch, then collect the logs that reveal them. Identity, EDR and cloud audit logs come first.
• Treat detections as code: version-controlled, reviewed, tested against sample events, mapped to MITRE ATT&CK, and each linked to a runbook.
• Precision per rule is the metric that keeps a SOC alive. A rule that is wrong most of the time trains analysts to ignore it.
• Alert on silence. A log source that stops sending is a blind spot, and sometimes the first sign of an attack.
• Ingestion cost scales with volume. Route noisy, low-value logs to cheap storage and keep the SIEM for what you detect on.
• Round-the-clock coverage is a staffing problem before it is a tooling problem. For most companies, MDR plus in-house detection engineering is the realistic model.

The vocabulary, briefly

Most platforms now blur these lines: SIEMs ship automation, XDR products ingest third-party logs, MDR providers bring their own SIEM. Decide what you need by capability, not by acronym.

Start with the questions, not the logs

The common failure is to point every log source at the SIEM, receive an invoice, and then wonder what to detect. Reverse it. List the scenarios you most need to catch, given how you are built:

• An account taken over through phishing or a stolen session.
• A leaked cloud key used from outside your network.
• An admin disabling logging or security tooling.
• Data pulled in bulk from a database, bucket or SaaS app.
• Malware or a hands-on attacker on an endpoint or server.
• Malicious code in your CI pipeline or dependencies.

For each scenario, write down the log events that would reveal it. That list is your collection plan, and the scenarios become your first detections.

Log sources in priority order

Two things matter as much as the sources. Time: everything in UTC, from hosts with synchronized clocks, or your correlations quietly fail. Identity: the same person appears as an email in the IdP, a username on a host and a role session in the cloud. Map them to one entity, or every investigation starts with manual joins.

Normalize to a common schema so rules work across sources. The Elastic Common Schema and the Open Cybersecurity Schema Framework (OCSF, which AWS Security Lake uses) are the two you will meet most.

Architecture options

The hybrid pattern is common and sensible: detection-grade sources go to the SIEM, bulky sources (flow logs, full DNS, verbose application logs) go to object storage in an open format, queryable when an investigation needs them. Many SIEMs also have cheaper tiers for logs you search rarely, such as the Microsoft Sentinel data lake tier.

Plan retention around investigations and obligations. Breaches are often discovered long after they began, and some standards set minimums: PCI DSS, for example, requires a year of audit log history with the most recent three months immediately available.

Detection engineering as code

Detections deserve the same discipline as application code. Keep them in a repository, review changes, test them, and deploy through CI.

Sigma is a vendor-neutral rule format that converts to most SIEM query languages, which keeps your detections portable. A rule for someone switching off CloudTrail:

title: CloudTrail logging stopped, deleted or modified
status: stable
description: Detects attempts to disable or change an AWS CloudTrail trail, a common step to hide activity.
logsource:
  product: aws
  service: cloudtrail
detection:
  selection:
    eventSource: cloudtrail.amazonaws.com
    eventName:
      - StopLogging
      - DeleteTrail
      - UpdateTrail
  condition: selection
falsepositives:
  - Planned changes by the platform team through infrastructure as code
level: high
tags:
  - attack.defense-impairment
  - attack.t1685.002

Convert it for your backend with sigma-cli (for example sigma convert -t splunk -p PIPELINE rule.yml, with the target and processing pipeline that match your platform), and commit the generated query alongside the source.

Not everything fits Sigma. Scheduled queries in your SIEM's own language are fine, as long as they live in the repository too. An example in Athena over CloudTrail, finding console sign-ins by IAM users without MFA:

SELECT eventtime, useridentity.arn, sourceipaddress, useragent
FROM cloudtrail_logs
WHERE eventname = 'ConsoleLogin'
  AND useridentity.type = 'IAMUser'
  AND json_extract_scalar(additionaleventdata, '$.MFAUsed') = 'No'
  AND eventtime > to_iso8601(date_add('day', -1, current_timestamp));

Every detection in the repository should carry:

• An owner who tunes it and answers for its false positives.
• ATT&CK technique tags, so you can see coverage against the techniques that matter for you. ATT&CK Navigator turns those tags into a heatmap.
• Known false positives, written down when the rule is created.
• A runbook link: what the analyst checks first, and what containment looks like.
• A test: sample events that must trigger it and sample events that must not, run in CI on every change.

Validate detections end to end, too. Periodically reproduce the behavior safely in a test account (stop a test trail, sign in without MFA on a test user) and confirm the alert arrives where people will see it. Rules that pass unit tests can still fail because a log source stopped flowing or a field was renamed.

Alert tuning: precision keeps the SOC alive

Alert fatigue is not a morale problem. It is a measurement problem. Track, for each rule, how many alerts it raised and how many turned out to matter. Then act on the numbers:

• Route by confidence and urgency. Only high-confidence, time-sensitive alerts page someone. Medium alerts go to a queue with a response target. Low-confidence signals feed correlation and hunting, not humans.
• Aggregate. Group alerts on the same user, host or key within a window into one incident with a timeline. Ten alerts about one compromised account are one problem.
• Enrich automatically. Asset owner, user's department and manager, whether the IP belongs to your VPN, recent changes to the account. Every manual lookup an analyst does is a candidate for automation.
• Allowlist with a reason and an expiry. Permanent exceptions accumulate until they hide exactly the activity you care about.
• Fix or retire low-precision rules. Narrow the logic, raise the threshold, add context, or turn the rule into a hunting query that someone runs deliberately.

Automation that analysts trust

Automate the repetitive parts first, and keep humans in the loop for anything disruptive until the automation has earned trust.

Each automated containment step needs a clear undo, and the people who might be affected (an engineer whose key was deactivated, an executive whose session was revoked) need to know what happened and why. The response itself belongs in your incident response playbooks.

People and coverage

A single 24/7 seat is 168 hours a week. That is more than four full-time people before anyone takes leave, falls ill, trains or quits, and a seat staffed by one tired person at 4 a.m. is not much of a seat. That arithmetic is why most companies should not build a round-the-clock SOC early.

Realistic models:

• MDR provider plus an internal owner. The provider watches around the clock; an internal engineer owns the relationship, the detections specific to your product and the response decisions.
• Business-hours SOC plus on-call. Analysts during the day, with only high-confidence alerts paging an on-call engineer at night. Works when automation handles enrichment and the paging rules are tight.
• Follow the sun. Teams in different time zones hand over. Needs disciplined handover notes and shared runbooks.

When buying MDR, ask: Do we keep our raw logs and can we export them? Can we see and add to the detection rules? What can the provider do on its own authority at 3 a.m., and who does it call when it needs us? Are response times contractual?

Not every signal is a security log

Some intrusions show up first as operational symptoms: a server pinned at full CPU because someone installed a miner, a disk filling up, a service restarting in a loop, traffic far above normal. Your SOC should see those signals, or at least share an on-call path with the people who do. A self-hosted option for Linux fleets is our Trinetra, which runs a small agent per server and sends threshold alerts to Telegram and other channels; Prometheus with Alertmanager or Netdata do the same job at different scales. Our observability guide covers the metrics and logs side in more depth.

Metrics that drive improvement

• Time to detect, from first malicious activity to the alert, measured on real incidents and exercises.
• Time to respond, from alert to containment.
• Precision per rule, and the trend for the whole rule set.
• Coverage of your priority scenarios, not a percentage of all ATT&CK techniques. Full coverage of the techniques that matter to you beats thin coverage of everything.
• Detections validated end to end in the last quarter.
• Log source health: sources that went silent, and for how long.

For compliance programs such as SOC 2, these same records are the evidence auditors ask for: that logs are collected, alerts are reviewed and incidents are handled.

What goes wrong in production

• A log source silently stops. A forwarder crashes, an API token expires, a vendor changes an endpoint. Detections over that source go quiet, and quiet looks like safety. Alert when any source's volume drops below its normal range.
• A format change breaks parsing. The vendor renames a field and every rule that used it stops matching, with no error. Tests against real sample events in CI, refreshed periodically, catch this.
• The ingest bill explodes. Someone enables verbose DNS or flow logs straight into the SIEM. Put budgets and alerts on ingestion per source, and route bulky sources to the data lake by default.
• Clocks disagree. A host a few minutes off breaks correlation windows and confuses timelines during an incident. Synchronize time everywhere and store UTC.
• The SIEM is a soft target. It holds sensitive data and admin access to response actions. Protect its admin accounts with phishing-resistant MFA and log access to the SIEM itself.
• Retention runs out mid-investigation. The breach started months ago and the logs that show the entry point have aged out. Set retention from investigation needs, not from the default.
• Dashboards replace detections. A wall of charts nobody watches at night is not monitoring. Anything that needs a response should be an alert with an owner.

Frequently Asked Questions

What is a SIEM and how does it work?

A SIEM (security information and event management) system collects logs from identity providers, endpoints, cloud accounts, network devices and applications, normalizes them into a common format, and runs detection rules over them to raise alerts. It also stores the logs so analysts can search them during an investigation. The detections and the people who respond to them decide its value, not the platform itself.

What is the difference between a SIEM and a SOC?

A SIEM is a tool. A SOC (security operations center) is the team and process that monitors alerts, investigates them and responds to incidents, usually using a SIEM as its main data source. You can buy a SIEM in a day; building a SOC means detections, runbooks, coverage hours and people who know your environment.

What logs should be sent to a SIEM first?

Start with identity provider sign-in and audit logs, EDR telemetry, cloud control plane logs such as AWS CloudTrail, Azure activity logs and Google Cloud audit logs, and email security logs. Those cover the most common ways attackers get in and escalate. Add SaaS admin logs, VPN, DNS and network flow data next, and route high-volume, low-value sources to cheaper storage.

What is the difference between SIEM, SOAR and XDR?

A SIEM collects and correlates logs from many sources and runs detections over them. SOAR automates the response side: enrichment, ticketing and containment steps triggered by alerts. XDR is a vendor's integrated detection and response across its own products, typically endpoint, identity and email, and many platforms now bundle all three.

Should a small company build a SOC or use an MDR provider?

Most companies without a dedicated security team should start with a managed detection and response (MDR) provider, because round-the-clock coverage needs several people before you count leave and turnover. Keep ownership of your log data and make sure the provider can act, or reach someone who can, at any hour. Build in-house detection engineering for what is specific to your product as the team grows.

How do you reduce SIEM alert fatigue?

Measure precision for every rule and fix or retire the rules that are mostly false positives. Group related alerts into one incident per user or host, enrich alerts automatically so analysts do not do manual lookups, and send only high-confidence, urgent alerts to a pager. Allowlist known-good activity with a documented reason and an expiry date rather than permanently.

Need help with security operations?

We run production systems and carry the pager for them, so we build detection the way we would want to be woken up by it: few alerts, each one worth acting on, with a runbook attached. Senior engineers set up the log pipeline, the detections and the response, whether you run them yourself or pair them with an MDR provider.

• Cybersecurity and compliance
• Cloud architecture and infrastructure
• Talk to an engineer