Live Cyber Attack Lab 🎯 Watch our IR team detect & respond to a rogue insider trying to steal data! Choose a Session

X

Sysmon Threat Analysis Guide

IT Pros, Threat Detection

Over the course of my pen testing experiments for the IOS blog, I’ll often “go to the video” —  that is take a peek into the Windows Event logs — to see what gets tracked when Windows process auditing is turned on. In my various scenarios, I pretend to be a blue team defender and try to work out the attack. If you have good security eyes, you can search for unusual activities  in the raw logs — say a PowerShell script running a DownloadString cmdlet or a VBS script disguised as a Word doc file — by scrolling through recent activity in the Windows Viewer.

Part I: Introduction to Sysmon Log Analysis

Part II: Deeper into Sysmon and Threat Detection 

 

But under real-word battle conditions, IT security pros would have to deal with zillions of log entries and perform threat analysis on complex attacks.

One big issue with relying on the out-of-the-box  Windows Events log is that there’s a lot of context unavailable in the entries, making the analysis  more than a little challenging.  For example, the  Windows Event logs are missing parent process information, and so process hierarchies can’t be worked out. In Part II, we’ll take full advantage of this parent process information to create more complex mapping structure. In Part I, let’s see what we can do with basic Sysmon information.

Part I: Introduction to Sysmon Log Analysis

What’s the answer to event log confusion? Ultimately, a SIEM solution would help normalize the event information, making it more amenable to analysis.

But you don’t have to go that far, at least initially. A first step in understanding what SIEM can do is to try Windows wonderful freebie tool Sysmon. It’s surprisingly easy to work with. Go Microsoft!

What’s So Great About Sysmon?

Sysmon installs as a device driver and service — more here — and it’s key advantage is that it takes log entries from multiple log sources, correlates some of the information, and puts the resulting entries into one folder in the Event Viewer, found under Microsoft->Windows->Sysmon->Operational.

In my own hair-raising expeditions through the Windows logs, I’ve had to bounce between, say, the PowerShell log folder, and then back to the Security folder and then scroll through separate Event Viewers heroically trying to match up entries. It ain’t easy to analyze and as I learned, it’s helpful to have a couple of aspirins in reserve.

Sysmon then provides a solid first-step in providing useful — or actionable, as vendors like to say — information to help you begin to understand the underlying causes. For example, I remotely launched a stealthy wmiexec session simulating a clever insider lateral move. Below you can see what I found in  the Windows Events log:

Cmd shell launching a “cd” command and redirecting output to a file. Seems strange, no?

For an IT pro, with some hacking knowledge, this particular command line should trigger some suspicions. Using cmd.exe to then run another command while redirecting the output to a strangely named file is the stuff of some command-and-control (C2) software: it’s a way to create a pseudo-shell using the WMI services.

Let’s take a look at the equivalent entry in Sysmon, and gaze upon the wealth of extra information contained in a single log entry:

Command shell launched by WMI?? J’accuse!

Not only can we see the actual command line, but also the file name and path of the executable, what Windows knows about it (“Windows Command Processor”), the process id of the parent, the command line of the parent which launched the Windows cmd shell, and the actual file name behind the parent process. We have it all, finally.

From the Sysmon log, we can conclude with good confidence that the strange command line found in the raw files is not something crafted by an employee doing normal work. Instead it was generated by a C2-like process — the wmiexec I mentioned above — and spawned directly by the WMI service process (WmiPrvSe). We now have the smoking gun that a remote attacker or insider is trying to probe the corporate IT system.

Introducing Get-Sysmonlogs

It’s wonderful that Sysmon puts all this log information in one place. Wouldn’t it even be more wonderful if we could access all the individual fields in the log programmatically, say, by a PowerShell cmdlet? We could then  write a little PowerShell script to automate the process of finding potential threats!

I’m not the only one who’s had this idea. And thankfully there are some posts in forums and github projects that explain how to use PowerShell to parse the Sysmon log. However, I wanted to avoid having to write separate lines of PS parsing script for each Sysmon field. Instead I took the lazy person’s approach to parsing Sysmon entries and I think I came up with something clever.

The first important point is that the convenient Get-WinEvent cmdlet can read the Sysmon logs and put the results into a PS variable, like below:

$events = Get-WinEvent  -LogName "Microsoft-Windows-Sysmon/Operational" | where { $_.id -eq 1 -or $_.id -eq 11}

If you’re trying this at home, by displaying the contents of the first element of the $events array, $events[0].Message,  you’ll notice it’s a series of one-line text strings following a very simple format: name of Sysmon field, a colon, and then the value.

Eureka! JSON-ready output from the Symon log.

Are you thinking what I’m thinking? With a little more effort we can convert this into a JSON-formatted string and then load the string directly into a PS object using the powerful ConverFrom-Json cmdlet.

I’ll show the actual PowerScript — it’s very simple — to do this in the next section. For now, let’s take a look at what my new cmdlet, called get-sysmonlogs and which I installed as a PS module, can accomplish.

Instead of having to dive into the Sysmon logs using the clunky Event Viewer interface, we can effortlessly search for incriminating activity from a PowerShell session, and use the PS where cmdlet (aliased by ?) to narrow down the results:

A list of all the cmd shells run by WMI. Threat analysis on the cheap with our own get-sysmonlogs.

Amazing!  I’ve created a tool that lets you query the Sysmon log as if it were a database. Those of you who recall my brief post on EQL know that this is exactly what this cool software does as well, though, more formally through a real SQL-like interface. Yeah, EQL is neat and we will explore it more later on.

Sysmon and Graph Analysis

Let’s sit back and appreciate what we’ve done. We now effectively have a database of Windows event information accessible through PowerShell. As I pointed out above, there are connections or links between those entries — through the ParentProcessId — so it would be possible to capture complete process hierarchies.

If you’ve followed our series on fileless malware, you know that hackers are fond of creating confusing multi-step attacks with each process performing one small  part of the attack and then spawning the next step. This ain’t easy to detect looking at the raw logs.

But with my get-sysmonlogs cmdlet and an additional data structure which we’ll get into next time — a graph naturally — we have a practical way to detect threats. We just need to conduct the right searches among the nodes. Next time I’ll explain how to do this.

As always with our DIY blog projects, the far more important point is that by working out some of the details of threat analysis on a smaller scale, you begin to appreciate how complicated it is to do enterprise-level threat detection.

We’ll start getting a sense of the interesting complexities in Part II, when we begin to weave together Sysmon events into more complicated structures.

Part II: Deeper into Sysmon and Threat Detection 

Working With Get-Sysmonlogs

Let’s now take a closer look at my amazing cmdlet that translates from Sysmon events into PowerShell objects. I’m somewhat proud of the fact I didn’t have to manually code a line for each of the fields. And here’s the great reveal:

$events = Get-WinEvent  -LogName "Microsoft-Windows-Sysmon/Operational" | where { $_.id -eq 1 }
 
foreach ($event in $events)  {
    $ev = $event.Message -split "`r`n"
    $jsons="{ "
    foreach ($line in $ev) {
        $line=$line -replace "\\","\\" `
               -replace "\{"," " `
               -replace "\}"," " `
               -replace '"','\"' `
               -replace "`n"," " 
        $line=$line -replace '(\s*[\w\s]+):\s*(.*)', '"$1":"$2",'
        $jsons = $jsons + $line } 

        $jsons =$jsons + '"blah" : "blah" }' 
        ConvertFrom-Json -InputObject $jsons 
    }
}

The whole shebang is now on github, and you can  download and import it as a sysmon module for your own project. The only wonkiness involves removing a few unpleasant characters — brackets, backslashes, line feeds, quotes — to make it more Json friendly.

Let’s do more with  get-sysmonlogs.  So … a classic signal of an intruder poking around the  system is the  use of the command “whoami”— often followed by “hostname”.  A hacker (or perhaps insider) who’s obtained someone else’s credential wants to make sure their impersonation is working so they’ll type the aforementioned commands when they’ve landed on a victim’s server.  For the rest of us, whoami and hostname are not words we’d enter into our shell sessions — if we ever even use a shell.

With my neat cmdlet giving access to all the Sysmon log entries, we can easily cook up a pipeline that filters on the process name (as we did in Part I). And we can get a little more granular with Sysmon, and look at the parent’s command line.

Usually when hackers get in and have shell access, it’s often with the legacy cmd shell — which is what happens, by the way, when hacking with psexec or smbexec. With the output of get-symonlogs, we can hunt for whoami processes that have been spawned by  these creaky shells — good evidence pointing to a threat.

Warning: Whoami launched by a legacy cmd shell.

For practical purposes searching through the very raw Windows Event logs and connecting processes is not possible. As we just saw, Sysmon log entries can open up lots of threat analysis possibilities. Let’s continue our exploration by mapping the Sysmon information into more complicated structures.

Data Structures 101: Lists and Graphs

Not only do the Sysmon logs entries give us the parent command line, but also the parent’s process id!

You’re probably one step ahead, but let me spell it out: we can now connect together processes into hierarchies and, dare I say, networks. Tapping into basic Comp Sci 101 knowledge, there are some natural data structure we use to capture this information. Linked lists and graphs immediately come to mind.

At first, I thought I’d have to dust off my copy of “Data Structures for Poets and Sous Chefs” but the Intertoobz came to my rescue. I stumbled onto Doug Finke’s wondrous collection of core algorithms on Gihub, written in PowerShell. Thanks Doug!

After getting past the minor learning curve, I was able to use his data structure algorithms to start organizing my Sysmon event data. I built list and graph data structures, and then using the APIs, wrote up a PowerShell function to search for a command and list out the process hierarchy. Cool.

I called it show-threat-path. It does a depth first search on the process hierarchy and lists out the application names and the associated command lines for a root application given by a parameter. For my first test, I searched on “whoami.exe”. Like so …

Something looks awry with process 2542!

Extra points if you noticed in the output snippet above that the whoami associated with the process Id of 2542 was spawned by the legacy cmd shell, which in turn was run by a weirdly named executable in the Windows directory.

Hmmm. If you’re familiar with the mechanics of a remote psexec — covered here — then sirens should be going off. I’ll let you in on a secret: playing the part of the hacker, I previously launched this whoami from a remote Linux server using the Impacket python scripts.

So the point is that with the nutrient rich Sysmon logs and some PowerShell, you can cook up practical threat hunting tools, like what I just did with show-threat-path.

Threat Hunting With Directed Graphs

Now it’s time to get a little wonkier. With all this process trace information obtained from Sysmon, I can look at the connections in a more general way. I want to think of the applications that get launched — powershell.exe, explorer.exe, etc — as nodes in a graph, and then makes connections to whatever app had in turned launched it. In effect I’m creating a roadmap that shows how all the apps communicate with each other (rather than creating  separate nodes for each process instance.)

Technically, I’m talking about a directed graph where the road, so to speak , is one-way from the app to its parent.

At this point it probably makes sense to gaze upon a graphic of what I’m talking about. Fortunately, there’s a wonderfully delicious PowerShell graph visualization tool known as GraphViz that has drop-dead easy wrappers available through PSQuickGraph.  So with this little bit of code …

#Let's graph it!!!
$gv = New-Graph -Type BiDirectionalGraph # PSQuickGraph
foreach ($e in $g.getAllEdges() )  { $g from Doug Fink's functions
    $vs= $e.startvertex
   $ve= $e.endvertex
    PSQuickGraph\Add-Edge -From $vs.value.Key -To $ve.value.Key -Graph $gv |Out-Null
}
Show-GraphLayout -Graph $gv

…  we can visualize the complicated relationships between applications through GraphViz’s interface:

Meet my application neighbors. A must-have idea for a Security Operations Center (SOC), no?

Where is this leading? There’s a graph-y way to spot threats! Instead of looking for specific text signatures as I did above with my show-threat-path cmdlet, we can try to find irregularities in a graph.

The idea is that you work out what are normal neighborhoods or sub-graphs — usually they appear entangled in a visualization — and then try to spot what appear to be less connected nodes. In fact your eyes are pretty good at doing this. Thankfully, there are also some simple algorithms that can discover neighborhoods and find intruders.

The advantage for threat hunters with this approach is that hackers can change their techniques and obfuscate their attacks, but they can’t easily hide their graph patterns.

Check this space next week for a continuation of this discussion. If you’re so motivated, you can do a search on graph anomalies cyber threats to get a sense of how researchers deal with this gnarly topic.

And once upon a time, yours truly wrote up a post about using one technique to work out neighborhoods in a social network associated with …  a famous Revolutionary War hero.

Andy Green

Andy Green

Andy blogs about data privacy and security regulations. He also loves writing about malware threats and what it means for IT security.

 

Does your cybersecurity start at the heart?

Get a highly customized data risk assessment run by engineers who are obsessed with data security.