Varonis announces strategic partnership with Microsoft to accelerate the secure adoption of Copilot.

Learn more

Practical PowerShell for IT Security, Part IV:  Security Scripting Platform (SSP)

In the previous post in this series, I suggested that it may be possible to unify my separate scripts — one for event handling, the other for classification — into...
Michael Buckbee
10 min read
Published May 10, 2017
Last updated June 16, 2023

In the previous post in this series, I suggested that it may be possible to unify my separate scripts — one for event handling, the other for classification — into a single system. Dare I say it, a security platform based on pure PowerShell code?

After I worked out a few details, mostly having to do with migraine-inducing PowerShell events, I was able to declare victory and register my patent for SSP, the Security Scripting Platform ©.

Get the Free PowerShell and Active Directory Essentials Video Course

I'd recommend this for both new and advanced PowerShell users. Building an AD tool is a great learning experience.

United States of PowerShell

While I’m having an unforgettable PowerShell adventure, I realize that a few of you may not be able to recall my recent scripting handiwork.

Let’s review together.

In the first post, I introduced the amazing one line of PowerShell that watched for file events and triggered a PS-style script block — that is, a piece of scripting code than runs in its own memory space.

  1. Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action
Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action

With a little more scripting sauce, I then cooked up what I called a File Access Analytics (FAA) app. Effectively, it tallies up the access event and displays some basic stats, and also detects bursts of access activity, which could indicate hacking behavior.

It’s a simplified version of User Behavior Analytics (UBA) technology that we’re keen on here at Varonis.

So far, so good.

Then in the third post, I showed how relatively easy it is with PowerShell to scan and classify files in a folder. Since this is a disk-intensive activity, it makes incredible sense to use PowerShell’s multi-tasking capability, known as Runspaces to speed up the classification work.

In the real-world of file event handling and data classification, say Varonis’s Data Classification Framework, a more optimized approach to categorizing file content is to feed file events into the classification engine.

Why?

Because then you don’t have to reclassify file content from scratch: you only look at the files that have changed. My classifier script would therefor benefit greatly by knowing something about file modification events.

That’s the approach I took with SSP.

Varonis’s own agents, which catch Linux or Windows file events, are finely tuned low-level code. For this kind of work, you want the code to be lean, mean, and completely focused on collecting events and to quickly pass this info to other apps that can do higher-level processing.

So I took my original event handling script, streamlined it and removed all the code to display statistics. I then reworked the classifier to scan for specific files that have been modified.

Basically, it’s a classic combination of a back-end engine coupled with a front-end interface.

The question is how to connect the two scripts: how do I tell the classifier that there’s been a file event?

Messages and Events

After spending a few long afternoons scanning the dev forums, I eventually stumbled upon PowerShell’s Register-EngineEvent.

What is this PowerShell cmdlet?

In my mind, it’s a way to pass messages using a named event that you can share between scripts. It works a little bit differently from traditional system messaging and queues since the received message asynchronously triggers a PowerShell script block. This’ll become clearer below.

In any case, register-EngineEvent has two faces. With the -forward parameter, it acts as a publisher.  And without the -forward ,parameter it takes on the role of a receiver.

Got that?

I used the event name Delta — technically the SourceIdentifer — to coordinate between my event handler script, which pushes out event messages, and my classifier script, which receives these message.

In the first of two scripts snippets below, I show how I register the public Delta event name with -Register-EngineEvent -forward, and then wait for internal file access events. When one comes in, I then send the internal file event message — in PowerShell-speak, it’s forwarded — to the corresponding Register-EngineEventin the classifier script in the second snippet.

  1. Register-EngineEvent -SourceIdentifier Delta -Forward
  2. While ($true) {
  3. $args=Wait-Event -SourceIdentifier Access # wait on internal file event
  4. Remove-Event -SourceIdentifier Access
  5. if ($args.MessageData -eq "Access") {
  6. #do some plain access processing
  7. New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData #send event to classifier via forwarding
  8. }
  9. elseif ($args.MessageData -eq "Burst") {
  10. #do some burst processing
  11. New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData #send event to classifier via forwarding
  12. }
  13. }
Register-EngineEvent -SourceIdentifier Delta -Forward
While ($true) {
   $args=Wait-Event -SourceIdentifier Access  # wait on internal file event
    Remove-Event -SourceIdentifier Access
    if ($args.MessageData -eq "Access") {  
       #do some plain access processing 
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData  #send event to classifier via forwarding
     }
    elseif ($args.MessageData -eq "Burst") {
       #do some burst processing
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs  -MessageData $args.MessageData #send event to classifier via forwarding
     }
}

On the receiving side, I leave out the -forward parameter and instead pass in a PowerShell script bock, which asynchronously handles the event. You can see this below.

  1. Register-EngineEvent -SourceIdentifier Delta -Action {
  2.  
  3. Remove-Event -SourceIdentifier Delta
  4. if($event.MessageData -eq "Access") {
  5. $filename = $args[0] #got file!
  6. Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load
  7. }
  8. elseif ($event.Messagedata -eq "Burst") {
  9. #do something
  10. }
  11.  
  12. }
Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file!
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load            
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something     
      }

}

Confused? And have I mentioned recently that file event handling is not easy, and that my toy scripts won’t stand up to business-level processing?

This gets messy because the New-Event and Wait-Eventcmdlets for internal event messaging are different from the external event messaging provided by Register-EngineEvent.

More Messiness

The full classification script is presented below. I’ll talk a little more about it in the next and final post in this series. In the meantime, gaze upon it in all its event-handling and multi-tasking glory.

  1. Import-Module -Name .\pslock.psm1 -Verbose
  2. function updatecnts {
  3. Param (
  4. [parameter(position=1)]
  5. $match,
  6. [parameter(position=2)]
  7. $obj
  8. )
  9.  
  10. for($j=0; $j -lt $match.Count;$j=$j+2) {
  11. switch -wildcard ($match[$j]) {
  12. 'Top*' { $obj| Add-Member -Force -type NoteProperty -Name Secret -Value $match[$j+1] }
  13. 'Sens*' { $obj| Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
  14. 'Numb*' { $obj| Add-Member -Force -type NoteProperty -Name Numbers -Value $match[$j+1] }
  15. }
  16.  
  17. }
  18.  
  19. return $obj
  20. }
  21.  
  22. $scan = {
  23. $name=$args[0]
  24. function scan {
  25. Param (
  26. [parameter(position=1)]
  27. [string] $Name
  28. )
  29. $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
  30.  
  31. $data = Get-Content $Name
  32.  
  33. $cnts= @()
  34.  
  35. if($data.Length -eq 0) { return $cnts}
  36.  
  37. foreach ($key in $classify.Keys) {
  38.  
  39. $m=$classify[$key].matches($data)
  40.  
  41. if($m.Count -gt 0) {
  42. $cnts+= @($key,$m.Count)
  43. }
  44. }
  45. $cnts
  46. }
  47. scan $name
  48. }
  49.  
  50.  
  51. $outarray = @() #where I keep classification stats
  52. $deltafile = [hashtable]::Synchronized(@{}) #hold file events for master loop
  53.  
  54. $list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"
  55.  
  56.  
  57. #long list --let's multithread
  58.  
  59. #runspace
  60. $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
  61. $RunspacePool.Open()
  62. $Tasks = @()
  63.  
  64.  
  65. foreach ($item in $list) {
  66.  
  67. $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
  68. $Task.RunspacePool = $RunspacePool
  69.  
  70. $status= $Task.BeginInvoke()
  71. $Tasks += @($status,$Task,$item.Name)
  72. }
  73.  
  74.  
  75. Register-EngineEvent -SourceIdentifier Delta -Action {
  76.  
  77. Remove-Event -SourceIdentifier Delta
  78. if($event.MessageData -eq "Access") {
  79. $filename = $args[0] #got file
  80. Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
  81. }
  82. elseif ($event.Messagedata -eq "Burst") {
  83. #do something
  84. }
  85. }
  86.  
  87. while ($Tasks.isCompleted -contains $false){
  88.  
  89. }
  90.  
  91. #check results of tasks
  92. for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
  93. $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
  94.  
  95.  
  96. if ($match.Count -gt 0) { # update clasafication array
  97. $obj = New-Object System.Object
  98. $obj | Add-Member -type NoteProperty -Name File -Value $Tasks[$i+2]
  99. #defaults
  100. $obj| Add-Member -type NoteProperty -Name Secret -Value 0
  101. $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
  102. $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
  103.  
  104. $obj=updatecnts $match $obj
  105. $outarray += $obj
  106. }
  107. $Tasks[$i+1].Dispose()
  108.  
  109. }
  110.  
  111. $outarray | Out-GridView -Title "Content Classification" #display
  112.  
  113. #run event handler as a separate job
  114. Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1}) #run event handler in background
  115.  
  116.  
  117. while ($true) { #the master executive loop
  118.  
  119.  
  120. Start-Sleep -seconds 10
  121. Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
  122. foreach ($key in $deltafile.Keys) {
  123.  
  124. $filename=$key
  125.  
  126. if($deltafile[$key] -eq 0) { continue} #nothing new
  127.  
  128. $deltafile[$key]=0
  129. $match = & $scan $filename #run scriptblock
  130. #incremental part
  131.  
  132. $found=$false
  133. $class=$false
  134. if($match.Count -gt 0)
  135. {$class =$true} #found sensitive data
  136. if($outarray.File -contains $filename)
  137. {$found = $true} #already in the array
  138. if (!$found -and !$class){continue}
  139.  
  140. #let's add/update
  141. if (!$found) {
  142.  
  143. $obj = New-Object System.Object
  144. $obj | Add-Member -type NoteProperty -Name File -Value $Tasks[$i+2]
  145. #defaults
  146. $obj| Add-Member -type NoteProperty -Name Secret -Value 0
  147. $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
  148. $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
  149.  
  150. $obj=updatecnts $match $obj
  151.  
  152. }
  153. else {
  154. $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_}
  155. }
  156. $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"
  157.  
  158. } #foreach
  159.  
  160. } #lock
  161. }#while
  162.  
  163. Write-Host "Done!"
Import-Module -Name .\pslock.psm1 -Verbose
function updatecnts {
Param ( 
        [parameter(position=1)]  
        $match, 
        [parameter(position=2)]
        $obj
        )

for($j=0; $j -lt $match.Count;$j=$j+2) {    
        switch -wildcard ($match[$j]) {
          'Top*'  { $obj| Add-Member -Force -type NoteProperty -Name Secret   -Value $match[$j+1] }
          'Sens*' { $obj|  Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
          'Numb*' { $obj|  Add-Member -Force -type NoteProperty -Name Numbers  -Value $match[$j+1] }           
         }
         
      }

  return $obj
}
  
$scan = {
$name=$args[0]
function scan {
   Param (
      [parameter(position=1)]
      [string] $Name
   )
      $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
     
      $data = Get-Content $Name
      
      $cnts= @()
      
      if($data.Length -eq 0) { return $cnts} 
      
      foreach ($key in $classify.Keys) {
       
        $m=$classify[$key].matches($data)           
           
        if($m.Count -gt 0) {
           $cnts+= @($key,$m.Count)  
        }
      }   
 $cnts   
}
scan $name
}

 
$outarray = @() #where I keep classification stats
$deltafile = [hashtable]::Synchronized(@{})  #hold file events for master loop 

$list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"  


#long list --let's multithread

#runspace
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
$RunspacePool.Open()
$Tasks = @()


foreach ($item in $list) {
  
  $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
  $Task.RunspacePool = $RunspacePool
  
  $status= $Task.BeginInvoke()
  $Tasks += @($status,$Task,$item.Name)
}


Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something
      }
}

while ($Tasks.isCompleted -contains $false){
  
}

#check results of tasks
for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
   $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
   
  
   if ($match.Count -gt 0) {  # update clasafication array 
      $obj = New-Object System.Object
      $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
      #defaults
      $obj| Add-Member -type NoteProperty -Name Secret -Value 0
      $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
      $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

      $obj=updatecnts $match $obj
      $outarray += $obj
   } 
   $Tasks[$i+1].Dispose()
   
}

$outarray | Out-GridView -Title "Content Classification" #display

#run event handler as a separate job
Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1})  #run event handler in background


while ($true) { #the master executive loop
   
   
      Start-Sleep -seconds 10
      Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
        foreach ($key in $deltafile.Keys) {  
    
        $filename=$key
       
        if($deltafile[$key] -eq 0) { continue} #nothing new

        $deltafile[$key]=0
        $match = & $scan $filename  #run scriptblock
                                        #incremental part
       
        $found=$false
        $class=$false
        if($match.Count -gt 0) 
            {$class =$true} #found sensitive data
        if($outarray.File -contains $filename) 
                {$found = $true} #already in the array  
        if (!$found -and !$class){continue}
 
        #let's add/update
        if (!$found) {

            $obj = New-Object System.Object
            $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
            #defaults
            $obj| Add-Member -type NoteProperty -Name Secret -Value 0
            $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
            $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

            $obj=updatecnts $match $obj

        }
        else {
            $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_} 
        }
        $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"   
        
       } #foreach

    } #lock
}#while

Write-Host "Done!"

In short, the classifier does an initial sweep of the files in a folder, stores the classification results in $outarray, and then when a file modification event shows up, it updates and displays $outarray with new classification data. In other words, incremental scanning.

There’s a small side issue of having to deal with updates to $outarray that can happen at any time while in another part of the classification script I’m actually looking to see what’s changed in this hashtable variable.

It’s a classic race condition. And the way I chose to handle it is to use PowerShell’s synchronized variables.

I’ll talk more about this mystifying PowerShell feature in the next post, and conclude with some words of advice on rolling-your-own solutions.

What you should do now

Below are three ways we can help you begin your journey to reducing data risk at your company:

  1. Schedule a demo session with us, where we can show you around, answer your questions, and help you see if Varonis is right for you.
  2. Download our free report and learn the risks associated with SaaS data exposure.
  3. Share this blog post with someone you know who'd enjoy reading it. Share it with them via email, LinkedIn, Reddit, or Facebook.

Try Varonis free.

Get a detailed data risk report based on your company’s data.
Deploys in minutes.

Keep reading

Varonis tackles hundreds of use cases, making it the ultimate platform to stop data breaches and ensure compliance.

practical-powershell-for-it-security,-part-v: security-scripting-platform-gets-a-makeover
Practical PowerShell for IT Security, Part V: Security Scripting Platform Gets a Makeover
A few months ago, I began a mission to prove that PowerShell can be used as a security monitoring tool. I left off with this post, which had PowerShell code...
practical-powershell-for-it-security,-part-iii:-classification-on-a-budget
Practical PowerShell for IT Security, Part III: Classification on a Budget
Last time, with a few lines of PowerShell code, I launched an entire new software category, File Access Analytics (FAA). My 15-minutes of fame is almost over, but I was...
practical-powershell-for-it-security,-part-ii:-file-access-analytics-(faa)
Practical PowerShell for IT Security, Part II: File Access Analytics (FAA)
In working on this series, I almost feel that with PowerShell we have technology that somehow time-traveled back from the future. Remember on Star Trek – the original of course...
practical-powershell-for-it-security,-part-i:-file-event-monitoring
Practical PowerShell for IT Security, Part I: File Event Monitoring
Back when I was writing the ultimate penetration testing series to help humankind deal with hackers, I came across some interesting PowerShell cmdlets and techniques. I made the remarkable discovery...