Inside Out Security Blog   /  

Practical PowerShell for IT Security, Part IV:  Security Scripting Platform (SSP)

Practical PowerShell for IT Security, Part IV:  Security Scripting Platform (SSP)

Contents

    In the previous post in this series, I suggested that it may be possible to unify my separate scripts — one for event handling, the other for classification — into a single system. Dare I say it, a security platform based on pure PowerShell code?

    After I worked out a few details, mostly having to do with migraine-inducing PowerShell events, I was able to declare victory and register my patent for SSP, the Security Scripting Platform ©.

    Get the Free PowerShell and Active Directory Essentials Video Course

    I'd recommend this for both new and advanced PowerShell users. Building an AD tool is a great learning experience.

    United States of PowerShell

    While I’m having an unforgettable PowerShell adventure, I realize that a few of you may not be able to recall my recent scripting handiwork.

    Let’s review together.

    In the first post, I introduced the amazing one line of PowerShell that watched for file events and triggered a PS-style script block — that is, a piece of scripting code than runs in its own memory space.

    1. Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action
    Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action

    With a little more scripting sauce, I then cooked up what I called a File Access Analytics (FAA) app. Effectively, it tallies up the access event and displays some basic stats, and also detects bursts of access activity, which could indicate hacking behavior.

    It’s a simplified version of User Behavior Analytics (UBA) technology that we’re keen on here at Varonis.

    So far, so good.

    Then in the third post, I showed how relatively easy it is with PowerShell to scan and classify files in a folder. Since this is a disk-intensive activity, it makes incredible sense to use PowerShell’s multi-tasking capability, known as Runspaces to speed up the classification work.

    In the real-world of file event handling and data classification, say Varonis’s Data Classification Framework, a more optimized approach to categorizing file content is to feed file events into the classification engine.

    Why?

    Because then you don’t have to reclassify file content from scratch: you only look at the files that have changed. My classifier script would therefor benefit greatly by knowing something about file modification events.

    That’s the approach I took with SSP.

    Varonis’s own agents, which catch Linux or Windows file events, are finely tuned low-level code. For this kind of work, you want the code to be lean, mean, and completely focused on collecting events and to quickly pass this info to other apps that can do higher-level processing.

    So I took my original event handling script, streamlined it and removed all the code to display statistics. I then reworked the classifier to scan for specific files that have been modified.

    Basically, it’s a classic combination of a back-end engine coupled with a front-end interface.

    The question is how to connect the two scripts: how do I tell the classifier that there’s been a file event?

    Messages and Events

    After spending a few long afternoons scanning the dev forums, I eventually stumbled upon PowerShell’s Register-EngineEvent.

    What is this PowerShell cmdlet?

    In my mind, it’s a way to pass messages using a named event that you can share between scripts. It works a little bit differently from traditional system messaging and queues since the received message asynchronously triggers a PowerShell script block. This’ll become clearer below.

    In any case, register-EngineEvent has two faces. With the -forward parameter, it acts as a publisher.  And without the -forward ,parameter it takes on the role of a receiver.

    Got that?

    I used the event name Delta — technically the SourceIdentifer — to coordinate between my event handler script, which pushes out event messages, and my classifier script, which receives these message.

    In the first of two scripts snippets below, I show how I register the public Delta event name with -Register-EngineEvent -forward, and then wait for internal file access events. When one comes in, I then send the internal file event message — in PowerShell-speak, it’s forwarded — to the corresponding Register-EngineEventin the classifier script in the second snippet.

    1. Register-EngineEvent -SourceIdentifier Delta -Forward
    2. While ($true) {
    3. $args=Wait-Event -SourceIdentifier Access # wait on internal file event
    4. Remove-Event -SourceIdentifier Access
    5. if ($args.MessageData -eq "Access") {
    6. #do some plain access processing
    7. New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData #send event to classifier via forwarding
    8. }
    9. elseif ($args.MessageData -eq "Burst") {
    10. #do some burst processing
    11. New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData #send event to classifier via forwarding
    12. }
    13. }
    Register-EngineEvent -SourceIdentifier Delta -Forward
    While ($true) {
       $args=Wait-Event -SourceIdentifier Access  # wait on internal file event
        Remove-Event -SourceIdentifier Access
        if ($args.MessageData -eq "Access") {  
           #do some plain access processing 
           New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData  #send event to classifier via forwarding
         }
        elseif ($args.MessageData -eq "Burst") {
           #do some burst processing
           New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs  -MessageData $args.MessageData #send event to classifier via forwarding
         }
    }

    On the receiving side, I leave out the -forward parameter and instead pass in a PowerShell script bock, which asynchronously handles the event. You can see this below.

    1. Register-EngineEvent -SourceIdentifier Delta -Action {
    2. Remove-Event -SourceIdentifier Delta
    3. if($event.MessageData -eq "Access") {
    4. $filename = $args[0] #got file!
    5. Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load
    6. }
    7. elseif ($event.Messagedata -eq "Burst") {
    8. #do something
    9. }
    10. }
    Register-EngineEvent -SourceIdentifier Delta -Action {
        
          Remove-Event -SourceIdentifier Delta
          if($event.MessageData -eq "Access") {
            $filename = $args[0] #got file!
             Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load            
          }
          elseif ($event.Messagedata -eq "Burst") {
            #do something     
          }
    
    }

    Confused? And have I mentioned recently that file event handling is not easy, and that my toy scripts won’t stand up to business-level processing?

    This gets messy because the New-Event and Wait-Eventcmdlets for internal event messaging are different from the external event messaging provided by Register-EngineEvent.

    More Messiness

    The full classification script is presented below. I’ll talk a little more about it in the next and final post in this series. In the meantime, gaze upon it in all its event-handling and multi-tasking glory.

    1. Import-Module -Name .\pslock.psm1 -Verbose
    2. function updatecnts {
    3. Param (
    4. [parameter(position=1)]
    5. $match,
    6. [parameter(position=2)]
    7. $obj
    8. )
    9. for($j=0; $j -lt $match.Count;$j=$j+2) {
    10. switch -wildcard ($match[$j]) {
    11. 'Top*' { $obj| Add-Member -Force -type NoteProperty -Name Secret -Value $match[$j+1] }
    12. 'Sens*' { $obj| Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
    13. 'Numb*' { $obj| Add-Member -Force -type NoteProperty -Name Numbers -Value $match[$j+1] }
    14. }
    15. }
    16. return $obj
    17. }
    18. $scan = {
    19. $name=$args[0]
    20. function scan {
    21. Param (
    22. [parameter(position=1)]
    23. [string] $Name
    24. )
    25. $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
    26. $data = Get-Content $Name
    27. $cnts= @()
    28. if($data.Length -eq 0) { return $cnts}
    29. foreach ($key in $classify.Keys) {
    30. $m=$classify[$key].matches($data)
    31. if($m.Count -gt 0) {
    32. $cnts+= @($key,$m.Count)
    33. }
    34. }
    35. $cnts
    36. }
    37. scan $name
    38. }
    39. $outarray = @() #where I keep classification stats
    40. $deltafile = [hashtable]::Synchronized(@{}) #hold file events for master loop
    41. $list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"
    42. #long list --let's multithread
    43. #runspace
    44. $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
    45. $RunspacePool.Open()
    46. $Tasks = @()
    47. foreach ($item in $list) {
    48. $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
    49. $Task.RunspacePool = $RunspacePool
    50. $status= $Task.BeginInvoke()
    51. $Tasks += @($status,$Task,$item.Name)
    52. }
    53. Register-EngineEvent -SourceIdentifier Delta -Action {
    54. Remove-Event -SourceIdentifier Delta
    55. if($event.MessageData -eq "Access") {
    56. $filename = $args[0] #got file
    57. Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
    58. }
    59. elseif ($event.Messagedata -eq "Burst") {
    60. #do something
    61. }
    62. }
    63. while ($Tasks.isCompleted -contains $false){
    64. }
    65. #check results of tasks
    66. for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
    67. $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
    68. if ($match.Count -gt 0) { # update clasafication array
    69. $obj = New-Object System.Object
    70. $obj | Add-Member -type NoteProperty -Name File -Value $Tasks[$i+2]
    71. #defaults
    72. $obj| Add-Member -type NoteProperty -Name Secret -Value 0
    73. $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
    74. $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
    75. $obj=updatecnts $match $obj
    76. $outarray += $obj
    77. }
    78. $Tasks[$i+1].Dispose()
    79. }
    80. $outarray | Out-GridView -Title "Content Classification" #display
    81. #run event handler as a separate job
    82. Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1}) #run event handler in background
    83. while ($true) { #the master executive loop
    84. Start-Sleep -seconds 10
    85. Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
    86. foreach ($key in $deltafile.Keys) {
    87. $filename=$key
    88. if($deltafile[$key] -eq 0) { continue} #nothing new
    89. $deltafile[$key]=0
    90. $match = & $scan $filename #run scriptblock
    91. #incremental part
    92. $found=$false
    93. $class=$false
    94. if($match.Count -gt 0)
    95. {$class =$true} #found sensitive data
    96. if($outarray.File -contains $filename)
    97. {$found = $true} #already in the array
    98. if (!$found -and !$class){continue}
    99. #let's add/update
    100. if (!$found) {
    101. $obj = New-Object System.Object
    102. $obj | Add-Member -type NoteProperty -Name File -Value $Tasks[$i+2]
    103. #defaults
    104. $obj| Add-Member -type NoteProperty -Name Secret -Value 0
    105. $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
    106. $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
    107. $obj=updatecnts $match $obj
    108. }
    109. else {
    110. $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_}
    111. }
    112. $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"
    113. } #foreach
    114. } #lock
    115. }#while
    116. Write-Host "Done!"
    Import-Module -Name .\pslock.psm1 -Verbose
    function updatecnts {
    Param ( 
            [parameter(position=1)]  
            $match, 
            [parameter(position=2)]
            $obj
            )
    
    for($j=0; $j -lt $match.Count;$j=$j+2) {    
            switch -wildcard ($match[$j]) {
              'Top*'  { $obj| Add-Member -Force -type NoteProperty -Name Secret   -Value $match[$j+1] }
              'Sens*' { $obj|  Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
              'Numb*' { $obj|  Add-Member -Force -type NoteProperty -Name Numbers  -Value $match[$j+1] }           
             }
             
          }
    
      return $obj
    }
      
    $scan = {
    $name=$args[0]
    function scan {
       Param (
          [parameter(position=1)]
          [string] $Name
       )
          $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
         
          $data = Get-Content $Name
          
          $cnts= @()
          
          if($data.Length -eq 0) { return $cnts} 
          
          foreach ($key in $classify.Keys) {
           
            $m=$classify[$key].matches($data)           
               
            if($m.Count -gt 0) {
               $cnts+= @($key,$m.Count)  
            }
          }   
     $cnts   
    }
    scan $name
    }
    
     
    $outarray = @() #where I keep classification stats
    $deltafile = [hashtable]::Synchronized(@{})  #hold file events for master loop 
    
    $list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"  
    
    
    #long list --let's multithread
    
    #runspace
    $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
    $RunspacePool.Open()
    $Tasks = @()
    
    
    foreach ($item in $list) {
      
      $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
      $Task.RunspacePool = $RunspacePool
      
      $status= $Task.BeginInvoke()
      $Tasks += @($status,$Task,$item.Name)
    }
    
    
    Register-EngineEvent -SourceIdentifier Delta -Action {
        
          Remove-Event -SourceIdentifier Delta
          if($event.MessageData -eq "Access") {
            $filename = $args[0] #got file
             Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
          }
          elseif ($event.Messagedata -eq "Burst") {
            #do something
          }
    }
    
    while ($Tasks.isCompleted -contains $false){
      
    }
    
    #check results of tasks
    for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
       $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
       
      
       if ($match.Count -gt 0) {  # update clasafication array 
          $obj = New-Object System.Object
          $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
          #defaults
          $obj| Add-Member -type NoteProperty -Name Secret -Value 0
          $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
          $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
    
          $obj=updatecnts $match $obj
          $outarray += $obj
       } 
       $Tasks[$i+1].Dispose()
       
    }
    
    $outarray | Out-GridView -Title "Content Classification" #display
    
    #run event handler as a separate job
    Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1})  #run event handler in background
    
    
    while ($true) { #the master executive loop
       
       
          Start-Sleep -seconds 10
          Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
            foreach ($key in $deltafile.Keys) {  
        
            $filename=$key
           
            if($deltafile[$key] -eq 0) { continue} #nothing new
    
            $deltafile[$key]=0
            $match = & $scan $filename  #run scriptblock
                                            #incremental part
           
            $found=$false
            $class=$false
            if($match.Count -gt 0) 
                {$class =$true} #found sensitive data
            if($outarray.File -contains $filename) 
                    {$found = $true} #already in the array  
            if (!$found -and !$class){continue}
     
            #let's add/update
            if (!$found) {
    
                $obj = New-Object System.Object
                $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
                #defaults
                $obj| Add-Member -type NoteProperty -Name Secret -Value 0
                $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
                $obj| Add-Member -type NoteProperty -Name Numbers -Value 0
    
                $obj=updatecnts $match $obj
    
            }
            else {
                $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_} 
            }
            $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"   
            
           } #foreach
    
        } #lock
    }#while
    
    Write-Host "Done!"

    In short, the classifier does an initial sweep of the files in a folder, stores the classification results in $outarray, and then when a file modification event shows up, it updates and displays $outarray with new classification data. In other words, incremental scanning.

    There’s a small side issue of having to deal with updates to $outarray that can happen at any time while in another part of the classification script I’m actually looking to see what’s changed in this hashtable variable.

    It’s a classic race condition. And the way I chose to handle it is to use PowerShell’s synchronized variables.

    I’ll talk more about this mystifying PowerShell feature in the next post, and conclude with some words of advice on rolling-your-own solutions.

    What you should do now

    Below are three ways we can help you begin your journey to reducing data risk at your company:

    1. Schedule a demo session with us, where we can show you around, answer your questions, and help you see if Varonis is right for you.
    2. Download our free report and learn the risks associated with SaaS data exposure.
    3. Share this blog post with someone you know who'd enjoy reading it. Share it with them via email, LinkedIn, Twitter, Reddit, or Facebook.

    We're Varonis.

    We've been keeping the world's most valuable data out of enemy hands since 2005 with our market-leading data security platform.

    How it works