PowerShell Workflow Foreach Parallel limited to 5 parallel threads

Some time ago Aidan Finn, one of my Hyper-V heroes, contacted me with a PowerShell question. Until that time I was able to help him quickly with every question he has thrown at me the last year… but he finally had one that was a real braincracker

ProblemPowerShell v3 has introduced Workflow which is based on the Windows Workflow Foundation.

One way you can use this is by creating a workflow there the -parallel parameter can be used with the ForEach statement, for example:

Workflow New-VMParallel
{
  Param ([parameter(Mandatory=$true)][String[]] $VMList)
  ForEach -Parallel ($VM in $VMList) 
  {
    New-VM –Name $VM –MemoryStartupBytes 512MB
  } 
}

Now this can be used very easy, for example by creating a $VMList array with a bunch of numbers from 1 to 200:

$VMList = 1..200

So Aidan had noticed, and why he contacted me, was that is a maximum of 5 parallel threads!
No matter what we did, we were not able to exceed this maximum when using Workflow.
Trust me when I say I had gotten rather frustrated about this!

My first try in finding a workaround was rather simple… it was actually one of my older scripts with a small change:

Workflow New-VMParallel
{
  Param ([parameter(Mandatory=$true)][String[]] $VMList)
  $first = $VMList[0]..$VMList[4]
  $second = $VMList[5]..$VMList[9]
  $third = $VMList[10]..$VMList[14]
  $fourth = $VMList[15]..$VMList[19]
  $fifth = $VMList[20]..$VMList[24]
  parallel
  {
    if ($first) {ForEach -Parallel ($VM in $First) {New-VM –Name VM$VM –MemoryStartupBytes 512MB} }
    if ($second) {ForEach -Parallel ($VM in $second) {New-VM –Name VM$VM –MemoryStartupBytes 512MB} }
    if ($third) {ForEach -Parallel ($VM in $third) {New-VM –Name VM$VM –MemoryStartupBytes 512MB} }
    if ($fourth) {ForEach -Parallel ($VM in $fourth) {New-VM –Name VM$VM –MemoryStartupBytes 512MB} }
    if ($fifth) {ForEach -Parallel ($VM in $fifth) {New-VM –Name VM$VM –MemoryStartupBytes 512MB} }
  } 
}

So in theory the code above would execute all five tasks  in parallel when each of those tasks would also execute its five tasks parallel. So 5*5=25 tasks in parallel… nasty, but if it had worked it would have been a quick-n-dirty workaround. No such luck… I experienced the same issue, a maximum of 5 parallel threads

Workflow is an great technology and the easy of which it can be used with the ForEach statement is just amazing… so a non-customizable limit of a maximum parallel threads doesn’t sound logical to me.
The task itself doesn’t matter but in the example given above it should only be limited by your environment (disks, CPU, memory, network, etc.) and not by the code.
I’ve also tried it with a copy task of 50 1GB files… same thing, maximum of 5 parallel threads. So the cause wasn’t in Hyper-V.

Until that time I was able to do everything with PowerShell, solve any problem and always find a solution or workaround… it may have taken me days to do it (since also I am still learning) but I was always able to find a way… and something as simple as this would break my stride? Like hell it will!
* Yes, frustration can be a strong motivator… 

So, when the limit seems to be inside Workflow itself, the logical next step would be not to use Workflow.

So I’m very happy with my workaround (yes, I see it as a workaround… it should be ‘fixed’ in Workflow… or at least I think that scripters have to be given the option to customize the maximum parallel threads with a parameter or whatever). Writing the workaround was also a great learning experience for me personally ’cause I’m using PowerShell in ways that I’m not used to (i.e. the BEGIN-PROCESS-END) but are very powerfull nevertheless.
Simply put:, I’ve written a function, Foreach-Parallel.

<#
.Synopsis
   This function can be used to execute tasks in parallel.
.DESCRIPTION
   This function can be used to execute tasks in parallel with more than 5 parallel tasks at once.
   The number of parallel tasks can be defined by parameter. The input is accepted by defining it by using
   the -InputObject parameter which also accepts input from the pipeline.
.EXAMPLE
   Get-ChildItem -Path D:\Files | ForEach-Parallel -MaxThreads 100 -ScriptBlock {Copy-Item -Path $_.FullName -Destination E:\Company\Files}
.EXAMPLE
   1..500 | Foreach-Parallel -MaxThreads 20 -ScriptBlock {New-VM –Name VM$_ –MemoryStartupBytes 512MB}
.EXAMPLE
   ForEach-Parallel -InputObject (Get-ChildItem -Path "D:\Files") -MaxThreads 100 -ScriptBlock {Copy-Item -Path $_.FullName -Destination E:\Company\Files}
#>

function ForEach-Parallel
{
  param (
    [Parameter(Mandatory=$true,ValueFromPipeline=$true)][Alias("Input")][ValidateNotNullOrEmpty()][PSObject]$InputObject,
    [Parameter(Mandatory=$false)][Alias("Threads")][ValidateNotNullOrEmpty()][int]$MaxThreads=5,
    [Parameter(Mandatory=$true,Position=0)][Alias("Script")][ValidateNotNullOrEmpty()][System.Management.Automation.ScriptBlock]$ScriptBlock
  )
  BEGIN 
  {
    $InitialSessionState = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
    $RunspacePool = [Runspacefactory]::CreateRunspacePool(1, $MaxThreads, $InitialSessionState, $Host)
    $RunspacePool.Open()
    $Threads = @()
    $ScriptBlock = $ExecutionContext.InvokeCommand.NewScriptBlock("param(`$_)`r`n" + $Scriptblock.ToString())
  }
  PROCESS
  {
    $PowerShell = ::Create().AddScript($ScriptBlock).AddArgument($InputObject)
    $PowerShell.RunspacePool=$RunspacePool
    $Threads+= @{instance = $PowerShell;handle = $PowerShell.BeginInvoke()}
  }
  END
  {
    $Running = $true
    while ($Running)
    {
      $Running = $false
      for ($i=0; $i -lt $Threads.Count; $i++)
      {
        $Thread = $Threads[$i]
        if ($Thread.handle.iscompleted) 
        {
          $Thread.instance.endinvoke($Thread.handle)
          $Thread.instance.dispose()
          $Thread[$i] = $null
        }
        else 
        {
          $Running = $true
        }
      }
    }
  }
}

Note: As it seems the plugin I use on my blog for posting code doesn’t like it when I use a spcific code… place [] with powershell in between in front of ::Create().AddScript($ScriptBlock).AddArgument($InputObject) and it will work

Please don’t get me the wrong way… I encourage you to start or remain using Workfow; it is still a brilliant piece of technology

And this blogpost is why I love helping the community… people get enthousiastic about PowerShell, they encounter issues and contact me asking for information or help… and every once in a while something like this comes along

The funny thing however is that I don’t think that I would have been able to write this script if the community itself wasn’t here… The TechNet website was a great source of information for me and also a lot, and i do mean a log, of blogs I found by using Google helped me out big time Also the RunspacePool was new to me, more about that in a post that I’ll write in the future.