multithreading - .net Task Scalability Problems -

i have application launches multiple tasks (1000+) , should scale 10k+ task area. these tasks launched gradually @ rate of ~100 per minute. each task downloading data piece piece, buffering it, , saving it.

i have 1 simple monitor loop (also task):

while 1     console.writeline("still running! tasks:" & task_count & " threads:" process.threadcount)     await task.delay(500) end while 

at ~3k tasks, runs fine. @ ~5k tasks, starts hitting fan. await task.delay(500) starts taking minutes execute. application thread count starts jumping 100 300 randomly. starts freezing , hanging. download speed goes 0. disk write speed goes 0. cpu usage goes 0, it's @ ~10%.

so solve issue, split application 5 separate process limited 1k tasks each. if need more, spin new process. communicate have central process communicate with, using thousands of wcf named pipes stay open hours if want.

but works! splitting multiple processes delivers better performance having many tasks in 1 process. haven't changed inside code tasks execute. split tasks multiple processes, , dandy.

isn't whole idea behind tasks abstract threads , make efficient concurrency easy? why seem scale poorly in single process had split in few?

because using bare tasks hard.

starting thousands of tasks @ same time not optimal solution. there better options on top of tpl plinq , tpl dataflow.

let's take dataflow example. can create block part of process. configure appropriate degree of parallelism , start posting items (you can bound size limit memory usage). instead of creating thousands of tasks limit amount of running tasks should reduce scheduling overhead , improve efficiency of resource usage.

here's example:

var block = new actionblock<item>(     item => processitem(item),     new executiondataflowoptions { maxdegreeofparallelism = 100 });  foreach (var item in getitems()) {     block.post(item); }  block.complete(); await block.completion; 

at end if processes running on same machine using same resources (i.e. cpu cores , networking stack). means improving efficiency of single process can achieve similar results.