Adaptive batching of streams to enhance throughput and to support dynamic load balancing