It would be interesting to know how another strategy performs here. You could first read the CSV data into a list that supports efficient splitting (knows its size, supports random access. e.g. ArrayList) and then stream process this list. The workload would be split evenely in as many chunks as you have CPUs. So CPU utilization should be perfect.
The downside is that processing can only start after the I/O operation is complete. The upside is that you don't have to figure out the optimal perfect batch size for every task.
It would be interesting to know how another strategy performs here. You could first read the CSV data into a list that supports efficient splitting (knows its size, supports random access. e.g. ArrayList) and then stream process this list. The workload would be split evenely in as many chunks as you have CPUs. So CPU utilization should be perfect.
The downside is that processing can only start after the I/O operation is complete. The upside is that you don't have to figure out the optimal perfect batch size for every task.
Arandur 3 points · Jan 2017
The other downside is that if your data is truly massive, you'll run out of heap space completely.
negromanusinc 1 points · Jan 2017
Very true. Thanks for pointing that out.
It's a bit sad that the stream API is only intended and really useful for processing data that is already in memory.
I haven't quite studied reactive streams yet. Maybe the upcoming Java 9 Flow API or RxJava are better suited.
http://download.java.net/java/jdk9/docs/api/java/util/concurrent/Flow.html
https://github.com/ReactiveX/RxJava