Star-P's data parallel mode allows the user to run familiar M commands on very large datasets. One of the simplest, most widely use commands available to M users is sort. Star-P users have a parallel sort right at their fingertips. Lets take it for a spin. These results are on a shared memory Opteron system with 12 cores running Star-P.
We create a vector with a billion random elements and sort it. This vector uses 8 GB of memory.
>> a = rand(1e9*p, 1);
>> tic; b = sort(a); toc
Elapsed time is 40.712581 seconds.
Often, its not the sorted elements one wants, but the permutation that sorts the input. This takes a little longer, but is equally easily achieved with:
>> tic; [b, perm] = sort(a); toc
Elapsed time is 217.236630 seconds.
We performed some scalability tests on a couple of large systems. The following graph is results from sorting 100 billion elements on an Altix with 256 processors. This vector takes up almost 1TB of memory. Thats right. A TERABYTE. Star-P sorts it in a few minutes.
We do not use shared memory systems exclusively for our scalability testing. The next graph shows results from a cluster with Gigabit Ethernet. Note, how the yellow line that shows communication time is higher than in the previous graph. On the Altix, communication is irrelevant. On a cluster, it is important to minimize communication. That is why we went to great lengths to minimize the amount of communication, which is almost optimal.
If you are curious, you can see our paper on parallel sorting, or download the psort code (free for non-commercial, personal and research use) from which the Star-P sort is derived.