Possible approaches to sort MapReduce reducer inputs by key either before or after reducer execution?

Thank you to the MathWorks developers for implementing a version of the MapReduce technique. I am trying to use it as an alternative to the Hadoop implementation but have noticed the difference that there is no "shuffle and sort" step between the mapper and the reducer. As a result, the reducer method is not called on its input keys in sorted order, and thus the order in which the reducer outputs are written to the output datastore does not necessarily match the order in which they would be written in the Hadoop implementation.

I'm trying to figure out a way to account for this difference and obtain output that exactly matches what Hadoop would return.

Could you please let me know whether any or all of them could work and how to implement them if so?

1. Intervene in the MapReduce algorithm to introduce a sorting step of the reducer outputs before the output datastore is written to disk.
2. Sort the output datastore, whether before or after it is written to disk. I could envision using the approach outlined here, but would this be Big Data "compatible," either by default or via modifications, i.e. would I need to create a tall array at some point and sort that?
3. Intervene in the MapReduce algorithm to introduce a "shuffle and sort" step between the mapper and the reducer.

Thanks again,

Dmitri

Answer 1

[Ayush]

Hi Dmitri,

To match Hadoop’s output order, load the output datastore as a tall array in MATLAB and use sortrows to sort by key. This approach is fully compatible with big data workflows.

If you need Hadoop’s exact behavior, you can also deploy your MATLAB code to a Hadoop cluster as described in the following document, ensuring each worker has access to the MATLAB Runtime (MCR) and all environment variables are set:

https://www.mathworks.com/help/releases/r2025b/matlab-parallel-server/configure-a-hadoop-cluster.html

Hope this helps!