What to do when you really ARE out of memory?

What is the solution for optimizing code when you really are just working with too large of a dataset?

Currently I need to perform triscatteredinterp using 3 vectors all (100,000,000 x 1).

scatteredInterpolant does not work any better in this instance.

Answer 1

[the cyclist]

For very large datasets, processing a random sample of the data will often give satisfactory results.

Answer 2

[Walter Roberson]

Store the data in hierarchies such as octrees that allow you to extract a subset that fits within working memory to do the fine-grained work on.

Answer 3

[Robert Jenkins]

The solution I used in the end was to break up each of my vectors into manageable chunks and execute TriScatteredInterp on each of them.

The interpolant is valid on the new grid but only gives solutions inside of the block from which it came. I then simply stitched these back together.

Thank you both for your input!