I am working with some large data sets ( N rows of data with 1 parameter varied, each consisting of M points) for which it is assumed that there exists a function that is able to accurately describe each of these rows of data. This function consists of P fit parameters and the one that I vary.
Now, M is a very large number and I cannot afford to use my fitting routine on all N rows of data. Fortunately, my fitting function can be integrated, such that I can instead consider the much smaller single-row data set consisting of just N points.
Getting a nice fit through the integrated quantities goes fast and gives me physically realistic values for my P fit parameters. However, when I then plug in the fit parameters in my original function to compare it to one of the N rows of M points, the result can be way off...
So what I now want to do is make a routine where I consider e.g. 2 out of my N rows, as well as the integrated data for my fitting routine. I tried to simply concatenate everything, but the values and numbers of points may differ significantly and in the end I get similar results as when I consider just a single row of M points at the cost of a slower routine.
How can I realize this combined fitting routine and make everything equally important, independent of the big difference in N and M?