findchangepts use for multiple files

Hi,

how do I use findchangepts to compare each line of a file to the correspondig line in another file? I have 50k files with 50k lines, each file is for one timestep.

ipt = findchangepts(x, 'MaxNumChanges',10,'Statistic','rms') compares one line in a file with the next line. Any suggestions how to implement this?

Answer 1

[Brahmadev]

Hi @Daniel Stein,

For comparing each line of a file to the corresponding line in another file for 50,000 files with 50,000 lines each, you are essentially looking to perform a pairwise comparison between two sequences at each timestep. Since "findchangepts" isn't designed for this purpose, you would need to implement a custom comparison.

Refer to the psuedocode below for an example on how this can be achieved:

% Assuming the files are named in a systematic way that allows you to match them
for t = 1:50000
    % Construct file names for the current timestep
    file1 = sprintf('path_to_files/timestep_%d_file1.txt', t);
    file2 = sprintf('path_to_files/timestep_%d_file2.txt', t);
    
    % Open the files
    fid1 = fopen(file1, 'r');
    fid2 = fopen(file2, 'r');
    
    % Initialize an array to hold RMS values for this timestep
    rms_values = zeros(50000, 1);
    
    % Read and compare each line
    for line = 1:50000
        % Read lines from both files
        line1 = fgetl(fid1);
        line2 = fgetl(fid2);
        
        % Convert lines to numerical arrays if necessary
        % This step depends on the format of your data
        data1 = str2num(line1); % Example conversion
        data2 = str2num(line2); % Example conversion
        
        % Calculate the RMS of the difference
        rms_values(line) = sqrt(mean((data1 - data2).^2));
    end
    
    % Close the files
    fclose(fid1);
    fclose(fid2);

end

Further the RMS values can be analysed for significant changes as required to do similar analysis to "findchangepts".

Hope this helps!