why is Matlab slow in for loop with large number of iterations but fast with a small number of iterations??

Hi,

I am running a function to extract some information from 100,000+ patient xray dicom files. the files are stored within a veracrypt encryption container for security purposes.

when i run the function on a small sample of files it performs really quickly, however when i run the function on the entire dataset it is very slow in comparison, going from several files per second to 1 file per second (approximately).

i was wandering why this is happening? i have tried storing the data on an ssd and on a normal hard drive and get the same sort of slow down when using a larger dataset compared to a small one.

i have added the code below for reference but haven't commented it fully yet.. this is for my thesis so i will do it once i get the extraction finished..

thanks for any help.

function [ DB, corrupted_files ] = extract_from_dcm( folder_name )
%EXTRACT_FROM_DCM Summary of this function goes here
%   Detailed explanation goes here

if nargin == 0
  folder_name = 'G:\medium2';
end

Database_Check = strcat(folder_name, '\DataBase.mat');

if exist(Database_Check, 'file')

    load(Database_Check);
    entry_start = length(DB) + 1;

else

    entry_start = 1;
    [ found_dicoms ] = recursive_search( folder_name );

end

mat_file_location = strcat(folder_name, '\DataBase.mat');
excel_DB_file = strcat(folder_name, '\DataBase.xlsx');
excel_Corrupted_file = strcat(folder_name, '\Corrupted_Files.xlsx');

% the recursive search creates a struct with the path for each
% dcm file found. the list is then recursivly used to locate
% the image and extract the relevant information from it.

fprintf('---------------------------------------------\n');
fprintf('Start Patient Data Extraction\n');
tic
h = waitbar(0,'','Name','Patient Data Extraction');
entry_end = length(found_dicoms);

if entry_end == 0

    % error message goes here.

else

    for  kk = entry_start : entry_end
        progress = kk/entry_end;
        progress_percent = round(progress * 100);
        waitbar(progress,h, sprintf('%d%%  %d/%d of images processed', progress_percent, kk,   entry_end));

        img_full_path = found_dicoms(kk).name;

        %         search_path = folder_name;
        %         img_full_path = strrep(img_full_path, search_path, '');

        try              %# Attempt to perform some computation
            dicom_info = dicominfo(img_full_path); %# The operation you are trying to perform goes here
        catch exception  %# Catch the exception
            continue       %# Pass control to the next loop iteration
        end

        try              %# Attempt to perform some computation
            dicom_read = dicomread(dicom_info); %# The operation you are trying to perform goes here
        catch exception  %# Catch the exception
            continue       %# Pass control to the next loop iteration
        end

        old = dicominfo(img_full_path);

        DB(kk).StudyDate = old.StudyDate;
        DB(kk).StudyTime = old.StudyTime;

        if isfield(old.PatientName, 'FamilyName')
            DB(kk).Forename = old.PatientName.FamilyName;
        else
            DB(kk).Forename = 'NA';
        end

        if isfield(old.PatientName, 'GivenName')
            DB(kk).LastName = old.PatientName.GivenName;
        else
            DB(kk).LastName = 'NA';
        end

        if isfield(old, 'PatientSex')
            DB(kk).PatientSex = old.PatientSex;
        else
            DB(kk).PatientSex = 'NA';
        end

        if isempty(old.PatientBirthDate)
            DB(kk).PatientBirthDate = '00000000';
        else
            DB(kk).PatientBirthDate = old.PatientBirthDate;
        end

        if strcmp(old.Manufacturer, 'Philips Medical Systems')
            DB(kk).Van = '1';
        else
            DB(kk).Van = '0';% section to represent organising by different vans
        end

        DB(kk).img_Path = img_full_path;

        save(mat_file_location,'DB','found_dicoms');

    end

end

[ corrupted_files, DB ] = corruption_check( DB, found_dicoms, folder_name );

toc
fprintf('End Patient Data Extraction\n');
fprintf('---------------------------------------------\n');

fprintf('---------------------------------------------\n');
fprintf('Start Saving Extracted Data \n');
tic

save(mat_file_location,'DB','corrupted_files','found_dicoms');

if isempty(DB)
  msg = sprintf('No Dicom Files Found');
  msgbox(strcat(msg));
else
  DB_table = struct2table(DB);
  writetable(DB_table, excel_DB_file);
end

if corrupted_files.num_corrupted > 0 
  Corrupted_Files = struct2table(corrupted_files);
  writetable(Corrupted_Files, excel_Corrupted_file);
else
  msg = sprintf('No Corrupted Files Found');
  msgbox(strcat(msg));
end

close(h);

toc
fprintf('End Saving Extracted Data \n');
fprintf('---------------------------------------------\n');

end

Answer 1

[bilalY]

My problem was the saving at the end of each iteration but the biggest problem was the line where i run the dicomread function. i changed the saving to occur for every 20 images processed.

I also preallocated DB

... i just need to find a solution for dicomread (which i was using as a way to check if the file was corrupt or not).