reading a file name from a list of files in a text file line by line and open the files one at a time

I have a list of files in a summary text file. Each line is a name with no spaces. I want to read the files in, one at a time, open one, get input from that file, then move on to the next. I have tried fgetl, fgets, and textscan. fgetl returns the correct file name, but fed into an fopen file, it will no open the file. fgets returns the name with a carraige return (not what I want) textscan returns all of the summary text file names as a single input. All of the files exist in the same directory as the summary textfile. Examples are

Sumary test file:

inputfile1

inputfile2

inputfile50

Each file has the same number of ascii text lines.

fid1=fopen('Summary test file')

for 1;1:3

aline=fgetl(fid1)

aline % prints out correct name as text with no spaces)

fid2=fopen(aline) % this wont open, gives fid= -1

end

How do I do this?

[Sudhakar Shinde]

Check for formatspec with textscan. If you could add example text file here, would be more possibly to get answer.

[richg]

Here are versions of these files with .txt added to their name so I could attach them. The filesin.txt is actually filesin. The dumpH_0gpa_100K.10100.custom.txt is actually dumpH_0gpa_100K.10100.custom. The earlier program could red the dumpH_0gpa_100K.custom file as input when I created a text variable with the file name in the program. Read the same file name in as a line in filesin, and it wont open. The outpue from the Matlab run though shows aline='dumpH_0gpa_100K.10100.custom' as you would expect.

[richg]

It added an extra .txt on the file when I attached it. This response panel is awkward and frustrating, I think.

[richg]

More detals. The file I attached is only a test example. The actual list of file names is 12505 names long, one name on each line. The file names have no spaces, but vary in length, so I am not sure what to do with a forma specifier. Specifically the file names are like this: first 2501 lines, dumpH_0gpa_100K.0.custom to dumpH_0gpa_100K.50000.custom, with the number between the dots incremented every 20 steps. (e.g. after - 0.custom, next is dumpH_0gpa_100K.20.custom, and so on.Each file listed has the same format, but different numerical values.

[Stephen23]

"The filesin.txt is actually filesin. The dumpH_0gpa_100K.10100.custom.txt is actually dumpH_0gpa_100K.10100.custom."

"It added an extra .txt on the file when I attached it. This response panel is awkward and frustrating, I think."

Is it possible that you are using Windows with the "Hide extensions for known file types" option selected (which unfortunately is the default):

This matters to your question, because the filenames listed in filesin.txt do not have extensions, so we need to know if the extension needs to be added or not. In any case, it is easy for us to find out, so lets take a look at the actual filenames (not just what Windows File Explorer shows you, which you should never trust anyway). Make sure that your current directory is the folder where those files are saved, and then run this command:

dir()

and show us the exact output (if there are lots of lines, only the first few please).

I also recommend that you deselect the awful "Hide extensions for known file types" option.

[Stephen23]

It is also worth noting that the files listed in filesin.txt do not match the name of the data file that you uploaded:

 dump_0gpa_100K.10100.custom.txt % filename of uploaded file
dumpH_0gpa_100K.10100.custom.txt % filename listed in FILESIN.TXT

Even disregarding the file-extension confusion (see previous comment) these are clearly different filenames, and what you have explained in your comments thus implies that none of the actual filenames match with the name listed in FILESIN.TXT.

As I wrote in my last comment, showing us the output of

dir()

would answer a lot of these questions (and mean that we would actually be able to help you).

[richg]

here is the dir list

10/28/2020 01:51 AM <DIR> .

10/28/2020 01:51 AM <DIR> ..

10/28/2020 01:51 AM 0 dir_list

10/27/2020 10:12 PM 52,544 dumpH_0.back

10/26/2020 11:24 PM 52,544 dumpH_0gpa_100K.10100.custom

10/26/2020 11:24 PM 20,910 dumpH_0gpa_100K.10100.custom.gz

10/26/2020 11:24 PM 52,564 dumpH_0gpa_100K.10120.custom

10/26/2020 11:24 PM 52,574 dumpH_0gpa_100K.10140.custom

10/26/2020 11:24 PM 52,528 dumpH_0gpa_100K.10160.custom

10/26/2020 11:24 PM 52,531 dumpH_0gpa_100K.10180.custom

10/27/2020 10:15 PM 20,910 dumpH_0gpa_100K.custom.zip

10/27/2020 10:18 PM 52,544 dump_0gpa_100K.10100.custom.txt.txt

10/27/2020 07:37 PM 145 filesin

10/27/2020 10:19 PM 145 filesin.txt

10/27/2020 07:09 PM 95 test_text1

13 File(s) 410,034 bytes

2 Dir(s) 1,346,096,627,712 bytes free

and here is the filesin file:

dumpH_0gpa_100K.10100.custom

dumpH_0gpa_100K.10120.custom

dumpH_0gpa_100K.10140.custom

dumpH_0gpa_100K.10160.custom

dumpH_0gpa_100K.10180.custom

[richg]

The hidden windoew extension option was on. That is why there are two .txt endings. I turned it off. Thanks.

Answer 1

[Mathieu NOE]

hi

you can try this - I tested it on dummys file. just found that my matlab is pretty slow for such simple tasks....

% read files listed in  filesin.txt file
fileID = fopen('filesin.txt');
C = textscan(fileID,'%s');
fclose(fileID);
celldisp(C)

numfiles = numel(C{1});
mydata = cell(1, numfiles);

for k = 1:numfiles
  myfilename = char(C{1}{k});
  mydata{k} = importdata(myfilename);
end

celldisp(mydata)

[richg]

I hae done this before. it works, but you have fo store the entire list file as a single array ot cells, then run through them (there are 12505 rows in the array. I will try it though and see if ti solves the problem.

[richg]

I tried your solution. It did not work. Same pattern. It can read the main file, However, whenI put the file into a loop, and triy to open the first name (myfilename in your code), it says

Error using importdata (line 139)

Unable to open file.

Error in text_input1 (line 15)

mydata{i} = importdata(myfilename);

(I am using i instaead of k as my loop counter).

How did you generate your filesin.txt? Was it a straight export of my attachment, did you type it line-by-line?

I generated the text file it "filesin.txt" using Linux UBUNTU running on Windows 10, with Matlab on Windows 10 OS. LINUX UBNUNTU here is a special add-in to Microsoft Windows 10 that runs bash in the command prompt. It is from Microsoft. LINUX normally uses a slightly different format than Windows on text files (differient ending). I need to try to figure out if that is the problem. Also, my file that it is reading does not have the .txt ending, but ends after .custom. I had to add the .txt to the file so I could attach it to the MATALB answers GUI. The attachment method in the MATLAB answers GUI states you cannot add files, excecpt certain specific types, even it the file is just an ASCII text file, It refussed the attacment without the correct suffix. It was my attempts to try to get something to post that led me to try saviing as txt, .gz, and .zip file types, which is why you see those in my dir() listing. That limit with the GUI was what I was so frustrated with in my earlier post.

There is a tool DOS2UNIX and UNIX2DOS that will convert bakc and forth. I am going ro try that tool and see if the file lengths are different. I am also going to look for wat to diiplay hidden characters.

How did you generate you filesin.txt? Was it a straight export of my attachment, did you type it line-by-line?

[Mathieu NOE]

hi

I simply used your "filesin.txt" as you provided

as explained I created from scratch some dummy data inside every "... .custom" file - but I don't pretend your file could be open as I did, so if you coud send some real "... .custom" files for testing purpose, we can further work your issues.

all the best

[richg]

I found mu problem. The problem was the filein file name included he full directory path. The line that it read in did not., it was simple the file name. I added the full directory path to the name read in form the filesin file name and concatenated them.

like this:

d1="full_path/"

for i=1:5

aline=fgetl(filesin)

fileinFull=[d1,ailine]

fid1=fopen(fileinFull)

% rest of processing of fid1 .eg. aline=fgetl(fid1)

fclose(fid1)

end

You couldn't see this because you were processing the example on another machine.

Thanks for your help.

[Stephen23]

Rather than using string concatenation, the recommended (and correct tool for the task) is to use fullfile:

fileinFull = fullfile(d1,ailine)

[richg]

Thanks - I see it. It allows me to move form Windows to LINUX and back. I do that a lot while writing and debugging.