MATLAB Answers

Why do I get "Out of memory." when reading only 16 chars?

7 views (last 30 days)
Ed Frank
Ed Frank on 25 Mar 2020
Answered: Guillaume on 29 Apr 2020
Dear Matlab community,
I want to read specific parts from a large (> 20 GB) binary file. However, the command
tmpString=fread(fid,[1,16],'char=>char');
fails with "Out of memory." The command is applied very near to the beginning of the file (offset is 20 bytes).
Why do I get this error and how can I successfully read in my file?
Thank you for your suggestions,
Ed

  17 Comments

Guillaume
Guillaume on 26 Mar 2020
"However what I could think of is that Matlab tries to guess the encoding"
No, as walter mentioned, as of R2020a, if you don't specify the encoding, it's UTF8. Prior to UTF8 it was whatever native encoding your system use. Matlab never tried to guess the encoding.
As walter said, it's very unlikely that your file uses UTF8 encoding for text (unless the text is prefixed by length information). It possibly doesn't use your native encoding either. When reading text from a binary file, it's always safer to read it as bytes (uint8) and then convert to the correct encoding with native2unicode.
Walter Roberson
Walter Roberson on 22 Apr 2020
It turns out that R2020a, fopen now tries to do encoding detection; https://www.mathworks.com/help/matlab/ref/fopen.html#btrnibn-1-encodingIn
Historically, encoding detection for text being read by readtable() used to examine the first 10 kilobytes of the file; matters might be different for fopen()
Ed Frank: if you are still interested, could you try starting by reading (say) one character, and timing the fopen() and the first short fread(), and then fread() of the next 79, to see whether the long time is at the fopen() or at the first fread() of character data, or if the position somehow triggers the delay ?
Guillaume
Guillaume on 29 Apr 2020
Sorry, I've been a bit too busy to follow answers recently but indeed I had conversations with Mathworks recently on text file parsing and indeed 2020a does automatic character set detection which is most likely the issue here. I'll post the details I've got from mathworks support in an answer.

Sign in to comment.

Accepted Answer

Guillaume
Guillaume on 29 Apr 2020
"However what I could think of is that Matlab tries to guess the encoding"
I've had discussions with Mathworks support about this. The whole process is not properly documented unfortunately, which I told them can be a problem.
Indeed, if you open a file without specifying a character encoding, matlab will try to guess the file encoding the first time you either:
  • use any character reading function such as fgetl, fgets, fscanf, etc.
  • use fread with a 'char' or '*char' precision
  • ask for the encoding with the multi-output version of fopen.
I haven't been given the full process of character set detection, but it does read the whole file which indeed can be an issue for large files. If any byte sequence in the file is not a valid UTF8 code point, then the algorithm uses some heuristics to see if it's a CJK encoding and if it still doesn't match, it assumes the local encoding.
To prevent this autodetection to take place, you have to specify an encoding when you fopen the file. If you don't know what the encoding is for your binary file, I'd suggest using 'US-ASCII'. As we mentioned in the comment, it's unlikely that a binary file uses UTF8 unless it prefixes the text by a length.
Unfortunately, it's not easy to go back to pre-2020a behaviour of automatically using the native encoding whatever it is, as R2020a has lost the ability of easily getting the local encoding. On the other hand, relying on native encoding when reading a binary file is asking for trouble.

  0 Comments

Sign in to comment.

More Answers (0)

Products


Translated by