How to read\open a csv file with millions of rows and hundreds of coloumns to compare/delete and save

Hi

i have csv files with millions of rows and hundreds of coloumns that i want to open\read in order to compare the files, remove duplicates and save the new file as csv also, and many other modifications..

when i used csvreader the PC stuck! so any help here

[Image Analyst]

How long did you wait for it? How did you create it in the first place? Does it need to be a csv file? Maybe you can just save it as a mat file. Or maybe try textscan() or fread().

[Shayma]

sorry i used "importadat" for the files (0.5-3 GB) because its numeric+string file , i waited for about two hours, the files have been download from "Zinc db", and yes it has to stay in its format a csv file

[Guillaume]

Text formats such as csv are very good for humans to read but are very slow for computers to read.

If you want speed, you need to use a binary (which will also use less space).

Saying that, it only took a few minutes for my machine to read a csv files with 5 million rows and 100 columns (~4 GB file), using csvread.

[Shayma]

How can i do it ?

[George]

Can you post a sample of a CSV file?

Can you use datastore to import the data?

[Shayma]

This is an example of my file ... i have 16 like that which i'm trying to combine to one, but no success till now . i tried to use datastore with one file and i got an error message Undefined function 'tabularTextDatastore' for input arguments of type 'char'.

so i don't know now how to proceed with that

[dbmn]

Which Matlab Version are you using? 'tabularTextDatastore' was introduced in R2014b...

Oh and you might be interested in this Matlab webinar to get the concepts of reading chunks of a huge dataset.

http://de.mathworks.com/videos/tackling-big-data-with-matlab-97711.html?s_tid=srchtitle

[Shayma]

o.k it have been a long time now, but i got the R2014b,i tried to use datastore and it works, at least it opens the first chunk thank you :)

[Shayma]

How long it suppose to take reading 100000 lines each chunk from 13.5 GB files ?

Answer 1

[George]

You can do this with textscan, but your formatSpec is going to be pretty gnarly.

fid = fopen('data.csv');
% your formatSpec will be very long because of the number of fields
formatSpec = '%s %s %f %d';  % reads a string, a string, a float, an integer
A = textscan(fid, formatSpec, 'HeaderLines', 1, 'Delimiter', ',');
fclose(fid);

csvread

[Shayma]

because i have 186 columns, i don't prefer to use this method ! thank you