TabularTextDataStore wont store more than 20000 rows
1 visualizzazione (ultimi 30 giorni)
Mostra commenti meno recenti
SkarletSam
il 21 Ott 2020
Commentato: Walter Roberson
il 21 Ott 2020
Im trying to import a large csv file (60,000 x 171). tabularTextDataStore is only storing up to 20,000 rows. Is there a more efficient way to import all of the data?
0 Commenti
Risposta accettata
Kevin Gurney
il 21 Ott 2020
The amount of data returned on each call to the read method of a TabularTextDatastore is controlled by the value of the ReadSize property.
If you want to read more than 20,000 rows on each call to read (20,000 is the default value), then you can adjust the value of ReadSize like so:
trainSet = tabularTextDatastore("data\trainingSet.csv");
trainSet.ReadSize = 60000; % You can adjust this value to your liking.
trainSet.TreatAsMissing = 'na';
trainSetR = read(trainSet)
2 Commenti
Walter Roberson
il 21 Ott 2020
There seems to be an internal BUFFER_UPPERLIMIT property that is set to 32 megabytes, with 64 megabytes used for HADOOP . I do not see any method to set that property.
Più risposte (0)
Vedere anche
Categorie
Scopri di più su Whos in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!