How to run a program on an csv file data in batches

1 visualizzazione (ultimi 30 giorni)
Hello
I am trying to import data from csv file with size (1046520 1). Using the importdata function.
K = load('caamel.csv');
It is taking over an hour to compute my program.
How do I run my code on the excel file in batches instead of one whole data which is taking forever to see result
I will appreciate your help
thanks
Tino
  12 Commenti
Tino
Tino il 20 Giu 2019
Hi Walter, How do I then implement the computation
Thanks in advance
Tino
Walter Roberson
Walter Roberson il 20 Giu 2019
You implied that the reason that your code is taking so long is that import (but really load) is slow. I showed timing on a file 50 times larger than yours could take about 20 minutes to load() so your file would take about half a minute to load(). If you were to switch to textscan then it should take roughly 250/50 = 5 seconds. At that point you might as well read in all the data and loop over portions of it.
textscan also permits you to to specify a count to read, if you find that easier than loading everything and looping indexing.

Accedi per commentare.

Risposte (0)

Categorie

Scopri di più su Large Files and Big Data in Help Center e File Exchange

Tag

Prodotti


Release

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by