Need help with "Out of memory" error

3 views (last 30 days)
Sara Bruschi
Sara Bruschi on 4 Jul 2022
Commented: Sara Bruschi on 4 Jul 2022
Hi everyone, I'm working with Matlab and InfluxDB in need to make them communicate and work with files containing millions of rows.
When I have to import the csv file with readtable I get no memory problem, but when I get to work with influxdb through matlab I get that a lot. I managed to resolve the problem when writing the data in the database just segmenting the timetable, but when I use the commands that helps me extract the data from the database (around 5 millions of rows) i get that error and I can't find a way around it.
Can someone please help me? Thanks in advance.

Answers (1)

Sam Chak
Sam Chak on 4 Jul 2022
5 millions of rows...
Even Excel cannot handle more than 1,048,576 rows. What is the size of your data array?
N = 5e6;
n = round(sqrt(N));
A = rand(n)
A = 2236×2236
0.2871 0.2892 0.2362 0.8154 0.6719 0.9486 0.0286 0.0167 0.0037 0.9809 0.2334 0.3200 0.6854 0.6888 0.1695 0.9808 0.2442 0.7251 0.9363 0.5723 0.9718 0.5492 0.7839 0.9613 0.7380 0.8300 0.1917 0.6323 0.4454 0.1851 0.3742 0.5974 0.1385 0.3654 0.7016 0.4632 0.2757 0.1444 0.1932 0.9558 0.1683 0.4811 0.3795 0.9344 0.0477 0.0531 0.9365 0.8929 0.8917 0.5186 0.2335 0.4736 0.3354 0.0314 0.2693 0.9257 0.4876 0.6192 0.8157 0.0835 0.9680 0.8536 0.5761 0.4397 0.7299 0.4963 0.4466 0.7446 0.1443 0.3364 0.2759 0.5051 0.8596 0.1811 0.9695 0.0002 0.8675 0.0702 0.0458 0.7376 0.0446 0.7774 0.4039 0.0414 0.3167 0.7822 0.3230 0.2219 0.2134 0.8739 0.0221 0.9181 0.7088 0.0396 0.4419 0.8038 0.9004 0.5114 0.8154 0.0024 0.5422 0.6169 0.3261 0.2753 0.1199 0.0807 0.5046 0.4376 0.1845 0.8173 0.7529 0.4920 0.7203 0.8673 0.4909 0.5040 0.0431 0.2915 0.2074 0.6698 0.2228 0.5988 0.0644 0.4645 0.8457 0.3008 0.7301 0.7456 0.3475 0.8645 0.0832 0.5660 0.6291 0.3286 0.5295 0.7372 0.3966 0.8460 0.6487 0.0219 0.9833 0.0456 0.9380 0.3492 0.3861 0.3639 0.9372 0.4164 0.7274 0.6656 0.3047 0.7029 0.6693 0.6465 0.2201 0.7117 0.0093 0.2530 0.9470 0.2857 0.5328 0.3144 0.2076 0.8893 0.1145 0.7776 0.9828 0.7616 0.6983 0.9311 0.2531 0.9205 0.4533 0.5781 0.7889 0.1680 0.8151 0.9566 0.5768 0.7956 0.1690 0.4041 0.2299 0.0682 0.1962 0.4454 0.7184 0.8617 0.5937 0.0149 0.5761 0.2546 0.8043 0.2645 0.7105 0.1647 0.2057 0.4319 0.6486 0.3131 0.0807 0.6536 0.2650 0.6225 0.6905 0.0867 0.1694 0.8908 0.2788 0.9686 0.5583 0.2647 0.6359 0.0701 0.1135 0.1764 0.6828 0.2149 0.9057 0.2459 0.4584 0.2944 0.1062 0.8637 0.2237 0.4451 0.7978 0.6307 0.7105 0.6784 0.0101 0.0640 0.7792 0.0225 0.7677 0.6073 0.2113 0.8333 0.0855 0.1213 0.5186 0.2191 0.3643 0.3197 0.7782 0.6828 0.2533 0.6939 0.0983 0.1220 0.0086 0.6968 0.3239 0.0517 0.4423 0.6635 0.5764 0.5497 0.8355 0.9435 0.8975 0.6853 0.8019 0.0649 0.0176 0.7949 0.8224 0.2203 0.2157 0.3511 0.9187 0.2796 0.7165 0.5693 0.8563 0.6711 0.7393 0.8317 0.8806 0.9828 0.9553 0.1042 0.5307 0.3902 0.9207 0.3834 0.4921 0.5256 0.6935 0.4608 0.2148 0.2966 0.0602 0.5499 0.5002 0.3270 0.8936 0.9960 0.0837 0.0307
B = ones(N, 1)
B = 5000000×1
1 1 1 1 1 1 1 1 1 1
whos
Name Size Bytes Class Attributes A 2236x2236 39997568 double B 5000000x1 40000000 double N 1x1 8 double cmdout 1x33 66 char n 1x1 8 double
  3 Comments
Sara Bruschi
Sara Bruschi on 4 Jul 2022
I alredy checked that and it was already set as in your photo.
For let matlab and influxdb communicate I'm using a user code found on github: https://github.com/wardgssens/influxdb-matlab/tree/v2-compatibility.
After writing my timetable to influx via the write command I'm trying to call them back with the query command but I get this error if I don't limit the rows.
It's strange also because if I limit the query to 999'999 rows it works fine, but limiting it to 1'000'000 rows I get the same error..

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by