Download all hours of all days of the year from CMORPH
1 visualizzazione (ultimi 30 giorni)
Mostra commenti meno recenti
Augusto Gabriel da Costa Pereira
il 16 Dic 2022
Commentato: VBBV
il 20 Dic 2022
This script below wants to download all hours of all days of the year from 2000 to 2005, but the following error appears:
I want to download all time data for 2000 2005 from this link: Index of /data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km (noaa.gov)
clc
clearvars
% Download the CPC data used in the script below
which_years = (2000:2005); % years to download
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name
%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year
for cc = 1:numel(which_years) % loop over years
this_year = which_years(cc);
for ci = 1:numel(mm) % loop over monthes
days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year
for ck = 1:days_per_month % loop over days
ymd_str = datenum(this_year, mm(ck)+1, 1) - datenum(this_year, mm(ck), 1); % year month day string
for ch = 1:ymd_str
hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/");
filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,".nc"); % create filename according to list above
disp(strcat(filename, " is processed"));
dataUrl = strcat(Url, filename);
data = webread(dataUrl);
filename_tmp=strcat("data_5", ymd_str ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
filename_out = websave(filename_tmp,dataUrl); % save downloaded data
% ncdisp(filename_tmp,'/','min'); % commented, just to avoid filling the command window
precip=ncread(filename_tmp,'cmorph'); % Dimensions: lon,lat,time
long=ncread(filename_tmp,'lon');
lat=ncread(filename_tmp,'lat');
time=ncread(filename_tmp,'time');
end
end
end
end
The error reported is this:
Error using matlab.internal.webservices.HTTPConnector/copyContentToByteArray (line 396)
The server returned the status 404 with message "Not Found" in response to the request to
URL
https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/2000/01/CMORPH_V1.0_ADJ_8km-30min_20000101.nc.
Error in readContentFromWebService (line 46)
byteArray = copyContentToByteArray(connection);
Error in webread (line 125)
[varargout{1:nargout}] = readContentFromWebService(connection, options);
Error in Untitled (line 36)
data = webread(dataUrl);
1 Commento
Peter Perkins
il 19 Dic 2022
Two things:
1) You probably want to look at using a datastore for this. They are specifically designed for this kind of multiple file scenario.
2) You are not doing yourself any favors by using datenums. Use datetimes.
Risposta accettata
Augusto Gabriel da Costa Pereira
il 18 Dic 2022
Modificato: Augusto Gabriel da Costa Pereira
il 19 Dic 2022
3 Commenti
Augusto Gabriel da Costa Pereira
il 18 Dic 2022
Modificato: Augusto Gabriel da Costa Pereira
il 18 Dic 2022
VBBV
il 20 Dic 2022
Ok, The problem was with error located in the line
data = webread(dataUrl);
which was resolved, Regarding the download of data, did you look at my updated comment ? . You need to use { } to store all files for years as shown below
filename_out{i+1} = websave(filename_tmp,dataUrl); % save /downloaded all data
Più risposte (1)
VBBV
il 17 Dic 2022
Modificato: VBBV
il 17 Dic 2022
Note the following changes
clc
clearvars
% Download the CPC data used in the script below
which_years = (2000:2005); % years to download
httpsUrl = "https://www.ncei.noaa.gov/data/cmorph-high-resolution-global-precipitation-estimates/access/30min/8km/";
% filename_tmp="temp.nc"; % temporary filename (for loop); data will be always stored under this temp name
%% main loop
mm = (1:12); % all monthes in year
% mm = [1 6]; % process only some specific monthes in year
for cc = 1:numel(which_years) % loop over years
this_year = which_years(cc);
for ci = 1:numel(mm) % loop over monthes
days_per_month = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % compute how many days this month/year
for ck = 1:days_per_month % loop over days
ymd_str = datenum(this_year, mm(ci)+1, 1) - datenum(this_year, mm(ci), 1); % year month day string
for ch = 1:ymd_str
hora_str = strcat(sprintf('%04d',this_year), sprintf('%02d', mm(ci)), sprintf('%02d', ch));
Url = strcat(httpsUrl, sprintf('%04d', this_year), "/", sprintf('%02d', mm(ci)),"/",sprintf('%02d',ch),'/');
for i = 0:23
filename = strcat("CMORPH_V1.0_ADJ_8km-30min_", hora_str ,sprintf('%02d',i),".nc"); % create filename according to list above
disp(strcat(filename, " is processed"));
dataUrl = strcat(Url, filename);
data = webread(dataUrl);
filename_tmp=strcat("data_5", num2str(ymd_str) ,".nc"); % temporary filename (for loop); data will be always stored under this temp name
filename_out = websave(filename_tmp,dataUrl); % save downloaded data
% ncdisp(filename_tmp,'/','min'); % commented, just to avoid filling the command window
precip=ncread(filename_tmp,'cmorph'); % Dimensions: lon,lat,time
long=ncread(filename_tmp,'lon');
lat=ncread(filename_tmp,'lat');
time=ncread(filename_tmp,'time');
end
end
end
end
end
8 Commenti
Vedere anche
Categorie
Scopri di più su Downloads in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!