Automatic Download from a Url when the file of the date is inputted by the user

19 views (last 30 days)
I have written a code to download a specific data from cosmic satellite website, the code is as follows
url_https='https://data.cosmic.ucar.edu';
dataUrl=strcat(url_https,"/gnss-ro/cosmic2/nrt/level1b/2022/004/podTc2_nrt_2022_004.tar.gz");
dataFile="podTc2_nrt_2022_005.tar.gz";
FileFullPath=websave(dataFile,dataUrl);
Files=untar("F:\podtc2_data\podTc2_nrt_2022_005.tar.gz",'podTc2_nrt_2022_005');
Now this code only works for one day of the data that is provided by the data portal. is there a way such that I could create a loop that modifies a string of characters corresponent to the desired date that I am looking to work with, then the web function can read the string automatically created by MATLAB.
I was wondering if a web crawler or something similar has been done on matlab?
Overall, I have currently created a code that download a specific date of data, I am looking to code a set of loops where if I enter a specific date the code would automatically crawl the web and download the specific file that I am looking for.
I have been stuck on this section for a quite a long time, hopefully you guys could give me some advice and ideas on this matter.
  2 Comments
Tianchu Lu
Tianchu Lu on 10 Aug 2022
Thank you very much for your answer benearth, so the part that should be modified would be the part of _005, as 005 means 5th of Jan 2022 and hence 001 is 1st of Jan and so on and so forth. So the pusedocode that I had in mind was that the user( myself) would enter a specific date that the I would like to investigate into, and then a loop where matlab would urlread (https://data.cosmic.ucar.edu/gnss-ro/cosmic2/nrt/level1b/2022/) and identify the date that the user want and would then download the specific zipped file of that date. so a loop where it would urlread and identify the date inputted and download the file correspond to the inputted information. But I will defo have a play with the code that you submitted below and let you know how I got on.

Sign in to comment.

Accepted Answer

Jan
Jan on 10 Aug 2022
This can be implemented with sprintf() or compose() easily.
url_https='https://data.cosmic.ucar.edu';
for year = 2020:2022 % ??? Bold guess
for day = 1:365 % ??? Bold guess
dataUrl = sprintf('%s/gnss-ro/cosmic2/nrt/level1b/%d/%03d/podTc2_nrt_%d_%03d.tar.gz', ...
url_https, year, day, year, day);
dataFile = sprintf('podTc2_nrt_%d_%03d.tar.gz', year, day);
FileFullPath = websave(dataFile, dataUrl);
end
end
  3 Comments
Débora Rodrigues
Débora Rodrigues on 20 Sep 2022 at 8:20
Edited: Débora Rodrigues on 20 Sep 2022 at 8:45
Hello! So i solved my problem!
I changed the function websave for urlwrite, and it worked! I am living it in here in case someone with similar problens appear haha.
url = 'http://ftp1.cptec.inpe.br/modelos/tempo/MERGE/GPM/DAILY';
for ano= 2010:2021
for mes= 1:12
for dia=1:30
data_url = sprintf('%s/%3d/%02d/MERGE_CPTEC_%d%02d%02d.grib2', ...
url, ano, mes, ano, mes, dia);
data_file= sprintf('MERGE_CPTEC_%d%02d%02d.grib2', ano, mes, dia);
%data = websave(data_url,data_file);
a= urlwrite(data_url,data_file);
end
end
end

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by