how to download data from website?

85 visualizzazioni (ultimi 30 giorni)
Lilya
Lilya il 23 Mar 2019
Modificato: Andrietta il 7 Ott 2024
Dear all,
I am trying to download data from the following website
https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/
my problem is I can not get the files, meaning only the html been located to my computer. (below what I used to locat the link into my machine)
url='https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
filename='A2019.nc'
outfilename=websave(filename,url)
what I need is getting the files separtly and read them.
Thanks for the help.
  2 Commenti
Juan
Juan il 27 Mag 2023
To download data from a website, follow these steps:
Identify the data you want to download on the website.
Right-click on the data or the download link.
Select the "Save link as" or "Save target as" option from the context menu.
Choose the destination folder on your computer where you want to save the downloaded data.
Click "Save" to initiate the download.
Wait for the download to complete. The time taken will depend on the size of the data and your internet connection speed.
Once the download is finished, you can access the downloaded data from the specified destination folder on your computer.
It's important to note that downloading data from websites should be done in compliance with applicable laws, website terms of service, and copyright restrictions. Ensure that you have the necessary rights and permissions to download and use the data obtained from websites.
SASSA
SASSA il 15 Ago 2024
repository.kaust.edu.sa/bitstream/handle/10754/673855/PhD%20Dissertation-Final_Lina%20Eyouni.pdf?sequence=4
The website you linked provides access to oceanographic data, but directly downloading individual files through code might be tricky. Here's a breakdown of what you're encountering and alternative approaches:
Understanding the Download Challenge:
  • The website likely uses a different download method than websave expects. It might require authentication or interact with the server differently.
Alternative Approaches:
  1. Manual Download:
  1. Command-line tools (for advanced users):
  • Tools like wget can be used to download files from websites. However, it might require additional configuration for NASA's specific setup. Refer to the wget documentation and NASA's download instructions (if available) for this approach.
Additional Tips:
  • NASA's OceanColor website provides information on download methods: SASSA Status
  • Consider using Python libraries specifically designed for downloading scientific data like earthpy or pydap. These libraries might offer better compatibility with NASA's data access methods.

Accedi per commentare.

Risposta accettata

Akira Agata
Akira Agata il 26 Mar 2019
How about the following?
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/';
str = webread(url);
links = regexp(str,'https://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)?(\.nc)','match')';
data = cell(size(links));
for kk = 1:numel(links)
data{kk} = webread(links{kk});
end
By running this, all the .nc files are downloaded and stored in the cell array data.
  12 Commenti
Belinda Finlay
Belinda Finlay il 12 Mag 2020
Thank you for your assistance Akira-san.
Soo Mee Kim
Soo Mee Kim il 11 Giu 2020
Hi, I have the same problem to to download nc files website.
When I tried the following code, both codes gave html file.
Please let me know how to download nc file from the website.
[my own]
fname = 'A2019152.L3m_DAY_PAR_par_4km.nc';
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
websave(fname, downloadURL, options);
[From Akira Agata's answer]
downloadURL = 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/A2019152.L3m_DAY_PAR_par_4km.nc';
options = weboptions('Username', '<username>', 'Password', '<password>');
data = webread(downloadURL, options);

Accedi per commentare.

Più risposte (7)

Tracy
Tracy il 1 Ago 2023
It looks like you are using MATLAB's websave function to download the file from the URL. However, the websave function is primarily used to download a file and save it to a specified location. It seems like you are trying to download multiple files, and it won't work as you expect because you are providing a URL of a directory, not a direct link to a specific file.
To download multiple files from the website, you will need to loop through the links in the directory and download each file individually. Additionally, you can use Python with the requests library to achieve this task more easily. Below is a Python script that demonstrates how to download multiple files from the website using the requests library:
python
import requests
import os
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/'
output_directory = './downloaded_files/'
# Create the output directory if it doesn't exist
os.makedirs(output_directory, exist_ok=True)
response = requests.get(url)
if response.status_code == 200:
# Parse the HTML content to find links to files
file_links = []
lines = response.text.split('\n')
for line in lines:
if '<a href="' in line:
start_index = line.find('<a href="') + len('<a href="')
end_index = line.find('">', start_index)
file_link = line[start_index:end_index]
if file_link.endswith('.nc'): # Only consider links that point to .nc files
file_links.append(file_link)
# Download each file and save it to the output directory
for file_link in file_links:
file_url = url + file_link
out_file_path = os.path.join(output_directory, file_link)
response = requests.get(file_url)
if response.status_code == 200:
with open(out_file_path, 'wb') as f:
f.write(response.content)
print(f"Downloaded: {file_link}")
else:
print(f"Failed to download: {file_link}")
else:
print("Failed to fetch the URL.")
print("All files downloaded successfully.")

Loria Smith
Loria Smith il 5 Set 2023
There are several ways of manual web scraping.
  1. Code a web scraper with Python. It is possible to quickly build software with any general-purpose programming language like Java, JavaScript, PHP, C, C#, and so on. ...
  2. Use a data service. ...
  3. Use Excel for data extraction. ...
  4. Web scraping tools

Hitesh
Hitesh il 3 Ott 2023
import requests
# Define the URL of the file you want to download
url = 'https://oceandata.sci.gsfc.nasa.gov/MODIS-Aqua/Mapped/Daily/4km/sst/2019/A2019.nc'
# Specify the local filename where you want to save the downloaded file
filename = 'A2019.nc'
# Send an HTTP GET request to the URL
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
# Open a local file with write-binary ('wb') mode and save the content of the response
with open(filename, 'wb') as file:
file.write(response.content)
print(f"File '{filename}' downloaded successfully.")
else:
print(f"Failed to download file. Status code: {response.status_code}")
There are various methods for manually collecting data from websites:
  1. Coding a Web Scraper with Python: You can create a web scraper using Python, a versatile programming language. Python offers libraries like BeautifulSoup and Scrapy that make web scraping easier.
  2. Utilizing a Data Service: Another option is to use a data service or API provided by the website, if available. This method allows you to access structured data without the need for web scraping.
  3. Leveraging Excel for Data Extraction: Microsoft Excel can also be used for data extraction. You can import data from web pages into Excel and then manipulate and analyze it as needed.
  4. Web Scraping Tools: There are various web scraping tools and software applications designed specifically for data extraction from websites. These tools often provide a user-friendly interface for collecting data without extensive coding.

ABDUL
ABDUL il 7 Ott 2024
Great insights on downloading data from websites! It’s fascinating how tools like MATLAB can simplify the process. Just as many of us need to navigate the SASSA appeal process to secure our benefits, having efficient methods to gather data online can be equally crucial in various projects. Thanks for sharing these tips!

ABDUL
ABDUL il 7 Ott 2024
This post is super helpful for anyone looking to download data from websites! The techniques you’ve outlined are great. It reminds me of how important it is to stay organized, much like when we’re going through a having the right data can make all the difference. Thanks for sharing these valuable insights! For more

Andrietta
Andrietta il 7 Ott 2024
"If you’re trying to download data from a website, web scraping techniques can help! There are some great resources for using MATLAB’s web functions to automate data retrieval. By the way, I was also searching for 'sassa payment dates 2024,' and realized combining this with web scraping could make tracking updates easier."

Andrietta
Andrietta il 7 Ott 2024
Modificato: Andrietta il 7 Ott 2024
"Using MATLAB’s webread or websave functions is a great way to download data from websites. It really helped me out when I was looking to pull info related to sassa appeal for an automation project. Anyone working on something similar might find these functions useful!"

Categorie

Scopri di più su Downloads in Help Center e File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by