Generating URLs from MATLAB

I am trying to write a script that generates URLs from 'aa' to 'zz' and pings the website, returning the <TITLE> of the page if the URL exists.

This is what I have done so far. I have no idea what to do next. Could you please help?

v={'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
subfolder='www';
s=sprintf('GET http://www.%.edu')

s =

'GET http://www.'

s=sprintf('GET http://www.v%.edu')

s =

'GET http://www.v'

[Adam Danz]

Your latest question is nearly the same as this one.

https://www.mathworks.com/matlabcentral/answers/616808-how-to-generate-a-matlab-code-that-generates-a-url

[Ntombikayise Bhengu]

Thank you so much

Answer 1

[Image Analyst]

I imagine you'd strip off the "GET " and call webread() with the URL.

v = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
for k = 1 : length(v)
    url = sprintf('http://www.%s%s.edu', v{k}, v{k});
    fprintf('Now reading %s...\n', url);
    pageContents = webread(url);
end

That's untested. Please look up webread() in the help for the correct way to use it.

[Ntombikayise Bhengu]

Thank you much. This is what I get:

v = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};

for k = 1 : length(v)

url = sprintf('http://www.%s%s.edu', v{k}, v{k});

fprintf('Now reading %s...\n', url);

pageContents = webread(url);

end

Now reading http://www.aa.edu...

Now reading http://www.bb.edu...

Error using webread (line 122)

Could not access server. http://www.bb.edu.

The code does not generateany more URLs

[Ntombikayise Bhengu]

I will also look at webread. Thank you so much.

[Image Analyst]

If you want it to keep trying, put the webread() into a try/catch:

v = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
for k = 1 : length(v)
    url = sprintf('http://www.%s%s.edu', v{k}, v{k});
    fprintf('Now reading %s...\n', url);
    try
        pageContents = webread(url);
    catch ME
        % Goes here if web site does not exist, then continues with the loop.
        fprintf('    Error : the web site "%s" does not exist or did not repond.\n', url);
    end
end

[Ntombikayise Bhengu]

Thank you so much very helpful. how does one generage a code with a combination of letters as well? in sucha way there are 676 possible websites with two letters?

tried this nothing happens.

 v = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
z = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
for k = 1 : length(v)
    
for m = 1 : length(z)
url = sprintf('http://www.%s%s.edu', v{k}, z{m});
    fprintf('Now reading %s...\n', url);
    try
        pageContents = webread(url);
    catch ME
        % Goes here if web site does not exist, then continues with the loop.
        fprintf('    Error : the web site "%s" does not exist or did not repond.\n', url);
    end
end

[Rik]

Where is your last end?

[Image Analyst]

Try it this way:

v = {'a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};
z = v;
for k = 1 : length(v)	
	for m = 1 : length(z)
		url = sprintf('http://www.%s%s.edu', v{k}, z{m});
		fprintf('Now reading %s...\n', url);
		try
			pageContents = webread(url);
		catch ME
			% Goes here if web site does not exist, then continues with the loop.
			fprintf('    Error : the web site "%s" does not exist or did not respond.\n', url);
		end
	end
end

[Ntombikayise Bhengu]

Thank you so much. It works. I really appreciate you taking the time to help me.