search large amount of words

Hi all, I have a string from URL in CHAR form, and i want to execute a search of large amount of words, does someone have an idea how to do that? thank you very much!

[Walter Roberson]

Will you be searching the same words many times for different URLs?

Is it possible that the URL is percent-encoded and that needs to be taken into account? Is it possible the words to search are percent encoded but a different way?

Is it possible that the character string is in UTF-8 or UTF-16 and contains encoded Unicode characters that might contain composed characters that might be composed differently or represented in decomposed form in the words to be searched?

Is it possible that the character string contains non-English characters and needs to be matched in a case-insensitive manner against the words that might be case-sensitive? For example 'The Greek upper-case letter "Σ" has two different lower-case forms: "ς" in word-final position and "σ" elsewhere' (wikipedia)

[Walter Roberson]

The words to search: are they somehow stored with distinct boundaries, such as cell array of strings or struct with a character vector field? Or are they a continuous string of text, perhaps in sentence-like form?

Or is the question the wrong way around, and the task is to use the URL to read a web page and then you want to search the content of the web page for any of a number of strings?

[Rafi Kandli]

yes, the task to use the URL (input) to read a web page and I want to search the content of the web page for any of a number of strings.

search of large amount of words in txt.

html = urlread('http://www.ynet.co.il');
txt = regexprep(html,'<script.*?/script>','');
txt = regexprep(txt,'<style.*?/style>','');
txt = regexprep(txt,'<.*?>','');

[Walter Roberson]

I see that Unicode is a factor here.

So the text could be with or without niqqud but mostly without?

[Rafi Kandli]

yes, especially without niqqud. thank you!

[Jan]

It matters if "large" means 1e3 or 1e9. It is either more efficient to search the stored list of words in the text, or extract the words of the text at first and search them in the list. If the string is obtains from an URL, a text file or obtained by OCR from an image does not matter at all. So please post the relevant details.

[Rafi Kandli]

Thank you all, I have a string of about 80 words and i want to check if at least one word from the string is contained in the text that is extracted from a website.

[Walter Roberson]

matchpattern = ['(?<=\W)', strjoin(WordListCell, '(?=\W)|(?<=\W)'), '(?=\W)' ];
results = regexp(txt, matchpattern, 'match');

This would be for the case of exact matches of words -- so for example plurals would not be matched, and words with article prefix characters would not be matched. I do not promise that \W knows about the proper classification of Hebrew characters.

Answer 1

[KSSV]

Read about strfind, strcmp, strcmpi and regexp.

Answer 2

[Image Analyst]

What about simply using the contains() function?

Description

TF = contains(str,pattern) returns 1 (true) if str contains the specified pattern, and returns 0 (false) otherwise.

If pattern is an array containing multiple patterns, then contains returns 1 if it finds any element of pattern in str.

[Walter Roberson]

Note: contains() does not check word boundaries. For example it would find "his" in "this".