Size: 107
Comment:
|
Size: 2452
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 5: | Line 5: |
* ClusteringAlgorithms | |
Line 7: | Line 8: |
* LexicalAnalysis * ["Precision"] * ["Recall"] * RelevanceFeedback * SignatureFile * StemmingConflation * StopWords * SuperImposedCoding * TruncationConflation * ThesaurusConflation '''This link provides a nice glossary of terms http://www.cs.jhu.edu/~weiss/glossary.html''' == Description == Examples: Library catalogs Generally the '''data''' are organized as a collection of '''documents'''. == Querying == Querying of unstructured textual data is referred to as '''Information Retrieval'''. It covers the following areas: * Querying based on keywords * The relevance of documents to the query * The analysis, classification and indexing of documents. Queries are formed using keywords and logical connectives ''and, or,'' and ''not'' where the ''and'' connective is implicit. '''Full Text''' --> All words in a document are ''keywords''. We use '''term''' to refer to words in a document, since all words are keywords. Given a document ''d'', and a term ''t'' one way of defining the relavence ''r'' is [[latex2($$r(d,t)=\log\left(1+\frac{n(d,t)}{n(d)}\right)$$)]] n(d) denotes the number of terms in the document, and n(d,t) denotes the number of occurrences of term t in the document d. KEY: In the inromration retrieval community, the relevance of a document to a term is referred to as '''term frequency''', regardless of the exact formula used. Inverse Document frequency defined as: [[latex2($$IDF = \frac{1}{n(t)}$$)]] where n(t) denotes the number of documents that contain the term t. Here we have a low IDF if the word is found in many of the documents. If it is found in only a few, then it is probably a good term to use! Thus the '''relavance''' of a document ''d'' to a set of terms ''Q'' is then defined as [[latex2($$r(d,Q)=\sum_{t \in Q}\frac{r(d,t)}{n(t)}$$)]] [[latex2($$r(d,Q)=\sum_{t \in Q}\frac{w(t) r(d,t)}{n(t)}$$)]] where w(t) is a weight specified by the user. '''KEY: Stop words''' are words that are not indexed such as ''and, or the, a'' etc. '''Proximity''': if a the terms occur close to each other in the document, the document would be ranked higher than if they occur far apart. We could (although we don't) modify the formula r(d,Q) to take proximity into account. |
|
Line 9: | Line 64: |
Back to ComputerTerms
Terms
- ["Precision"]
- ["Recall"]
This link provides a nice glossary of terms http://www.cs.jhu.edu/~weiss/glossary.html
Description
Examples: Library catalogs
Generally the data are organized as a collection of documents.
Querying
Querying of unstructured textual data is referred to as Information Retrieval. It covers the following areas:
- Querying based on keywords
- The relevance of documents to the query
- The analysis, classification and indexing of documents.
Queries are formed using keywords and logical connectives and, or, and not where the and connective is implicit.
Full Text --> All words in a document are keywords. We use term to refer to words in a document, since all words are keywords.
Given a document d, and a term t one way of defining the relavence r is
latex2($$r(d,t)=\log\left(1+\frac{n(d,t)}{n(d)}\right)$$)
n(d) denotes the number of terms in the document, and n(d,t) denotes the number of occurrences of term t in the document d.
KEY: In the inromration retrieval community, the relevance of a document to a term is referred to as term frequency, regardless of the exact formula used.
Inverse Document frequency defined as:
latex2($$IDF = \frac{1}{n(t)}$$) where n(t) denotes the number of documents that contain the term t.
Here we have a low IDF if the word is found in many of the documents. If it is found in only a few, then it is probably a good term to use!
Thus the relavance of a document d to a set of terms Q is then defined as
latex2($$r(d,Q)=\sum_{t \in Q}\frac{r(d,t)}{n(t)}$$)
latex2($$r(d,Q)=\sum_{t \in Q}\frac{w(t) r(d,t)}{n(t)}$$) where w(t) is a weight specified by the user.
KEY: Stop words are words that are not indexed such as and, or the, a etc.
Proximity: if a the terms occur close to each other in the document, the document would be ranked higher than if they occur far apart. We could (although we don't) modify the formula r(d,Q) to take proximity into account.
Back to ComputerTerms