|
Size: 2617
Comment:
|
Size: 2423
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 9: | Line 9: |
| * ["Precision"] * ["Recall"] |
* [[Precision]] * [[Recall]] |
| Line 41: | Line 41: |
| {{{ n(d,t) r(d,t) = log( 1 + ------ ) n(d) |
$$$r(d,t)=\log\left(1+\frac{n(d,t)}{n(d)}\right)$$$ |
| Line 46: | Line 43: |
| n(d) denotes the number of terms in the document, and n(d,t) denotes the number of occurrences of term t in the document d. }}} |
n(d) denotes the number of terms in the document, and n(d,t) denotes the number of occurrences of term t in the document d. |
| Line 54: | Line 49: |
| {{{ 1 IDF = ---- where n(t) denotes the number of documents that contain the term t. n(t) }}} |
$$$IDF = \frac{1}{n(t)}$$$ where n(t) denotes the number of documents that contain the term t. |
| Line 64: | Line 57: |
| {{{ --- r(d,t) r(d,Q) = \ ------ /__ n(t) |
$$$r(d,Q)=\sum_{t \in Q}\frac{r(d,t)}{n(t)}$$$ |
| Line 69: | Line 59: |
| t element of Q | $$$r(d,Q)=\sum_{t \in Q}\frac{w(t) r(d,t)}{n(t)}$$$ |
| Line 71: | Line 61: |
| --- w(t) r(d,t) r(d,Q) = \ ----------- where w(t) is a weight specified by the user. /__ n(t) t element of Q }}} |
where w(t) is a weight specified by the user. |
| Line 82: | Line 65: |
| '''Proximity''': if a the terms occur close to each other in the document, the document would be ranked higher than if they occur far apart. We could (although we don't) modify the formula r(d,Q) to take proximity into account. |
'''Proximity''': if a the terms occur close to each other in the document, the document would be ranked higher than if they occur far apart. We could (although we don't) modify the formula $$r(d,Q)$$ to take proximity into account. |
| Line 88: | Line 68: |
Back to ComputerTerms
Terms
This link provides a nice glossary of terms http://www.cs.jhu.edu/~weiss/glossary.html
Description
Examples: Library catalogs
Generally the data are organized as a collection of documents.
Querying
Querying of unstructured textual data is referred to as Information Retrieval. It covers the following areas:
- Querying based on keywords
- The relevance of documents to the query
- The analysis, classification and indexing of documents.
Queries are formed using keywords and logical connectives and, or, and not where the and connective is implicit.
Full Text --> All words in a document are keywords. We use term to refer to words in a document, since all words are keywords.
Given a document d, and a term t one way of defining the relavence r is
$$$r(d,t)=\log\left(1+\frac{n(d,t)}{n(d)}\right)$$$
n(d) denotes the number of terms in the document, and n(d,t) denotes the number of occurrences of term t in the document d.
KEY: In the inromration retrieval community, the relevance of a document to a term is referred to as term frequency, regardless of the exact formula used.
Inverse Document frequency defined as:
$$$IDF = \frac{1}{n(t)}$$$
where n(t) denotes the number of documents that contain the term t.
Here we have a low IDF if the word is found in many of the documents. If it is found in only a few, then it is probably a good term to use!
Thus the relavance of a document d to a set of terms Q is then defined as
$$$r(d,Q)=\sum_{t \in Q}\frac{r(d,t)}{n(t)}$$$
$$$r(d,Q)=\sum_{t \in Q}\frac{w(t) r(d,t)}{n(t)}$$$
where w(t) is a weight specified by the user.
KEY: Stop words are words that are not indexed such as and, or the, a etc.
Proximity: if a the terms occur close to each other in the document, the document would be ranked higher than if they occur far apart. We could (although we don't) modify the formula $$r(d,Q)$$ to take proximity into account.
Back to ComputerTerms