summaryrefslogtreecommitdiff
path: root/contrib/tsearch2/README.tsearch2
diff options
context:
space:
mode:
authorTom Lane <tgl@sss.pgh.pa.us>2007-11-13 21:02:29 +0000
committerTom Lane <tgl@sss.pgh.pa.us>2007-11-13 21:02:29 +0000
commit90e3f2aca716e562f81a64dba5fb5232b140e4d7 (patch)
tree2040468fc9a6e902614d48b03912da6859e95bf7 /contrib/tsearch2/README.tsearch2
parent4ea3210a04478163edf1ffd7d2547b62e82df4f0 (diff)
downloadpostgresql-90e3f2aca716e562f81a64dba5fb5232b140e4d7.tar.gz
Replace the now-incompatible-with-core contrib/tsearch2 module with a
compatibility package. This supports importing dumps from past versions using tsearch2, and provides the old names and API for most functions that were changed. (rewrite(ARRAY[...]) is a glaring omission, though.) Pavel Stehule and Tom Lane
Diffstat (limited to 'contrib/tsearch2/README.tsearch2')
-rw-r--r--contrib/tsearch2/README.tsearch2210
1 files changed, 0 insertions, 210 deletions
diff --git a/contrib/tsearch2/README.tsearch2 b/contrib/tsearch2/README.tsearch2
deleted file mode 100644
index 5d7b19d55f..0000000000
--- a/contrib/tsearch2/README.tsearch2
+++ /dev/null
@@ -1,210 +0,0 @@
-Tsearch2 - full text search extension for PostgreSQL
-
- [1]Online version of this document is available
-
- Tsearch2 - is the full text engine, fully integrated into PostgreSQL
- RDBMS.
-
-Main features
-
- * Full online update
- * Supports multiple table driven configurations
- * flexible and rich linguistic support (dictionaries, stop words),
- thesaurus
- * full multibyte (UTF-8) support
- * Sophisticated ranking functions with support of proximity and
- structure information (rank, rank_cd)
- * Index support (GiST and Gin) with concurrency and recovery support
- * Rich query language with query rewriting support
- * Headline support (text fragments with highlighted search terms)
- * Ability to plug-in custom dictionaries and parsers
- * Template generator for tsearch2 dictionaries with [2]snowball
- stemmer support
- * It is mature (5 years of development)
-
- Tsearch2, in a nutshell, provides FTS operator (contains) for the new
- data types, representing document (tsvector) and query (tsquery).
- Table driven configuration allows creation of custom searches using
- standard SQL commands.
-
- tsvector is a searchable data type, representing document. It is a set
- of unique words along with their positional information in the
- document, organized in a special structure optimized for fast access
- and lookup. Each entry could be labelled to reflect its importance in
- document.
-
- tsquery is a data type for textual queries with support of boolean
- operators. It consists of lexemes (optionally labelled) with boolean
- operators between.
-
- Table driven configuration allows to specify:
- * parser, which used to break document onto lexemes
- * what lexemes to index and the way they are processed
- * dictionaries to be used along with stop words recognition.
-
-OpenFTS vs Tsearch2
-
- [3]OpenFTS is a middleware between application and database. OpenFTS
- uses tsearch2 as a storage and database engine as a query executor
- (searching). Everything else, i.e. parsing of documents, query
- processing, linguistics, carry outs on client side. That's why OpenFTS
- has its own configuration table (fts_conf) and works with its own set
- of dictionaries. OpenFTS is more flexible, because it could be used in
- multi-server architecture with separate machines for repository of
- documents (documents could be stored in filesystem), database and
- query engine.
-
- See [4]Documentation Roadmap for links to documentation.
-
-Authors
-
- * Oleg Bartunov <oleg@sai.msu.su>, Moscow, Moscow University, Russia
- * Teodor Sigaev <teodor@sigaev.ru>, Moscow,Moscow University,Russia
-
-Contributors
-
- * Robert John Shepherd and Andrew J. Kopciuch submitted
- "Introduction to tsearch" (Robert - tsearch v1, Andrew - tsearch
- v2)
- * Brandon Craig Rhodes wrote "Tsearch2 Guide" and "Tsearch2
- Reference" and proposed new naming convention for tsearch V2
-
-Sponsors
-
- * ABC Startsiden - compound words support
- * University of Mannheim for UTF-8 support (in 8.2)
- * jfg:networks ([5]http:www.jfg-networks.com/) for Gin - Generalized
- Inverted index (in 8.2)
- * Georgia Public Library Service and LibLime, Inc. for Thesaurus
- dictionary
- * PostGIS community - GiST Concurrency and Recovery
-
- The authors are grateful to the Russian Foundation for Basic Research
- and Delta-Soft Ltd., Moscow, Russia for support.
-
-Limitations
-
- * Length of lexeme < 2K
- * Length of tsvector (lexemes + positions) < 1Mb
- * The number of lexemes < 4^32
- * 0< Positional information < 16383
- * No more than 256 positions per lexeme
- * The number of nodes ( lexemes + operations) in tsquery < 32768
-
-References
-
- * GiST development site -
- [6]http://www.sai.msu.su/~megera/postgres/gist
- * GiN development - [7]http://www.sigaev.ru/gin/
- * OpenFTS home page - [8]http://openfts.sourceforge.net/
- * Mailing list -
- [9]http://sourceforge.net/mailarchive/forum.php?forum=openfts-gene
- ral
-
-Documentation Roadmap
-
- * Several docs are available from docs/ subdirectory
- + "Tsearch V2 Introduction" by Andrew Kopciuch
- + "Tsearch2 Guide" by Brandon Rhodes
- + "Tsearch2 Reference" by Brandon Rhodes
- * Readme.gendict in gendict/ subdirectory
- + Also, check [10]Gendict tutorial
- * Check [11]tsearch2 Wiki pages for various documentation
-
-Support
-
- Authors urgently recommend people to use [12]openfts-general or
- [13]pgsql-general mailing lists for questions and discussions.
-
-Development History
-
- Latest news
-
- To the PostgreSQL 8.2 release we added:
- * multibyte (UTF-8) support
- * Thesaurus dictionary
- * Query rewriting
- * rank_cd relevation function now support different weights of
- lexemes
- * GiN support adds scalability of tsearch2
-
- Pre-tsearch era
- Development of OpenFTS began in 2000 after realizing that we
- need a search engine optimized for online updates with access
- to metadata from the database. This is essential for online
- news agencies, web portals, digital libraries, etc. Most search
- engines available utilize an inverted index which is very fast
- for searching but very slow for online updates. Incremental
- updates of an inverted index is a complex engineering task
- while we needed something light, free and with the ability to
- access metadata from the database. The last requirement was
- very important because in a real life application search engine
- should always consult metadata ( topic, permissions, date
- range, version, etc.). We extensively use PostgreSQL as a
- database backend and have no intention to move from it, so the
- problem was to find a data structure and a fast way to access
- it. PostgreSQL has rather unique data type for storing sets
- (think about words) - arrays, but lacks index access to them.
- During our research we found a paper of Joseph Hellerstein, who
- introduced an interesting data structure suitable for sets -
- RD-tree (Russian Doll tree). Further research lead us to the
- idea to use GiST for implementing RD-tree, but at that time the
- GiST code was untouched for a long time and contained several
- bugs. After work on improving GiST for version 7.0.3 of
- PostgreSQL was done, we were able to implement RD-Tree and use
- it for index access to arrays of integers. This implementation
- was ideally suited for small arrays and eliminated complex
- joins, but was practically useless for indexing large arrays.
- The next improvement came from an idea to represent a document
- by a single bit-signature, a so-called superimposed signature
- (see "Index Structures for Databases Containing Data Items with
- Set-valued Attributes", 1997, Sven Helmer for details). We
- developed the contrib/intarray module and used it for full
- text indexing.
-
- tsearch v1
- It was inconvenient to use integer id's instead of words, so we
- introduced a new data type called 'txtidx' - a searchable data
- type (textual) with indexed access. This was a first step of
- our work on an implementation of a built-in PostgreSQL full
- text search engine. Even though tsearch v1 had many features of
- a search engine it lacked configuration support and relevance
- ranking. People were encouraged to use OpenFTS, which provided
- relevance ranking based on positional information and flexible
- configuration. OpenFTS v.0.34 is the last version based on
- tsearch v1.
-
- tsearch V2
- People recognized tsearch as a powerful tool for full text
- searching and insisted on adding ranking support, better
- configurability, etc. We already thought about moving most of
- the features of OpenFTS to tsearch, and in the early 2003 we
- decided to work on a new version of tsearch. We abandoned
- auxiliary index tables which were used by OpenFTS to store
- positional information and modified the txtidx type to store
- them internally. We added table-driven configuration, support
- of ispell dictionaries, snowball stemmers and the ability to
- specify which types of lexemes to index. Now, it's possible to
- generate headlines of documents with highlighted search terms.
- These changes make tsearch more user friendly and turn it into
- a really powerful full text search engine. Brandon Rhodes
- proposed to rename tsearch functions for consistency and we
- renamed txtidx type to tsvector and other things as well. To
- allow users of tsearch v1 smooth upgrade, we named the module
- as tsearch2. Since version 0.35 OpenFTS uses tsearch2.
-
-References
-
- 1. http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/Tsearch_V2_Readme.html
- 2. http://snowball.tartarus.org/
- 3. http://openfts.sourceforge.net/
- 4. file://localhost/u/megera/WWW/postgres/gist/tsearch/V2/docs/Tsearch_V2_Readme82.html#dm
- 5. http:www.jfg-networks.com/
- 6. http://www.sai.msu.su/~megera/postgres/gist
- 7. http://www.sigaev.ru/gin/
- 8. http://openfts.sourceforge.net/
- 9. http://sourceforge.net/mailarchive/forum.php?forum=openfts-general
- 10. http://www.sai.msu.su/~megera/wiki/Gendict
- 11. http://www.sai.msu.su/~megera/wiki/Tsearch2
- 12. http://sourceforge.net/mailarchive/forum.php?forum=openfts-general
- 13. http://archives.postgresql.org/pgsql-general/