diff options
| author | Tom Lane <tgl@sss.pgh.pa.us> | 2007-11-13 21:02:29 +0000 |
|---|---|---|
| committer | Tom Lane <tgl@sss.pgh.pa.us> | 2007-11-13 21:02:29 +0000 |
| commit | 90e3f2aca716e562f81a64dba5fb5232b140e4d7 (patch) | |
| tree | 2040468fc9a6e902614d48b03912da6859e95bf7 /contrib/tsearch2/README.tsearch2 | |
| parent | 4ea3210a04478163edf1ffd7d2547b62e82df4f0 (diff) | |
| download | postgresql-90e3f2aca716e562f81a64dba5fb5232b140e4d7.tar.gz | |
Replace the now-incompatible-with-core contrib/tsearch2 module with a
compatibility package. This supports importing dumps from past versions
using tsearch2, and provides the old names and API for most functions
that were changed. (rewrite(ARRAY[...]) is a glaring omission, though.)
Pavel Stehule and Tom Lane
Diffstat (limited to 'contrib/tsearch2/README.tsearch2')
| -rw-r--r-- | contrib/tsearch2/README.tsearch2 | 210 |
1 files changed, 0 insertions, 210 deletions
diff --git a/contrib/tsearch2/README.tsearch2 b/contrib/tsearch2/README.tsearch2 deleted file mode 100644 index 5d7b19d55f..0000000000 --- a/contrib/tsearch2/README.tsearch2 +++ /dev/null @@ -1,210 +0,0 @@ -Tsearch2 - full text search extension for PostgreSQL - - [1]Online version of this document is available - - Tsearch2 - is the full text engine, fully integrated into PostgreSQL - RDBMS. - -Main features - - * Full online update - * Supports multiple table driven configurations - * flexible and rich linguistic support (dictionaries, stop words), - thesaurus - * full multibyte (UTF-8) support - * Sophisticated ranking functions with support of proximity and - structure information (rank, rank_cd) - * Index support (GiST and Gin) with concurrency and recovery support - * Rich query language with query rewriting support - * Headline support (text fragments with highlighted search terms) - * Ability to plug-in custom dictionaries and parsers - * Template generator for tsearch2 dictionaries with [2]snowball - stemmer support - * It is mature (5 years of development) - - Tsearch2, in a nutshell, provides FTS operator (contains) for the new - data types, representing document (tsvector) and query (tsquery). - Table driven configuration allows creation of custom searches using - standard SQL commands. - - tsvector is a searchable data type, representing document. It is a set - of unique words along with their positional information in the - document, organized in a special structure optimized for fast access - and lookup. Each entry could be labelled to reflect its importance in - document. - - tsquery is a data type for textual queries with support of boolean - operators. It consists of lexemes (optionally labelled) with boolean - operators between. - - Table driven configuration allows to specify: - * parser, which used to break document onto lexemes - * what lexemes to index and the way they are processed - * dictionaries to be used along with stop words recognition. - -OpenFTS vs Tsearch2 - - [3]OpenFTS is a middleware between application and database. OpenFTS - uses tsearch2 as a storage and database engine as a query executor - (searching). Everything else, i.e. parsing of documents, query - processing, linguistics, carry outs on client side. That's why OpenFTS - has its own configuration table (fts_conf) and works with its own set - of dictionaries. OpenFTS is more flexible, because it could be used in - multi-server architecture with separate machines for repository of - documents (documents could be stored in filesystem), database and - query engine. - - See [4]Documentation Roadmap for links to documentation. - -Authors - - * Oleg Bartunov <oleg@sai.msu.su>, Moscow, Moscow University, Russia - * Teodor Sigaev <teodor@sigaev.ru>, Moscow,Moscow University,Russia - -Contributors - - * Robert John Shepherd and Andrew J. Kopciuch submitted - "Introduction to tsearch" (Robert - tsearch v1, Andrew - tsearch - v2) - * Brandon Craig Rhodes wrote "Tsearch2 Guide" and "Tsearch2 - Reference" and proposed new naming convention for tsearch V2 - -Sponsors - - * ABC Startsiden - compound words support - * University of Mannheim for UTF-8 support (in 8.2) - * jfg:networks ([5]http:www.jfg-networks.com/) for Gin - Generalized - Inverted index (in 8.2) - * Georgia Public Library Service and LibLime, Inc. for Thesaurus - dictionary - * PostGIS community - GiST Concurrency and Recovery - - The authors are grateful to the Russian Foundation for Basic Research - and Delta-Soft Ltd., Moscow, Russia for support. - -Limitations - - * Length of lexeme < 2K - * Length of tsvector (lexemes + positions) < 1Mb - * The number of lexemes < 4^32 - * 0< Positional information < 16383 - * No more than 256 positions per lexeme - * The number of nodes ( lexemes + operations) in tsquery < 32768 - -References - - * GiST development site - - [6]http://www.sai.msu.su/~megera/postgres/gist - * GiN development - [7]http://www.sigaev.ru/gin/ - * OpenFTS home page - [8]http://openfts.sourceforge.net/ - * Mailing list - - [9]http://sourceforge.net/mailarchive/forum.php?forum=openfts-gene - ral - -Documentation Roadmap - - * Several docs are available from docs/ subdirectory - + "Tsearch V2 Introduction" by Andrew Kopciuch - + "Tsearch2 Guide" by Brandon Rhodes - + "Tsearch2 Reference" by Brandon Rhodes - * Readme.gendict in gendict/ subdirectory - + Also, check [10]Gendict tutorial - * Check [11]tsearch2 Wiki pages for various documentation - -Support - - Authors urgently recommend people to use [12]openfts-general or - [13]pgsql-general mailing lists for questions and discussions. - -Development History - - Latest news - - To the PostgreSQL 8.2 release we added: - * multibyte (UTF-8) support - * Thesaurus dictionary - * Query rewriting - * rank_cd relevation function now support different weights of - lexemes - * GiN support adds scalability of tsearch2 - - Pre-tsearch era - Development of OpenFTS began in 2000 after realizing that we - need a search engine optimized for online updates with access - to metadata from the database. This is essential for online - news agencies, web portals, digital libraries, etc. Most search - engines available utilize an inverted index which is very fast - for searching but very slow for online updates. Incremental - updates of an inverted index is a complex engineering task - while we needed something light, free and with the ability to - access metadata from the database. The last requirement was - very important because in a real life application search engine - should always consult metadata ( topic, permissions, date - range, version, etc.). We extensively use PostgreSQL as a - database backend and have no intention to move from it, so the - problem was to find a data structure and a fast way to access - it. PostgreSQL has rather unique data type for storing sets - (think about words) - arrays, but lacks index access to them. - During our research we found a paper of Joseph Hellerstein, who - introduced an interesting data structure suitable for sets - - RD-tree (Russian Doll tree). Further research lead us to the - idea to use GiST for implementing RD-tree, but at that time the - GiST code was untouched for a long time and contained several - bugs. After work on improving GiST for version 7.0.3 of - PostgreSQL was done, we were able to implement RD-Tree and use - it for index access to arrays of integers. This implementation - was ideally suited for small arrays and eliminated complex - joins, but was practically useless for indexing large arrays. - The next improvement came from an idea to represent a document - by a single bit-signature, a so-called superimposed signature - (see "Index Structures for Databases Containing Data Items with - Set-valued Attributes", 1997, Sven Helmer for details). We - developed the contrib/intarray module and used it for full - text indexing. - - tsearch v1 - It was inconvenient to use integer id's instead of words, so we - introduced a new data type called 'txtidx' - a searchable data - type (textual) with indexed access. This was a first step of - our work on an implementation of a built-in PostgreSQL full - text search engine. Even though tsearch v1 had many features of - a search engine it lacked configuration support and relevance - ranking. People were encouraged to use OpenFTS, which provided - relevance ranking based on positional information and flexible - configuration. OpenFTS v.0.34 is the last version based on - tsearch v1. - - tsearch V2 - People recognized tsearch as a powerful tool for full text - searching and insisted on adding ranking support, better - configurability, etc. We already thought about moving most of - the features of OpenFTS to tsearch, and in the early 2003 we - decided to work on a new version of tsearch. We abandoned - auxiliary index tables which were used by OpenFTS to store - positional information and modified the txtidx type to store - them internally. We added table-driven configuration, support - of ispell dictionaries, snowball stemmers and the ability to - specify which types of lexemes to index. Now, it's possible to - generate headlines of documents with highlighted search terms. - These changes make tsearch more user friendly and turn it into - a really powerful full text search engine. Brandon Rhodes - proposed to rename tsearch functions for consistency and we - renamed txtidx type to tsvector and other things as well. To - allow users of tsearch v1 smooth upgrade, we named the module - as tsearch2. Since version 0.35 OpenFTS uses tsearch2. - -References - - 1. http://www.sai.msu.su/~megera/postgres/gist/tsearch/V2/docs/Tsearch_V2_Readme.html - 2. http://snowball.tartarus.org/ - 3. http://openfts.sourceforge.net/ - 4. file://localhost/u/megera/WWW/postgres/gist/tsearch/V2/docs/Tsearch_V2_Readme82.html#dm - 5. http:www.jfg-networks.com/ - 6. http://www.sai.msu.su/~megera/postgres/gist - 7. http://www.sigaev.ru/gin/ - 8. http://openfts.sourceforge.net/ - 9. http://sourceforge.net/mailarchive/forum.php?forum=openfts-general - 10. http://www.sai.msu.su/~megera/wiki/Gendict - 11. http://www.sai.msu.su/~megera/wiki/Tsearch2 - 12. http://sourceforge.net/mailarchive/forum.php?forum=openfts-general - 13. http://archives.postgresql.org/pgsql-general/ |
