summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
* meson: add docs, docs_pdf optionsAndres Freund2023-04-044-16/+86
| | | | | | | | | | | | | Detect and report if the tools necessary to build documentation are available during configure. This is represented as two new options 'docs' and 'docs_pdf', both defaulting to 'auto'. This should also fix a meson error about the installdocs target, when none of the doc tools are found. Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> Discussion: https://postgr.es/m/20230325201414.sh7c6xlut2fpunnv@awork3.anarazel.de Discussion: https://postgr.es/m/ZB8331v5IhUA/pNu@telsasoft.com
* meson: docs: Preparatory cleanupsAndres Freund2023-04-041-17/+23
| | | | | | | | These are just minor prerequisite changes for later commits. Kept separate for easier review. Discussion: https://postgr.es/m/3fc3bb9b-f7f8-d442-35c1-ec82280c564a@enterprisedb.com Discussion: https://postgr.es/m/20230329224132.fnymznyxmta5ugrs@awork3.anarazel.de
* Add Copyright notice in 001_basic.pl and 002_pg_upgrade.pl.Amit Kapila2023-04-052-0/+4
| | | | | Author: Kuroda Hayato Discussion: https://postgr.es/m/TYCPR01MB587073D91E372B8EF719931EF5929@TYCPR01MB5870.jpnprd01.prod.outlook.com
* docs: Remove support for 'htmlhelp' formatAndres Freund2023-04-043-68/+0
| | | | | | | | | | | | | We had partial support for generating documentation suitable for .chm files. However, we only had wired up generating the input files using docbook-xsl, not generating an actual .chm file. Nor did we document how to do so. Additionally, it was very slow to generate htmlhelp, as we never applied the docbook-xsl stylesheet performance improvements to htmlhelp. It doesn't look like there's any interest in the htmlhelp output, so remove it, instead of spending cycles to finish the support. Discussion: https://postgr.es/m/20230324165822.wcrj3akllbqquy7u@awork3.anarazel.de
* sequences: Lock buffer before initializing pageAndres Freund2023-04-041-3/+1
| | | | | | | | | | | | | | | | | fill_seq_fork_with_data(), used to initialize a new sequence relation, only locked the buffer after calling PageInit(), even though PageInit() modifies page contents. This is unlikely to cause real-world issues, as the relation is exclusively locked at that point, and the buffer not yet marked dirty, so other processes should not access the buffer. This issue looks to have been present since the introduction of sequences in e8647c45d66a. Given the low risk, it does not seem worth backpatching the fix. Discussion: https://postgr.es/m/20230404185501.wdkmo3k7bedlx6qk@awork3.anarazel.de
* doc: Add more details about pg_stat_get_xact_blocks_{fetched,hit}Michael Paquier2023-04-051-4/+8
| | | | | | | | | | | | | | | | The explanation describing the dependency to system read() calls for these two functions has been removed in ddfc2d9. And after more discussion about d69c404, we have concluded that adding more details makes them easier to understand. While on it, use the term "block read requests" (maybe found in cache) rather than "buffers fetched" and "buffer hits". Per discussion with Melanie Plageman, Kyotaro Horiguchi, Bertrand Drouvot and myself. Discussion: https://postgr.es/m/CAAKRu_ZmdiScT4q83OAbfmR5AH-L5zWya3SXjaxiJvhCob-e2A@mail.gmail.com Backpatch-through: 11
* Fix MSVC warning introduced in ea1db8ae70.Jeff Davis2023-04-042-0/+2
| | | | | Discussion: https://postgr.es/m/CA+hUKGJR1BhCORa5WdvwxztD3arhENcwaN1zEQ1Upg20BwjKWA@mail.gmail.com Reported-by: Thomas Munro
* Remove comment obsoleted by 11c2d6fd.Thomas Munro2023-04-051-3/+2
| | | | | Reported-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://postgr.es/m/1604497.1680637072%40sss.pgh.pa.us
* Canonicalize ICU locale names to language tags.Jeff Davis2023-04-0410-27/+258
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Convert to BCP47 language tags before storing in the catalog, except during binary upgrade or when the locale comes from an existing collation or template database. The resulting language tags can vary slightly between ICU versions. For instance, "@colBackwards=yes" is converted to "und-u-kb-true" in older versions of ICU, and to the simpler (but equivalent) "und-u-kb" in newer versions. The process of canonicalizing to a language tag also understands more input locale string formats than ucol_open(). For instance, "fr_CA.UTF-8" is misinterpreted by ucol_open() and the region is ignored; effectively treating it the same as the locale "fr" and opening the wrong collator. Canonicalization properly interprets the language and region, resulting in the language tag "fr-CA", which can then be understood by ucol_open(). This commit fixes a problem in prior versions due to ucol_open() misinterpreting locale strings as described above. For instance, creating an ICU collation with locale "fr_CA.UTF-8" would store that string directly in the catalog, which would later be passed to (and misinterpreted by) ucol_open(). After this commit, the locale string will be canonicalized to language tag "fr-CA" in the catalog, which will be properly understood by ucol_open(). Because this fix affects the resulting collator, we cannot change the locale string stored in the catalog for existing databases or collations; otherwise we'd risk corrupting indexes. Therefore, only canonicalize locales for newly-created (not upgraded) collations/databases. For similar reasons, do not backport. Discussion: https://postgr.es/m/8c7af6820aed94dc7bc259d2aa7f9663518e6137.camel@j-davis.com Reviewed-by: Peter Eisentraut
* Add a way to get the current function's OID in pl/pgsql.Tom Lane2023-04-048-4/+77
| | | | | | | | | | | | | Invent "GET DIAGNOSTICS oid_variable = PG_ROUTINE_OID". This is useful for avoiding the maintenance nuisances that come with embedding a function's name in its body, as one might do for logging purposes for example. Typically users would cast the result to regproc or regprocedure to get something human-readable, but we won't pre-judge whether that's appropriate. Pavel Stehule, reviewed by Kirk Wolak and myself Discussion: https://postgr.es/m/CAFj8pRA4zMd5pY-B89Gm64bDLRt-L+akOd34aD1j4PEstHHSVQ@mail.gmail.com
* Add a run_as_owner option to subscriptions.Robert Haas2023-04-0412-92/+376
| | | | | | | | | | | | | | | | | | This option is normally false, but can be set to true to obtain the legacy behavior where the subscription runs with the permissions of the subscription owner rather than the permissions of the table owner. The advantages of this mode are (1) it doesn't require that the subscription owner have permission to SET ROLE to each table owner and (2) since no role switching occurs, the SECURITY_RESTRICTED_OPERATION restrictions do not apply. On the downside, it allows any table owner to easily usurp the privileges of the subscription owner - basically, to take over their account. Because that's generally quite undesirable, we don't make this mode the default, but we do make it available, just in case the new behavior causes too many problems for someone. Discussion: http://postgr.es/m/CA+TgmoZ-WEeG6Z14AfH7KhmpX2eFh+tZ0z+vf0=eMDdbda269g@mail.gmail.com
* Perform logical replication actions as the table owner.Robert Haas2023-04-049-109/+250
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Up until now, logical replication actions have been performed as the subscription owner, who will generally be a superuser. Commit cec57b1a0fbcd3833086ba686897c5883e0a2afc documented hazards associated with that situation, namely, that any user who owns a table on the subscriber side could assume the privileges of the subscription owner by attaching a trigger, expression index, or some other kind of executable code to it. As a remedy, it suggested not creating configurations where users who are not fully trusted own tables on the subscriber. Although that will work, it basically precludes using logical replication in the way that people typically want to use it, namely, to replicate a database from one node to another without necessarily having any restrictions on which database users can own tables. So, instead, change logical replication to execute INSERT, UPDATE, DELETE, and TRUNCATE operations as the table owner when they are replicated. Since this involves switching the active user frequently within a session that is authenticated as the subscription user, also impose SECURITY_RESTRICTED_OPERATION restrictions on logical replication code. As an exception, if the table owner can SET ROLE to the subscription owner, these restrictions have no security value, so don't impose them in that case. Subscription owners are now required to have the ability to SET ROLE to every role that owns a table that the subscription is replicating. If they don't, replication will fail. Superusers, who normally own subscriptions, satisfy this property by default. Non-superusers users who own subscriptions will need to be granted the roles that own relevant tables. Patch by me, reviewed (but not necessarily in its entirety) by Jelte Fennema, Jeff Davis, and Noah Misch. Discussion: http://postgr.es/m/CA+TgmoaSCkg9ww9oppPqqs+9RVqCexYCE6Aq=UsYPfnOoDeFkw@mail.gmail.com
* Add missing XML ID attributesPeter Eisentraut2023-04-042-2/+2
| | | | | Author: Brar Piening <brar@gmx.de> Discussion: https://www.postgresql.org/message-id/dc813a6f-60d9-991f-eecd-675a0921de11@gmx.de
* Code review for recent SQL/JSON commitsAlvaro Herrera2023-04-045-172/+266
| | | | | | | | | | | | | | | | | | | | | | | | - At the last minute and for no particularly good reason, I changed the WITHOUT token to be marked especially for lookahead, from the one in WITHOUT TIME to the one in WITHOUT UNIQUE. Study of upcoming patches (where a new WITHOUT ARRAY WRAPPER clause is added) showed me that the former was better, so put it back the way the original patch had it. - update exprTypmod() for JsonConstructorExpr to return the typmod of the RETURNING clause, as a comment there suggested. Perhaps it's possible for this to make a difference with datetime types, but I didn't try to build a test case. - The nodeFuncs.c support code for new nodes was calling walker() directly instead of the WALK() macro as introduced by commit 1c27d16e6e5c. Modernize that. Also add exprLocation() support for a couple of nodes that missed it. Lastly, reorder the code more sensibly. The WITHOUT_LA -> WITHOUT change means that stored rules containing either WITHOUT TIME ZONE or WITHOUT UNIQUE KEYS would change representation. Therefore, bump catversion. Discussion: https://postgr.es/m/20230329181708.e64g2tpy7jyufqkr@alvherre.pgsql
* bufmgr: Remove buffer-write-dirty tracepointsAndres Freund2023-04-033-29/+0
| | | | | | | | | The trace point was using the relfileno / fork / block for the to-be-read-in buffer. Some upcoming work would make that more expensive to provide. We still have buffer-flush-start/done, which can serve most tracing needs that buffer-write-dirty could serve. Discussion: https://postgr.es/m/f5164e7a-eef6-8972-75a3-8ac622ed0c6e@iki.fi
* Make SP-GiST redirect cleanup more aggressive.Peter Geoghegan2023-04-031-2/+1
| | | | | | | | | | | | | | Commit 61b313e4 made VACUUM pass down a heaprel to index AM bulkdelete and vacuumcleanup routines. Although this was primarily intended as preparation for logical decoding on standbys, it also made it easy to correct an old deficiency in how we determine how to cleanup SP-GiST redirect and placeholder tuples. Pass the heaprel to GlobalVisTestFor() during cleanup of redirect and placeholder tuples, rather than pessimistically passing NULL. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Discussion: https://postgr.es/m/02392033-f030-a3c8-c7d0-5c27eb529fec@gmail.com
* Recycle deleted nbtree pages more aggressively.Peter Geoghegan2023-04-033-10/+8
| | | | | | | | | | | | | | | | | | | | | | | | Commit 61b313e4 made nbtree consistently pass down a heaprel to low level routines like _bt_getbuf(). Although this was primarily intended as preparation for logical decoding on standbys, it also made it easy to correct an old deficiency in how nbtree VACUUM determines whether or not it's now safe to recycle deleted pages. Pass the heaprel to GlobalVisTestFor() in nbtree routines that deal with recycle safety. nbtree now makes less pessimistic assumptions about recycle safety within non-catalog relations. This enhancement complements the recycling enhancement added by commit 9dd963ae25. nbtree remains just as pessimistic as ever when it comes to recycle safety within indexes on catalog relations. There is no fundamental reason why we need to treat catalog relations differently, though. The behavioral inconsistency is a consequence of the way that nbtree uses nextXID values to implement what Lanin and Shasha call "the drain technique". Note in particular that it has nothing to do with whether or not index tuples might still be required for an older MVCC snapshot. Author: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Discussion: https://postgr.es/m/CAH2-WzkaiDxCje0yPuH=3Uh2p1V_2pFGY==xfbZoZu7Ax_NB8g@mail.gmail.com
* Move heaprel struct field next to index rel field.Peter Geoghegan2023-04-035-6/+6
| | | | | | | | | Commit 61b313e4 added a heaprel struct member to IndexVacuumInfo, but placed it last. Move the heaprel struct member next to the index struct member to improve the code's readability. Author: Peter Geoghegan <pg@bowt.ie> Discussion: https://postgr.es/m/CAH2-WznG=TV6S9d3VA=y0vBHbXwnLs9_LLdiML=aNJuHeriwxg@mail.gmail.com
* Fix possible logical replication crash.Robert Haas2023-04-031-1/+3
| | | | | | | | | | | Commit c3afe8cf5a1e465bd71e48e4bc717f5bfdc7a7d6 added a new password_required option but forgot that you need database access to check whether an arbitrary role ID is a superuser. Report and patch by Hou Zhijie. I added a comment. Thanks to Alexander Lakhin for devising a way to reproduce the crash. Discussion: http://postgr.es/m/OS0PR01MB5716BFD7EC44284C89F40808948F9@OS0PR01MB5716.jpnprd01.prod.outlook.com
* When using valgrind, log the current query after an error is detected.Tom Lane2023-04-031-0/+66
| | | | | | | | | | | | In USE_VALGRIND builds, add code to print the text of the current query to the valgrind log whenever the valgrind error count has increased during the query. Valgrind will already have printed its report, if the error is distinct from ones already seen, so that this works out fairly well as a way of annotating the log. Onur Tirtir and Tom Lane Discussion: https://postgr.es/m/AM9PR83MB0498531E804DC8DF8CFF0E8FE9D09@AM9PR83MB0498.EURPRD83.prod.outlook.com
* Revert 764da7710bAlexander Korotkov2023-04-031-35/+13
| | | | Discussion: https://postgr.es/m/20230323003003.plgaxjqahjgkuxrk%40awork3.anarazel.de
* Revert 11470f544eAlexander Korotkov2023-04-036-285/+186
| | | | Discussion: https://postgr.es/m/20230323003003.plgaxjqahjgkuxrk%40awork3.anarazel.de
* Rename BufferAccessStrategyData.ring_size to nbuffersDavid Rowley2023-04-031-9/+9
| | | | | | | | | | | | The new name better reflects what the field is - the size of the buffers[] array. ring_size sounded more like it is in units of bytes. An upcoming commit allows a BufferAccessStrategy of custom sizes, so it seems relevant to improve this beforehand. Author: Melanie Plageman Reviewed-by: David Rowley Discussion: https://postgr.es/m/CAAKRu_YefVbhg4rAxU2frYFdTap61UftH-kUNQZBvAs%2BYK81Zg%40mail.gmail.com
* Disable vacuum's use of a buffer access strategy during failsafeDavid Rowley2023-04-033-5/+18
| | | | | | | | | | | | | | | | | | | | | | Traditionally, vacuum always makes use of a buffer access strategy 32 buffers in size. This means that running vacuums tend not to cause too many shared buffers to become dirty, however, this can cause vacuums to run much more slowly than they otherwise could as WAL flushes will occur more frequently due to having to flush WAL out to the LSN of the dirty page before that page can be written to disk. When we are performing failsafe VACUUMs (as added in 1e55e7d17), we really want to make the vacuum work go as quickly as possible, so here we disable the buffer access strategy when entering failsafe mode while vacuuming a relation. Per idea and analyis from Andres Freund. In passing, also include some changes I had intended for 32fbe0239. Author: Melanie Plageman Reviewed-by: Justin Pryzby, David Rowley Discussion: https://postgr.es/m/20230111182720.ejifsclfwymw2reb%40awork3.anarazel.de
* Fix typo in CI READMEDaniel Gustafsson2023-04-031-1/+1
| | | | s/fron/from/
* Only make buffer strategy for vacuum when it's likely neededDavid Rowley2023-04-031-1/+3
| | | | | | | | | | | | | | | | VACUUM FULL and VACUUM ONLY_DATABASE_STATS will not use the vacuum strategy ring created in vacuum(), so don't waste effort making it in those cases. There are other conceivable cases where the buffer strategy also won't be used, but those are probably less common and not worth troubling over too much. For example VACUUM (PROCESS_MAIN false, PROCESS_TOAST false). There are other cases too, but many of these are only discovered once inside vacuum_rel(). Author: Melanie Plageman Reviewed-by: David Rowley Discussion: https://postgr.es/m/CAAKRu_ZLRuzkM3zKogiZAz2hUony37yLY4aaLb8fPf8fgqs5Og@mail.gmail.com
* pg_basebackup: Correct type of WalSegSzPeter Eisentraut2023-04-032-2/+2
| | | | | | | | | The pg_basebackup code had WalSegSz as uint32, whereas the rest of the code has it as int. This seems confusing, and using the extra range wouldn't actually work. Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Discussion: https://www.postgresql.org/message-id/flat/1bf15c7a-0acd-1864-081e-7a28814310fe%40enterprisedb.com
* Remove some global variables from vacuum.cDavid Rowley2023-04-031-23/+20
| | | | | | | | | | | | | | Using global variables because we don't want to pass these values around as parameters does not really seem like a great idea, so let's remove these two global variables and adjust a few functions to accept these values as parameters instead. This is part of a wider patch which intends to allow the size of the buffer access strategy that vacuum uses to be adjusted. Author: Melanie Plageman Reviewed-by: Bharath Rupireddy Discussion: https://postgr.es/m/CAAKRu_b1q_07uquUtAvLqTM%3DW9nzee7QbtzHwA4XdUo7KX_Cnw%40mail.gmail.com
* Doc: update pgindent/README.Tom Lane2023-04-021-3/+3
| | | | I missed updating this when we pulled pg_bsd_indent into the tree.
* Add info in WAL records in preparation for logical slot conflict handlingAndres Freund2023-04-0215-29/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit only implements one prerequisite part for allowing logical decoding. The commit message contains an explanation of the overall design, which later commits will refer back to. Overall design: 1. We want to enable logical decoding on standbys, but replay of WAL from the primary might remove data that is needed by logical decoding, causing error(s) on the standby. To prevent those errors, a new replication conflict scenario needs to be addressed (as much as hot standby does). 2. Our chosen strategy for dealing with this type of replication slot is to invalidate logical slots for which needed data has been removed. 3. To do this we need the latestRemovedXid for each change, just as we do for physical replication conflicts, but we also need to know whether any particular change was to data that logical replication might access. That way, during WAL replay, we know when there is a risk of conflict and, if so, if there is a conflict. 4. We can't rely on the standby's relcache entries for this purpose in any way, because the startup process can't access catalog contents. 5. Therefore every WAL record that potentially removes data from the index or heap must carry a flag indicating whether or not it is one that might be accessed during logical decoding. Why do we need this for logical decoding on standby? First, let's forget about logical decoding on standby and recall that on a primary database, any catalog rows that may be needed by a logical decoding replication slot are not removed. This is done thanks to the catalog_xmin associated with the logical replication slot. But, with logical decoding on standby, in the following cases: - hot_standby_feedback is off - hot_standby_feedback is on but there is no a physical slot between the primary and the standby. Then, hot_standby_feedback will work, but only while the connection is alive (for example a node restart would break it) Then, the primary may delete system catalog rows that could be needed by the logical decoding on the standby (as it does not know about the catalog_xmin on the standby). So, it’s mandatory to identify those rows and invalidate the slots that may need them if any. Identifying those rows is the purpose of this commit. Implementation: When a WAL replay on standby indicates that a catalog table tuple is to be deleted by an xid that is greater than a logical slot's catalog_xmin, then that means the slot's catalog_xmin conflicts with the xid, and we need to handle the conflict. While subsequent commits will do the actual conflict handling, this commit adds a new field isCatalogRel in such WAL records (and a new bit set in the xl_heap_visible flags field), that is true for catalog tables, so as to arrange for conflict handling. The affected WAL records are the ones that already contain the snapshotConflictHorizon field, namely: - gistxlogDelete - gistxlogPageReuse - xl_hash_vacuum_one_page - xl_heap_prune - xl_heap_freeze_page - xl_heap_visible - xl_btree_reuse_page - xl_btree_delete - spgxlogVacuumRedirect Due to this new field being added, xl_hash_vacuum_one_page and gistxlogDelete do now contain the offsets to be deleted as a FLEXIBLE_ARRAY_MEMBER. This is needed to ensure correct alignment. It's not needed on the others struct where isCatalogRel has been added. This commit just introduces the WAL format changes mentioned above. Handling the actual conflicts will follow in future commits. Bumps XLOG_PAGE_MAGIC as the several WAL records are changed. Author: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com> Author: Andres Freund <andres@anarazel.de> (in an older version) Author: Amit Khandekar <amitdkhan.pg@gmail.com> (in an older version) Reviewed-by: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Robert Haas <robertmhaas@gmail.com> Reviewed-by: Fabrízio de Royes Mello <fabriziomello@gmail.com> Reviewed-by: Melanie Plageman <melanieplageman@gmail.com>
* Fix copy-pasto in contrib/auth_delay/meson.build variable name.Noah Misch2023-04-021-2/+2
|
* Use PG_TEST_TIMEOUT_DEFAULT in 019_replslot_limit.pl.Noah Misch2023-04-021-5/+4
| | | | | | | | | Oversight in f2698ea02ca8a56f38935d2b300ac54936712558, which introduced the variable. This lowers some 1000s timeouts to the configurable default of 180s, due to a lack of evidence for needing the longer timeout. The alternative was 6*PG_TEST_TIMEOUT_DEFAULT, which we can adopt if the need arises. Given the lack of observed trouble with these timeouts, no back-patch.
* Pass down table relation into more index relation functionsAndres Freund2023-04-0126-170/+214
| | | | | | | | | | | | This is done in preparation for logical decoding on standby, which needs to include whether visibility affecting WAL records are about a (user) catalog table. Which is only known for the table, not the indexes. It's also nice to be able to pass the heap relation to GlobalVisTestFor() in vacuumRedirectAndPlaceholder(). Author: "Drouvot, Bertrand" <bertranddrouvot.pg@gmail.com> Discussion: https://postgr.es/m/21b700c3-eecf-2e05-a699-f8c78dd31ec7@gmail.com
* Assert only valid flag bits are passed to visibilitymap_set()Andres Freund2023-04-011-1/+1
| | | | | | | | | If visibilitymap_set() is called with flags containing a higher bit than VISIBILITYMAP_ALL_FROZEN, the state of neighboring pages is affected. While there was an assertion that *some* valid bits were set, it did not check that *only* valid bits were. Change that. Discussion: https://postgr.es/m/20230331043300.gux3s5wzrapqi4oe@awork3.anarazel.de
* hio: Release extension lock before initializing page / pinning VMAndres Freund2023-04-011-7/+7
| | | | | | | | | | | | | | | | PageInit() while holding the extension lock is unnecessary after 0d1fe9f74e3 started to use RBM_ZERO_AND_LOCK - nobody can look at the new page before we release the page lock. PageInit() zeroes the page, which isn't that cheap, so deferring it until after the extension lock is released seems like a good idea. Doing visibilitymap_pin() while holding the extension lock, introduced in 7db0cd2145f2, looks like an accident. Due to the restrictions on HEAP_INSERT_FROZEN it's unlikely to be a performance issue, but it still seems better to move it out. We also are doing the visibilitymap_pin() while holding the buffer lock, which will be fixed in a separate commit. Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi> Discussion: http://postgr.es/m/419312fd-9255-078c-c3e3-f0525f911d7f@iki.fi
* pg_dump: Use only LZ4 frame format for compressionTomas Vondra2023-04-011-211/+343
| | | | | | | | | | | | | | | | | | | | | | | | After 0da243fed0 got committed, it was reported that in some cases the compression ratio is rather poor - particularly for custom format with narrow tables - due to writing the LZ4 header/footer for each row. This commit switches to LZ4F (LZ4 frame format), eliminating most of the overhead and greatly improving the compression ratio. This makes the compressed size about the same for plain and custom formats (just like for gzip, for example). LZ4F is now used by both compression APIs, which allowed refactoring and reusing more of the code. For consistency this also renames the LZ4File struct to LZ4State, and a number of functions are now prefixed with LZ4Stream_ (instead of LZ4File_). Patch by Georgios Kokolatos, based on report and initial patch by Justin Pryzby. Review and minor cleanups by me. Author: Georgios Kokolatos, Justin Pryzby Reported-by: Justin Pryzby Reviewed-by: Tomas Vondra Discussion: https://postgr.es/m/20230227044910.GO1653%40telsasoft.com
* Doc: add Buffer Access Strategy to the glossaryDavid Rowley2023-04-011-0/+28
| | | | | | | | | | It seems useful to add this to the glossary as there's discussion around adding an option to VACUUM to disable and adjust the size of the buffer access strategy that VACUUM uses. Author: Melanie Plageman Reviewed-by: Justin Pryzby, David Rowley Discussion: https://postgr.es/m/ZBYDTrD1kyGg%2BHkS%40telsasoft.com
* Add show_data option to pg_get_wal_block_info.Peter Geoghegan2023-03-316-24/+39
| | | | | | | | | | | | | | | | | | | | | | | Allow users to opt out of returning FPI data and block data from pg_get_wal_block_info as an optimization. Testing has shown that this can make function execution over twice as fast in some cases. When pg_get_wal_block_info is called with "show_data := false", it always returns NULL values for its block_data and block_fpi_data bytea output parameters. Nothing else changes. In particular, the function will still return the usual per-block summary of block data/FPI space overhead. Use of "show_data := false" is therefore feasible with all queries that don't specifically require these raw binary strings. Follow-up to recent work in commit 122376f0. There still hasn't been a stable release with the pg_get_wal_block_info function, so no bump in the pg_walinspect extension version. Per suggestion from Melanie Plageman. Author: Peter Geoghegan <pg@bowt.ie> Discussion: https://postgr.es/m/CAAKRu_bJvbcYBRj2cN6G2xV7B7-Ja+pjTO1nEnEhRR8OXYiABA@mail.gmail.com Discussion: https://postgr.es/m/CAH2-Wzm9shOkEDM10_+qOZkRSQhKVxwBFiehH6EHWQQRd_rDPw@mail.gmail.com
* SQL/JSON: support the IS JSON predicateAlvaro Herrera2023-03-3125-67/+1030
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch introduces the SQL standard IS JSON predicate. It operates on text and bytea values representing JSON, as well as on the json and jsonb types. Each test has IS and IS NOT variants and supports a WITH UNIQUE KEYS flag. The tests are: IS JSON [VALUE] IS JSON ARRAY IS JSON OBJECT IS JSON SCALAR These should be self-explanatory. The WITH UNIQUE KEYS flag makes these return false when duplicate keys exist in any object within the value, not necessarily directly contained in the outermost object. Author: Nikita Glukhov <n.gluhov@postgrespro.ru> Author: Teodor Sigaev <teodor@sigaev.ru> Author: Oleg Bartunov <obartunov@gmail.com> Author: Alexander Korotkov <aekorotkov@gmail.com> Author: Amit Langote <amitlangote09@gmail.com> Author: Andrew Dunstan <andrew@dunslane.net> Reviewers have included (in no particular order) Andres Freund, Alexander Korotkov, Pavel Stehule, Andrew Alsup, Erik Rijkers, Zihong Yu, Himanshu Upadhyaya, Daniel Gustafsson, Justin Pryzby. Discussion: https://postgr.es/m/CAF4Au4w2x-5LTnN_bxky-mq4=WOqsGsxSpENCzHRAzSnEd8+WQ@mail.gmail.com Discussion: https://postgr.es/m/cd0bb935-0158-78a7-08b5-904886deac4b@postgrespro.ru Discussion: https://postgr.es/m/20220616233130.rparivafipt6doj3@alap3.anarazel.de Discussion: https://postgr.es/m/abd9b83b-aa66-f230-3d6d-734817f0995d%40postgresql.org
* Further tweaking of width_bucket() edge cases.Tom Lane2023-03-314-28/+88
| | | | | | | | | | | | | | | | | | I realized that the third overflow case I posited in commit b0e9e4d76 actually should be handled in a different way: rather than tolerating the idea that the quotient could round to 1, we should clamp so that the output cannot be more than "count" when we know that the operand is less than bound2. That being the case, we don't need an overflow-aware increment in that code path, which leads me to revert the movement of the pg_add_s32_overflow() call. (The diff in width_bucket_float8 might be easier to read by comparing against b0e9e4d76^.) What's more, width_bucket_numeric also has this problem of the quotient potentially rounding to 1, so add a clamp there too. As before, I'm not quite convinced that a back-patch is warranted. Discussion: https://postgr.es/m/391415.1680268470@sss.pgh.pa.us
* Reject system columns as elements of foreign keys.Tom Lane2023-03-313-10/+23
| | | | | | | | | | | | | | | | | | | | | | Up through v11 it was sensible to use the "oid" system column as a foreign key column, but since that was removed there's no visible usefulness in making any of the remaining system columns a foreign key. Moreover, since the TupleTableSlot rewrites in v12, such cases actively fail because of implicit assumptions that only user columns appear in foreign keys. The lack of complaints about that seems like good evidence that no one is trying to do it. Hence, rather than trying to repair those assumptions (of which there are at least two, maybe more), let's just forbid the case up front. Per this patch, a system column in either the referenced or referencing side of a foreign key will draw this error; however, putting one in the referenced side would have failed later anyway, since we don't allow unique indexes to be made on system columns. Per bug #17877 from Alexander Lakhin. Back-patch to v12; the case still appears to work in v11, so we shouldn't break it there. Discussion: https://postgr.es/m/17877-4bcc658e33df6de1@postgresql.org
* Ensure acquire_inherited_sample_rows sets its output parameters.Tom Lane2023-03-311-2/+4
| | | | | | | | | | | | The totalrows/totaldeadrows outputs were left uninitialized in cases where we find no analyzable child tables of a partitioned table. This could lead to setting the partitioned table's pg_class.reltuples value to garbage. It's not clear that that would have any very bad effects in practice, but fix it anyway because it's making valgrind unhappy. Reported and diagnosed by Alexander Lakhin (bug #17880). Discussion: https://postgr.es/m/17880-9282037c923d856e@postgresql.org
* pg_regress: Emit TAP compliant outputDaniel Gustafsson2023-03-314-262/+325
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This converts pg_regress output format to emit TAP compliant output while keeping it as human readable as possible for use without TAP test harnesses. As verbose harness related information isn't really supported by TAP this also reduces the verbosity of pg_regress runs which makes scrolling through log output in buildfarm/CI runs a bit easier as well. As the meson TAP parser conumes whitespace, the leading indentation for differentiating parallel tests from sequential tests has been changed to a single character prefix. This patch has been around for an extended period of time, reviewers listed below may have been involved in reviewing a version quite different from the version in this commit. The original idea for this patch was a hacking session with Jinbao Chen. TAP format testing is also enabled in meson as of this. Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Reviewed-by: Nikolay Shaplov <dhyan@nataraj.su> Reviewed-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com> Discussion: https://postgr.es/m/BD4B107D-7E53-4794-ACBA-275BEB4327C9@yesql.se Discussion: https://postgr.es/m/20220221164736.rq3ornzjdkmwk2wo@alap3.anarazel.de
* Move ExecEvalJsonConstructor new function to a more natural placeAlvaro Herrera2023-03-312-39/+39
| | | | | Commit 7081ac46ace8 put it at the end of the file, but that doesn't look very nice.
* No need to add FORMAT to the keyword precedence listAlvaro Herrera2023-03-311-1/+0
| | | | Commit 7081ac46ace8 put it there. Remove it.
* Add XML ID attributes to create_publication.sgml.Amit Kapila2023-03-315-51/+68
| | | | | | | | | | This commit adds XML ID attributes to all varlistentries in create_publication.sgml. This allows us to include links to refer to publication options, making documents more readable. Author: Kuroda Hayato Reviewed-by: Peter Smith, Amit Kapila Discussion: https://postgr.es/m/TYAPR01MB58668219FEA4EC231486A433F58E9@TYAPR01MB5866.jpnprd01.prod.outlook.com
* Bump PGSTAT_FILE_FORMAT_ID, omitted in 8aaa04b32d7Andres Freund2023-03-301-1/+1
| | | | | | I forgot to do so in the referenced commit. While the consequences of omitting the version change are likely to be harmless (besides discarding stats, as a PGSTAT_FILE_FORMAT_ID bump also does), it still seems worth doing.
* Track shared buffer hits in pg_stat_ioAndres Freund2023-03-3012-47/+109
| | | | | | | | | | | | Among other things, this should make it easier to calculate a useful cache hit ratio by excluding buffer reads via buffer access strategies. As buffer access strategies reuse buffers (and thus evict the prior buffer contents), it is normal to see reads on repeated scans of the same data. Author: Melanie Plageman <melanieplageman@gmail.com> Reviewed-by: Bertrand Drouvot <bertranddrouvot.pg@gmail.com> Reviewed-by: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/CAAKRu_beMa9Hzih40%3DXPYqhDVz6tsgUGTrhZXRo%3Dunp%2Bszb%3DUA%40mail.gmail.com
* Fix List memory issue in transformColumnDefinitionDavid Rowley2023-03-311-0/+3
| | | | | | | | | | | | | | | | | | | | | When calling generateSerialExtraStmts(), we would pass in the constraint->options. In some cases, generateSerialExtraStmts() would modify the referenced List to remove elements from it, but doing so is invalid without assigning the list back to all variables that point to it. In the particular reported problem case, the List became empty, in which cases it became NIL, but the passed in constraint->options didn't get to find out about that and was left pointing to free'd memory. To fix this, just perform a list_copy() inside generateSerialExtraStmts(). We could just do a list_copy() just before we perform the delete from the list, however, that seems less robust. Let's make sure the generated CreateSeqStmt gets a completely different copy of the list to be safe. Bug: #17879 Reported-by: Fei Changhong Diagnosed-by: Fei Changhong Discussion: https://postgr.es/m/17879-b7dfb5debee58ff5@postgresql.org Backpatch-through: 11, all supported versions
* Parallel Hash Full Join.Thomas Munro2023-03-317-48/+323
| | | | | | | | | | | | | | | | | | | | | | | | | | Full and right outer joins were not supported in the initial implementation of Parallel Hash Join because of deadlock hazards (see discussion). Therefore FULL JOIN inhibited parallelism, as the other join strategies can't do that in parallel either. Add a new PHJ phase PHJ_BATCH_SCAN that scans for unmatched tuples on the inner side of one batch's hash table. For now, sidestep the deadlock problem by terminating parallelism there. The last process to arrive at that phase emits the unmatched tuples, while others detach and are free to go and work on other batches, if there are any, but otherwise they finish the join early. That unfairness is considered acceptable for now, because it's better than no parallelism at all. The build and probe phases are run in parallel, and the new scan-for-unmatched phase, while serial, is usually applied to the smaller of the two relations and is either limited by some multiple of work_mem, or it's too big and is partitioned into batches and then the situation is improved by batch-level parallelism. Author: Melanie Plageman <melanieplageman@gmail.com> Author: Thomas Munro <thomas.munro@gmail.com> Reviewed-by: Thomas Munro <thomas.munro@gmail.com> Discussion: https://postgr.es/m/CA%2BhUKG%2BA6ftXPz4oe92%2Bx8Er%2BxpGZqto70-Q_ERwRaSyA%3DafNg%40mail.gmail.com
View Lorry Controller Status