summaryrefslogtreecommitdiff
path: root/src/bin/pg_rewind
Commit message (Collapse)AuthorAgeFilesLines
* Fix overridden callbacks in pg_rewind.Daniel Gustafsson2023-05-022-2/+2
| | | | | | | | | | | | | The <source>_traverse_files functions take a callback for processing files, but both the local and libpq source implementations called the function directly without using the callback argument. While there is no bug right now as the function called is the same as the callback, fix by calling the callback to reduce the risk of subtle bugs in the future. Author: Junwang Zhao <zhjwpku@gmail.com> Reviewed-by: Richard Guo <guofenglinux@gmail.com> Discussion: https://postgr.es/m/CAEG8a3Jdwgh+PZr2zh1=t8apA4Yz8tKq+uubPqoCt14nvWKHEw@mail.gmail.com
* Fix typos in commentsMichael Paquier2023-05-021-1/+1
| | | | | | | | | The changes done in this commit impact comments with no direct user-visible changes, with fixes for incorrect function, variable or structure names. Author: Alexander Lakhin Discussion: https://postgr.es/m/e8c38840-596a-83d6-bd8d-cebc51111572@gmail.com
* Add missing source files to nls.mkPeter Eisentraut2023-04-141-0/+1
|
* Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.Thomas Munro2023-04-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to have the option to use O_DIRECT/FILE_FLAG_NO_BUFFERING in a later commit, we need the addresses of user space buffers to be well aligned. The exact requirements vary by OS and file system (typically sectors and/or memory pages). The address alignment size is set to 4096, which is enough for currently known systems: it matches modern sectors and common memory page size. There is no standard governing O_DIRECT's requirements so we might eventually have to reconsider this with more information from the field or future systems. Aligning I/O buffers on memory pages is also known to improve regular buffered I/O performance. Three classes of I/O buffers for regular data pages are adjusted: (1) Heap buffers are now allocated with the new palloc_aligned() or MemoryContextAllocAligned() functions introduced by commit 439f6175. (2) Stack buffers now use a new struct PGIOAlignedBlock to respect PG_IO_ALIGN_SIZE, if possible with this compiler. (3) The buffer pool is also aligned in shared memory. WAL buffers were already aligned on XLOG_BLCKSZ. It's possible for XLOG_BLCKSZ to be configured smaller than PG_IO_ALIGNED_SIZE and thus for O_DIRECT WAL writes to fail to be well aligned, but that's a pre-existing condition and will be addressed by a later commit. BufFiles are not yet addressed (there's no current plan to use O_DIRECT for those, but they could potentially get some incidental speedup even in plain buffered I/O operations through better alignment). If we can't align stack objects suitably using the compiler extensions we know about, we disable the use of O_DIRECT by setting PG_O_DIRECT to 0. This avoids the need to consider systems that have O_DIRECT but can't align stack objects the way we want; such systems could in theory be supported with more work but we don't currently know of any such machines, so it's easier to pretend there is no O_DIRECT support instead. That's an existing and tested class of system. Add assertions that all buffers passed into smgrread(), smgrwrite() and smgrextend() are correctly aligned, unless PG_O_DIRECT is 0 (= stack alignment tricks may be unavailable) or the block size has been set too small to allow arrays of buffers to be all aligned. Author: Thomas Munro <thomas.munro@gmail.com> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> Discussion: https://postgr.es/m/CA+hUKGK1X532hYqJ_MzFWt0n1zt8trz980D79WbjwnT-yYLZpg@mail.gmail.com
* meson: add install-{quiet, world} targetsAndres Freund2023-03-231-1/+1
| | | | | | | To define our own install target, we need dependencies on the i18n targets, which we did not collect so far. Discussion: https://postgr.es/m/3fc3bb9b-f7f8-d442-35c1-ec82280c564a@enterprisedb.com
* Break up long GETTEXT_FILES listsPeter Eisentraut2023-03-081-1/+14
| | | | | | One file per line seems best. We already did this in some cases. This adopts the same format everywhere (except in some cases where the list reasonably fits on one line).
* pg_rewind: Fix determining TLI when server was just promoted.Heikki Linnakangas2023-02-234-59/+64
| | | | | | | | | | | | | | | | | | | | | | | If the source server was just promoted, and it hasn't written the checkpoint record yet, pg_rewind considered the server to be still on the old timeline. Because of that, it would claim incorrectly that no rewind is required. Fix that by looking at minRecoveryPointTLI in the control file in addition to the ThisTimeLineID on the checkpoint. This has been a known issue since forever, and we had worked around it in the regression tests by issuing a checkpoint after each promotion, before running pg_rewind. But that was always quite hacky, so better to fix this properly. This doesn't add any new tests for this, but removes the previously-added workarounds from the existing tests, so that they should occasionally hit this codepath again. This is arguably a bug fix, but don't backpatch because we haven't really treated it as a bug so far. Also, the patch didn't apply cleanly to v13 and below. I'm sure sure it could be made to work on v13, but doesn't seem worth the risk and effort. Reviewed-by: Kyotaro Horiguchi, Ibrar Ahmed, Aleksander Alekseev Discussion: https://www.postgresql.org/message-id/9f568c97-87fe-a716-bd39-65299b8a60f4%40iki.fi
* Add wait_for_replay_catchup wrapper to Cluster.pmAlvaro Herrera2023-02-131-1/+1
| | | | | | | This simplifies a few lines of Perl test code a bit. Author: Bertrand Drouvot Discussion: https://postgr.es/m/846724b5-0723-f4c2-8b13-75301ec7509e@gmail.com
* Update copyright for 2023Bruce Momjian2023-01-0226-26/+26
| | | | Backpatch-through: 11
* Add copyright notices to meson filesAndrew Dunstan2022-12-202-0/+4
| | | | Discussion: https://postgr.es/m/222b43a5-2fb3-2c1b-9cd0-375d376c8246@dunslane.net
* meson: Add windows resource filesAndres Freund2022-10-051-0/+6
| | | | | | | | | | | | | The generated resource files aren't exactly the same ones as the old buildsystems generate. Previously "InternalName" and "OriginalFileName" were mostly wrong / not set (despite being required), but that was hard to fix in at least the make build. Additionally, the meson build falls back to a "auto-generated" description when not set, and doesn't set it in a few cases - unlikely that anybody looks at these descriptions in detail. Author: Andres Freund <andres@anarazel.de> Author: Nazir Bilal Yavuz <byavuz81@gmail.com> Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
* Revert 56-bit relfilenode change and follow-up commits.Robert Haas2022-09-281-3/+3
| | | | | | | | There are still some alignment-related failures in the buildfarm, which might or might not be able to be fixed quickly, but I've also just realized that it increased the size of many WAL records by 4 bytes because a block reference contains a RelFileLocator. The effect of that hasn't been studied or discussed, so revert for now.
* Increase width of RelFileNumbers from 32 bits to 56 bits.Robert Haas2022-09-271-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | RelFileNumbers are now assigned using a separate counter, instead of being assigned from the OID counter. This counter never wraps around: if all 2^56 possible RelFileNumbers are used, an internal error occurs. As the cluster is limited to 2^64 total bytes of WAL, this limitation should not cause a problem in practice. If the counter were 64 bits wide rather than 56 bits wide, we would need to increase the width of the BufferTag, which might adversely impact buffer lookup performance. Also, this lets us use bigint for pg_class.relfilenode and other places where these values are exposed at the SQL level without worrying about overflow. This should remove the need to keep "tombstone" files around until the next checkpoint when relations are removed. We do that to keep RelFileNumbers from being recycled, but now that won't happen anyway. However, this patch doesn't actually change anything in this area; it just makes it possible for a future patch to do so. Dilip Kumar, based on an idea from Andres Freund, who also reviewed some earlier versions of the patch. Further review and some wordsmithing by me. Also reviewed at various points by Ashutosh Sharma, Vignesh C, Amul Sul, Álvaro Herrera, and Tom Lane. Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com
* meson: Add initial version of meson based build systemAndres Freund2022-09-212-0/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Autoconf is showing its age, fewer and fewer contributors know how to wrangle it. Recursive make has a lot of hard to resolve dependency issues and slow incremental rebuilds. Our home-grown MSVC build system is hard to maintain for developers not using Windows and runs tests serially. While these and other issues could individually be addressed with incremental improvements, together they seem best addressed by moving to a more modern build system. After evaluating different build system choices, we chose to use meson, to a good degree based on the adoption by other open source projects. We decided that it's more realistic to commit a relatively early version of the new build system and mature it in tree. This commit adds an initial version of a meson based build system. It supports building postgres on at least AIX, FreeBSD, Linux, macOS, NetBSD, OpenBSD, Solaris and Windows (however only gcc is supported on aix, solaris). For Windows/MSVC postgres can now be built with ninja (faster, particularly for incremental builds) and msbuild (supporting the visual studio GUI, but building slower). Several aspects (e.g. Windows rc file generation, PGXS compatibility, LLVM bitcode generation, documentation adjustments) are done in subsequent commits requiring further review. Other aspects (e.g. not installing test-only extensions) are not yet addressed. When building on Windows with msbuild, builds are slower when using a visual studio version older than 2019, because those versions do not support MultiToolTask, required by meson for intra-target parallelism. The plan is to remove the MSVC specific build system in src/tools/msvc soon after reaching feature parity. However, we're not planning to remove the autoconf/make build system in the near future. Likely we're going to keep at least the parts required for PGXS to keep working around until all supported versions build with meson. Some initial help for postgres developers is at https://wiki.postgresql.org/wiki/Meson With contributions from Thomas Munro, John Naylor, Stone Tickle and others. Author: Andres Freund <andres@anarazel.de> Author: Nazir Bilal Yavuz <byavuz81@gmail.com> Author: Peter Eisentraut <peter@eisentraut.org> Reviewed-By: Peter Eisentraut <peter.eisentraut@enterprisedb.com> Discussion: https://postgr.es/m/20211012083721.hvixq4pnh2pixr3j@alap3.anarazel.de
* Harmonize more parameter names in bulk.Peter Geoghegan2022-09-202-3/+3
| | | | | | | | | | | | | | | | Make sure that function declarations use names that exactly match the corresponding names from function definitions in optimizer, parser, utility, libpq, and "commands" code, as well as in remaining library code. Do the same for all code related to frontend programs (with the exception of pg_dump/pg_dumpall related code). Like other recent commits that cleaned up function parameter names, this commit was written with help from clang-tidy. Later commits will handle ecpg and pg_dump/pg_dumpall. Author: Peter Geoghegan <pg@bowt.ie> Reviewed-By: David Rowley <dgrowleyml@gmail.com> Discussion: https://postgr.es/m/CAH2-WznJt9CMM9KJTMjJh_zbL5hD9oX44qdJ4aqZtjFi-zA3Tg@mail.gmail.com
* Clean up inconsistent use of fflush().Tom Lane2022-08-291-0/+1
| | | | | | | | | | | | | | | | | | | | | | More than twenty years ago (79fcde48b), we hacked the postmaster to avoid a core-dump on systems that didn't support fflush(NULL). We've mostly, though not completely, hewed to that rule ever since. But such systems are surely gone in the wild, so in the spirit of cleaning out no-longer-needed portability hacks let's get rid of multiple per-file fflush() calls in favor of using fflush(NULL). Also, we were fairly inconsistent about whether to fflush() before popen() and system() calls. While we've received no bug reports about that, it seems likely that at least some of these call sites are at risk of odd behavior, such as error messages appearing in an unexpected order. Rather than expend a lot of brain cells figuring out which places are at hazard, let's just establish a uniform coding rule that we should fflush(NULL) before these calls. A no-op fflush() is surely of trivial cost compared to launching a sub-process via a shell; while if it's not a no-op then we likely need it. Discussion: https://postgr.es/m/2923412.1661722825@sss.pgh.pa.us
* Replace pgwin32_is_junction() with lstat().Thomas Munro2022-08-061-4/+0
| | | | | | | | | | | | | | | | | | | Now that lstat() reports junction points with S_IFLNK/S_ISLINK(), and unlink() can unlink them, there is no need for conditional code for Windows in a few places. That was expressed by testing for WIN32 or S_ISLNK, which we can now constant-fold. The coding around pgwin32_is_junction() was a bit suspect anyway, as we never checked for errors, and we also know that errors can be spuriously reported because of transient sharing violations on this OS. The lstat()-based code has handling for that. This also reverts 4fc6b6ee on master only. That was done because lstat() didn't previously work for symlinks (junction points), but now it does. Tested-by: Andrew Dunstan <andrew@dunslane.net> Discussion: https://postgr.es/m/CA%2BhUKGLfOOeyZpm5ByVcAt7x5Pn-%3DxGRNCvgiUPVVzjFLtnY0w%40mail.gmail.com
* Remove configure probes for symlink/readlink, and dead code.Thomas Munro2022-08-051-5/+0
| | | | | | | | | | | | | | | | | | | symlink() and readlink() are in SUSv2 and all targeted Unix systems have them. We have partial emulation on Windows. Code that raised runtime errors on systems without it has been dead for years, so we can remove that and also references to such systems in the documentation. Define HAVE_READLINK and HAVE_SYMLINK macros on Unix. Our Windows replacement functions based on junction points can't be used for relative paths or for non-directories, so the macros can be used to check for full symlink support. The places that deal with tablespaces can just use symlink functions without checking the macros. (If they did check the macros, they'd need to provide an #else branch with a runtime or compile time error, and it'd be dead code.) Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Reviewed-by: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/CA+hUKGJ3LHeP9w5Fgzdr4G8AnEtJ=z=p6hGDEm4qYGEUX5B6fQ@mail.gmail.com
* Revert "Use wildcards instead of manually-maintained file lists in */nls.mk."Tom Lane2022-07-131-1/+1
| | | | | | | This reverts commit 617d69141220f277170927e03a19d2f1b77aed77. While I still think the basic idea is attractive, we need to sort out what happens with built .c files, and there also seem to be VPATH issues.
* Use wildcards instead of manually-maintained file lists in */nls.mk.Tom Lane2022-07-131-1/+1
| | | | | | | | | | | | | | | | | | The backend already used a mechanically-generated list of *.c files, but everywhere else we had a manually-written-out list of files in which to seek translatable messages. Commit b0a55e432 contains the latest in a long line of failures to update those lists. Rather than manually fix its oversight, let's change to using "$(wildcard *.c)" in all these nls.mk files. Many of these files also have manual references to some *.c files in other directories, most often src/common/. Perhaps we should try to improve that situation too; but it's a bit less clear how, so for now just fix the local file references. Kyotaro Horiguchi and Tom Lane Discussion: https://postgr.es/m/20220713.160853.453362706160476128.horikyota.ntt@gmail.com
* NLS: Put list of available languages into LINGUAS filesPeter Eisentraut2022-07-132-1/+1
| | | | | | | | | | | | | | | | | | | | | This moves the list of available languages from nls.mk into a separate file called po/LINGUAS. Advantages: - It keeps the parts notionally managed by programmers (nls.mk) separate from the parts notionally managed by translators (LINGUAS). - It's the standard practice recommended by the Gettext manual nowadays. - The Meson build system also supports this layout (and of course doesn't know anything about our custom nls.mk), so this would enable sharing the list of languages between the two build systems. (The MSVC build system currently finds all po files by globbing, so it is not affected by this change.) Reviewed-by: Andres Freund <andres@anarazel.de> Discussion: https://www.postgresql.org/message-id/flat/557a9f5c-e871-edc7-2f58-a4140fb65b7b@enterprisedb.com
* Change internal RelFileNode references to RelFileNumber or RelFileLocator.Robert Haas2022-07-065-24/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We have been using the term RelFileNode to refer to either (1) the integer that is used to name the sequence of files for a certain relation within the directory set aside for that tablespace/database combination; or (2) that value plus the OIDs of the tablespace and database; or occasionally (3) the whole series of files created for a relation based on those values. Using the same name for more than one thing is confusing. Replace RelFileNode with RelFileNumber when we're talking about just the single number, i.e. (1) from above, and with RelFileLocator when we're talking about all the things that are needed to locate a relation's files on disk, i.e. (2) from above. In the places where we refer to (3) as a relfilenode, instead refer to "relation storage". Since there is a ton of SQL code in the world that knows about pg_class.relfilenode, don't change the name of that column, or of other SQL-facing things that derive their name from it. On the other hand, do adjust closely-related internal terminology. For example, the structure member names dbNode and spcNode appear to be derived from the fact that the structure itself was called RelFileNode, so change those to dbOid and spcOid. Likewise, various variables with names like rnode and relnode get renamed appropriately, according to how they're being used in context. Hopefully, this is clearer than before. It is also preparation for future patches that intend to widen the relfilenumber fields from its current width of 32 bits. Variables that store a relfilenumber are now declared as type RelFileNumber rather than type Oid; right now, these are the same, but that can now more easily be changed. Dilip Kumar, per an idea from me. Reviewed also by Andres Freund. I fixed some whitespace issues, changed a couple of words in a comment, and made one other minor correction. Discussion: http://postgr.es/m/CA+TgmoamOtXbVAQf9hWFzonUo6bhhjS6toZQd7HZ-pmojtAmag@mail.gmail.com Discussion: http://postgr.es/m/CA+Tgmobp7+7kmi4gkq7Y+4AM9fTvL+O1oQ4-5gFTT+6Ng-dQ=g@mail.gmail.com Discussion: http://postgr.es/m/CAFiTN-vTe79M8uDH1yprOU64MNFE+R3ODRuA+JWf27JbhY4hJw@mail.gmail.com
* Translation updatesPeter Eisentraut2022-05-1610-1663/+2977
| | | | | Source-Git-URL: https://git.postgresql.org/git/pgtranslation/messages.git Source-Git-Hash: dde45df385dab9032155c1f867b677d55695310c
* Pre-beta mechanical code beautification.Tom Lane2022-05-124-11/+16
| | | | | Run pgindent, pgperltidy, and reformat-dat-files. I manually fixed a couple of comments that pgindent uglified.
* Fix typos and grammar in code and test commentsMichael Paquier2022-05-112-2/+2
| | | | | | | | This fixes the grammar of some comments in a couple of tests (SQL and TAP), and in some C files. Author: Justin Pryzby Discussion: https://postgr.es/m/20220511020334.GH19626@telsasoft.com
* Remove extraneous blank lines before block-closing bracesAlvaro Herrera2022-04-131-1/+0
| | | | | | | | | These are useless and distracting. We wouldn't have written the code with them to begin with, so there's no reason to keep them. Author: Justin Pryzby <pryzby@telsasoft.com> Discussion: https://postgr.es/m/20220411020336.GB26620@telsasoft.com Discussion: https://postgr.es/m/attachment/133167/0016-Extraneous-blank-lines.patch
* Make XLogRecGetBlockTag() throw error if there's no such block.Tom Lane2022-04-111-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | All but a few existing callers assume without checking that this function succeeds. While it probably will, that's a poor excuse for not checking. Let's make it return void and instead throw an error if it doesn't find the block reference. Callers that actually need to handle the no-such-block case must now use the underlying function XLogRecGetBlockTagExtended. In addition to being a bit less error-prone, this should also serve to suppress some Coverity complaints about XLogRecGetBlockRefInfo. While at it, clean up some inconsistency about use of the XLogRecHasBlockRef macro: make XLogRecGetBlockTagExtended use that instead of open-coding the same condition, and avoid calling XLogRecHasBlockRef twice in relevant code paths. (That is, calling XLogRecHasBlockRef followed by XLogRecGetBlockTag is now deprecated: use XLogRecGetBlockTagExtended instead.) Patch HEAD only; this doesn't seem to have enough value to consider a back-branch API break. Discussion: https://postgr.es/m/425039.1649701221@sss.pgh.pa.us
* Put new command-line options into alphabetical order in help outputPeter Eisentraut2022-04-111-2/+2
|
* Improve frontend error logging style.Tom Lane2022-04-085-42/+28
| | | | | | | | | | | | | | | | | | | | | | | | Get rid of the separate "FATAL" log level, as it was applied so inconsistently as to be meaningless. This mostly involves s/pg_log_fatal/pg_log_error/g. Create a macro pg_fatal() to handle the common use-case of pg_log_error() immediately followed by exit(1). Various modules had already invented either this or equivalent macros; standardize on pg_fatal() and apply it where possible. Invent the ability to add "detail" and "hint" messages to a frontend message, much as we have long had in the backend. Except where rewording was needed to convert existing coding to detail/hint style, I have (mostly) resisted the temptation to change existing message wording. Patch by me. Design and patch reviewed at various stages by Robert Haas, Kyotaro Horiguchi, Peter Eisentraut and Daniel Gustafsson. Discussion: https://postgr.es/m/1363732.1636496441@sss.pgh.pa.us
* Custom WAL Resource Managers.Jeff Davis2022-04-061-4/+7
| | | | | | | | | | | | | Allow extensions to specify a new custom resource manager (rmgr), which allows specialized WAL. This is meant to be used by a Table Access Method or Index Access Method. Prior to this commit, only Generic WAL was available, which offers support for recovery and physical replication but not logical replication. Reviewed-by: Julien Rouhaud, Bharath Rupireddy, Andres Freund Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com
* pgstat: remove stats_temp_directory.Andres Freund2022-04-061-3/+2
| | | | | | | | | | | | | | | | With stats now being stored in shared memory, the GUC isn't needed anymore. However, the pg_stat_tmp directory and PG_STAT_TMP_DIR define are kept, as pg_stat_statements (and some out-of-core extensions) store data in it. Docs will be updated in a subsequent commit, together with the other pending docs updates due to shared memory stats. Author: Andres Freund <andres@anarazel.de> Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Reviewed-By: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://postgr.es/m/20220330233550.eiwsbearu6xhuqwe@alap3.anarazel.de Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de
* Add option --config-file to pg_rewindMichael Paquier2022-04-072-4/+32
| | | | | | | | | | | | | | | | | | | | | This option is useful to do a rewind with the server configuration file (aka postgresql.conf) located outside the data directory, which is something that some Linux distributions and some HA tools like to rely on. As a result, this can simplify the logic around a rewind by avoiding the copy of such files before running pg_rewind. This option affects pg_rewind when it internally starts the target cluster with some "postgres" commands, adding -c config_file=FILE to the command strings generated, when: - retrieving a restore_command using a "postgres -C" command for -c/--restore-target-wal. - forcing crash recovery once to get the cluster into a clean shutdown state. Author: Gunnar "Nick" Bluth Reviewed-by: Michael Banck, Alexander Kukushkin, Michael Paquier, Alexander Alekseev Discussion: https://postgr.es/m/7c59265d-ac50-b0aa-ca1e-65e8bd27642a@pro-open.de
* Remove exclusive backup modeStephen Frost2022-04-061-3/+3
| | | | | | | | | | | | | | | | | | | | | | Exclusive-mode backups have been deprecated since 9.6 (when non-exclusive backups were introduced) due to the issues they can cause should the system crash while one is running and generally because non-exclusive provides a much better interface. Further, exclusive backup mode wasn't really being tested (nor was most of the related code- like being able to log in just to stop an exclusive backup and the bits of the state machine related to that) and having to possibly deal with an exclusive backup and the backup_label file existing during pg_basebackup, pg_rewind, etc, added other complexities that we are better off without. This patch removes the exclusive backup mode, the various special cases for dealing with it, and greatly simplifies the online backup code and documentation. Authors: David Steele, Nathan Bossart Reviewed-by: Chapman Flack Discussion: https://postgr.es/m/ac7339ca-3718-3c93-929f-99e725d1172c@pgmasters.net https://postgr.es/m/CAHg+QDfiM+WU61tF6=nPZocMZvHDzCK47Kneyb0ZRULYzV5sKQ@mail.gmail.com
* Fix compilerwarning in logging size_tDaniel Gustafsson2022-04-051-2/+2
| | | | | | | | | | | The pg_fatal log which included filesizes were using UINT64_FORMAT for the size_t variables, which failed on 32 bit buildfarm animals. Change to using plain int instead, which is in line with how digestControlFile is doing it already. Per buildfarm animals florican and lapwing. Discussion: https://postgr.es/m/13C2BF64-4A6D-47E4-9181-3A658F00C9B7@yesql.se
* pg_rewind: Fetch small files according to new size.Daniel Gustafsson2022-04-055-14/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There's a race condition if a file changes in the source system after we have collected the file list. If the file becomes larger, we only fetched up to its original size. That can easily result in a truncated file. That's not a problem for relation files, files in pg_xact, etc. because any actions on them will be replayed from the WAL. However, configuration files are affected. This commit mitigates the race condition by fetching small files in whole, even if they have grown. A test is added in which an extra file copied is concurrently grown with the output of pg_rewind thus guaranteeing it to have changed in size during the operation. This is not a full fix: we still believe the original file size for files larger than 1 MB. That should be enough for configuration files, and doing more than that would require big changes to the chunking logic in libpq_source.c. This mitigates the race condition if the file is modified between the original scan of files and copying the file, but there's still a race condition if a file is changed while it's being copied. That's a much smaller window, though, and pg_basebackup has the same issue. This race can be seen with pg_auto_failover, which frequently uses ALTER SYSTEM, which updates postgresql.auto.conf. Often, pg_rewind will fail, because the postgresql.auto.conf file changed concurrently and a partial version of it was copied to the target. The partial file would fail to parse, preventing the server from starting up. Author: Heikki Linnakangas Reviewed-by: Cary Huang Discussion: https://postgr.es/m/f67feb24-5833-88cb-1020-19a4a2b83ac7%40iki.fi
* Add new block-by-block strategy for CREATE DATABASE.Robert Haas2022-03-291-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Because this strategy logs changes on a block-by-block basis, it avoids the need to checkpoint before and after the operation. However, because it logs each changed block individually, it might generate a lot of extra write-ahead logging if the template database is large. Therefore, the older strategy remains available via a new STRATEGY parameter to CREATE DATABASE, and a corresponding --strategy option to createdb. Somewhat controversially, this patch assembles the list of relations to be copied to the new database by reading the pg_class relation of the template database. Cross-database access like this isn't normally possible, but it can be made to work here because there can't be any connections to the database being copied, nor can it contain any in-doubt transactions. Even so, we have to use lower-level interfaces than normal, since the table scan and relcache interfaces will not work for a database to which we're not connected. The advantage of this approach is that we do not need to rely on the filesystem to determine what ought to be copied, but instead on PostgreSQL's own knowledge of the database structure. This avoids, for example, copying stray files that happen to be located in the source database directory. Dilip Kumar, with a fairly large number of cosmetic changes by me. Reviewed and tested by Ashutosh Sharma, Andres Freund, John Naylor, Greg Nancarrow, Neha Sharma. Additional feedback from Bruce Momjian, Heikki Linnakangas, Julien Rouhaud, Adam Brusselback, Kyotaro Horiguchi, Tomas Vondra, Andrew Dunstan, Álvaro Herrera, and others. Discussion: http://postgr.es/m/CA+TgmoYtcdxBjLh31DLxUXHxFVMPGzrU5_T=CYCvRyFHywSBUQ@mail.gmail.com
* Remove more unused module imports from TAP testsDaniel Gustafsson2022-03-271-1/+0
| | | | | | | | | | | | | | | | | This is a follow-up to commit 7dac61402 which removed a set of unused modules from the TAP test. The Config references in the pg_ctl and pg_rewind tests were removed in commit 1c6d46293. Fcntl ':mode' and File::stat in the pg_ctl test were added in c37b3d08c which was probably a leftover from an earlier version of the patch, as the function using these was added to another module in that commit. The Config reference in the ldap test was added in ee56c3b21 which in turn use $^O instead of interrogating Config. Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> Discussion: https://postgr.es/m/87lewyqk45.fsf@wibble.ilmari.org
* pg_upgrade: Upgrade an Assert to a real 'if' testAlvaro Herrera2022-03-231-1/+3
| | | | | | | | | It seems possible for the condition being tested to be true in production, and nobody would never know (except when some data eventually becomes corrupt?). Author: Álvaro Herrera <alvherre@alvh.no-ip.org> Discussion: https://postgr.es/m//202109040001.zky3wgv2qeqg@alvherre.pgsql
* Add circular WAL decoding buffer, take II.Thomas Munro2022-03-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Teach xlogreader.c to decode the WAL into a circular buffer. This will support optimizations based on looking ahead, to follow in a later commit. * XLogReadRecord() works as before, decoding records one by one, and allowing them to be examined via the traditional XLogRecGetXXX() macros and certain traditional members like xlogreader->ReadRecPtr. * An alternative new interface XLogReadAhead()/XLogNextRecord() is added that returns pointers to DecodedXLogRecord objects so that it's now possible to look ahead in the WAL stream while replaying. * In order to be able to use the new interface effectively while streaming data, support is added for the page_read() callback to respond to a new nonblocking mode with XLREAD_WOULDBLOCK instead of waiting for more data to arrive. No direct user of the new interface is included in this commit, though XLogReadRecord() uses it internally. Existing code doesn't need to change, except in a few places where it was accessing reader internals directly and now needs to go through accessor macros. Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Reviewed-by: Tomas Vondra <tomas.vondra@enterprisedb.com> Reviewed-by: Andres Freund <andres@anarazel.de> (earlier versions) Discussion: https://postgr.es/m/CA+hUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq=AovOddfHpA@mail.gmail.com
* Rework internal command generation of pg_rewindMichael Paquier2022-03-011-10/+33
| | | | | | | | | | | | | | | | | | | | | | | | pg_rewind generates and executes internally up to two commands to work on the target cluster, depending on the options given by its caller: - postgres -C to retrieve the value of restore_command, when using -c/--restore-target-wal. - postgres --single to enforce recovery once and get the target cluster in a clean shutdown state. Both commands have been applying incorrect quoting rules, which could lead to failures when for example using a target data directory with unexpected characters like CRLFs. Those commands are now generated with PQExpBuffer, making use of string_utils.h to quote those commands as they should. We may extend those commands in the future with more options, so this makes any upcoming additions easier. This is arguably a bug fix, but nobody has complained about the existing code being a problem either, so no backpatch is done. Extracted from a larger patch by the same author. Author: Gunnar "Nick" Bluth Discussion: https://postgr.es/m/7c59265d-ac50-b0aa-ca1e-65e8bd27642a@pro-open.de
* Use test functions in pg_rewind test moduleDaniel Gustafsson2022-02-231-15/+4
| | | | | | | | | | Commit 61081e75c introduced pg_rewind along with the test suite, which ensured that subroutines didn't incur more than one test to plan. Now that we no longer explicitly plan tests (since 549ec201d), we can use the usual Test::More functions. Reviewed-by: Andrew Dunstan <andrew@dunslane.net> Discussion: https://postgr.es/m/AA527525-F0CC-4AA2-AF98-543CABFDAF59@yesql.se
* Remove most msys special processing in TAP testsAndrew Dunstan2022-02-201-1/+0
| | | | | | | | | | Following migration of Windows buildfarm members running TAP tests to use of ucrt64 perl for those tests, special processing for msys perl is no longer necessary and so is removed. Backpatch to release 10 Discussion: https://postgr.es/m/c65a8781-77ac-ea95-d185-6db291e1baeb@dunslane.net
* Silence minor compiler warnings.Tom Lane2022-02-131-1/+1
| | | | | | | | | | | | | | Depending on compiler version and optimization level, we might get a complaint that lazy_scan_heap's "freespace" is used uninitialized. Compilers not aware that ereport(ERROR) doesn't return complained about bbsink_lz4_new(). Assigning "-1" to a uint64 value has unportable results; fortunately, the value of xlogreadsegno is unimportant when xlogreadfd is -1. (It looks to me like there is no need for xlogreadsegno to be static in the first place, but I didn't venture to change that.)
* Replace Test::More plans with done_testingDaniel Gustafsson2022-02-118-14/+19
| | | | | | | | | | | | | | | | | | | Rather than doing manual book keeping to plan the number of tests to run in each TAP suite, conclude each run with done_testing() summing up the the number of tests that ran. This removes the need for maintaning and updating the plan count at the expense of an accurate count of remaining during the test suite runtime. This patch has been discussed a number of times, often in the context of other patches which updates tests, so a larger number of discussions can be found in the archives. Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Reviewed-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Reviewed-by: Michael Paquier <michael@paquier.xyz> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://postgr.es/m/DD399313-3D56-4666-8079-88949DAC870F@yesql.se
* Make logical decoding a part of the rmgr.Jeff Davis2022-01-191-1/+1
| | | | | | | | | | | Add a new rmgr method, rm_decode, and use that rather than a switch statement. In preparation for rmgr extensibility. Reviewed-by: Julien Rouhaud Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com Discussion: https://postgr.es/m/20220118095332.6xtlcjoyxobv6cbk@jrouhaud
* Clean up TAP tests' usage of wait_for_catchup().Tom Lane2022-01-162-9/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default, wait_for_catchup() waits for the replication connection to reach the primary's write LSN. That's fine, but in an apparent attempt to save one query round-trip, it was coded so that we executed pg_current_wal_lsn() again during each probe query. Thus, we presented the standby with a moving target to be reached. (While the test script itself couldn't be causing the write LSN to advance while it's blocked in wait_for_catchup(), it's plenty plausible that background activity such as autovacuum is emitting more WAL.) That could make the test take longer than necessary, and potentially it could mask bugs by allowing the standby to process more WAL than a strict interpretation of the test scenario allows. So, change wait_for_catchup() to do it "by the book", explicitly collecting the write LSN to wait for at the outset. Also, various call sites were instructing wait_for_catchup() to wait for the standby to reach the primary's insert LSN rather than its write LSN. This also seems like a bad idea. While in most test scenarios those are the same, if they are different then the inserted-but-not-yet-written WAL is not presently available to the standby. The test isn't doing anything to make it become so, so again we have the potential for unwanted test delay, perhaps even a test timeout. (Again, background activity would be needed to make this more than a hypothetical problem.) Hence, change the callers where necessary so that the wait target is always the primary's write LSN. While at it, simplify callers by making use of wait_for_catchup's default arguments wherever possible (the preceding change makes this possible in more places than it was before). And rewrite wait_for_catchup's documentation a bit. Patch by me; thanks to Julien Rouhaud for review. Discussion: https://postgr.es/m/2368336.1641843098@sss.pgh.pa.us
* Update copyright for 2022Bruce Momjian2022-01-0723-23/+23
| | | | Backpatch-through: 10
* Be more specific about OOM in XLogReaderAllocateAlvaro Herrera2021-11-221-3/+3
| | | | | | | | | | | A couple of spots can benefit from an added errdetail(), which matches what we were already doing in other places; and those that cannot withstand errdetail() can get a more descriptive primary message. Author: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Reviewed-by: Julien Rouhaud <rjuju123@gmail.com> Discussion: https://postgr.es/m/CALj2ACV+cX1eM03GfcA=ZMLXh5fSn1X1auJLz3yuS1duPSb9QA@mail.gmail.com
* Move Perl test modules to a better namespaceAndrew Dunstan2021-10-249-27/+27
| | | | | | | | | | | | | | | The five modules in our TAP test framework all had names in the top level namespace. This is unwise because, even though we're not exporting them to CPAN, the names can leak, for example if they are exported by the RPM build process. We therefore move the modules to the PostgreSQL::Test namespace. In the process PostgresNode is renamed to Cluster, and TestLib is renamed to Utils. PostgresVersion becomes simply PostgreSQL::Version, to avoid possible confusion about what it's the version of. Discussion: https://postgr.es/m/aede93a4-7d92-ef26-398f-5094944c2504@dunslane.net Reviewed by Erik Rijkers and Michael Paquier
* Fix incorrect format placeholdersPeter Eisentraut2021-09-151-7/+2
| | | | | | | | Also remove obsolete comments about why the 64-bit integers need to be printed in a separate buffer. The reason used to be portability, but now the remaining reason is that we need the string lengths for the progress displays. That is evident by looking at the code right below, so a new comment doesn't seem necessary.
View Lorry Controller Status