pgstat: store statistics in shared memory.

Previously the statistics collector received statistics updates via UDP and
shared statistics data by writing them out to temporary files regularly. These
files can reach tens of megabytes and are written out up to twice a
second. This has repeatedly prevented us from adding additional useful
statistics.

Now statistics are stored in shared memory. Statistics for variable-numbered
objects are stored in a dshash hashtable (backed by dynamic shared
memory). Fixed-numbered stats are stored in plain shared memory.

The header for pgstat.c contains an overview of the architecture.

The stats collector is not needed anymore, remove it.

By utilizing the transactional statistics drop infrastructure introduced in a
prior commit statistics entries cannot "leak" anymore. Previously leaked
statistics were dropped by pgstat_vacuum_stat(), called from [auto-]vacuum. On
systems with many small relations pgstat_vacuum_stat() could be quite
expensive.

Now that replicas drop statistics entries for dropped objects, it is not
necessary anymore to reset stats when starting from a cleanly shut down
replica.

Subsequent commits will perform some further code cleanup, adapt docs and add
tests.

Bumps PGSTAT_FILE_FORMAT_ID.

Author: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Author: Andres Freund <andres@anarazel.de>
Author: Melanie Plageman <melanieplageman@gmail.com>
Reviewed-By: Andres Freund <andres@anarazel.de>
Reviewed-By: Thomas Munro <thomas.munro@gmail.com>
Reviewed-By: Justin Pryzby <pryzby@telsasoft.com>
Reviewed-By: "David G. Johnston" <david.g.johnston@gmail.com>
Reviewed-By: Tomas Vondra <tomas.vondra@2ndquadrant.com> (in a much earlier version)
Reviewed-By: Arthur Zakirov <a.zakirov@postgrespro.ru> (in a much earlier version)
Reviewed-By: Antonin Houska <ah@cybertec.at> (in a much earlier version)
Discussion: https://postgr.es/m/20220303021600.hs34ghqcw6zcokdh@alap3.anarazel.de
Discussion: https://postgr.es/m/20220308205351.2xcn6k4x5yivcxyd@alap3.anarazel.de
Discussion: https://postgr.es/m/20210319235115.y3wz7hpnnrshdyv6@alap3.anarazel.de

2 files changed, 24 insertions, 57 deletions

diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index dc38e16405..566ecbf091 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1933,51 +1933,39 @@ PgFdwPathExtraData
 PgFdwRelationInfo
 PgFdwScanState
 PgIfAddrCallback
+PgStatShared_Archiver
+PgStatShared_BgWriter
+PgStatShared_Checkpointer
+PgStatShared_Common
+PgStatShared_Database
+PgStatShared_Function
+PgStatShared_HashEntry
+PgStatShared_Relation
+PgStatShared_ReplSlot
+PgStatShared_SLRU
+PgStatShared_Subscription
+PgStatShared_Wal
 PgStat_ArchiverStats
 PgStat_BackendFunctionEntry
+PgStat_BackendSubEntry
 PgStat_BgWriterStats
 PgStat_CheckpointerStats
 PgStat_Counter
+PgStat_EntryRef
+PgStat_EntryRefHashEntry
+PgStat_FetchConsistency
 PgStat_FunctionCallUsage
 PgStat_FunctionCounts
-PgStat_FunctionEntry
-PgStat_GlobalStats
+PgStat_HashKey
 PgStat_Kind
-PgStat_Msg
-PgStat_MsgAnalyze
-PgStat_MsgAnlAncestors
-PgStat_MsgArchiver
-PgStat_MsgAutovacStart
-PgStat_MsgBgWriter
-PgStat_MsgCheckpointer
-PgStat_MsgChecksumFailure
-PgStat_MsgConnect
-PgStat_MsgDeadlock
-PgStat_MsgDisconnect
-PgStat_MsgDropdb
-PgStat_MsgDummy
-PgStat_MsgFuncpurge
-PgStat_MsgFuncstat
-PgStat_MsgHdr
-PgStat_MsgInquiry
-PgStat_MsgRecoveryConflict
-PgStat_MsgReplSlot
-PgStat_MsgResetcounter
-PgStat_MsgResetreplslotcounter
-PgStat_MsgResetsharedcounter
-PgStat_MsgResetsinglecounter
-PgStat_MsgResetslrucounter
-PgStat_MsgResetsubcounter
-PgStat_MsgSLRU
-PgStat_MsgSubscriptionDrop
-PgStat_MsgSubscriptionError
-PgStat_MsgTabpurge
-PgStat_MsgTabstat
-PgStat_MsgTempFile
-PgStat_MsgVacuum
-PgStat_MsgWal
+PgStat_KindInfo
+PgStat_LocalState
 PgStat_PendingDroppedStatsItem
+PgStat_ReplSlotStats
 PgStat_SLRUStats
+PgStat_ShmemControl
+PgStat_Snapshot
+PgStat_SnapshotEntry
 PgStat_StatDBEntry
 PgStat_StatFuncEntry
 PgStat_StatReplSlotEntry
@@ -1985,7 +1973,6 @@ PgStat_StatSubEntry
 PgStat_StatTabEntry
 PgStat_SubXactStatus
 PgStat_TableCounts
-PgStat_TableEntry
 PgStat_TableStatus
 PgStat_TableXactStatus
 PgStat_WalStats
@@ -2533,7 +2520,6 @@ StartReplicationCmd
 StartupStatusEnum
 StatEntry
 StatExtEntry
-StatMsgType
 StateFileChunk
 StatisticExtInfo
 Stats
@@ -2647,8 +2633,6 @@ TXNEntryFile
 TYPCATEGORY
 T_Action
 T_WorkerStatus
-TabStatHashEntry
-TabStatusArray
 TableAmRoutine
 TableAttachInfo
 TableDataInfo
@@ -3433,6 +3417,7 @@ pgssHashKey
 pgssSharedState
 pgssStoreKind
 pgssVersion
+pgstat_entry_ref_hash_hash
 pgstat_page
 pgstattuple_type
 pgthreadlock_t
diff --git a/src/tools/valgrind.supp b/src/tools/valgrind.supp
index e3a179d210..4e8c482757 100644
--- a/src/tools/valgrind.supp
+++ b/src/tools/valgrind.supp
@@ -15,24 +15,6 @@
 # those bytes as padding, this is harmless.
 
 {
-	padding_pgstat_send
-	Memcheck:Param
-	socketcall.send(msg)
-
-	fun:*send*
-	fun:pgstat_send
-}
-
-{
-	padding_pgstat_sendto
-	Memcheck:Param
-	socketcall.sendto(msg)
-
-	fun:*send*
-	fun:pgstat_send
-}
-
-{
 	padding_pgstat_write
 	Memcheck:Param
 	write(buf)