Skip to content

Commit

Permalink
v5.0.1 fix edge case condition when upgrading to version 5
Browse files Browse the repository at this point in the history
  • Loading branch information
keithf4 committed Dec 20, 2023
1 parent 1ec65d6 commit ee8653f
Show file tree
Hide file tree
Showing 10 changed files with 362 additions and 158 deletions.
17 changes: 9 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
5.0.0
=====
5.0.0 & 5.0.1
=============
UPGRADE NOTES
-------------
- The upgrade to 5.x had to be split into two parts to allow certain transaction limiting conditions to be met. Please see [pg_partman_5.0.1_upgrade.md](doc/pg_partman_5.0.1_upgrade.md) if you run into any errors concerning triggers when trying to update. There is no standalone version of 5.0.0 for pg_partman any longer and all users should update to 5.0.1.
- If you have already successfully upgraded to version 5.0.0, the update to 5.0.1 will not introduce any changes. The new minor version is to allow for the staged upgrade.
- It is recommended that you take a backup of the `part_config` and `part_config_sub` tables before upgrading just to ensure they can be restored in case there are any issues. These tables are recreated as part of the upgrade.
- If you see any errors about the following tables existing during an upgrade attempt, please review their content and ensure you do not need any of the backed up pre-5.x configuration data they contain. Drop them if not needed and try the upgrade again: `part_config_pre_500_data`, `part_config_sub_pre_500_data`
- For CHANGELOG notes prior to version 5.0.0, please see [CHANGELOG-pre-5.0.0.txt](CHANGELOG-pre-5.0.0.txt).
- Many thanks to all the people that have helped with testing and code review during 5.x development. In particular...
- Leigh Downs and the team at Crunchy Data for extensive testing
- vitaly-burovoy on Github for some amazing optimizations and code review
Expand All @@ -13,15 +14,15 @@ UPGRADE NOTES

BREAKING CHANGES
----------------
- Removed trigger-based partitioning support. All partitioning is now done using built-in declarative partitioning. The partitioning `type` in pg_partman will now refer to the types of delcarative partitioning that are supported. As of 5.0.0, only `range` is supported, but others are in development (Github Issue #490).
- Removed trigger-based partitioning support. All partitioning is now done using built-in declarative partitioning. The partitioning `type` in pg_partman will now refer to the types of delcarative partitioning that are supported. As of 5.0.1, only `range` is supported, but others are in development (Github Issue #490).
- See [migrate_to_declarative.md](doc/migrate_to_declarative.md) for assistance on migrating off trigger-based partitioning.
- Many functions have had their parameters altered, renamed, rearranged or removed. These should be more consistent across the code-base now. Please review ALL calls to pg_partman functions to ensure that your parameter names and values have been updated to match the changes.
- The `part_config` and `part_config_sub` tables have had some columns removed and rearranged to only contain supported features.
- Due to a majority of extension objects being dropped & recreated, privileges on the extension objects ARE NOT being preserved as they have been done with past extension updates. Ensure existing privileges are recorded before upgrading pg_partman and are regranted/revoked after the upgrade is complete. Check the following system catalogs for privilege information for pg_partman objects: `information_schema.routine_privileges` & `information_schema.table_privileges`
- Some specialized time-based interval types have been deprecated. See [pg_partman_5.0.0_upgrade.md](doc/pg_partman_5.0.0_upgrade.md) for additional guidance on migrating unsupported partition methods to supported ones.
- Some specialized time-based interval types have been deprecated. See [pg_partman_5.0.1_upgrade.md](doc/pg_partman_5.0.1_upgrade.md) for additional guidance on migrating unsupported partition methods to supported ones.
- All time-based interval values must now be valid values for the interval data type. The previous weekly, hourly, daily, etc interval values are no longer supported.
- Removed specialized quarterly partitioning (see [5.0.0 migration doc](doc/pg_partman_5.0.0_upgrade.md)).
- Removed specialized weekly partitioning with ISO style week numbers (see [5.0.0 migration doc](doc/pg_partman_5.0.0_upgrade.md)).
- Removed specialized quarterly partitioning (see [5.0.1 migration doc](doc/pg_partman_5.0.1_upgrade.md)).
- Removed specialized weekly partitioning with ISO style week numbers (see [5.0.1 migration doc](doc/pg_partman_5.0.1_upgrade.md)).
- Hourly partitioning now has seconds on the child partition suffix. Migration for this is not necessary, but just be aware that any new partition sets created with this interval may look different than existing ones from prior pg_partman versions.
- The minimum required version of PostgreSQL is now 14
- Required for calling procedures via background worker (Github PR #242).
Expand All @@ -30,7 +31,7 @@ BREAKING CHANGES
NEW FEATURES
------------
- Changed all usage of the term "native" to "declarative" to better match upstream PostgreSQL terminology for built-in partitioning.
- Simplified all time-based partitioning suffixes to `YYYYMMDD` for intervals greater than or equal to 1 day and `YYYYMMDD_HH24MISS` for intervals less than 1 day. Removed extra underscores to allow longer base partition names. Existing partition suffixes will still be supported, but newly created partition sets will use the new naming patterns by default. It is recommended that migration to the new suffixes is done when possible to ensure future support of possible pg_partman changes. The [documentation](doc/pg_partman_5.0.0_upgrade.md) on migrating the old specialized weekly/quarterly partition sets to be supported in 5.0.0 can be used as guidance for migrating other child table names as well.
- Simplified all time-based partitioning suffixes to `YYYYMMDD` for intervals greater than or equal to 1 day and `YYYYMMDD_HH24MISS` for intervals less than 1 day. Removed extra underscores to allow longer base partition names. Existing partition suffixes will still be supported, but newly created partition sets will use the new naming patterns by default. It is recommended that migration to the new suffixes is done when possible to ensure future support of possible pg_partman changes. The [documentation](doc/pg_partman_5.0.1_upgrade.md) on migrating the old specialized weekly/quarterly partition sets to be supported in 5.0.1 can be used as guidance for migrating other child table names as well.
- By default, data in the default partition is now ignored when calculating new child partitions to create. If a new child table's boundaries would include data that exists in the default, this will cause an error during maintenance and must be manually resolved by either removing that data from the default or partitioning it out to the proper child table using the partition_data function/procedure.
- A flag is available to take default data into consideration, but this should only be used in rare circumstances to correct maintenance issues and should not be left permanently enabled for ideal partition maintenance.
- Added option to make default table optional during partition creation (`p_default_table`) (Github Issue #489).
Expand Down
6 changes: 3 additions & 3 deletions META.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "pg_partman",
"abstract": "Extension to manage partitioned tables by time or ID",
"version": "5.0.0",
"version": "5.0.1",
"maintainer": [
"Keith Fiske <keith@keithf4.com>"
],
Expand All @@ -20,9 +20,9 @@
},
"provides": {
"pg_partman": {
"file": "sql/pg_partman--5.0.0.sql",
"file": "sql/pg_partman--5.0.1.sql",
"docfile": "doc/pg_partman.md",
"version": "5.0.0",
"version": "5.0.1",
"abstract": "Extension to manage partitioned tables by time or ID"
}
},
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
PostgreSQL Partition Manager
====================

pg_partman is an extension to create and manage both time-based and number-based table partition sets. As of version 5.0.0, only built-in, declarative partitioning is supported and the older trigger-based methods have been deprecated.
pg_partman is an extension to create and manage both time-based and number-based table partition sets. As of version 5.0.1, only built-in, declarative partitioning is supported and the older trigger-based methods have been deprecated.

The declarative partitioning built into PostgreSQL provides the commands to create a partitioned table and its children. pg_partman uses the built-in declarative features that PostgreSQL provides and builds upon those with additional features and enhancements to make managing partitions easier. One key way that pg_partman extends partitioning in Postgres is by providing a means to automate the child table maintenance over time (Ex. adding new children, dropping old ones based on a retention policy). pg_partman also has features to turn an existing table into a partitioned table or vice versa.

Expand All @@ -24,16 +24,16 @@ The following list of files is found in the [doc](doc) folder of the pg_partman
| [pg_partman_howto.md](doc/pg_partman_howto.md) | A How-To guide for general usage of pg_partman. Provides examples for setting up new partition sets and migrating existing tables to partitioned tables. |
| [migrate_to_partman.md](doc/migrate_to_partman.md) | How to migrate existing partition sets to being managed by pg_partman. |
| [migrate_to_declarative.md](doc/migrate_to_declarative.md) | How to migrate from trigger-based partitioning to declarative partitioning. |
| [pg_partman_5.0.0_upgrade.md](doc/pg_partman_5.0.0_upgrade.md) | If pg_partman is being upgraded to version 5.x from any prior version, special considerations may need to be made. Please carefully review this document before performing any upgrades to 5.x or higher. |
| [fix_missing_procedures.md](doc/fix_missing_procedures.md) | If pg_partman had been installed prior to PostgreSQL 11 and upgraded since then, it may be missing procedures. This document outlines how to restore those procedures and preserve the current configration. |
| [pg_partman_5.0.1_upgrade.md](doc/pg_partman_5.0.1_upgrade.md) | If pg_partman is being upgraded to version 5.x from any prior version, special considerations may need to be made. Please carefully review this document before performing any upgrades to 5.x or higher. |
| [fix_missing_procedures.md](doc/fix_missing_procedures.md) | If pg_partman had been installed prior to PostgreSQL 11 and upgraded since then, it may be missing procedures. This document outlines how to restore those procedures and preserve the current configuration. |

INSTALLATION
------------
Requirement:
Requirement:

* PostgreSQL >= 14

Recommended:
Recommended:

* [pg_jobmon](https://github.com/omniti-labs/pg_jobmon) (>=v1.4.0). PG Job Monitor will automatically be used if it is installed and setup properly.

Expand Down Expand Up @@ -106,7 +106,7 @@ ALTER EXTENSION pg_partman UPDATE TO '<latest version>';

If you are doing a `pg_dump`/`pg_restore` and you've upgraded pg_partman in place from previous versions, it is recommended you use the `--column-inserts` option when dumping and/or restoring pg_partman's configuration tables. This is due to ordering of the configuration columns possibly being different (upgrades just add the columns onto the end, whereas the default of a new install may be different).

If upgrading between any major versions of pg_partman (4.x -> 5.x, etc), please carefully read all intervening version notes in the [CHANGELOG](CHANGELOG.txt), especially those notes for the major version. There are often additional instructions and other important considerations for the updates. Extra special considerations are needed if you are upgrading to 5+ from any version less than 5.0.0. Please see [pg_partman_5.0.0_upgrade.md](doc/pg_partman_5.0.0_upgrade.md).
If upgrading between any major versions of pg_partman (4.x -> 5.x, etc), please carefully read all intervening version notes in the [CHANGELOG](CHANGELOG.txt), especially those notes for the major version. There are often additional instructions and other important considerations for the updates. Extra special considerations are needed if you are upgrading to 5+ from any version less than 5.0.0. Please see [pg_partman_5.0.1_upgrade.md](doc/pg_partman_5.0.1_upgrade.md).

IMPORTANT NOTE: Some updates to pg_partman must drop and recreate its own database objects. If you are revoking PUBLIC privileges from functions/procedures, that can be added back to objects that are recreated as part of an update. If restrictions from PUBLIC use are desired for pg_partman, it is recommended to install it into its own schema as shown above and the revoke undesired access to that schema. Otherwise you may have to add an additional step to your extension upgrade procedures to revoke PUBLIC access again.

Expand Down
2 changes: 1 addition & 1 deletion doc/migrate_to_partman.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ Rename the existing partitions to new naming convention. pg_partman uses a stati
_pYYYYMMDD_HH24MISS - All time intervals less than 1 day
_p##### - Serial/ID partition has a suffix that is the value of the lowest possible entry in that table (Ex: _p10, _p20000, etc)

You can use custom datetime_string formats to change the suffix the children will get with pg_partman 5.0.0 and greater, but that is not covered in this tutorial. This only covers how to convert to using pg_partman's default suffixes and pg_partman always adds `_p` to the beginning to distinguish the suffix boundary.
You can use custom datetime_string formats to change the suffix the children will get with pg_partman 5.0.1 and greater, but that is not covered in this tutorial. This only covers how to convert to using pg_partman's default suffixes and pg_partman always adds `_p` to the beginning to distinguish the suffix boundary.

Step 3a
-------
Expand Down
2 changes: 1 addition & 1 deletion doc/pg_partman.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ About
-----
PostgreSQL Partition Manager is an extension to help make managing time or number/id based table partitioning easier. It has many options, but usually only a few are needed, so it's much easier to use than it may first appear (and definitely easier than implementing it yourself).

As of version 5.0.0, the minimum version of PostgreSQL required is 14 and trigger-based partitioning is no longer supported. All partitioning is done using built-in declarative partitioning. Currently only ranged partitioning is supported for time- and number-based intervals. Version 4.x of pg_partman, which still has trigger-based support, is no longer in active development and will only be receiving critical bug fixes for a limited time. If partitioning is a critical part of your infrastructure, please make plans to upgrade in the near future.
As of version 5.0.1, the minimum version of PostgreSQL required is 14 and trigger-based partitioning is no longer supported. All partitioning is done using built-in declarative partitioning. Currently only ranged partitioning is supported for time- and number-based intervals. Version 4.x of pg_partman, which still has trigger-based support, is no longer in active development and will only be receiving critical bug fixes for a limited time. If partitioning is a critical part of your infrastructure, please make plans to upgrade in the near future.

A default partition to catch data outside the existing child boundaries is automatically created for all partition sets. The `check_default()` function provides monitoring for any data getting inserted into the default table and the `partition_data_`* set of functions can easily partition that data for you if it is valid data. That is much easier than automatically creating new child tables on demand and having to clean up potentially hundreds or thousands of unwanted partitions. And also better than throwing an error and losing the data!

Expand Down
Loading

0 comments on commit ee8653f

Please sign in to comment.