Repository navigation
feature#451: Partition mac_track_ips and rebuild port and IP partitioning with cli/upgrade_partitioning.php - #450
Merged
Conversation
…artitioning.php Replace mactrack_convert.php with cli/upgrade_partitioning.php, which checks mac_track_ports and mac_track_ips against the selected scheme (the Port Partition Size setting, or --partition=daily|weekly|disabled), says so when there is no drift and rebuilds each drifted table in turn: rename it to <table>_rebuild, create it again with the retention and partitioning scheme, copy each scan date, then drop the original. - Partition mac_track_ips by scan date like mac_track_ports: fresh installs, maintenance (partition drop, creation ahead, recovery of stranded rows) and the Utilities status page. - Register the run in the process registry (24 hour timeout), hold the scan lock, resume an interrupted run from the last scan date and print progress. - Show a maintenance message on the MAC and IP address views, defer the poller and maintenance while a rebuild is unfinished. - Save the scheme chosen with --partition as the Port Partition Size setting, which can now be Disabled; drop --keep-old and the -d, -h and -v options from the help (-h and -v still work). - Tombstone mactrack_convert.php in manifest.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unguarded history mutations, maintenance utilities, and database upgrades can operate on partial rebuild tables, restoring deleted data, corrupting derived records, or marking migrations complete before all rows are migrated.
4 open issues
What changed in this PR
Replaces the port-only conversion tool with resumable partition rebuilding for both MAC and IP history, extending MacTrack’s existing scan-date partitioning and retention support.
Changes:
- Adds a CLI that detects partitioning drift and rebuilds tables using daily, weekly, or disabled partitioning.
- Integrates rebuild state with polling, maintenance, Utilities, and search-view notices; adds IP partition maintenance.
- Updates tests, documentation, translation strings, and packaging for the replacement tool.
| File | Description |
|---|---|
tests/Unit/PartitionWebTest.php |
Tests Utilities options, CLI launches, and interrupted-run handling. |
tests/Unit/PartitionSettingTest.php |
Adds the Disabled setting expectation. |
tests/Unit/PartitionRebuildTest.php |
Covers rebuild planning, copying, recovery, and setup. |
tests/Unit/PartitionMaintainTest.php |
Adds IP partition retention and recovery coverage. |
tests/Unit/PartitionConvertTest.php |
Removes the superseded conversion test suite. |
tests/Unit/ArpDot1xRetentionTest.php |
Updates the shared IP-retention assertion. |
tests/Unit/ActivityTest.php |
Covers interrupted rebuild and poller-deferral reporting. |
tests/Security/PreparedStatementConsistencyTest.php |
Removes the retired script’s baseline entry. |
tests/bin/patch-coverage.php |
Removes the retired conversion entry point. |
setup.php |
Adds Disabled partitioning and updates setting guidance. |
README.md |
Documents rebuild commands, recovery, and operational constraints. |
poller_mactrack.php |
Defers polling while rebuild source tables remain. |
phpunit.xml |
Includes replacement rebuild sources in coverage. |
manifest.json |
Marks the old CLI for upgrade removal. |
mactrack_view_macs.php |
Adds rebuild maintenance notices to MAC views. |
mactrack_view_arp.php |
Adds the rebuild maintenance notice to IP searching. |
mactrack_utilities.php |
Launches the new CLI and displays IP partition status. |
mactrack_convert.php |
Removes the old conversion CLI. |
locales/po/cacti.pot |
Refreshes partitioning and maintenance-message translations. |
lib/mactrack_partitions.php |
Generalizes partition SQL helpers to allowlisted history tables. |
lib/mactrack_partition_rebuild.php |
Adds shared setup, rebuild-state, notice, and retention helpers. |
lib/mactrack_partition_ops.php |
Extends partition maintenance to IP history. |
lib/mactrack_partition_convert.php |
Removes the former conversion implementation. |
lib/mactrack_functions.php |
Loads rebuild helpers and integrates maintenance deferral and IP retention. |
lib/mactrack_activity.php |
Reports active or unfinished partition rebuilds. |
includes/database.php |
Partitions newly created IP tables and generalizes process-status checks. |
cli/upgrade_partitioning.php |
Implements drift assessment and resumable table rebuilding. |
cli/upgrade_database.php |
Updates a comment to name the replacement CLI. |
CHANGELOG.md |
Records the feature and updates the earlier partitioning entry. |
.github/copilot-instructions.md |
Updates the documented project layout for CLI maintenance scripts. |
Review effort: Balanced
… after - Remember in mt_db_maint_missed that the daily database maintenance came due (or was skipped) while a rebuild was under way; the poller runs it in its first cycle after the rebuild instead of waiting for the next window. - Describe setting up and maintaining partitioning in the README. - Order the 5.0 change log security, issue, feature, dev and reference feature#451. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…areas The global, site and device Ports to Ignore (Regular Expression) fields are text areas with the same characteristics as the device Ports to Ignore (3 rows, 80 columns, textAreaNotes, 255 characters). Each line is its own expression, so a line break no longer becomes a literal that never matches a port name. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…a rebuild Address review of the partitioning rebuild: - Hold the partition lock, which a rebuild holds for its whole run, across every change to the history (device and site removal, MAC authorization and revocation, MacAuth save and removal, and the Utilities that truncate the tables, recreate the aggregated data or rebuild the scan dates), and refuse while a rebuild is running or unfinished, so nothing is lost or restored when the copy finishes. - Defer the background database upgrade while a rebuild table exists, and have the rebuild recheck for an upgrade once its locks are held. - Partition a recreated IP table by the saved Port Partition Size instead of forcing weekly: Disabled leaves it unpartitioned and a saved size is kept. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
bmfmancini
approved these changes
Nov 17, 2026
xmacan
approved these changes
Nov 17, 2026
3 of 6 tasks
somethingwithproof
left a comment
Member
There was a problem hiding this comment.
No blockers found in the final changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.





Description
Fixes #451.
Partitions the IP address table (
mac_track_ips) by scan date like the port table, and replacesmactrack_convert.phpwithcli/upgrade_partitioning.php, which checks how the MAC address table (mac_track_ports) and the IP address table are partitioned against the selected scheme and rebuilds the ones that have drifted.The script,
cli/upgrade_partitioning.php--partition=daily|weekly|disabled, or, with no option, the Port Partition Size setting.--partitionis saved withset_config_option()before the rebuild starts, so a failed run still leaves the setting as asked and a later run converges. The setting can now also be Disabled. The per-scan and hourly sizes stay with the poller; the script says so and changes nothing for them.dMaxValue) is reported and left alone.<table>_rebuild;scan_dateandINSERT ... SELECTit from the renamed table;register_process_start()(24 h timeout, well over the four hours asked for) and unregistered on every exit path, includingSIGTERM/SIGINT. An interrupted run resumes from the last scan date copied (the partly copied date is redone, and a partly filled new table with another scheme is dropped and created again).--debug, a line before each long step (create, verify, drop), every scan date with--debug.--keep-oldand-dare gone.--helplists--partition,--debug,--versionand--help;-h,-H,-vand-Vare still accepted.mactrack_convert.phpis deleted and added totombstonesinmanifest.json. Utilities (Configure Port Partitioning) starts the new script; its backup checkbox is gone.Pausing scanning and maintenance, and catching up
mactrack_db_maint_run()skips, while any<table>_rebuildtable exists, which covers an interrupted run.mt_db_maint_missed), a skipped maintenance records it too, and the poller runs the maintenance in its first cycle after the rebuild instead of waiting for the next window (the poller's own schedule would otherwise lose a window that crosses midnight). The flag is cleared when the maintenance runs.mt_partition_convertingflag no longer blocks starting the resume from Utilities (found in review); only the live process registration counts.Nothing changes the history while a rebuild is running or unfinished (review)
<table>_rebuildtable is left. Without that, a delete made during the copy would be undone when the original's rows were copied back, and derived tables would be built from a partial history. The lock is taken with a 2 second wait, so these actions never hang behind a rebuild, and they do not depend on the scan lock, so they still work while a scan runs.cli/upgrade_database.php) waits while a<table>_rebuildtable exists, and the rebuild rechecks for a running upgrade once it holds its locks, so a migration is never marked complete against a partly copied table.The IP table
mac_track_ipsis partitioned likemac_track_ports: fresh installs partition it; the daily maintenance drops expired partitions by the IP retention, creates the days ahead (daily or weekly; the per-scan and hourly sizes map to daily) and splits rows stranded indMaxValueinto a recovery partition; Utilities View Partition Status shows its layout. An unpartitioned IP table keeps the existing row-deleting retention.Ports to Ignore (Regular Expression) is a text area
The regular expression field is now a text area, like the device Ports to Ignore, at all three levels: the global Ports to Ignore While Scanning (Regular Expression) setting and the site and device forms. It has the same characteristics (3 rows, 80 columns,
textAreaNotes, 255 characters). Each line is its own expression and a port matching any line is ignored, so a line break cannot become a literal that never matches a port name; a single-line expression is compiled exactly as before. Validation, trimming and the invalid-expression handling are unchanged.Documentation and change log
README.mdnow has a Port and IP history partitioning section: how it works, setting it up (new and existing installs, the setting, retention, the script), checking and changing the scheme, what a rebuild does, what happens while it runs (and after an interruption), ongoing maintenance, and troubleshooting.CHANGELOG.md: the pending section is headed 5.0 and ordered security, issue, feature, dev (entries were interleaved before); the feature is recorded asfeature#451.Related Issue
#451, a feature issue for partitioning
mac_track_ipsand moving the conversion routine (fixed by this pull request). Follows feature#426.Motivation and Context
Changing the partition size or upgrading left tables that no longer match the setting, and the only way to correct them was a one-way copy that could not be resumed, covered the port table only, could not tell whether a table had drifted, and left database maintenance to the next window.
How Has This Been Tested?
tests/bin/patch-coverage.phpreports 100% of the changed lines (newPartitionRebuildTest.php; additions to the maintenance, web, activity and setting tests; the unit tests found a resume bug, fixed).check-i18n-pot.phpandvalidate-manifest.phppass;locales/po/cacti.potis the only catalog touched.develop): unpartitioned to weekly, a no-drift rerun, weekly to daily, to disabled,SIGTERMin the middle of a rebuild followed by a resume, a hand-made half-copied table and resume, fresh install partitioning both tables, and the maintenance message and Process Status state.Types of changes
Checklist:
Notes for reviewers:
mac_track_ports_oldbackups left by the previous--keep-oldare not touched.