Skip to content

feature#451: Partition mac_track_ips and rebuild port and IP partitioning with cli/upgrade_partitioning.php - #450

Merged
TheWitness merged 4 commits into
developfrom
feat/mactrack-upgrade-partitioning-cli
Nov 17, 2026
Merged

TheWitness merged 4 commits into
developfrom
feat/mactrack-upgrade-partitioning-cli

Conversation

@TheWitness

@TheWitness TheWitness commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #451.

Partitions the IP address table (mac_track_ips) by scan date like the port table, and replaces mactrack_convert.php with cli/upgrade_partitioning.php, which checks how the MAC address table (mac_track_ports) and the IP address table are partitioned against the selected scheme and rebuilds the ones that have drifted.

The script, cli/upgrade_partitioning.php

  • Scheme: --partition=daily|weekly|disabled, or, with no option, the Port Partition Size setting. --partition is saved with set_config_option() before the rebuild starts, so a failed run still leaves the setting as asked and a later run converges. The setting can now also be Disabled. The per-scan and hourly sizes stay with the poller; the script says so and changes nothing for them.
  • No drift: a table that is partitioned as selected (right size, valid layout, nothing in dMaxValue) is reported and left alone.
  • Rebuild, serially (MAC address table, then IP table), for each drifted table:
    1. rename it to <table>_rebuild;
    2. create it again with the retention and partitioning scheme (history older than the retention window goes into the first partition, so nothing is lost and a long history does not exceed the partition limit);
    3. loop over each distinct scan_date and INSERT ... SELECT it from the renamed table;
    4. verify the row counts, then drop the renamed table.
  • Process control: registered with register_process_start() (24 h timeout, well over the four hours asked for) and unregistered on every exit path, including SIGTERM/SIGINT. An interrupted run resumes from the last scan date copied (the partly copied date is redone, and a partly filled new table with another scheme is dropped and created again).
  • Output: a progress line at least every 30 seconds without --debug, a line before each long step (create, verify, drop), every scan date with --debug.
  • Options: --keep-old and -d are gone. --help lists --partition, --debug, --version and --help; -h, -H, -v and -V are still accepted.
  • mactrack_convert.php is deleted and added to tombstones in manifest.json. Utilities (Configure Port Partitioning) starts the new script; its backup checkbox is gone.

Pausing scanning and maintenance, and catching up

  • The script holds the scan lock, so scans (and therefore the poller's maintenance, which runs inside the poller) cannot start. The poller also exits early, and mactrack_db_maint_run() skips, while any <table>_rebuild table exists, which covers an interrupted run.
  • Maintenance that comes due meanwhile is not lost: each deferred poller cycle records that the daily window passed (mt_db_maint_missed), a skipped maintenance records it too, and the poller runs the maintenance in its first cycle after the rebuild instead of waiting for the next window (the poller's own schedule would otherwise lose a window that crosses midnight). The flag is cleared when the maintenance runs.
  • Maintenance mode message: the MAC address and IP address views show a maintenance-mode notice while a run is active or was interrupted, and Process Status shows that MacTrack is repartitioning.
  • A dead run's mt_partition_converting flag no longer blocks starting the resume from Utilities (found in review); only the live process registration counts.

Nothing changes the history while a rebuild is running or unfinished (review)

  • Device and site removal, MAC authorization and revocation, MacAuth save and removal, and the Utilities that remove all scan results, recreate or purge the aggregated data and rebuild the scan dates hold the partition lock, which the rebuild holds for its whole run, across the check and the whole change, and are refused with a message while a rebuild is running or a <table>_rebuild table is left. Without that, a delete made during the copy would be undone when the original's rows were copied back, and derived tables would be built from a partial history. The lock is taken with a 2 second wait, so these actions never hang behind a rebuild, and they do not depend on the scan lock, so they still work while a scan runs.
  • The background database upgrade (cli/upgrade_database.php) waits while a <table>_rebuild table exists, and the rebuild rechecks for a running upgrade once it holds its locks, so a migration is never marked complete against a partly copied table.
  • A recreated IP table (setup also runs when the plugin is re-registered during an upgrade) is partitioned by the saved Port Partition Size: Disabled leaves it unpartitioned, Daily creates daily partitions, and a saved size is never overwritten with weekly.

The IP table

mac_track_ips is partitioned like mac_track_ports: fresh installs partition it; the daily maintenance drops expired partitions by the IP retention, creates the days ahead (daily or weekly; the per-scan and hourly sizes map to daily) and splits rows stranded in dMaxValue into a recovery partition; Utilities View Partition Status shows its layout. An unpartitioned IP table keeps the existing row-deleting retention.

Ports to Ignore (Regular Expression) is a text area

The regular expression field is now a text area, like the device Ports to Ignore, at all three levels: the global Ports to Ignore While Scanning (Regular Expression) setting and the site and device forms. It has the same characteristics (3 rows, 80 columns, textAreaNotes, 255 characters). Each line is its own expression and a port matching any line is ignored, so a line break cannot become a literal that never matches a port name; a single-line expression is compiled exactly as before. Validation, trimming and the invalid-expression handling are unchanged.

Documentation and change log

  • README.md now has a Port and IP history partitioning section: how it works, setting it up (new and existing installs, the setting, retention, the script), checking and changing the scheme, what a rebuild does, what happens while it runs (and after an interruption), ongoing maintenance, and troubleshooting.
  • CHANGELOG.md: the pending section is headed 5.0 and ordered security, issue, feature, dev (entries were interleaved before); the feature is recorded as feature#451.

Related Issue

#451, a feature issue for partitioning mac_track_ips and moving the conversion routine (fixed by this pull request). Follows feature#426.

Motivation and Context

Changing the partition size or upgrading left tables that no longer match the setting, and the only way to correct them was a one-way copy that could not be resumed, covered the port table only, could not tell whether a table had drifted, and left database maintenance to the next window.

How Has This Been Tested?

  • Pest: 1095 tests pass; tests/bin/patch-coverage.php reports 100% of the changed lines (new PartitionRebuildTest.php; additions to the maintenance, web, activity and setting tests; the unit tests found a resume bug, fixed). check-i18n-pot.php and validate-manifest.php pass; locales/po/cacti.pot is the only catalog touched.
  • Real run of the script against MariaDB 10.5 with a scratch Cacti (core develop): unpartitioned to weekly, a no-drift rerun, weekly to daily, to disabled, SIGTERM in the middle of a rebuild followed by a resume, a hand-made half-copied table and resume, fresh install partitioning both tables, and the maintenance message and Process Status state.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation repository accordingly.

Notes for reviewers:

  • Searching is intentionally empty or partial between the rename and the end of the copy; that is what the maintenance message covers. Scans stay paused for the same time.
  • A rebuild needs disk space for a second copy of each table and its indexes.
  • mac_track_ports_old backups left by the previous --keep-old are not touched.

…artitioning.php

Replace mactrack_convert.php with cli/upgrade_partitioning.php, which checks
mac_track_ports and mac_track_ips against the selected scheme (the Port
Partition Size setting, or --partition=daily|weekly|disabled), says so when
there is no drift and rebuilds each drifted table in turn: rename it to
<table>_rebuild, create it again with the retention and partitioning scheme,
copy each scan date, then drop the original.

- Partition mac_track_ips by scan date like mac_track_ports: fresh installs,
  maintenance (partition drop, creation ahead, recovery of stranded rows) and
  the Utilities status page.
- Register the run in the process registry (24 hour timeout), hold the scan
  lock, resume an interrupted run from the last scan date and print progress.
- Show a maintenance message on the MAC and IP address views, defer the poller
  and maintenance while a rebuild is unfinished.
- Save the scheme chosen with --partition as the Port Partition Size setting,
  which can now be Disabled; drop --keep-old and the -d, -h and -v options
  from the help (-h and -v still work).
- Tombstone mactrack_convert.php in manifest.json.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 14:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unguarded history mutations, maintenance utilities, and database upgrades can operate on partial rebuild tables, restoring deleted data, corrupting derived records, or marking migrations complete before all rows are migrated.

4 open issues
What changed in this PR

Replaces the port-only conversion tool with resumable partition rebuilding for both MAC and IP history, extending MacTrack’s existing scan-date partitioning and retention support.

Changes:

  • Adds a CLI that detects partitioning drift and rebuilds tables using daily, weekly, or disabled partitioning.
  • Integrates rebuild state with polling, maintenance, Utilities, and search-view notices; adds IP partition maintenance.
  • Updates tests, documentation, translation strings, and packaging for the replacement tool.
File Description
tests/​Unit/​PartitionWebTest.php Tests Utilities options, CLI launches, and interrupted-run handling.
tests/​Unit/​PartitionSettingTest.php Adds the Disabled setting expectation.
tests/​Unit/​PartitionRebuildTest.php Covers rebuild planning, copying, recovery, and setup.
tests/​Unit/​PartitionMaintainTest.php Adds IP partition retention and recovery coverage.
tests/​Unit/​PartitionConvertTest.php Removes the superseded conversion test suite.
tests/​Unit/​ArpDot1xRetentionTest.php Updates the shared IP-retention assertion.
tests/​Unit/​ActivityTest.php Covers interrupted rebuild and poller-deferral reporting.
tests/​Security/​PreparedStatementConsistencyTest.php Removes the retired script’s baseline entry.
tests/​bin/​patch-coverage.php Removes the retired conversion entry point.
setup.php Adds Disabled partitioning and updates setting guidance.
README.md Documents rebuild commands, recovery, and operational constraints.
poller_mactrack.php Defers polling while rebuild source tables remain.
phpunit.xml Includes replacement rebuild sources in coverage.
manifest.json Marks the old CLI for upgrade removal.
mactrack_view_macs.php Adds rebuild maintenance notices to MAC views.
mactrack_view_arp.php Adds the rebuild maintenance notice to IP searching.
mactrack_utilities.php Launches the new CLI and displays IP partition status.
mactrack_convert.php Removes the old conversion CLI.
locales/​po/​cacti.pot Refreshes partitioning and maintenance-message translations.
lib/​mactrack_partitions.php Generalizes partition SQL helpers to allowlisted history tables.
lib/​mactrack_partition_rebuild.php Adds shared setup, rebuild-state, notice, and retention helpers.
lib/​mactrack_partition_ops.php Extends partition maintenance to IP history.
lib/​mactrack_partition_convert.php Removes the former conversion implementation.
lib/​mactrack_functions.php Loads rebuild helpers and integrates maintenance deferral and IP retention.
lib/​mactrack_activity.php Reports active or unfinished partition rebuilds.
includes/​database.php Partitions newly created IP tables and generalizes process-status checks.
cli/​upgrade_partitioning.php Implements drift assessment and resumable table rebuilding.
cli/​upgrade_database.php Updates a comment to name the replacement CLI.
CHANGELOG.md Records the feature and updates the earlier partitioning entry.
.github/​copilot-instructions.md Updates the documented project layout for CLI maintenance scripts.

Review effort: Balanced

Comment thread cli/upgrade_partitioning.php Resolved
Comment thread cli/upgrade_partitioning.php Resolved
Comment thread cli/upgrade_partitioning.php Resolved
Comment thread lib/mactrack_partition_rebuild.php Resolved
… after

- Remember in mt_db_maint_missed that the daily database maintenance came
  due (or was skipped) while a rebuild was under way; the poller runs it in
  its first cycle after the rebuild instead of waiting for the next window.
- Describe setting up and maintaining partitioning in the README.
- Order the 5.0 change log security, issue, feature, dev and reference
  feature#451.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@TheWitness TheWitness changed the title feat(partitions): rebuild port and IP partitioning with cli/upgrade_partitioning.php feature#451: Partition mac_track_ips and rebuild port and IP partitioning with cli/upgrade_partitioning.php Nov 17, 2026
TheWitness and others added 2 commits November 17, 2026 09:19
…areas

The global, site and device Ports to Ignore (Regular Expression) fields are
text areas with the same characteristics as the device Ports to Ignore (3 rows,
80 columns, textAreaNotes, 255 characters). Each line is its own expression, so
a line break no longer becomes a literal that never matches a port name.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…a rebuild

Address review of the partitioning rebuild:

- Hold the partition lock, which a rebuild holds for its whole run, across
  every change to the history (device and site removal, MAC authorization and
  revocation, MacAuth save and removal, and the Utilities that truncate the
  tables, recreate the aggregated data or rebuild the scan dates), and refuse
  while a rebuild is running or unfinished, so nothing is lost or restored
  when the copy finishes.
- Defer the background database upgrade while a rebuild table exists, and
  have the rebuild recheck for an upgrade once its locks are held.
- Partition a recreated IP table by the saved Port Partition Size instead of
  forcing weekly: Disabled leaves it unpartitioned and a saved size is kept.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

@TheWitness
TheWitness merged commit 85b58be into develop Nov 17, 2026
9 of 14 checks passed
@TheWitness
TheWitness deleted the feat/mactrack-upgrade-partitioning-cli branch November 17, 2026 14:42

@somethingwithproof somethingwithproof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers found in the final changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature: Partition mac_track_ips and move the partition conversion to cli/upgrade_partitioning.php

5 participants