IDEX l1a event messages by lacoak21 · Pull Request #2870 · IMAP-Science-Operations-Center/imap_processing

lacoak21 · 2026-03-25T22:44:21Z

Change Summary

Overview

Parse and write all IDEX event messages out to the msg l1a CDF

File changes

imap_processing/cdf/config/imap_idex_global_cdf_attrs.yaml
- update descriptor

imap_processing/cdf/config/imap_idex_l1a_variable_attrs.yaml
- add attributes for messages
imap_processing/idex/evt_msg_decode_utils.py
- Add a regex repacer function to fill in event message placeholders with the given parameters
imap_processing/idex/idex_evt_msg_parsing_dictionaries.json
- Lookup tables provided by the IDEX team
imap_processing/idex/idex_l1a.py
- Parse the l1a packet into messages

Testing

Update l1a event message test

Copilot

Pull request overview

Adds support for decoding IDEX event-message packets into rendered, human-readable message strings and writing them to a dedicated L1A “msg” CDF product, along with the needed CDF metadata and updated tests.

Changes:

Introduces template-based rendering for IDEX event messages using IDEX-provided lookup dictionaries.
Updates IDEX L1A processing to generate an imap_idex_l1a_msg dataset with a messages variable.
Extends CDF global/variable attribute configs and updates the L1A event-message test expectations.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
imap_processing/tests/idex/test_idex_l1a.py	Updates EVT test to expect “msg” product naming and validates rendered messages against example data.
imap_processing/idex/idex_l1a.py	Adds message-product processing (`process_idex_msg_data`) and renders event messages into a new dataset variable.
imap_processing/idex/idex_evt_msg_parsing_dictionaries.json	Adds IDEX-provided dictionaries/templates used for decoding event messages.
imap_processing/idex/evt_msg_decode_utils.py	Adds placeholder/template rendering utility for event message strings.
imap_processing/cdf/config/imap_idex_l1a_variable_attrs.yaml	Adds CDF variable attributes for the new `messages` variable.
imap_processing/cdf/config/imap_idex_global_cdf_attrs.yaml	Renames/updates global attributes for msg products and their metadata fields.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-25T22:50:04Z

imap_processing/tests/idex/test_idex_l1a.py

+    # Validate the messages with the IDEX team example data
+    example_data = pd.read_csv(
+        f"{TEST_DATA_DIR}/idex_event_messages.csv", skiprows=1, header=None
+    )



idex_event_messages.csv is referenced here but does not exist anywhere in the repository (and it is not present under imap_processing/tests/idex/test_data). This will fail at runtime. Add the CSV file to the repo/test_data or adjust the test to use an existing reference file (e.g., a bundled CDF/H5) instead.

Suggested change

# Validate the messages with the IDEX team example data

example_data = pd.read_csv(

f"{TEST_DATA_DIR}/idex_event_messages.csv", skiprows=1, header=None

)

# Validate the messages with the IDEX team example data (if available)

csv_path = Path(TEST_DATA_DIR) / "idex_event_messages.csv"

if not csv_path.exists():

pytest.skip("idex_event_messages.csv not available; skipping message validation")

example_data = pd.read_csv(csv_path, skiprows=1, header=None)

Copilot · 2026-03-25T22:50:05Z

imap_processing/cdf/config/imap_idex_global_cdf_attrs.yaml

+imap_idex_l1a_msg:
   <<: *instrument_base
-   Data_type: L1A_EVT>Level-1A Event Message Data
-   Logical_source: imap_idex_l1a_evt
+   Data_type: L1A_msg>Level-1A Event Message Data
+   Logical_source: imap_idex_l1a_msg
   Logical_source_description: IMAP Mission IDEX Instrument Level-1A Event Message Data.


Data_type values use lowercase msg (e.g. L1A_msg, L1B_msg) while the other IDEX product Data_type entries are uppercase (e.g. L1A_CATLST, L1A_SCI-1WEEK). If Data_type is intended to follow an ISTP/mission naming convention, update these to the correct canonical form (likely L1A_MSG / L1B_MSG) to avoid metadata validation issues.

Copilot · 2026-03-25T22:50:05Z

imap_processing/cdf/config/imap_idex_global_cdf_attrs.yaml

+imap_idex_l1b_msg:
   <<: *instrument_base
-   Data_type: L1B_EVT>Level-1B Event Message Data
-   Logical_source: imap_idex_l1b_evt
+   Data_type: L1B_msg>Level-1B Event Message Data
+   Logical_source: imap_idex_l1b_msg
   Logical_source_description: IMAP Mission IDEX Instrument Level-1B Event Message Data.


This renames the L1B event message product key from imap_idex_l1b_evt to imap_idex_l1b_msg, but the rest of the repo still references imap_idex_l1b_evt (e.g. tests and CLI inputs). Either update those references consistently in the same PR, or avoid renaming the L1B product here to prevent mismatched logical sources/filenames.

Copilot · 2026-03-25T22:50:05Z

imap_processing/idex/idex_l1a.py

+            else:
+                # If no template exists for an event ID, fall back to a message
+                # that still preserves the event name and raw parameter bytes.
+                phex = ", ".join(f"0x{x:02X}" for x in params)


In the no-template fallback, phex is built from params (the full 2-D array) instead of the current event’s parameters (params[idx]). Iterating params yields numpy arrays, so f"0x{x:02X}" will raise a TypeError, and even if it didn’t it would render the wrong values. Build the hex string from params[idx] (or params[idx].tolist()).

Suggested change

phex = ", ".join(f"0x{x:02X}" for x in params)

phex = ", ".join(f"0x{x:02X}" for x in params[idx])

Copilot · 2026-03-25T22:50:05Z

imap_processing/idex/idex_l1a.py

        for level, dataset in datasets_by_level.items():
-            if IDEXAPID.IDEX_EVT in dataset:
-                logger.info(f"Processing IDEX {level} Event Message data")
+            # Only produce l1a products for event messages. L1a will be processed in a


The comment says “Only produce l1a products for event messages. L1a will be processed in another job.” but this branch is explicitly processing L1A event messages here (and skipping L1B). Please correct the comment to match the actual behavior (likely “L1B will be processed in another job” or similar).

Suggested change

# Only produce l1a products for event messages. L1a will be processed in a

# Only produce l1a products for event messages. L1b will be processed in

Copilot · 2026-03-25T22:50:06Z

imap_processing/idex/evt_msg_decode_utils.py

+        # to combine)
+        n_bytes = int(m.group(2)) if m.group(2) else 1


+0 placeholders exist in the provided message templates (e.g. {p1+0|idexModeLCDictionary}), but n_bytes is parsed as 0 in that case, producing value=0 and ignoring the actual parameter. Treat +0 as a single-byte field (or clamp n_bytes to at least 1) so templates decode correctly.

Suggested change

# to combine)

n_bytes = int(m.group(2)) if m.group(2) else 1

# to combine). Treat +0 as a single-byte field by clamping to at least 1.

n_bytes = int(m.group(2)) if m.group(2) else 1

if n_bytes < 1:

n_bytes = 1

Copilot · 2026-03-25T22:50:06Z

imap_processing/tests/idex/test_idex_l1a.py

 from unittest import mock

 import numpy as np
+import pandas as pd


This test now depends on pandas, but pyproject.toml lists pandas only under the optional tools extra (it is not included in the test extra). In environments that install only test dependencies, this import will fail. Either avoid pandas here (use the stdlib csv module) or add pandas to the test extra.

Suggested change

import pandas as pd

write out all event messages

0cbfae3

lacoak21 added this to the March 2026 milestone Mar 25, 2026

lacoak21 requested a review from Copilot March 25, 2026 22:44

lacoak21 self-assigned this Mar 25, 2026

lacoak21 added this to IMAP Mar 25, 2026

Copilot started reviewing on behalf of lacoak21 March 25, 2026 22:44 View session

lacoak21 requested review from tech3371 and vmartinez-cu March 25, 2026 22:45

Copilot AI reviewed Mar 25, 2026

View reviewed changes

add validation csv

c1bdac6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IDEX l1a event messages#2870

IDEX l1a event messages#2870
lacoak21 wants to merge 2 commits intoIMAP-Science-Operations-Center:devfrom
lacoak21:idex_event_message_products

lacoak21 commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    # Validate the messages with the IDEX team example data
-    example_data = pd.read_csv(
-        f"{TEST_DATA_DIR}/idex_event_messages.csv", skiprows=1, header=None
-    )
+    # Validate the messages with the IDEX team example data (if available)
+    csv_path = Path(TEST_DATA_DIR) / "idex_event_messages.csv"
+    if not csv_path.exists():
+        pytest.skip("idex_event_messages.csv not available; skipping message validation")
+    example_data = pd.read_csv(csv_path, skiprows=1, header=None)

	phex = ", ".join(f"0x{x:02X}" for x in params)
	phex = ", ".join(f"0x{x:02X}" for x in params[idx])

	# Only produce l1a products for event messages. L1a will be processed in a
	# Only produce l1a products for event messages. L1b will be processed in

Conversation

lacoak21 commented Mar 25, 2026

Change Summary

Overview

File changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants