Skip to content

Use a supplied communicator group and not MPI_COMM_WORLD #237

Closed
#300
Closed
Use a supplied communicator group and not MPI_COMM_WORLD#237
#300
Assignees
ss421mo-jonasganderton
Labels
enhancementNew feature or request

Description

@ss421
Steven Sandbach (ss421)
opened on Mar 5, 2025

While testing malak with XIOS in server mode, I encounter an error towards the end of the application that appear to be related to the VarObs and CX Writer. The error is not very helpful so I tested without the writer filters and the application runs. I suspect this is to do with the communicator group because when running in server mode, the jedi application does not use the MPI_COMM_WORLD comm, instead it uses a split communicator where some PEs are used by the server.

I did a search for MPI_COMM_WORLD and found a few references, the following in particular:

/// \note This filter must only be used with ObsSpaces using the \c MPI_COMM_WORLD communicator,
/// otherwise a deadlock will occur while writing the VarObs file. This is due to a limitation of
/// the \c Ops_WriteVarobs function, which could be removed by replacing \c mpl_comm_world in the
/// call to \c Ops_Mpl_Gatherv by \c mpi_group (for consistency with all other MPI calls in \c
/// Ops_WriteVarobs).

see: https://github.com/MetOffice/opsinputs/blob/2720b0b5d3ec2a129e27475d0fc6911547b4de17/src/opsinputs/VarObsWriter.h#L46C1-L50C22

suggest that there would be some additional work that is required if we want XIOS server + varWriter. I don't imagine that this is too much work. Adding a few people that may know:

Wojciech Śmigaj (@wsmigaj) David Simonin (@DavidSimonin) Chris Thomas (@ctgh) Michael Cooke (@mikecooke77) Adam Maycock (@adammaycock) David Davies (@DJDavies2)

if others have an ideas then please comment below.

Thanks, Steve.

Activity

  1. Steven Sandbach (ss421)
    self-assigned this
    on Mar 5, 2025
  2. Steven Sandbach (ss421)
    added
    enhancementNew feature or request
    on Mar 5, 2025

  3. mikecooke77 commented on Mar 5, 2025

    @mikecooke77
    Michael Cooke (mikecooke77)
    on Mar 5, 2025
    Collaborator

    Given that XIOS is only going to be with LFRic this will not be a problem operationally. However we will still want this for testing VAR vs JADA and therefore I support this work.

    Do you know if this is a problem with both the VarObs writer and CXWriter? Or is it just the varobs?

    What you suggest makes perfect sense. It would be good to get someone with some experience of working to debug mpi issues to work on this (or perhaps someone who wants to learn a bit more about MPI).

  4. ss421 commented on Mar 5, 2025

    @ss421
    Steven Sandbach (ss421)
    on Mar 5, 2025
    ContributorAuthor

    Given that XIOS is only going to be with LFRic this will not be a problem operationally. However we will still want this for testing VAR vs JADA and therefore I support this work.

    Do you know if this is a problem with both the VarObs writer and CXWriter? Or is it just the varobs?

    What you suggest makes perfect sense. It would be good to get someone with some experience of working to debug mpi issues to work on this (or perhaps someone who wants to learn a bit more about MPI).

    This is only an issue if the server is used. I am led to understand that we will need that when we get to C896 but I don't know any details at the moment.

    I didn't dig too far so I'm not sure but I suspect both. The initial error appeared to be related to CX (I used a diff from a working test case) but the comment above is in the varobs. I could do a test to see if each is working.

    I don't think it should be too difficult, my guess is the MPI_WORLD_COMM is hard coded somewhere likely in a module, it would just need to be passed the actual group.

  5. ss421 commented on Mar 5, 2025

    @ss421
    Steven Sandbach (ss421)
    on Mar 5, 2025
    ContributorAuthor

    Doing a diff on a working Verizon (no-server) and the version with the server enabled. I can see that the error appears during the varobs processing. For context, in the working version, the following is printed:

     VarobPGEs % RadialVelocity Allocate OK
PGE structure after Ops_VarobPGEs
 ************** START PRINT OF ModelOB header ***************
 ModelOB: Validity Time =-32768-32768-32768-32768     0     0-32768
 ModelOB: ObsGroup = 38
         ModelOB columns           1         248         495         742         989        1236        1483
              BG % Tskin -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000
 BG % RadialVelocity   1 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000 -1073741824.0000000
 ************** END PRINT OF ModelOB header ***************
BG % Tskin Deallocate OK
BG % RadialVelocity Deallocate OK
1672 observations output to varobs file
247000 observations not output to varobs file
LatLon array resized from 3344 to 4096
Times array resized from 1672 to 2048
<a href="http://www-nwp/~opsrc/OPS/view/dev/doc/ops_get_env_character.html">ops_get_env_character</a>
Environment variable OPS_CREATE_VAROBS_DONE_FILE hasn't been set
Using default value:
Ob % Latitude Deallocate OK
Ob % Longitude Deallocate OK
Ob % Time Deallocate OK

    while with the server enabled, the following (omitting the lines above 247000 observations not...) is printed:

    247000 observations not output to varobs file
Application 327001397 exit codes: 139
Application 327001397 exit signals: Killed
Application 327001397 resources: utime ~156s, stime ~149s, Rss ~1883948, inblocks ~12753617, outblocks ~2758561
[WARN] lfricjedi_hofx.x returned an error 137
[INFO] Checking that output has been produced, otherwise task fails
Check JEDI output ............ FAIL

    Removing the write for both varobs and CX (see: https://github.com/MetOffice/jjdocs/commit/5082277dc246ad817762804377ba4360d6d6f482) and the application runs.

    When adding back the CX writer (https://github.com/MetOffice/jjdocs/commit/726e53a6ad3eb8eac7026e0b1b9c20473db35ebd) I get a time-out failure rather than a hard failure like this. The failure in that case is likely due to miss-match in communicator group - I think it is likely waiting for synchronization across the "world comm" which is clearly not going to happen.

    switching to just VARobs writer (https://github.com/MetOffice/jjdocs/commit/be21419c8060767ae1b82efe7b2434006a4fe284) and I get the same failure as documented above.

  6. ctgh commented on Mar 4, 2026

    @ctgh
    Chris Thomas (ctgh)
    on Mar 4, 2026
    Collaborator

    Steven Sandbach (@ss421) Is this still a problem for malak?

  7. mikecooke77 commented on Mar 13, 2026

    @mikecooke77
    Michael Cooke (mikecooke77)
    on Mar 13, 2026
    Collaborator

    I think this is an issue should we want to run VAR with LFRic in server mode for XIOS. I guess the question is will we use server mode with JOPA? or just JADA?

  8. ss421 commented on Mar 13, 2026

    @ss421
    Steven Sandbach (ss421)
    on Mar 13, 2026
    ContributorAuthor

    My understanding is that we will want to use a server to read the C896 file but we have not done any testing related to that so Im not 100% sure. I think that would be an optimization rather than not being able to run it at all.

  9. mikecooke77 commented on Mar 20, 2026

    @mikecooke77
    Michael Cooke (mikecooke77)
    on Mar 20, 2026
    Collaborator

    Looking at the timings for the high resolution benchmark this will need to be done as a lot of the JOPA tasks are taking 50 mins rather than less than 10 mins. They need to be competitve with the UM model interface ones.

  10. Michael Cooke (mikecooke77)
    assigned
    Jonas Ganderton (mo-jonasganderton)
    on Sep 14, 2026
  11. Matt Shin (matthewrmshin)
    linked a pull request that will close this issueVarobs cx xios fix #300on Oct 13, 2026
  12. Matt Shin (matthewrmshin)
    closed this as completedin #300on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions