Backout behavior with Spring JMS and IBM MQ Broker - BO message does not get committed to BO Queue

[mwehrmeyer]

Hello,

we are using JMS to receive messages from an IBM MQ. We are receiving messages asynchronously via a JmsListener and process them, sometimes by sending other messages via a JmsTemplate and sometimes by doing JPA operations.

However, it is conceivable that our data source is unavailable temporarily, in which case our database accesses fail. It is one of our requirements however that we process all messages. In case of a database error, the message should be moved to a backout queue. To this end, we have set BOTHRESH and BOQNAME on our input queue.

If we process a message which, during processing, generates a JPA related exception (Constraint Violation for instance), our code passes that exception on to the JmsListener that services the input queue. After BOTHRESH attempts, our message is moved to the backout queue. There is a caveat, however, and that is that the message does not seem to be committed.

In our case, MQ Explorer will show that the length of our backout queue is greater than zero, browsing the messages in the queue will show an empty list of messages. MQ Explorer also shows that the backout queue has uncommitted messages. My expectation would be that any message that is moved to the backout queue is fully committed when it is moved there. When our application shuts down, the count of messages in the backout queue decreases by one and the count of messages in the input queue increases by one, for each uncommitted message.

IBM MQ documentation says that the removal to a backout queue, when handling messages asynchronously. is handled in an unrelated unit of work, which I interpret to mean that a developer has no means to manually commit a backout message to it's queue.

It is often mentioned that the fix is to add defaultListenerContainerFactory.setSessionTransacted(true) to the bean that instantiates our listener container. Doing this did not yield any progress.

I don't believe this is expected behavior, which is why I am filing this bug.

Thank you!

[sbrannen]

Hi @mwehrmeyer,

The backout-to-BOQNAME behavior is implemented in IBM MQ classes for JMS and its queue manager. Spring's message listener container only calls Session.commit() or Session.rollback() and has no visibility into MQ's internals.

DefaultMessageListenerContainer commits the session on every poll cycle, including when no message is received (which is intentional: to reset the transaction timeout), so at a first glance the commit/rollback handling looks correct on Spring's side.

In any case, I'd like to rule out a few things before we dig any deeper.

1. Could you please share your full message listener container configuration — container class, concurrency, sessionTransacted/acknowledge mode, and whether your listener uses a plain MessageListener or SessionAwareMessageListener?
2. What does your ConnectionFactory setup look like, and which MQ client version are you using?
3. Does the same behavior show up if you consume from the same queue with a plain JMS client (without Spring), using a synchronous receive() loop?

That last point matters because the note you quoted from IBM's docs about backout under "asynchronous" delivery landing in "an unrelated unit of work" describes IBM's own async MessageListener delivery model. Whereas, Spring's container actually polls with a synchronous receive() call under the hood, so it's worth confirming whether that caveat even applies here, or whether something else is going on.

Cheers,

Sam

[mwehrmeyer]

Hi @sbrannen, thank you for your response!

To answer your questions:

1: I should add one additional point: in our particular situation, we are getting messages from an adapter. During development we make use of a mock of this adapter which runs alongside our application and which is available in source. In order to insert a message to our application, we issue a curl command to a controller of the mock adapter, with a message file in one of the post parameters and some additional metadata. The adapter filters out this metadata, but passes the message in the message file to our application.

The mentioned controller in our mock uses jmsTemplate#send with the input queue of our application as destination in order to place the message we want our application to process. Therefore, the immediate code below describes the beans in our adapter for the sake of completeness. Also, both samples contain some jank from previous attempts to solve the issue in more unconventional ways, for example via an XA transaction aware MQ bean. Be assured that we tried MQConnectionFactory extensively.

@Bean(name = "ibmMqConnectionFactory")
  @ConditionalOnProperty(name = "mq.type", havingValue = "ibm-mq")
  @Primary
  public ConnectionFactory ibmMqConnectionFactory(MqProperties mqProperties) throws JMSException {
    Objects.requireNonNull(mqProperties.getHost(), "host missing");
    MQXAConnectionFactory factory = new MQXAConnectionFactory();
    factory.setConnectionNameList(mqProperties.getHost() + "(" + mqProperties.getPort()
        + ")" + ", " + mqProperties.getFallbackhost() + "("
        + mqProperties.getFallbackport() + ")");
    factory.setChannel(mqProperties.getChannel());
    factory.setQueueManager(mqProperties.getQueueManager());
    factory.setSSLCipherSuite("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384");
    factory.setStringProperty(WMQConstants.USERID, mqProperties.getUser());
    factory.setStringProperty(WMQConstants.PASSWORD, mqProperties.getPassword());
    factory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
    factory.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, true);

    // return new CachingConnectionFactory(factory);
    return factory;
}
  @Bean(name = "ibmMqListenerFactory")
  public DefaultJmsListenerContainerFactory ibmMqListenerFactory(ConnectionFactory ibmMqConnectionFactory) {
    DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
    factory.setConnectionFactory(ibmMqConnectionFactory);
    factory.setSessionTransacted(true);
    return factory;
  }

  @Bean()
  public JmsTemplate jmsTemplate(ConnectionFactory connectionFactory) {
    JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
    jmsTemplate.setSessionTransacted(true);
    return jmsTemplate;
  }

Our application mostly makes use of JmsListeners, because it is also a requirement to be as independent of technical details as possible. In our instant case, we receive incoming messages from our adapter. The way we receive messages is that we implement an observer and subscribe to a notification service from our adapter client code. This notification service implements a jakarta.jms.MessageListener which calls onMessage on reception of a message. It further wraps the incoming message in some business logic object and eventually delivers the message to our observer. Much of this is very abstract and seems quite immaterial to our problem.

Now to the beans in our application:

  @Bean()
  public JmsTemplate jmsTemplate(ConnectionFactory connectionFactory) {
    JmsTemplate jmsTemplate = new JmsTemplate(connectionFactory);
    jmsTemplate.setSessionTransacted(true);
    return jmsTemplate;
  }
  @Bean
  public DefaultJmsListenerContainerFactory jmsListenerContainerFactory(
      PlatformTransactionManager jtaTransactionManager,
      ConnectionFactory connectionFactory) {

    DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();

    factory.setConnectionFactory(connectionFactory);
    factory.setTransactionManager(jtaTransactionManager);
    factory.setSessionTransacted(true);
    factory.setConcurrency("1"); // an attempt to solve the problem

    return factory;
  }
  @Bean
  public ConnectionFactory ibmMqConnectionFactory() throws JMSException {
    MQXAConnectionFactory factory = new MQXAConnectionFactory();

    factory.setConnectionNameList("%s(%d), %s(%d)".formatted(queueProperties.getHost(), queueProperties.getPort(),
        queueProperties.getFallbackhost(), queueProperties.getFallbackport()));
    factory.setChannel(queueProperties.getChannel());
    factory.setQueueManager(queueProperties.getQueueManager());
    factory.setSSLCipherSuite("TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384");
    factory.setStringProperty(WMQConstants.USERID, queueProperties.getUser());
    factory.setStringProperty(WMQConstants.PASSWORD, queueProperties.getPassword());
    factory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
    factory.setBooleanProperty(WMQConstants.USER_AUTHENTICATION_MQCSP, true);
    /*
    CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(factory);
    cachingConnectionFactory.setCacheConsumers(false);
    return cachingConnectionFactory;
     */
    return factory;

We use mq-jms-spring-boot-starter version 4.1.0 in both our mock adapter and our application.

I hope this answers questions 1 and 2.

As regards 3: Our adapter can also be run in synchronous mode (called batch mode here), in which case it exposes a messagecontrol#receive() method. Curiously, the issue does not occur when using synchronous mode, messages are properly committed to the backout queue as far as I recall (though it has been about a week since I last checked). However, a member of our development team was concerned that our application might behave in subtly different ways if we were to receive messages synchronously, which is why it is currently considered undesirable. Given that is is a sunday, I do not think this member of our team is available for comment.