Missed Quartz Triggers After Spring Boot Upgrade to 3.4.5

[smitmistry77]

Hi Team,

We recently upgraded our application’s Spring Boot version from 3.4.2 to 3.4.5, which also included an update to the spring-boot-starter-quartz dependency.

Following the upgrade, we observed that approximately 2–3 scheduled triggers per day are intermittently not firing as expected. These triggers were functioning reliably prior to the upgrade.

Could you please advise if there are any known issues or changes in Quartz integration that might explain this behavior?

Thanks,
Smit Mistry

[bclozel]

Spring Boot has been using Quartz 2.3.2 for a long time (see spring-projects/spring-boot#18725), and I don't think the quartz auto-configuration evolved recently.

This might be related to another change in Spring Framework, or somewhere else? If you can provide a minimal sample we can have a look, but it sounds like it is hard to reproduce.

[smitmistry77]

Hi bclozel,
Triggers are missed no logs are there for it. Not sure what samples we can provide you

[bclozel]

I'm not sure what you're expecting from us then, if we can't reproduce the problem and if we're not aware of any issue nor change related to Quartz

[smitmistry77]

To validate, we reverted back to Spring Boot 3.4.2 approximately 24 hours ago, and since then, the issue has not reoccurred.

[smitmistry77]

package com.opentext.cms.quartz.execution;

import lombok.extern.slf4j.Slf4j;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.quartz.SchedulerException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.ApplicationContext;
import org.springframework.scheduling.quartz.SchedulerFactoryBean;

import java.time.Instant;
import java.util.Date;
import java.util.UUID;

@slf4j
public class TokenRefreshJob implements Job {

/**
 * Default scheduler trigger name.
 */
private static final String DEFAULT_TRIGGER_NAME = "Default_Token_Refresh_Trigger";
@Autowired
ApplicationContext applicationContext;
@Autowired
private TokenRefreshJobExecutor tokenRefreshJobExecutor;
private SchedulerFactoryBean factoryBean;
@Value("${tg.cms.trigger.schedulers}")
private boolean scheduleTrigger;

/**
 * {@inheritDoc}
 */
@Override
public void execute(final JobExecutionContext context) throws JobExecutionException {
    try {
        log.info("Executing TokenRefreshJob {} with scheduleTrigger={}", Date.from(Instant.now()), scheduleTrigger);
        if (scheduleTrigger) {
            factoryBean = getSchedulerFactoryBean();
            String correlationId = UUID.randomUUID().toString();
            String triggerName = null;
            String instanceId = null;
            try {
                instanceId = factoryBean.getScheduler().getSchedulerInstanceId();
            } catch (final SchedulerException e) {
                log.info("[{}] Unable to fetch the scheduler instance id. Error : {}", correlationId, e.getMessage());
            }

            if (context.isRecovering()) {
                triggerName = context.getRecoveringTriggerKey().getGroup();

                if (triggerName.equalsIgnoreCase(DEFAULT_TRIGGER_NAME)) {
                    log.info("[{}] Default trigger was in-progress when the application went down. Aborting its recovery re-run.", correlationId);
                    return;
                }

                log.info("[{}] Recovery job running for trigger : {} at {} in instance {}", correlationId, triggerName, context.getFireTime(), instanceId);
            } else {
                triggerName = context.getTrigger().getKey().getName();
                log.info("Running trigger : {} at {} in instance {}", triggerName, context.getFireTime(), instanceId);
            }

            tokenRefreshJobExecutor.executeRefreshJob(correlationId, triggerName);

            log.info("[{}] {} next scheduled @ {}", correlationId, triggerName, context.getNextFireTime());
        }
    } catch (Exception e) {
        log.error("Error while executing TokenRefreshJob {} at time {}", e, Date.from(Instant.now()));
        e.printStackTrace();
    } finally {
        log.info("Finally block of TokenRefreshJob {} at time {}", Date.from(Instant.now()));
    }
}

public SchedulerFactoryBean getSchedulerFactoryBean() {
    return applicationContext.getBean(SchedulerFactoryBean.class);
}

}
This is the sample code that is used for the job to run

[smitmistry77]

The problem is of missfire of 1-3 triggers on update to 3.4.5 version. Its not having any pattern

[bclozel]

The SchedulerFactoryBean class didn't change recently, but the fact that a quartz job is fetching a bean from the application context sounds like lifecycle issues that were recently worked on in Spring Framework.

Can you try following the advice from this comment and report back? I'm transferring the issue to Spring Framework, waiting for your feedback on the spring.locking.strict=true property.

[jhoeller]

I'm not really seeing locking involved here, in particular not within different 6.2.x versions, but also since these missed triggers seem to happen at regular application runtime (after bootstrap) where we are always using strict locking anyway.

So I'm afraid this will have to be debugged specifically. Something in the Quartz scheduler reacts differently, possibly to some state in the Spring application context but it's hard to see what this could be.

[smitmistry77]

So, just want to explain the way we are running the schedulers and how the triggers are configured,

1. Triggers are configured every 15 minutes, and there is a simple trigger for every 5 minutes.

2. We are running in Kubernetes with 2 pods of the application, and below properties are added to handle this.
   spring.quartz.job-store-type=jdbc
   spring.quartz.properties.org.quartz.jobStore.isClustered=true
   spring.quartz.properties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.oracle.OracleDelegate
   spring.quartz.properties.org.quartz.jobStore.dataSource=dataSource
   spring.quartz.properties.org.quartz.jobStore.class=org.springframework.scheduling.quartz.LocalDataSourceJobStore
   spring.quartz.properties.org.quartz.threadPool.threadCount=10
   spring.quartz.properties.org.quartz.jobStore.clusterCheckinInterval=20000
   spring.quartz.properties.org.quartz.jobStore.misfireThreshold=60000
   spring.quartz.properties.org.quartz.scheduler.instanceId=AUTO
   Observation:
   But, everything works fine when I am using the below configuration version
   Springboot: 3.4.2
   Spring Framework: 6.2.5

   The problem happens with the following version:
   Spring boot:3.4.5
   Spring Frameworkversion: 6.2.5

[smitmistry77]

Just want to add on here that after reverting only the springboot version didn't help. After 2 days again we see the miss in trigger firing. Now we are reverting the spring-framework version for spring-context jar as well.

[spring-projects-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[smitmistry77]

Hi ,
What is the information required ?

[bclozel]

This:

So I'm afraid this will have to be debugged specifically. Something in the Quartz scheduler reacts differently, possibly to some state in the Spring application context but it's hard to see what this could be.

[smitmistry77]

What exactly is required on this can you please let me know so I can do the needful ASAP.

[bclozel]

Ideally, a minimal sample application that we can run that reproduces the problem.
If this is not possible, we would need extensive information about what happens at runtime when the trigger is missed. Maybe through debug logs and debugging - anything that shows a significant difference of state when triggers aren't met.

It is hard for us to pinpoint an exact thing to do as we have no intuition about the problem right now.

[smitmistry77]

I have shared the class above which is TokenRefreshJob that is implementing job,
I am using the latest spring-boot-starter-quartz and org.quartz-scheduler.quartz - 2.5.0 version, Also I have configured the triggers every 15 mins.
As you can see in the class, first line only I have added the logs which will intimate the time it ran. we checked the logs and it has not even triggered it.
Attached the logs where 0530 trigger is missed.

[bclozel]

I'm closing this because this is not going anywhere and you haven't shown that this is a problem with Spring.
We can reopen this issue if you can provide the requested information and it shows a bug in Spring.