---
title: Watermarks
description: How a polling Mule flow remembers where it stopped, so the next poll picks up only what is new. The Database source that keeps the bookmark for you, the Object Store pattern for everything else, and what a redeploy or a second replica does to it. Four live diagrams and a three-question check.
date: 2026-10-21
level: Intermediate
tags: MuleSoft, Mule 4, Watermarks, Object Store, Polling
---

A watermark is the bookmark a polling flow keeps: the newest value it has seen, usually a last-modified timestamp, so the next poll asks only for what changed after it. Some Mule 4 sources keep that bookmark for you. Everywhere else you keep it yourself in an Object Store, and where that store lives decides what a redeploy does to it.

This lesson keeps the orders app from the first lesson and adds sync-orders, a flow that copies changed orders to a CRM every few minutes. Each diagram plays by itself while it's on screen. The buttons change the story; the arrows step through it.

## The source keeps the bookmark

A few Mule 4 sources poll and watermark on their own: the Database connector's On Table Row, the File, FTP and SFTP triggers for new or updated files, the Salesforce object sources, the HTTP Polling Source and the IMAP mail trigger. POP3 can't: the protocol doesn't support it.

On Table Row is the easiest to picture. It selects from a table at a regular interval and generates one message per row. Give it a watermark column and the values from that column filter the next poll, so only rows with a greater value come back.

<figure class="lab" data-lab="poll" data-desc="On Table Row polls the orders table every minute with updated_at as its watermark column. After a poll that ended at 09:12, order 102 is edited at 09:18 and order 104 arrives at 09:20. The next poll returns only 102 and 104, and the bookmark moves to 09:20. With no watermark column, the same poll returns all four rows and the CRM gets the unchanged orders 101 and 103 again.">
</figure>

Switch the watermark column to "none" and watch the messages row. Nothing filters the poll, so every row comes back every minute.

On Table Row also takes an ID column. With one set, it makes sure the same row isn't processed twice by concurrent polls, which matters when one poll is still running as the next one starts.

## Keep it yourself with an Object Store

Mule 3 had a `<watermark>` element inside `<poll>`. Neither exists in Mule 4: the Scheduler replaced the poll, and the watermark became four plain steps in your flow.

1. Object Store Retrieve reads the watermark into a variable, with a default value for the very first run.
2. The query asks for everything changed after that variable.
3. The flow does its work.
4. Object Store Store writes the highest timestamp it saw, ready for the next run.

```xml
<os:retrieve key="watermark" target="watermark">
    <os:default-value>2017-09-11T00:00:00.000Z</os:default-value>
</os:retrieve>
<db:select config-ref="ordersDb">
    <db:sql>SELECT * FROM orders WHERE updated_at > :since</db:sql>
    <db:input-parameters>#[{ since: vars.watermark }]</db:input-parameters>
</db:select>
<flow-ref name="send-to-crm"/>
<os:store key="watermark" failIfPresent="false" failOnNullValue="false">
    <os:value>#[max(payload map $.updated_at)]</os:value>
</os:store>
```

<figure class="lab" data-lab="store" data-desc="The flow sync-orders runs every five minutes: Object Store Retrieve, Database Select, an HTTP Request to the CRM, then Object Store Store. On the first run the store is empty and Retrieve returns its default value, so every order is read. On a later run it reads 09:20, finds two new orders and stores 09:34. When the CRM call fails, Store never runs and the next run reads the same orders again. When nothing is new, Store gets null: it throws OS:NULL_VALUE with failOnNullValue at its default of true, and skips the write when it is false.">
</figure>

Two defaults bite here. Store's `failIfPresent` must be false, or it only works the first time. And `failOnNullValue` is true by default, so a run that finds nothing throws `OS:NULL_VALUE`. Try "nothing new" with both settings.

Now pick "CRM fails". The flow stops at the error, Store never runs, and the next run reads the same orders again. That's what you want, as long as the CRM can take an order it has already seen: if the failure came halfway through, the first half goes out twice.

## Where the bookmark lives

An object store declared with `persistent="true"` writes to local disk. On a Mule runtime on your own servers that disk survives a restart. On CloudHub 2.0 it doesn't: replicas clear local disk when an app restarts or redeploys.

<figure class="lab" data-lab="restart" data-desc="sync-orders stored 09:34 as its watermark, then the app is redeployed. With the object store on your own servers, or in Object Store v2, Retrieve still reads 09:34 and the next Select returns only the new order 107. With a local object store on CloudHub 2.0, the redeploy wipes the store, Retrieve returns its default value and the CRM receives every order again.">
</figure>

The docs' fix is Object Store v2, which keeps the data outside the app. It has its own limits: keys last up to 30 days, reading one in its last seven days extends it, and it's rate limited. CloudHub 1.0 behaves like CloudHub 2.0 here.

## Two replicas, two bookmarks

Scale sync-orders to two replicas on CloudHub 2.0 and the question becomes who polls. Without clustering, schedules run on every replica independently, and each replica has its own object store. With clustering, only the primary node runs the schedule and the object store is shared.

<figure class="lab" data-lab="replicas" data-desc="sync-orders runs on two CloudHub 2.0 replicas. With clustering off, both Schedulers fire, each replica reads 09:34 from its own object store, and both send order 107, so the CRM gets it twice. With clustering on, only the primary replica runs the schedule, reads the shared watermark and sends 107 once.">
</figure>

The same goes for On Table Row. Its Primary Node Only setting has to be on in a cluster to prevent duplicate record processing, and most connector sources ship with it off. One more catch: during a rolling restart on CloudHub 2.0, no node polls until the cluster elects a new primary, and the trigger that falls in that gap doesn't fire.

## The traps

A few more from the docs that are easy to learn in production:

- By default the Scheduler doesn't wait for a slow run to finish before starting the next one. Set `disallowConcurrentExecution="true"` and it skips the tick instead.
- A missed schedule behaves differently per platform. CloudHub 1.0 runs it as soon as the app is back, CloudHub 2.0 waits for the next scheduled time.
- On Table Row's Max Rows limit silently drops the rows over it.
- The Salesforce connector warns that a record committed after the query's timestamp but before the next poll can be missed, and has a Query Time Buffer for exactly that.
- Object stores don't support transactions, so the work and the watermark can't be committed together.
- The File connector's watermark mode is off by default. Without it, the docs have you delete or move each file once it's read.

## Check yourself

Three questions on what you just ran. Pick an answer to see why.

<ol class="quiz" aria-label="Check yourself">
<li data-answer="1"><p class="quiz-q">sync-orders uses the Object Store pattern, and the CRM call fails halfway through a run. What does the next run read?</p><ul class="quiz-options"><li>Only the orders that didn't go out</li><li>The same orders as the failed run</li><li>Nothing: the run that failed is lost</li></ul><details class="quiz-why" id="quiz-why-1"><summary>Answer</summary><p>The same orders. The flow stopped before Store, so the watermark didn't move, and the next run queries from the same point. Orders sent before the failure go out again.</p></details></li>
<li data-answer="2"><p class="quiz-q">The watermark is in an object store with <code>persistent="true"</code>, and the app runs on CloudHub 2.0. What does Retrieve return after a redeploy?</p><ul class="quiz-options"><li>The last stored watermark</li><li>An <code>OS:KEY_NOT_FOUND</code> error</li><li>Its default value, so the whole table is read again</li></ul><details class="quiz-why" id="quiz-why-2"><summary>Answer</summary><p>The default value. CloudHub 2.0 clears local disk on a redeploy, and <code>persistent="true"</code> writes to local disk. Retrieve with a default value returns it instead of failing. Object Store v2 would have kept the watermark.</p></details></li>
<li data-answer="0"><p class="quiz-q">sync-orders runs on two CloudHub 2.0 replicas with clustering off. How many times does a new order reach the CRM?</p><ul class="quiz-options"><li>Twice: each replica polls with its own store</li><li>Once: the replicas share the watermark</li><li>Once: only the primary replica polls</li></ul><details class="quiz-why" id="quiz-why-3"><summary>Answer</summary><p>Twice. Without clustering, schedules run on every replica independently and object stores aren't shared between them, so both find the same order.</p></details></li>
</ol>

## Where the facts come from

Every claim in this lesson was checked against the MuleSoft documentation:

- The Database connector's [reference](https://docs.mulesoft.com/db-connector/latest/database-documentation), for On Table Row
- [Set up watermarks with an Object Store](https://docs.mulesoft.com/object-store-connector/latest/object-store-to-watermark) and the [Object Store connector reference](https://docs.mulesoft.com/object-store-connector/latest/object-store-connector-reference)
- [Migrating watermarks](https://docs.mulesoft.com/mule-runtime/4.4/migration-patterns-watermark) and [migrating the poll component](https://docs.mulesoft.com/mule-runtime/4.4/migration-core-poll), from the Mule 4.4 docs
- [Polling sources](https://docs.mulesoft.com/mule-sdk/latest/polling-sources) in the Mule SDK and the [Scheduler](https://docs.mulesoft.com/mule-runtime/latest/scheduler-concept)
- [Object stores on CloudHub](https://docs.mulesoft.com/cloudhub/managing-application-data-with-object-stores), [Object Store v2](https://docs.mulesoft.com/object-store/), [CloudHub 2.0 clustering](https://docs.mulesoft.com/cloudhub-2/ch2-clustering) and [CloudHub 2.0 schedules](https://docs.mulesoft.com/cloudhub-2/ch2-manage-schedules)
- [HA clusters](https://docs.mulesoft.com/on-premises/latest/mule-high-availability-ha-clusters), the [File connector reference](https://docs.mulesoft.com/file-connector/latest/file-documentation) and the [Salesforce connector reference](https://docs.mulesoft.com/salesforce-connector/latest/salesforce-connector-reference)

Two things the docs don't say, so the labs don't either: when On Table Row commits its new watermark, row by row or once per poll, and whether a row whose flow failed is picked up again. Next up is queues, VM and Anypoint MQ. If a lab does something Mule wouldn't, tell me. The contact links are right below.