Issue 

When running Redpanda Shadowing(Self Managed or Cloud) , monitoring tools may incorrectly report that topic/partitions which are empty on the source cluster as having a lag.
 

Detail 

When running the monitoring features as detailed in documentation 

#rpk shadow status

rpk shadow status shinylink
OVERVIEW
========
NAME   shinylink
UID    b2222a2d-9d81-4345-8095-04f16bfa2b63
STATE  ACTIVE


TASKS
=====
NAME                      BROKER_ID  SHARD  STATE   REASON
Source Topic Sync         0          0      ACTIVE  Source Topic Sync has started
Consumer Group Shadowing  0          0      ACTIVE  Consumer Group Shadowing has started
Consumer Group Shadowing  0          1      ACTIVE  Consumer Group Shadowing has started
Security Migrator Task    0          0      ACTIVE  Security Migrator Task has started


TOPICS
======
Name: howSoonIsNow, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          0         1000    1000      0
      
Name: humDrumTown, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          620      620      0        620

Name: theSmiths, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          231      231      0        231


Synced Shadow Topic Properties:
  - segment.ms
  - message.timestamp.type
  - max.message.bytes
  - cleanup.policy
  - retention.ms

#lag metrics > 0 

curl -s localhost:9644/public_metrics|grep redpanda_shadow_link_shadow_lag{|awk '$NF > 0.0 {print $0}' 
redpanda_shadow_link_shadow_lag{redpanda_partition="0",redpanda_topic="theSmiths",shadow_link_name="shinylink"} 231.000000
redpanda_shadow_link_shadow_lag{redpanda_partition="0",redpanda_topic="humDrumTown",shadow_link_name="shinylink"} 620.000000

Here we can see 2 topics are reporting a lag in this test Shadowing environment.

If we check the partition details on each of the topic/partitions on the source cluster ...

rpk topic describe humDrumTown  -p 
PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          0       2      [0]       620               620

rpk topic describe theSmiths  -p 
PARTITION  LEADER  EPOCH  REPLICAS  LOG-START-OFFSET  HIGH-WATERMARK
0          0       2      [0]       231               231

in both cases,  the LOG-START-OFFSET=HIGH-WATERMARK..  

This means there are no messages in these topic/partions.  Previous messages will have been deleted via topic retention or  via rpk topic trim-prefix /kafka delete api 

As the Shadowing monitoring commands are incorrectly reporting these as a having a lag , this is a Redpanda bug .
 

Resolution / Workaround 

This is a Redpanda bug logged as CORE-15308 and is fixed in Redpanda Release 25.3.7 and higher 

If you experience unexpected lag on earlier releases and cannot upgrade yet , check the source topic to see if the partitions are empty as detailed in this article. In which case, the lag can be safely ignored