Partition three is having a day

Reported: orders-events consumer group lag alert:

PARTITION  LAG
0          41
1          12
2          77
3          1,402,777
4..7       < 100

• Adding consumers and forcing a rebalance doesn't help: whichever consumer gets partition 3 lags, the rest sit idle.
• Last night, data engineering ran a backfill that replayed 2M historical orders through the same producer.
• Producer code: producer.send("orders-events", key=str(order.customer_ref), value=...).
• In historical data, customer_ref is None for guest checkouts, which was most orders before 2022.