Overview

This article targets a specific symptom pattern where user retrieval works, but listing/search is incomplete:

• User lookup works:
  • GET /api/2.1/users/<userId> returns the expected user record
  • GraphQL user(id:"user:<userId>") returns the expected user record
• User listing/search is incomplete:
  • GET /api/2.1/search?q=select * from users returns a small subset and/or excludes known existing users
  • GraphQL usersCount and/or users(first:N) returns an unexpectedly small number
  • GraphQL userSearch(searchTerm:"...") may return totalCount: 0 for known existing users

In this failure mode, the usual root cause is that the user search index (esuser) and its bookmarks are out of sync with the user feed table (search_user), often after an environment refresh or a partial indexing attempt.

Solution

1) Confirm the symptom via API if possible (requires a valid admin session key)

Use a known-good admin session key and send it as request header li-api-session-key. If the key ends with a trailing ., the dot is part of the key.

Run these checks:

GET /api/2.1/search?q=select%20count(*)%20from%20users
GET /api/2.1/search?q=select%20*%20from%20users%20LIMIT%20250
GET /api/2.1/search?q=select%20id,login%20from%20users%20where%20login%20=%20'<login>'
GET /api/2.1/users/<userId>

Expected healthy behavior:

• select count(*) from users returns a realistic community user count (not a tiny number like 10–50).
• select * from users LIMIT 250 returns data.size = 250.
• Filtering by login returns the correct record.

Auth sanity check:

• If you get HTTP 403 with JSON containing code: 203 and message “You are not authorized to make this request”, the session key is not valid for that environment/request.
• If GraphQL returns HTTP 401 with UNAUTHENTICATED, the session key is not valid.

2) Confirm ES user-index state in World (counts, bookmarks, backfill progress)

In World console for the affected community, run:

A) ES document count for the user index (esuser)

#set($idx = $community.getSearchManager().getIndex("esuser"))
$idx.getSize()

B) Current bookmarks

#set($idx = $community.getSearchManager().getIndex("esuser"))
$idx.getBookmarks()

C) Backfill progress (authoritative “is it still indexing?” indicator)

$community.getSearchManager().getIndexer("esuser").getBackfillFeedCount()
$community.getSearchManager().getIndexer("esuser").getBackfillFeedCount().getPercentDone()

How to interpret the backfill values:

• count is the remaining number of backfill feed rows based on the current bookmark position.
• total is the total number of backfill feed rows in the current backfill range.
• getPercentDone() should increase during indexing and end at 100 when complete.

3) Reset bookmarks safely (Elasticsearch v8-safe)

On Elasticsearch v8, older “in-place indexing” snippets may fail due to Bookmarks vs Bookmarks8 mismatches.

Common failure signature:

java.lang.ClassCastException: lithium.search.index.Bookmarks cannot be cast to lithium.elasticsearch.v8.index.Bookmarks8

Recommended approach (v8-safe): generate a reset bookmark via the feeder and set it via setVersionBookmarks:

#set($idx = $community.getSearchManager().getIndex("esuser"))
#set($indexer = $community.getSearchManager().getIndexer("esuser"))
#set($resetBm = $indexer.getFeeder().getBookmarksForReset())
$idx.setVersionBookmarks($idx.getBookmarks().getVersion(), $resetBm)

If setVersionBookmarks throws a NullPointerException: print the values first, then retry using those variables (to avoid passing nulls):

#set($idx = $community.getSearchManager().getIndex("esuser"))
#set($ver = $idx.getBookmarks().getVersion())
#set($indexer = $community.getSearchManager().getIndexer("esuser"))
#set($resetBm = $indexer.getFeeder().getBookmarksForReset())
$ver
$resetBm
$idx.setVersionBookmarks($ver, $resetBm)

4) Monitor until backfill completes

Re-run these periodically:

$community.getSearchManager().getIndexer("esuser").getBackfillFeedCount()
$community.getSearchManager().getIndexer("esuser").getBackfillFeedCount().getPercentDone()
#set($idx = $community.getSearchManager().getIndex("esuser"))
$idx.getSize()

When remediation is complete, backfill should show 100% done, and API user list/search should be consistent again.

5) Re-validate via API (if you have the key from step 1)

Repeat the API checks from Step 1. In a successful remediation you should now see:

• select count(*) from users returning the same value across API 2.1 and GraphQL usersCount (if you check GraphQL).
• select * from users LIMIT 250 returning 250 records.
• Previously missing users now present in unfiltered lists.

6) If it still fails: check for indexing-side failures

Look for these patterns in lithium.log (Sumo):

• Id=<id>, Index=esuser, Failed to get populated at IndexFeed#fillUpdate(IndexUpdate) method, DELETING id from search_xxx table
• Exception while writing bookmark with type esuser, bookmark = null

If these appear repeatedly, do not keep re-running bookmark writes blindly. Investigate the underlying exception and only then retry the bookmark reset.