This page documents significant known issues in Weaviate, their symptoms, and recommended resolutions. Use the table below to find issues that may be affecting you, and their status.

On a replicated cluster, an object can lose data on one or more replicas after it is read:

• An affected replica keeps only the properties that the query returned. The object's remaining properties are gone from that replica.
• An affected replica also loses the object's vector, so the object no longer matches vector searches that the replica answers.
• The object still exists and keeps its ID, and aggregate counts are unchanged, so object counts and existence checks still look correct.
• At consistency level ONE, the same object can come back with a different set of properties from one read to the next, depending on which replica answers.

In affected versions, when replicas disagreed about an object, Weaviate could repair that object at read time. If the query that triggered the repair asked for only some of the object's properties, the coordinator could treat that partial view as the source of truth and write it back to the replicas being repaired, replacing the stored object with the subset of properties the query returned.

This issue occurs when:

• Replicas hold different states for the object, for example because a node was offline while writes continued.
• The read uses consistency level QUORUM or ALL. Reads at ONE do not trigger repair-on-read.
• A query that returns only some of the object's properties matches the object.

A cluster whose replicas report that they are fully in sync is not exempt: the problem has been observed on clusters where background synchronization had already brought every replica up to date.

Upgrade to a release that contains the fix:

• If on 1.37.x → upgrade to 1.37.14 or higher
• If on 1.38.x → upgrade to 1.38.7 or higher

All 1.39 releases contain the fix.

The 1.36 line and earlier were never patched, and fall outside the supported version range. If you run 1.36 or earlier, upgrade to one of the releases listed above.

These conditions are narrow, but the resulting data loss is easy to miss: a truncated object keeps its ID and still counts toward aggregates. Upgrade even if your object counts and health checks look normal.

• Nodes fail to start and remain indefinitely in initialization
• Repeated log messages: waiting for database to be restored
• May show schema update errors during RAFT command replay:

  cmd_class":"Content_par_test","cmd_type":2,"cmd_type_name":"TYPE_UPDATE_CLASS","error":"updating schema: TYPE_UPDATE_CLASS: bad request :parse class update: property \"content\": property fields other than description cannot be updated through updating the class. Use the add property feature (e.g. \"POST /v1/schema/{className}/properties\")

A regression introduced in schema catch-up handling caused invalid RAFT commands to block database initialization. The system would continuously retry these invalid commands, preventing marking the database as ready.

Upgrade to the fixed version:

• If on 1.27.x → upgrade to 1.27.27 or higher
• If on 1.28.x → upgrade to 1.28.16 or higher
• If on 1.29.x → upgrade to 1.29.8 or higher
• If on 1.30.x → upgrade to 1.30.4 or higher

After downgrading from 1.28+ to older 1.27.x versions:

• RAFT snapshots fail to load
• Cluster cannot reach Ready state
• Node initialization failures

RAFT snapshot format changes introduced in 1.28.13, 1.29.5, and 1.30.2 are not backward compatible with 1.27 releases prior to 1.27.26.

When downgrading from 1.28.13+, 1.29.5+, or 1.30.2+:

• Ensure you downgrade to version 1.27.26 or higher
• Do not downgrade to 1.27.25 or earlier

• Nodes stuck in crash loop with consistent restart intervals
• Logs show schema catch-up in progress but never complete:

  Schema catching up: applying log entry: [X/Y]

• Startup probe failures in Kubernetes

During cluster initialization or node recovery, applying accumulated RAFT log entries (especially with large schemas or many collections) may exceed the default 600-second bootstrap timeout.

Increase bootstrap timeout and startup probe:

env:
  - name: RAFT_BOOTSTRAP_TIMEOUT
    value: "900" # 15 minutes

startupProbe:
  failureThreshold: 90 # 90 * 10 = 900 seconds
  periodSeconds: 10
  httpGet:
    path: /v1/.well-known/ready
    port: 8080

Calculate appropriate timeout:

• Small clusters (< 10 collections): 600s default usually sufficient
• Medium clusters (10-100 collections): 900-1800s
• Large clusters (100+ collections): 1800-3600s

Note: Versions 1.27.4+, 1.26.11+, and 1.25.26+ include optimizations that reduce schema rebuild time during catch-up.

• Monitor cluster scale and adjust timeouts proactively
• Use WCS tool: wcs startup-count <failure-threshold> --apply; wcs sync

• Frequent leader elections and cluster instability
• Logs showing timeout errors:

  heartbeat timeout reached, starting election
  Election timeout reached, restarting election
  memberlist: Failed fallback TCP ping: timeout 1s: read tcp [...]: i/o timeout

• High CPU usage, memory pressure, or goroutine counts in monitoring
• Performance degradation during normal operations

Under heavy load or network latency, nodes cannot respond to RAFT heartbeats and memberlist pings within default timeout windows. This causes false failure detection, unnecessary leader elections, and cascading performance issues.

The issue is typically a symptom of underlying resource pressure rather than a RAFT problem itself.

1. Adjust RAFT timeout multiplier:

# Production (default in 1.31+)
RAFT_TIMEOUTS_MULTIPLIER=5

# High-latency networks
RAFT_TIMEOUTS_MULTIPLIER=10

# Heavily loaded or unstable environments
RAFT_TIMEOUTS_MULTIPLIER=15

This multiplies all timeout values:

• Heartbeat timeout: 1s → 5s (with multiplier of 5)
• Election timeout: 1s → 5s
• Leader lease timeout: 0.5s → 2.5s
• Memberlist TCP timeout: 10s → 50s

2. Investigate root cause:

Check for underlying issues:

• Memory pressure or OOM events
• CPU saturation
• Network latency or packet loss
• Too many collections causing Go scheduler pressure

3. If too many collections:

Reduce Go scheduler load:

GOMAXPROCS=<value less than available CPUs>

• Start with default multiplier (5) for most environments
• Increase gradually if seeing frequent elections
• Monitor cluster stability after changes
• Address underlying resource issues rather than only masking with higher timeouts

Error message in logs:

dial tcp: address 99999999: invalid port

Often accompanied by memberlist instability messages:

memberlist: Suspect weaviate-0 has failed, no acks received
memberlist: Marking weaviate-0 as failed, suspect timeout reached

This is not a RAFT problem but a symptom of memberlist instability. The invalid port 99999999 is intentionally returned to prevent RAFT from communicating with nodes that are not part of the memberlist, which prevents cross-talk issues where RAFT might contact old IP addresses from previous cluster configurations.

The underlying cause is typically:

1. Too many collections causing Go scheduler slowdown and network I/O delays
2. Network connectivity issues preventing memberlist health checks

Address underlying causes:

1. If too many collections:

   GOMAXPROCS=<value less than available CPUs>

2. If network issues:
   • Check connectivity between all cluster nodes
   • Review network policies and firewall rules
   • Verify DNS resolution

Tenant deletion fails with timeout errors:

context deadline exceeded
session: fetching region failed: RequestCanceled: request context canceled
caused by: context deadline exceeded

Occurs only when tenant offloading module (offload-s3) is enabled.

When tenant offloading is enabled and AWS credentials are misconfigured, the deletion process attempts to delete cloud resources but times out waiting for AWS responses.

Temporary workaround (if upgrade not immediately possible):

Option 1: Disable tenant offloading

# Remove or disable tenant offloading module configuration

Option 2: Correct AWS credentials

# Provide valid AWS credentials for tenant offloading
AWS_ACCESS_KEY_ID=<valid_key>
AWS_SECRET_ACCESS_KEY=<valid_secret>

Permanent fix: Upgrade to fixed version:

• 1.26.x → 1.26.14 or higher
• 1.27.x → 1.27.11 or higher
• 1.28.x → 1.28.5 or higher

• Shard initialization failures during tenant activation
• Errors in logs:

  memory pressure: cannot init shard: not enough memory mappings
  broadcast: cannot reach enough replicas

• Tenant activation failures
• On versions that carry it, these entries have a docs_url=https://docs.weaviate.io/e/core-mem001 field, and the error returned to the client ends in the same link. Both lead to not enough memory mappings, which has the fix.

The system has reached the operating system limit for memory-mapped files (vm.max_map_count). Each shard requires multiple memory mappings, and the default OS limit may be insufficient for large multi-tenant deployments.

Platform-specific steps (Linux, Docker, Kubernetes, Weaviate Cloud) are in not enough memory mappings. In short:

Increase the system memory mapping limit:

# Check current value
sysctl vm.max_map_count

# Increase to 3-4x current value
# Example: 1048576 → 4194304
sysctl -w vm.max_map_count=4194304

Make the change persistent:

# Add to /etc/sysctl.conf
echo "vm.max_map_count=4194304" >> /etc/sysctl.conf

Restart affected pods to apply the new configuration.

Single-node clusters panic on startup with:

Recovered from panic: assignment to entry in nil map
[...stack trace...]
github.com/weaviate/weaviate/cluster/schema.(*schema).addClass

RAFT snapshots with no collections (previously called classes) cause a nil map assignment during restoration due to JSON unmarshaler omitempty behavior. This edge case occurs when snapshots are created before any collections are added.

Option 1: Upgrade (recommended)

Upgrade to a release that contains the fix:

• 1.29.9 and later 1.29 releases
• 1.30.9 and later 1.30 releases
• 1.31.2 and later 1.31 releases
• 1.32.0 and later

Two points to watch when you choose a target release:

• 1.31.0 and 1.31.1 do not contain the fix. Within the 1.31 line, upgrade to 1.31.2 or later.
• The 1.28 line was never patched. If you run 1.28, upgrade to one of the releases listed above.

Option 2: Remove empty snapshot

If you cannot upgrade yet, identify and remove the problematic snapshot:

# Navigate to RAFT directory
cd raft/snapshots/

# Find snapshot with empty classes
# Look for state.bin containing: {"node_id":"...","snapshot_id":"...","classes":{}}

# Remove the empty snapshot directory
rm -rf <snapshot-directory>

Example structure:

raft/
├── db_users/
├── raft.db
└── snapshots/
    ├── 4-4-1727456146194/    # Valid snapshot
    └── 5-6-1728681332462/    # Empty snapshot - remove this

This issue should not occur in normal operations. It typically happens only if a snapshot is created before any schema is defined.

Node stuck during bootstrap with error messages indicating snapshot threshold reached but unable to create snapshot. This typically occurs only during initial cluster setup with rapid configuration changes.

During bootstrap, many configuration changes in short succession increase RAFT log size and trigger snapshot threshold before the node has fully initialized. The cluster becomes stuck because:

1. It cannot create a snapshot (requires bootstrap completion)
2. It cannot apply new configurations (requires snapshot first)

This should be rare in normal operations.

Temporarily increase snapshot thresholds:

RAFT_SNAPSHOT_INTERVAL=600  # seconds (default: 120)
RAFT_SNAPSHOT_THRESHOLD=24576  # entries (default: 8192)

This allows the node to apply all RAFT log entries before triggering snapshot creation.

After node reports healthy:

1. Remove the custom configuration
2. Restart the node to return to defaults

• Avoid making many rapid configuration changes during initial cluster bootstrap
• Stage large schema deployments rather than applying all at once

Log entries showing:

failed to decode incoming command
error: unknown rpc type 71
remote-address: 10.0.104.114:42128

Note: 71 represents ASCII 'G' (GET), 80 represents ASCII 'P' (POST)

HTTP requests being sent to RAFT's internal TCP endpoint (port 8300). This commonly occurs when Prometheus or other monitoring tools auto-discover and attempt to scrape all open ports, including internal RAFT ports.

Configure monitoring to exclude RAFT ports:

Update Prometheus scrape configuration to skip internal cluster ports:

• Port 7000: Memberlist
• Port 7100-7103: Memberlist gossip
• Port 8300: RAFT

For Prometheus Operator:

additionalScrapeConfigs:
  - job_name: "weaviate"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        regex: "(7000|7100|7101|7102|7103|8300)"
        action: drop

This is informational only and does not impact cluster functionality.

If you encounter an issue not listed here:

1. Search GitHub Issues
2. Ask in the Weaviate Community Forum

If you can't find an existing issue, please open a new one. Try to include the following information:

• Weaviate version
• Deployment environment (cloud, on-prem, Kubernetes, etc.)
• Relevant log excerpts
• Steps to reproduce
• Impact on your workload