Redis Architecture & Config

Data Explorer utilizes Redis as a high-performance middleware layer for caching, background queues, and rate limiting.

Global Caching

Caches schema introspection results and metadata to reduce load on source databases.

BullMQ Queues

Handles queued background workflows, currently most visible in migration and other async work that should not block the main request path.

Rate Limiting

Uses ThrottlerStorageRedis to synchronize API request limits across the entire server cluster.

Redis Roles in the Ecosystem

In Data Explorer, Redis is more than a simple cache. It coordinates multiple backend capabilities: cache, BullMQ, throttling, notifications, presence, and search indexing. In other words, it is real runtime infrastructure rather than a nice-to-have accelerator.

Background Processing (BullMQ)

When a workflow should not keep a synchronous request open for too long, the backend can push a job into Redis so a worker handles it separately. In the current repo, the clearest example of this pattern is the migration queue and related internal async work.

State Synchronization

Redis ensures that if you run 3 backend servers (scaling), the 100 requests/min limit is correctly applied to that user across all 3 servers.

Internal Data Structure

Data Explorer organizes data in Redis using consistent prefixes for easier management and monitoring.

Prefix	Module	Data Type
cache:*	CacheManager	String / JSON (Schema metadata)
bull:*	BullMQ	Hash / List / Set (Job queue state)
throttler:*	Rate Limiter	Integer (Request counters)
presence:*	Presence service	Hash + TTL / HSCAN presence reads
search_index:*	Search service	Item payloads + token candidate indexes

`KEYS` is no longer the preferred large-keyspace path. `RedisService.keys()` now remains as a compatibility wrapper over `SCAN`, presence reads hashes with `HSCAN`, and global search resolves candidate indexes instead of parsing the full item set in the service layer.

BullMQ Job Lifecycle

Queue-based workflows such as migration or longer async tasks go through three main stages in Redis:

Waiting

Job is pushed to Redis List and waits for an idle worker.

Active

Worker is processing. Redis holds a lock to prevent duplicate processing.

Completed

Job finished, result is temporarily cached for frontend polling.

Redis Deployment with Docker

The fastest way to run Redis for dev or self-hosting is using Docker Compose.

docker-compose.yml (extract)

redis:

image: redis:7-alpine

ports:

- "6379:6379"

command: redis-server --save 60 1 --loglevel warning

Advanced Redis Features

Real-time Notifications

Uses Redis Pub/Sub combined with SSE (Server-Sent Events) to push instant notifications to the browser. Pub/Sub channels are partitioned per user (`notifications:user:<id>`) to reduce unnecessary fan-out when many users are online.

Global Schema Search

Redis stores metadata indexes for all database connections. This allows you to search for tables or views across dozens of different databases in just a few milliseconds.

ERD Designer Optimization

That same search/index layer is what lets surfaces such as ERD, global search, and table-picking flows stay responsive without re-querying source metadata on every keystroke:

Smart Suggestions: Enables suggestions for related tables or entities based on a previously synced index.
Low Re-query Pressure: Lookups happen against the Redis-backed index instead of constantly hitting the source database again, reducing latency and unnecessary load.

Current product boundary

Redis is infrastructure today, not a dedicated workspace

In the current codebase, Redis powers cache, queues, notification streams, presence, throttling, and the search index. It is not yet exposed as a full Redis Explorer for end users, so this section should be read as architecture and operations documentation rather than a guide to a dedicated Redis workspace.

Configuration & Connectivity

The backend uses the `REDIS_URL` environment variable for connectivity. If undefined, it defaults to `redis://localhost:6379`.

.env Configuration

# Full Redis connection URL

REDIS_URL=redis://user:password@your-redis-host:6379

# Using Redis TLS (for managed services like Upstash / AWS)

REDIS_URL=rediss://default:yourpassword@name.upstash.io:6379

Troubleshooting Redis

Backend fails to start due to Redis error?

Check if Redis is running (redis-cli ping). Ensure port 6379 is open in the firewall.

BullMQ jobs are stuck or not processing?

Confirm Redis is not out of memory (OOM). You might need to clear old jobs with FLUSHDB during development.

Max connections error?

Free cloud providers often limit connections to 10-20. Consider upgrading if the app scale increases.

PreviousTechnology Stack Next Security & Privacy

Ready to explore data?

Join the community on GitHub to contribute to the smartest local database IDE.

Launch App