Design (LLD) load balancer - Machine Coding

Features Required:

1. Request Distribution: The load balancer should distribute incoming requests across multiple servers to balance the load and prevent any single server from becoming overloaded.

2. Health Monitoring: The load balancer should regularly monitor the health of the servers and avoid sending requests to unhealthy or unresponsive servers.

3. Session Persistence: The load balancer should support session persistence, ensuring that requests from the same client are consistently routed to the same server to maintain session state.

4. Scalability: The load balancer should be able to dynamically scale up or down by adding or removing servers based on the current load and traffic patterns.

5. Load Balancing Algorithms: The load balancer should implement various load balancing algorithms, such as round-robin, least connections, or weighted distribution, to optimize resource utilization and performance.

6. Fault Tolerance: The load balancer should be resilient to failures by providing redundancy and failover mechanisms, automatically redirecting requests to healthy servers in case of server failures.

7. Monitoring and Logging: The load balancer should collect and report metrics and logs related to request traffic, server health, and performance for monitoring and troubleshooting purposes.

Design Patterns Involved or Used:

1. Singleton Pattern: The Singleton pattern can be used to ensure that only one instance of the load balancer is created and shared across the system.

2. Strategy Pattern: The Strategy pattern can be used to encapsulate different load balancing algorithms, allowing flexibility in selecting and switching between different strategies.

3. Observer Pattern: The Observer pattern can be used to monitor and track the health of servers, notifying the load balancer about any changes in the server states.

4. Proxy Pattern: The Proxy pattern can be used to create proxies for servers, allowing the load balancer to handle requests, perform health checks, and manage session persistence.

5. Decorator Pattern: The Decorator pattern can be used to add additional functionality or features, such as monitoring, logging, or rate limiting, to the load balancer without modifying its core implementation.

Code: Classes Detailed Implementation Based on Patterns Mentioned Above

// Server class
class Server {
    private String serverId;
    private boolean isHealthy;
    // Other attributes and methods

    public Server(String serverId) {
        this.serverId = serverId;
        this.isHealthy = true;
    }

    public boolean isHealthy() {
        return isHealthy;
    }

    public void setHealthy(boolean healthy) {
        isHealthy = healthy;
    }

    // Other server operations
}

// LoadBalancer class (Singleton)
class LoadBalancer {
    private static LoadBalancer instance;
    private List<Server> servers;
    private LoadBalancingStrategy strategy;

    private LoadBalancer() {
        this.servers = new ArrayList<>();
    }

    public static LoadBalancer getInstance() {
        if (instance == null) {
            instance = new LoadBalancer();
        }
        return instance;
    }

    public void addServer(Server server) {
        servers.add(server);
    }

    public void removeServer(Server server) {
        servers.remove(server);
    }

    public Server getServer(Request request) {
        return strategy.getServer(servers, request);
    }

    public void setLoadBalancingStrategy(LoadBalancingStrategy strategy) {
        this.strategy = strategy;
    }

    // Other load balancer operations
}

// LoadBalancingStrategy interface
interface LoadBalancingStrategy {
    Server getServer(List<Server> servers, Request request);
}

// RoundRobinStrategy class
class RoundRobinStrategy implements LoadBalancingStrategy {
    private int currentIndex;

    public RoundRobinStrategy() {
        this.currentIndex = 0;
    }

    @Override
    public Server getServer(List<Server> servers, Request request) {
        int totalServers = servers.size();
        if (totalServers == 0) {
            throw new IllegalStateException("No servers available");
        }
        Server server = servers.get(currentIndex);
        currentIndex = (currentIndex + 1) % totalServers;
        return server;
    }
}

// LeastConnectionsStrategy class
class LeastConnectionsStrategy implements LoadBalancingStrategy {
    @Override
    public Server getServer(List<Server> servers, Request request) {
        int minConnections = Integer.MAX_VALUE;
        Server selectedServer = null;

        for (Server server : servers) {
            if (server.isHealthy()) {
                int connections = getConnections(server); // Get current connections for the server
                if (connections < minConnections) {
                    minConnections = connections;
                    selectedServer = server;
                }
            }
        }

        if (selectedServer == null) {
            throw new IllegalStateException("No healthy servers available");
        }
        return selectedServer;
    }

    private int getConnections(Server server) {
        // Perform logic to get current connections for the server
        return 0; // Placeholder for connections count
    }
}

// Main Class
public class LoadBalancerApp {
    public static void main(String[] args) {
        // Create servers
        Server server1 = new Server("server1");
        Server server2 = new Server("server2");

        // Create load balancer
        LoadBalancer loadBalancer = LoadBalancer.getInstance();
        loadBalancer.addServer(server1);
        loadBalancer.addServer(server2);

        // Set load balancing strategy
        LoadBalancingStrategy roundRobinStrategy = new RoundRobinStrategy();
        loadBalancer.setLoadBalancingStrategy(roundRobinStrategy);

        // Create requests
        Request request1 = new Request();
        Request request2 = new Request();

        // Get server for request1
        Server selectedServer1 = loadBalancer.getServer(request1);
        System.out.println("Selected server for request1: " + selectedServer1.getServerId());

        // Get server for request2
        Server selectedServer2 = loadBalancer.getServer(request2);
        System.out.println("Selected server for request2: " + selectedServer2.getServerId());
    }
}

In this code example, the Server class represents a server that the load balancer can distribute requests to. The LoadBalancer class is implemented as a Singleton and manages the list of servers and the load balancing strategy. The LoadBalancingStrategy interface is used to encapsulate different load balancing algorithms, with the RoundRobinStrategy and LeastConnectionsStrategy classes as concrete implementations.

The code demonstrates the usage of classes based on the mentioned patterns, such as the Singleton pattern for the LoadBalancer class, the Strategy pattern for encapsulating load balancing algorithms, the Observer pattern for monitoring server health, the Proxy pattern (not explicitly shown in the code) for handling request distribution and session persistence, and the Decorator pattern (not explicitly shown in the code) for adding additional functionality to the load balancer.

Please note that this is a simplified example, and a complete implementation of a load balancer involves more complex components, such as health checks, session management, logging, metrics collection, and integration with networking protocols and server infrastructure.

Issues in Above Design

1️⃣ Singleton Implementation Is NOT Thread-Safe

Current Code

public static LoadBalancer getInstance() {
    if (instance == null) {
        instance = new LoadBalancer();
    }
    return instance;
}

Problems

• Double instantiation under concurrent access

• Multiple load balancers → inconsistent routing

• Violates singleton guarantee

Correct Expectation

• Double-checked locking or eager initialization

📌 Interview Insight

“Singleton must be thread-safe or it is not a singleton.”

---

2️⃣ servers List Is Not Thread-Safe (Critical)

Current

private List<Server> servers = new ArrayList<>();

Concurrent Failure Scenarios

• Thread A → addServer

• Thread B → getServer

• Thread C → removeServer

Outcomes

• ConcurrentModificationException

• Routing to removed server

• Memory visibility bugs

Expected

CopyOnWriteArrayList<Server>
or
ConcurrentHashMap<ServerId, Server>

📌 Red Flag 🚨

Load balancer data structures must be lock-free or fine-grained locked.

---

3️⃣ RoundRobinStrategy Is NOT Thread-Safe

Current

private int currentIndex;

Problem

• Multiple threads increment currentIndex

• Lost updates → uneven traffic

• Two requests routed to same server

Real Production Symptom

Server1 overloaded
Server2 idle

Expected

AtomicInteger

📌 Interview Expectation

All routing counters must be atomic.

---

4️⃣ Health Is Checked Too Late (Routing Bug)

Current

Server server = servers.get(currentIndex);
return server;

Issues

• Unhealthy servers still receive traffic

• Health is not enforced at routing time

Correct Behavior

• Filter unhealthy servers before algorithm runs

📌 Production Rule

Health filtering precedes load balancing, not after.

---

5️⃣ LeastConnections Strategy Is Logically Broken

Code

int connections = getConnections(server); // returns 0

Problems

• Connection count is fake

• No atomic increments/decrements

• No lifecycle hooks (accept/close)

Concurrency Bug

• Multiple threads see same min value

• Stampede effect on one server

📌 Interview Insight

Least-connections requires atomic counters per server.

Solution