smux

English | 中文

Introduction

Smux (Simple MUltipleXing) is a multiplexing library for Golang. It relies on an underlying connection to provide reliability and ordering, such as TCP or KCP, and provides stream-oriented multiplexing. This library was originally designed to power connection management for kcp-go.

Features

1. Token bucket controlled receiving, providing a smoother bandwidth graph (see picture below).
2. Session-wide receive buffer shared among streams for fully controlled overall memory usage.
3. Minimized header (8 bytes), maximized payload.
4. Battle-tested on millions of devices in kcptun.
5. Built-in fair queue traffic shaping.
6. Per-stream sliding window for congestion control (protocol version 2+).

Architecture

• Session: The main manager for a multiplexed connection. It manages the underlying io.ReadWriteCloser, handles stream creation/acceptance, and manages the shared receive buffer.
• Stream: A logical stream within a session. It implements the net.Conn interface, handling data buffering and flow control.
• Frame: The wire format for data transmission.

Frame Allocator

alloc.go implements a slab-style allocator tuned for frames up to 64 KB. Seventeen sync.Pool buckets cache power-of-two slice capacities, and a De Bruijn based msb() lookup picks the smallest bucket that can satisfy a request in constant time. Buffers are deliberately reused without zeroing, so each new frame simply overwrites the previous payload without paying the Go runtime's memclr cost.

Benefits for the session:

1. Bounded fragmentation (each request wastes < 50%) keeps the shared receive buffer predictable under load.
2. Reuse of pre-sized slices drastically lowers GC pressure and removes repeated zeroing work.
3. Constant-time bucket selection avoids locks or searches, so high-throughput sessions keep tail latency steady even with thousands of flows.

Documentation

For complete documentation, see the associated Godoc.

Benchmark

$ go test -v -run=^$ -bench .
goos: darwin
goarch: amd64
pkg: github.com/xtaci/smux
BenchmarkMSB-4           	30000000	        51.8 ns/op
BenchmarkAcceptClose-4   	   50000	     36783 ns/op
BenchmarkConnSmux-4      	   30000	     58335 ns/op	2246.88 MB/s	    1208 B/op	      19 allocs/op
BenchmarkConnTCP-4       	   50000	     25579 ns/op	5124.04 MB/s	       0 B/op	       0 allocs/op
PASS
ok  	github.com/xtaci/smux	7.811s

Specification

 +---------------+---------------+-------------------------------+
 |  VERSION (1B) |    CMD (1B)   |          LENGTH (2B)          |
 +---------------+---------------+-------------------------------+
 |                          STREAMID (4B)                        |
 +---------------------------------------------------------------+
 |                                                               |
 /                        DATA (Variable)                        /
 |                                                               |
 +---------------------------------------------------------------+

VALUES FOR LATEST VERSION:
VERSION:
    1/2
    
CMD:
    cmdSYN(0)
    cmdFIN(1)
    cmdPSH(2)
    cmdNOP(3)
    cmdUPD(4)	// only supported on version 2
    
STREAMID:
    client use odd numbers starts from 1
    server use even numbers starts from 0
    
cmdUPD:
    | CONSUMED(4B) | WINDOW(4B) |

Usage

func client() {
    // Get a TCP connection
    conn, err := net.Dial(...)
    if err != nil {
        panic(err)
    }

    // Setup client side of smux
    session, err := smux.Client(conn, nil)
    if err != nil {
        panic(err)
    }

    // Open a new stream
    stream, err := session.OpenStream()
    if err != nil {
        panic(err)
    }

    // Stream implements io.ReadWriteCloser
    stream.Write([]byte("ping"))
    stream.Close()
    session.Close()
}

func server() {
    // Accept a TCP connection
    conn, err := listener.Accept()
    if err != nil {
        panic(err)
    }

    // Setup server side of smux
    session, err := smux.Server(conn, nil)
    if err != nil {
        panic(err)
    }

    // Accept a stream
    stream, err := session.AcceptStream()
    if err != nil {
        panic(err)
    }

    // Listen for a message
    buf := make([]byte, 4)
    stream.Read(buf)
    stream.Close()
    session.Close()
}

Configuration

smux.Config allows tuning the session parameters:

• Version: Protocol version (1 or 2).
• KeepAliveInterval: Interval for sending NOP frames to keep the connection alive.
• KeepAliveTimeout: Timeout for closing the session if no data is received.
• MaxFrameSize: Maximum size of a frame.
• MaxReceiveBuffer: Maximum size of the shared receive buffer.
• MaxStreamBuffer: Maximum size of the per-stream buffer.

Reference

• hashicorp/yamux
• xtaci/kcp-go
• xtaci/kcptun