Introduction
When working on security, it's a pleasant surprise to accidentally achieve a performance boost. This is precisely what happened with Bijou64 – a variable-length integer encoding developed for the Subduction CRDT sync protocol. Initially designed to fix a subtle signature verification bug, Bijou64 proved to be not only more secure but also several times faster than the widely-used LEB128 varint encoding.
The Problem with Traditional Varint Encodings
Many binary protocols require a compact way to encode integers that are usually small but occasionally large. Variable-length integer encodings solve this issue. However, most designs treat canonicalization as an afterthought, often requiring a runtime check in the decoder rather than being built into the structure of the encoding.
LEB128 is a popular choice for many projects due to its simplicity and efficiency. It encodes a number as a sequence of 7-bit segments, with the high bit of each byte signaling if more bytes follow. This method avoids writing a full 8 bytes even when representing small numbers. However, a problem arises: each number can be represented in multiple ways in LEB128, complicating signed data management and compression.
Bijou64: An Elegant Solution
Bijou64 was created to address these limitations by ensuring that a number can only be represented in one unique way. This not only simplifies data storage and compression but also enhances security by minimizing signature verification errors. By avoiding multiple representations, Bijou64 facilitates deduplication of number sequences when stored.
Enhanced Performance
Recent tests show that Bijou64 is not only more secure but also more efficient than LEB128. Benchmarks indicate it is on average 2 to 3 times faster, a valuable gain in environments where processing speed is crucial.
Use Cases and Adoption
Bijou64 fits well in various applications where efficiency and security are paramount, such as distributed file systems or secure communication protocols. Its growing adoption demonstrates developers recognize its added value over traditional solutions.
Conclusion
In conclusion, Bijou64 represents a significant advancement in varint encodings. Its ability to combine security and performance makes it an indispensable choice for modern projects requiring efficient integer management. Let's discuss your project in 15 minutes.