When str.lower() Becomes a Security Vulnerability in Python

Introduction

At first glance, Python's str.lower() function seems simple and harmless. Yet, in certain specific situations, it can introduce a serious security vulnerability. To understand how, we need to delve into the specifics of Internet standards and Unicode.

Background and Internet Standards

Many Internet standards only support ASCII characters, while the world uses much more than the Latin alphabet. To address this, a conversion from Unicode to ASCII for domain names is required. This is where NamePrep comes into play, defined in RFC 3491, a crucial component of Internationalizing Domain Names in Applications, known as "IDNA 2003."

The Role of StringPrep and IDNA

The StringPrep algorithm, defined in RFC 3454, is critical in this process. StringPrep includes a "case folding" step (a form of lowercasing/uppercasing codepoints) enabling case-insensitive string comparisons. In Python, this step is implemented in the stringprep module of the standard library.

The Issue with str.lower()

The core issue lies in the use of str.lower() in Python's implementation of StringPrep. This function uses the Unicode data of the specific version of the Python interpreter, which can lead to inconsistencies with the conversion rules defined in RFC 3454 using Unicode 3.2.0.

Vulnerability Example

Consider the character 'Ꭰ' (U+13A0):

• RFC 3454 compliant value: "ᎠᎠ".encode("idna") yields 'xn--58da'
• Value using Unicode 17.0.0 case-folding: "ᎠᎠ".encode("idna") yields 'xn--kz9aa'

This discrepancy can have surprising and potentially dangerous consequences in critical applications.

How to Fix the Issue

To avoid this vulnerability, it is crucial to use Unicode 3.2.0 conversion rules and not the latest ones. Fortunately, Python provides access to the Unicode 3.2.0 database via unicodedata.ucd_3_2_0, allowing alignment with the RFC 3454 specification.

Conclusion

Handling characters and Unicode standards is a complex but essential aspect of ensuring application security and compatibility. By understanding the specifics of the str.lower() implementation, developers can avoid potential pitfalls.

Let's discuss your project in 15 minutes.