Python UnicodeError — What It Means & How to Fix It

What Does This Error Mean?

A Python UnicodeError means Python encountered a text character it could not handle using the current encoding. Python stores text as Unicode internally, but when reading files, printing to a terminal, or sending text over a network, it must convert to a specific encoding like UTF-8 or ASCII. If the text contains a character the chosen encoding cannot represent — or if the encoded bytes do not match the expected encoding — Python raises a UnicodeError. The two most common subtypes are UnicodeDecodeError (reading bytes that do not match the encoding) and UnicodeEncodeError (converting text to bytes for output).

Affected Models

Python 2.x
Python 3.x
All Python versions

Common Causes

Reading a file that was saved with a different encoding than you specified (for example, opening a Latin-1 file as UTF-8)
Trying to print a string containing non-ASCII characters to a terminal that only supports ASCII
A database or API returning byte strings with a different encoding than your code assumes
Python 2 mixing byte strings and unicode strings without explicit conversion
A text file containing a BOM (Byte Order Mark) at the start that the reader is not expecting

How to Fix It

When opening a file, always specify the encoding explicitly. Use open('filename.txt', 'r', encoding='utf-8') rather than relying on Python's default encoding, which varies by operating system.

On Windows, Python's default encoding is often cp1252 (Windows-1252), not UTF-8. Always specify encoding='utf-8' when working with files that might contain non-English characters.
If you do not know the file's encoding, use the 'chardet' library to detect it. Install it with: pip install chardet. Then use chardet.detect(raw_bytes) to identify the encoding before decoding.

chardet analyses the byte patterns in the file and makes a confident guess at the encoding. It is not perfect, but it handles the most common cases.
To ignore or replace characters that cause errors, add an errors parameter. Use open('file.txt', encoding='utf-8', errors='ignore') to skip bad characters, or errors='replace' to substitute them with a placeholder.

Use 'ignore' only when you accept that some characters will be lost. Use 'replace' when you want to see where problems occur. Neither is the same as fixing the root encoding mismatch.
If you see a UnicodeEncodeError when printing, your terminal may not support the character. On Windows, run: chcp 65001 in Command Prompt before running your script to switch the terminal to UTF-8 mode.

This is especially common on Windows where older terminals default to a limited code page that cannot display many international characters.
If you are working with a file that starts with a BOM (Byte Order Mark), open it with encoding='utf-8-sig' instead of 'utf-8'. The 'utf-8-sig' encoding automatically strips the BOM.

Files created in Windows Notepad or Excel often start with a hidden BOM. Without utf-8-sig, the BOM appears as junk characters at the start of your data.

When to Call a Professional

UnicodeErrors are always fixable in code. No external help is needed. The key is identifying whether the problem is at read time (decoding) or write time (encoding), and specifying the correct encoding for the data you are working with.

Frequently Asked Questions

Why does my Python script work on Mac but break on Windows with a UnicodeError?

Mac and Linux default to UTF-8 for almost everything. Windows uses different default encodings depending on the context — often cp1252 for files and a legacy code page for the terminal. The fix is to always specify encoding='utf-8' explicitly in your open() calls rather than relying on the system default.

What is the difference between UnicodeDecodeError and UnicodeEncodeError?

UnicodeDecodeError happens when you are reading bytes and Python cannot interpret them as valid characters using the specified encoding — the bytes coming in do not match what the encoding expects. UnicodeEncodeError happens when you are writing text out and Python cannot convert a character into bytes using the target encoding — the character exists in Python's internal format but cannot be expressed in the output encoding.

Should I just use errors='ignore' to stop the errors?

Only as a last resort. Using errors='ignore' silently drops characters from your data, which can corrupt the meaning of text. It is better to identify the correct encoding and open the file properly. Use 'ignore' only when you are sure the unreadable characters are garbage bytes and not real content.