please detect encoding before opening a file

[HaujetZhao]

Please introduce modules to detect file encoding.

chardet is an option.

The default behavior of rich-cli opening a file is using system prefered encoding. Windows in some different country have different locale encoding.

So rich-cli will raise error if attempt to open utf-8 files.

The solution is: open file in bytes mode, read the bytes, detect the encoding using chardet module, decode the bytes to text content.

[harkabeeparolus]

I have doubts that this would be a good idea. Personally, I've been using ftfy for years, and I trust it. And ftfy's recommendation is:

Don't use chardet

Encoding detection on raw bytes is not a good idea. It was important in the ’90s, during the rough transition to Unicode – and the most popular way of doing it, chardet, hasn’t changed since the ’90s.

A heuristic designed before there was multilingual social media, before there were emoji, is not going to work correctly in the 2020s.

I'd be happy to hear counterarguments, but I don't think that chardet in particular would be helpful here. If anything, I'd consider using, vendoring or reimplementing, the simplified heuristics that ftfy uses:

Though it’s not part of the usual operation of ftfy, ftfy does contain a byte-encoding-guesser that tries to be less terrible than other byte-encoding-guessers in common cases. Instead of using probabilistic heuristics, it picks up on very strong signals like “having a UTF-16 byte-order mark” or “decoding successfully as UTF-8”.

Something I think we can certainly do, however, is to ensure that we use the encoding "utf-8-sig" everywhere for reading files. Some Windows or Microsoft programs will generate UTF-8 files with an unnecessary BOM, and the "utf-8-sig" will correctly and safely skip it.

Thoughts?

[javs1979]

Notwithstanding the proposed implementations, I encountered a problem with rich --pager to display a text file containing a (Rich) Tree with diverse strings. For the most part, strings contain characters with accents.

Text file is certified UTF-8 (according to latest VS Code).

Environnement

• Windows 11
• VS Code 1.105.1.
• Windows Terminal 1.23.12811.0 with latest JetBrains Mono font (popular Nerd Font)
• rich-cli version 1.8.0.

Steps to reproduce

1. Generate a Rich Tree with some Python code. Output of Python script goes directly to a text file (using >). PYTHONIOENCODING is set to UTF-8 beforehand.
2. Save Tree to a text file, let's call it tree_example.txt.
3. Ensure text file is properly encoded to UTF-8. According to VS Code, it already is.
4. In Windows Terminal, run rich --pager tree_example.txt

Expected behavior

rich --pager shows file like this...

Entités
├── Sous-entité 1
├── Sous-entité 2
└── Sous-entité 3

Actual behavior

rich --pager shows file like that...

Entités
├── Sous-entité 1
├── Sous-entité 2
└── Sous-entité 3

(Comparing the actual output with different encodings, it appears file is displayed as if its encoding was windows-1252).

Conclusion

rich-cli needs to detect encoding before opening a file... anyhow.