WEBDIENSTE/ECC
1
0
Fork 0
mirror of https://github.com/affaan-m/ECC.git synced 2026-10-14 12:15:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix: address remaining CodeRabbit review comments

Browse Source 
- Add missing Urdu link to Bengali README language selectors
- Narrow zero-width char regex to only ZWSP (U+200B) and BOM (U+FEFF),
  preserving bidi controls and separators
- Add pg_encoding_to_char check for PostgreSQL UTF-8 verification

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Md Kishor Morol
2026-08-02 23:59:38 -04:00
co-authored by Claude Opus 4.6
parent 45a9780118
commit 2a31c83b0e
2 changed files with 6 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/bn/README.md
+2 -2
View File
@@ -1,4 +1,4 @@
**ভাষা:** [English](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | বাংলা
**ভাষা:** [English](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | [اردو](../ur/README.md) | বাংলা
# Everything Claude Code
@@ -15,7 +15,7 @@
**Language / ভাষা / 语言 / 語言 / 언어 / Dil / Язык / Ngôn ngữ**
[**English**](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | [বাংলা](README.md)
[**English**](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | [اردو](../ur/README.md) | [বাংলা](README.md)
</div>
skills/bengali-nlp/SKILL.md
+4 -3
View File
@@ -287,7 +287,7 @@ def prepare_bangla_dataset(texts: list[str]) -> list[str]:
# Normalize Unicode
text = unicodedata.normalize("NFC", text)
# Remove unwanted zero-width characters but preserve ZWNJ/ZWJ (important for conjuncts)
text = re.sub(r'[\u200b\u200e\u200f\u2028-\u202f\ufeff]', '', text)
text = re.sub(r'[\u200b\ufeff]', '', text)
# Normalize whitespace
text = re.sub(r'\s+', ' ', text).strip()
if text:
@@ -305,7 +305,8 @@ CREATE TABLE bengali_content (
body TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
);
-- PostgreSQL: UTF-8 is the default. Verify ICU collation availability first:
-- PostgreSQL: Verify UTF-8 encoding and ICU collation availability:
-- SELECT pg_encoding_to_char(encoding) FROM pg_database WHERE datname = current_database();
-- SELECT collname FROM pg_collation WHERE collname LIKE 'bn%';
-- If missing, create it (requires ICU-enabled PostgreSQL build):
-- CREATE COLLATION IF NOT EXISTS "bn-BD-x-icu" (provider = icu, locale = 'bn-BD');
@@ -370,5 +371,5 @@ title = "বাংলা ভাষা"
text = scraped_html.get_text()
# GOOD — strip unwanted zero-width chars, preserving ZWNJ (U+200C) and ZWJ (U+200D)
text = re.sub(r'[\u200b\u200e\u200f\u2028-\u202f\ufeff]', '', scraped_html.get_text())
text = re.sub(r'[\u200b\ufeff]', '', scraped_html.get_text())
```