Fix font loading: V1 support, V0 header fix, flexible discovery, doc cleanup

FONT_CONVERSION.md

@@ -1,37 +0,0 @@
-# Font Conversion Guide
-
-To use custom fonts with the CrossPoint Reader, you must convert standard `.ttf` or `.otf` font files into the specific `.epdfont` binary format used by the compiled firmware.
-
-We use a Python script located at `lib/EpdFont/scripts/fontconvert.py`.
-
-## Requirements
-- Python 3
-- `freetype-py` library (`pip install freetype-py`)
-
-## Usage
-
-Run the script from the project root:
-
-```bash
-python3 lib/EpdFont/scripts/fontconvert.py --binary [Family-Style-Size] [Size] [PathToFont]
-```
-
-### Arguments
-1. `name`: The output filename (without extension). **Convention:** `Family-Style-Size` (e.g. `Bookerly-Regular-12`).
-2. `size`: The integer point size (e.g. `12`).
-3. `fontstack`: Path to the source font file (e.g. `fonts/Bookerly-Regular.ttf`).
-4. `--binary`: **REQUIRED**. Flags the script to output the `.epdfont` binary instead of a C header.
-
-### Example
-
-To convert `Bookerly-Regular.ttf` to a size 12 font:
-
-```bash
-python3 lib/EpdFont/scripts/fontconvert.py --binary Bookerly-Regular-12 12 fonts/Bookerly-Regular.ttf
-```
-
-This will generate `Bookerly-Regular-12.epdfont` in your current directory.
-
-## Installing on Device
-1. Rename the file if necessary to match the pattern: `Family-Style-Size.epdfont`.
-2. Copy the `.epdfont` file to the `/fonts` directory on your SD card.

docs/CHANGES.md

@@ -73,3 +73,10 @@ The EPUB reader core was modified to improve stability, performance, and memory
 *   Rendering: `lib/GfxRenderer/GfxRenderer.{cpp,h}`, `lib/EpdFont/EpdFont.{cpp,h}`
 *   EPUB Engine: `lib/Epub/*` (various files optimized and cleaned)
 *   Tools: `lib/EpdFont/scripts/fontconvert.py`
+
+### Update: Enhanced Font Discovery & Format Support (2025-01-20)
+
+*   **V1 Format Support**: Added full support for the newer V1 `.epdfont` format (32-byte header, uint32 offsets) used by the web-based converter (`epdfont.clev.app`).
+*   **V0 Format Fix**: Fixed a regression in V0 font loading where the header read was truncated to 32 bytes (instead of 48), restoring support for `LibreBaskerville` and other legacy fonts.
+*   **Flexible Discovery**: Updated `FontManager` to support `Family_Style_Size` (underscore-separated) naming conventions, enabling compatibility with a wider range of auto-generated filenames.
+*   **Documentation**: Rewrote `FONT_CONVERSION.md` to cover both the Python script and the new web converter.

docs/CUSTOM_FONTS.md

@@ -19,7 +19,9 @@ To optimize for the limited RAM of the ESP32 and the specific requirements of E-
 *   **Input**: TTF/OTF files.
 *   **Output**: `.epdfont` binary file.
 *   **Format Details**:
-    *   **Header**: Contains metadata (magic "EPDF", version, metrics, offsets).
+    *   **Header**:
+        *   **Version 1 (New)**: 32-byte header, uint32 offsets. Compact and efficient.
+        *   **Version 0 (Legacy)**: 48-byte header, uint16 offsets. Retained for backward compatibility.
     *   **Intervals**: Unicode ranges supported by the font.
     *   **Glyphs**: Metrics for each character (width, height, advance, offsets).
     *   **Bitmaps**: 1-bit or 2-bit (antialiased) pixel data for glyphs.
@@ -29,9 +31,10 @@ To optimize for the limited RAM of the ESP32 and the specific requirements of E-
 Fonts are stored on the SD card in the `/fonts` directory.
 
 *   **Location**: `/fonts`
-*   **Naming Convention**: `Family-Style-Size.epdfont`
+*   **Naming Convention**:
-    *   Example: `Literata-Regular-14.epdfont`
+    *   **Standard**: `Family-Style-Size.epdfont` (e.g., `LibreBaskerville-Regular-14.epdfont`)
-    *   Example: `Literata-BoldItalic-14.epdfont`
+    *   **Web Converter**: `Family_Style_Size.epdfont` (e.g., `Aileron_Regular_18.epdfont`)
+    *   **Single File**: `Family.epdfont` (e.g., `Aileron.epdfont`) - automatically detected as Regular style.
 *   **Manager**: `src/managers/FontManager.cpp`
     *   **Scans** the `/fonts` directory on startup/demand.
     *   **Groups** files into `Family -> Size -> Styles (Regular, Bold, Italic, BoldItalic)`.

docs/FONT_CONVERSION.md

@@ -0,0 +1,73 @@
+# Font Conversion Guide
+
+To use custom fonts with the CrossPoint Reader, you must convert standard `.ttf` or `.otf` font files into the specific `.epdfont` binary format used by the compiled firmware.
+
+## Supported Formats
+
+The CrossPoint Reader supports two versions of the `.epdfont` format:
+1.  **Version 1 (Recommended):** The newer, optimized format generated by the web converter.
+2.  **Version 0 (Legacy):** The original format generated by the Python script.
+
+## Method 1: Web Converter (Recommended)
+
+The easiest way to generate compatible fonts is using the online converter.
+
+1.  Go to [https://epdfont.clev.app/](https://epdfont.clev.app/).
+2.  Select your `.ttf` or `.otf` file.
+3.  Choose the font size (e.g., 18).
+4.  Download the generated `.epdfont` file.
+
+### Filename Requirements
+
+The firmware scans for fonts in the `/fonts` directory on the SD card. It attempts to parse the filename to determine the Font Family, Style, and Size.
+
+For best results, rename your downloaded file to match one of these patterns:
+
+*   `Family_Style_Size.epdfont` (e.g., `Aileron_Regular_18.epdfont`)
+*   `Family-Style-Size.epdfont` (e.g., `LibreBaskerville-Bold-14.epdfont`)
+
+**Supported Styles:**
+*   `Regular`
+*   `Bold`
+*   `Italic`
+*   `BoldItalic`
+
+**Note:** If you download a file named just `Aileron.epdfont`, the reader will try to load it, but using the explicit naming convention above ensures the correct style and size are recognized.
+
+## Method 2: Python Script (Legacy)
+
+You can also use the included Python script located at `lib/EpdFont/scripts/fontconvert.py`.
+
+### Requirements
+- Python 3
+- `freetype-py` library (`pip install freetype-py`)
+
+### Usage
+
+Run the script from the project root:
+
+```bash
+python3 lib/EpdFont/scripts/fontconvert.py --binary [Family-Style-Size] [Size] [PathToFont]
+```
+
+### Arguments
+1. `name`: The output filename (without extension). **Convention:** `Family-Style-Size` (e.g. `Bookerly-Regular-12`).
+2. `size`: The integer point size (e.g. `12`).
+3. `fontstack`: Path to the source font file (e.g. `fonts/Bookerly-Regular.ttf`).
+4. `--binary`: **REQUIRED**. Flags the script to output the `.epdfont` binary instead of a C header.
+
+### Example
+
+To convert `Bookerly-Regular.ttf` to a size 12 font:
+
+```bash
+python3 lib/EpdFont/scripts/fontconvert.py --binary Bookerly-Regular-12 12 fonts/Bookerly-Regular.ttf
+```
+
+This will generate `Bookerly-Regular-12.epdfont` in your current directory.
+
+## Installing on Device
+
+1.  Copy your generated `.epdfont` files to the `/fonts` directory on your SD card.
+2.  Restart the CrossPoint Reader.
+3.  Go to **Settings** -> **Set Custom Font Family** to select your loaded fonts.

lib/EpdFont/CustomEpdFont.cpp

@@ -6,12 +6,13 @@
 #include <algorithm>
 
 CustomEpdFont::CustomEpdFont(const String& filePath, const EpdFontData* data, uint32_t offsetIntervals,
-                             uint32_t offsetGlyphs, uint32_t offsetBitmaps)
+                             uint32_t offsetGlyphs, uint32_t offsetBitmaps, int version)
     : EpdFont(data),
       filePath(filePath),
       offsetIntervals(offsetIntervals),
       offsetGlyphs(offsetGlyphs),
-      offsetBitmaps(offsetBitmaps) {
+      offsetBitmaps(offsetBitmaps),
+      version(version) {
   // Initialize bitmap cache
   for (size_t i = 0; i < BITMAP_CACHE_CAPACITY; i++) {
     bitmapCache[i].data = nullptr;
@@ -112,26 +113,8 @@ const EpdGlyph* CustomEpdFont::getGlyph(uint32_t cp, const EpdFontStyles::Style
     }
 
     if (foundInterval) {
-      // Calculate total glyphs to ensure bounds safety
+      uint32_t stride = (version == 1) ? 16 : 13;
-      uint32_t totalGlyphCount = (offsetBitmaps - offsetGlyphs) / 13;
+      uint32_t glyphFileOffset = offsetGlyphs + (glyphIndex * stride);
-      if (glyphIndex >= totalGlyphCount) {
-        Serial.printf("CustomEpdFont: Glyph index %u out of bounds (total %u)\n", glyphIndex, totalGlyphCount);
-        // If out of bounds, and we haven't tried fallback, try it.
-        if (!triedFallback) {
-          if (currentCp == 0x2018 || currentCp == 0x2019) {
-            currentCp = 0x0027;
-            triedFallback = true;
-            continue;
-          } else if (currentCp == 0x201C || currentCp == 0x201D) {
-            currentCp = 0x0022;
-            triedFallback = true;
-            continue;
-          }
-        }
-        return nullptr;
-      }
-
-      uint32_t glyphFileOffset = offsetGlyphs + (glyphIndex * 13);
 
       if (!fontFile.isOpen()) {
         if (!SdMan.openFileForRead("CustomFont", filePath.c_str(), fontFile)) {
@@ -146,22 +129,62 @@ const EpdGlyph* CustomEpdFont::getGlyph(uint32_t cp, const EpdFontStyles::Style
         return nullptr;
       }
 
-      uint8_t glyphBuf[13];
+      uint8_t w, h, adv, res = 0;
-      if (fontFile.read(glyphBuf, 13) != 13) {
+      int16_t l, t = 0;
-        Serial.println("CustomEpdFont: Read failed (glyph entry)");
+      uint32_t dLen, dOffset = 0;
-        fontFile.close();
-        return nullptr;
-      }
 
-      uint8_t w = glyphBuf[0];
+      if (version == 1) {
-      uint8_t h = glyphBuf[1];
+        // New format (16 bytes)
-      uint8_t adv = glyphBuf[2];
+        uint8_t glyphBuf[16];
-      int8_t l = (int8_t)glyphBuf[3];
+        if (fontFile.read(glyphBuf, 16) != 16) {
-      // glyphBuf[4] unused
+          Serial.println("CustomEpdFont: Read failed (glyph entry v1)");
-      int8_t t = (int8_t)glyphBuf[5];
+          fontFile.close();
-      // glyphBuf[6] unused
+          return nullptr;
-      uint16_t dLen = glyphBuf[7] | (glyphBuf[8] << 8);
+        }
-      uint32_t dOffset = glyphBuf[9] | (glyphBuf[10] << 8) | (glyphBuf[11] << 16) | (glyphBuf[12] << 24);
+
+        /*
+          view.setUint8(offset++, glyph.width);
+          view.setUint8(offset++, glyph.height);
+          view.setUint8(offset++, glyph.advanceX);
+          view.setUint8(offset++, 0);
+          view.setInt16(offset, glyph.left, true);
+          offset += 2;
+          view.setInt16(offset, glyph.top, true);
+          offset += 2;
+          view.setUint32(offset, glyph.dataLength, true);
+          offset += 4;
+          view.setUint32(offset, glyph.dataOffset, true);
+          offset += 4;
+        */
+
+        w = glyphBuf[0];
+        h = glyphBuf[1];
+        adv = glyphBuf[2];
+        res = glyphBuf[3];
+        l = (int16_t)(glyphBuf[4] | (glyphBuf[5] << 8));  // Little endian int16
+        t = (int16_t)(glyphBuf[6] | (glyphBuf[7] << 8));  // Little endian int16
+        dLen = glyphBuf[8] | (glyphBuf[9] << 8) | (glyphBuf[10] << 16) | (glyphBuf[11] << 24);
+        dOffset = glyphBuf[12] | (glyphBuf[13] << 8) | (glyphBuf[14] << 16) | (glyphBuf[15] << 24);
+
+      } else {
+        // Old format (13 bytes)
+        uint8_t glyphBuf[13];
+        if (fontFile.read(glyphBuf, 13) != 13) {
+          Serial.println("CustomEpdFont: Read failed (glyph entry)");
+          fontFile.close();
+          return nullptr;
+        }
+
+        w = glyphBuf[0];
+        h = glyphBuf[1];
+        adv = glyphBuf[2];
+        l = (int8_t)glyphBuf[3];
+        // glyphBuf[4] unused
+        t = (int8_t)glyphBuf[5];
+        // glyphBuf[6] unused
+        dLen = glyphBuf[7] | (glyphBuf[8] << 8);
+        dOffset = glyphBuf[9] | (glyphBuf[10] << 8) | (glyphBuf[11] << 16) | (glyphBuf[12] << 24);
+      }
 
       /*
       Serial.printf("[CEF] Parsed Glyph %u: Off=%u, Len=%u, W=%u, H=%u, L=%d, T=%d\n",

lib/EpdFont/CustomEpdFont.h

@@ -22,7 +22,7 @@ struct GlyphStructCacheEntry {
 class CustomEpdFont : public EpdFont {
  public:
   CustomEpdFont(const String& filePath, const EpdFontData* data, uint32_t offsetIntervals, uint32_t offsetGlyphs,
-                uint32_t offsetBitmaps);
+                uint32_t offsetBitmaps, int version = 0);
   ~CustomEpdFont() override;
 
   const EpdGlyph* getGlyph(uint32_t cp, const EpdFontStyles::Style style = EpdFontStyles::REGULAR) const override;
@@ -45,6 +45,7 @@ class CustomEpdFont : public EpdFont {
   mutable GlyphStructCacheEntry glyphCache[GLYPH_CACHE_CAPACITY];
 
   mutable uint32_t currentAccessCount = 0;
+  int version = 0;
 
   void clearCache() const;
 };

lib/EpdFont/EpdFontData.h

@@ -11,7 +11,7 @@ typedef struct {
   uint8_t advanceX;     ///< Distance to advance cursor (x axis)
   int16_t left;         ///< X dist from cursor pos to UL corner
   int16_t top;          ///< Y dist from cursor pos to UL corner
-  uint16_t dataLength;  ///< Size of the font data.
+  uint32_t dataLength;  ///< Size of the font data.
   uint32_t dataOffset;  ///< Pointer into EpdFont->bitmap
 } EpdGlyph;
 

src/managers/FontManager.cpp

@@ -56,10 +56,21 @@ void FontManager::scanFonts() {
 
       String name = String(filename);
       if (name.endsWith(".epdfont")) {
-        // Expected format: Family-Style-Size.epdfont
+        // Expected format: Family-Style-Size.epdfont or Family_Size.epdfont
-        int firstDash = name.indexOf('-');
+        // Or just Family.epdfont (V1 single file)
-        if (firstDash > 0) {
+
-          String family = name.substring(0, firstDash);
+        String family;
+        int separator = name.indexOf('-');
+        if (separator < 0) separator = name.indexOf('_');
+
+        if (separator > 0) {
+          family = name.substring(0, separator);
+        } else {
+          // No separator, take the whole name (minus .epdfont)
+          family = name.substring(0, name.length() - 8);
+        }
+
+        if (family.length() > 0) {
           if (std::find(availableFamilies.begin(), availableFamilies.end(), family.c_str()) ==
               availableFamilies.end()) {
             availableFamilies.push_back(family.c_str());
@@ -124,23 +135,83 @@ CustomEpdFont* loadFontFile(const String& path) {
     return nullptr;
   }
 
-  uint32_t intervalCount = buf[1];
+  Serial.printf("[FontMgr] Header Dump %s: ", path.c_str());
-  uint32_t fileSize = buf[2];
+  for (int i = 0; i < 12; i++) Serial.printf("%08X ", buf[i]);
-  uint32_t height = buf[3];
+  Serial.println();
-  uint32_t glyphCount = buf[4];
-  int32_t ascender = (int32_t)buf[5];
-  int32_t descender = (int32_t)buf[7];
 
-  uint32_t offsetIntervals = buf[9];
+  /*
-  uint32_t offsetGlyphs = buf[10];
+   * Version Detection Improved
-  uint32_t offsetBitmaps = buf[11];
+   *
+   * V1:
+   *   Offset 20 (buf[5]) is OffsetIntervals. It matches header size = 32.
+   *   Offset 4 (buf[1]) low 16 bits is Version = 1.
+   *
+   * V0:
+   *   Offset 36 (buf[9]) is OffsetIntervals. It matches header size = 48.
+   */
 
-  Serial.printf("[FontMgr] parsed header: intv=%u, glyphs=%u, fileSz=%u, h=%u, asc=%d, desc=%d\n", intervalCount,
+  int version = -1;
-                glyphCount, fileSize, height, ascender, descender);
+
-  Serial.printf("[FontMgr] offsets: intv=%u, gly=%u, bmp=%u\n", offsetIntervals, offsetGlyphs, offsetBitmaps);
+  // Check for V1
+  if (buf[5] == 32 && (buf[1] & 0xFFFF) == 1) {
+    version = 1;
+  }
+  // Check for V0
+  else if (buf[9] == 48) {
+    version = 0;
+  }
+  // Fallback: Use the old file size check if offsets are weird (detected from legacy files?)
+  else if (buf[2] > 10000) {
+    // V0 has fileSize at offset 8 (buf[2])
+    version = 0;
+  }
+
+  uint32_t intervalCount, fileSize, glyphCount, offsetIntervals, offsetGlyphs, offsetBitmaps;
+  uint8_t height, advanceY;
+  int32_t ascender, descender;
+  bool is2Bit;
+
+  if (version == 1) {
+    // V1 Parsing
+    uint8_t* b8 = (uint8_t*)buf;
+
+    is2Bit = (b8[6] != 0);
+    advanceY = b8[8];
+    ascender = (int8_t)b8[9];
+    descender = (int8_t)b8[10];
+
+    intervalCount = b8[12] | (b8[13] << 8) | (b8[14] << 16) | (b8[15] << 24);
+    glyphCount = b8[16] | (b8[17] << 8) | (b8[18] << 16) | (b8[19] << 24);
+    offsetIntervals = b8[20] | (b8[21] << 8) | (b8[22] << 16) | (b8[23] << 24);
+    offsetGlyphs = b8[24] | (b8[25] << 8) | (b8[26] << 16) | (b8[27] << 24);
+    offsetBitmaps = b8[28] | (b8[29] << 8) | (b8[30] << 16) | (b8[31] << 24);
+
+    height = advanceY;
+    fileSize = 0;  // Unknown
+
+  } else if (version == 0) {
+    // V0 Parsing
+    // We already read 48 bytes into buf
+    intervalCount = buf[1];
+    fileSize = buf[2];
+    height = buf[3];
+    glyphCount = buf[4];
+    ascender = (int32_t)buf[5];
+    descender = (int32_t)buf[7];
+    is2Bit = (buf[8] != 0);
+
+    offsetIntervals = buf[9];
+    offsetGlyphs = buf[10];
+    offsetBitmaps = buf[11];
+  } else {
+    Serial.printf("[FontMgr] Unknown version for %s\n", path.c_str());
+    f.close();
+    return nullptr;
+  }
 
   // Validation
-  if (offsetIntervals >= fileSize || offsetGlyphs >= fileSize || offsetBitmaps >= fileSize) {
+  // For V1, we trust offsets generated by the tool
+  if (offsetIntervals == 0 || offsetGlyphs == 0 || offsetBitmaps == 0) {
     Serial.println("[FontMgr] Invalid offsets in header");
     f.close();
     return nullptr;
@@ -178,11 +249,10 @@ CustomEpdFont* loadFontFile(const String& path) {
   fontData->advanceY = (uint8_t)height;
   fontData->ascender = ascender;
   fontData->descender = descender;
-  fontData->descender = descender;
+  fontData->is2Bit = is2Bit;
-  fontData->is2Bit = (buf[8] != 0);
   fontData->bitmap = nullptr;
 
-  return new CustomEpdFont(path, fontData, offsetIntervals, offsetGlyphs, offsetBitmaps);
+  return new CustomEpdFont(path, fontData, offsetIntervals, offsetGlyphs, offsetBitmaps, version);
 }
 
 EpdFontFamily* FontManager::getCustomFontFamily(const std::string& familyName, int fontSize) {
@@ -194,9 +264,29 @@ EpdFontFamily* FontManager::getCustomFontFamily(const std::string& familyName, i
   String sizeStr = String(fontSize);
 
   CustomEpdFont* regular = loadFontFile(basePath + "Regular-" + sizeStr + ".epdfont");
+
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Regular_" + sizeStr + ".epdfont");
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + "_" + sizeStr + ".epdfont");
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + ".epdfont");
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + "-Regular.epdfont");
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Regular.epdfont");
+  if (!regular) regular = loadFontFile("/fonts/" + String(familyName.c_str()) + "-" + sizeStr + ".epdfont");
+
   CustomEpdFont* bold = loadFontFile(basePath + "Bold-" + sizeStr + ".epdfont");
+  if (!bold) bold = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Bold_" + sizeStr + ".epdfont");
+  if (!bold) bold = loadFontFile("/fonts/" + String(familyName.c_str()) + "-Bold.epdfont");
+  if (!bold) bold = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Bold.epdfont");
+
   CustomEpdFont* italic = loadFontFile(basePath + "Italic-" + sizeStr + ".epdfont");
+  if (!italic) italic = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Italic_" + sizeStr + ".epdfont");
+  if (!italic) italic = loadFontFile("/fonts/" + String(familyName.c_str()) + "-Italic.epdfont");
+  if (!italic) italic = loadFontFile("/fonts/" + String(familyName.c_str()) + "_Italic.epdfont");
+
   CustomEpdFont* boldItalic = loadFontFile(basePath + "BoldItalic-" + sizeStr + ".epdfont");
+  if (!boldItalic)
+    boldItalic = loadFontFile("/fonts/" + String(familyName.c_str()) + "_BoldItalic_" + sizeStr + ".epdfont");
+  if (!boldItalic) boldItalic = loadFontFile("/fonts/" + String(familyName.c_str()) + "-BoldItalic.epdfont");
+  if (!boldItalic) boldItalic = loadFontFile("/fonts/" + String(familyName.c_str()) + "_BoldItalic.epdfont");
 
   if (!regular) {
     if (bold) regular = bold;