mirror of
https://github.com/daveallie/crosspoint-reader.git
synced 2026-02-04 14:47:37 +03:00
perf: optimize large EPUB indexing from O(n^2) to O(n) (#458)
## Summary Optimizes EPUB metadata indexing for large books (2000+ chapters) from ~30 minutes to ~50 seconds by replacing O(n²) algorithms with O(n log n) hash-indexed lookups. Fixes #134 ## Problem Three phases had O(n²) complexity due to nested loops: | Phase | Operation | Before (2768 chapters) | |-------|-----------|------------------------| | OPF Pass | For each spine ref, scan all manifest items | ~25 min | | TOC Pass | For each TOC entry, scan all spine items | ~5 min | | buildBookBin | For each spine item, scan ZIP central directory | ~8.4 min | Total: **~30+ minutes** for first-time indexing of large EPUBs. ## Solution Replace linear scans with sorted hash indexes + binary search: - **OPF Pass**: Build `{hash(id), len, offset}` index from manifest, binary search for each spine ref - **TOC Pass**: Build `{hash(href), len, spineIndex}` index from spine, binary search for each TOC entry - **buildBookBin**: New `ZipFile::fillUncompressedSizes()` API - single ZIP central directory scan with batch hash matching All indexes use FNV-1a hashing with length as secondary key to minimize collisions. Indexes are freed immediately after each phase. ## Results **Shadow Slave EPUB (2768 chapters):** | Phase | Before | After | Speedup | |-------|--------|-------|---------| | OPF pass | ~25 min | 10.8 sec | ~140x | | TOC pass | ~5 min | 4.7 sec | ~60x | | buildBookBin | 506 sec | 34.6 sec | ~15x | | **Total** | **~30+ min** | **~50 sec** | **~36x** | **Normal EPUB (87 chapters):** 1.7 sec - no regression. ## Memory Peak temporary memory during indexing: - OPF index: ~33KB (2770 items × 12 bytes) - TOC index: ~33KB (2768 items × 12 bytes) - ZIP batch: ~44KB (targets + sizes arrays) All indexes cleared immediately after each phase. No OOM risk on ESP32-C3. ## Note on Threshold All optimizations are gated by `LARGE_SPINE_THRESHOLD = 400` to preserve existing behavior for small books. However, the algorithms work correctly for any book size and are faster even for small books: | Book Size | Old O(n²) | New O(n log n) | Improvement | |-----------|-----------|----------------|-------------| | 10 ch | 100 ops | 50 ops | 2x | | 100 ch | 10K ops | 800 ops | 12x | | 400 ch | 160K ops | 4K ops | 40x | If preferred, the threshold could be removed to use the optimized path universally. ## Testing - [x] Shadow Slave (2768 chapters): 50s first-time indexing, loads and navigates correctly - [x] Normal book (87 chapters): 1.7s indexing, no regression - [x] Build passes - [x] clang-format passes ## Files Changed - `lib/Epub/Epub/parsers/ContentOpfParser.h/.cpp` - OPF manifest index - `lib/Epub/Epub/BookMetadataCache.h/.cpp` - TOC index + batch size lookup - `lib/ZipFile/ZipFile.h/.cpp` - New `fillUncompressedSizes()` API - `lib/Epub/Epub.cpp` - Timing logs <details> <summary><b>Algorithm Details</b> (click to expand)</summary> ### Phase 1: OPF Pass - Manifest to Spine Lookup **Problem**: Each `<itemref idref="ch001">` in spine must find matching `<item id="ch001" href="...">` in manifest. ``` OLD: For each of 2768 spine refs, scan all 2770 manifest items = 7.6M string comparisons NEW: While parsing manifest, build index: { hash("ch001"), len=5, file_offset=120 } Sort index, then binary search for each spine ref: 2768 × log₂(2770) ≈ 2768 × 11 = 30K comparisons ``` ### Phase 2: TOC Pass - TOC Entry to Spine Index Lookup **Problem**: Each TOC entry with `href="chapter0001.xhtml"` must find its spine index. ``` OLD: For each of 2768 TOC entries, scan all 2768 spine entries = 7.6M string comparisons NEW: At beginTocPass(), read spine once and build index: { hash("OEBPS/chapter0001.xhtml"), len=25, spineIndex=0 } Sort index, binary search for each TOC entry: 2768 × log₂(2768) ≈ 30K comparisons Clear index at endTocPass() to free memory. ``` ### Phase 3: buildBookBin - ZIP Size Lookup **Problem**: Need uncompressed file size for each spine item (for reading progress). Sizes are in ZIP central directory. ``` OLD: For each of 2768 spine items, scan ZIP central directory (2773 entries) = 7.6M filename reads + string comparisons Time: 506 seconds NEW: Step 1: Build targets from spine { hash("OEBPS/chapter0001.xhtml"), len=25, index=0 } Sort by (hash, len) Step 2: Single pass through ZIP central directory For each entry: - Compute hash ON THE FLY (no string allocation) - Binary search targets - If match: sizes[target.index] = uncompressedSize Step 3: Use sizes array directly (O(1) per spine item) Total: 2773 entries × log₂(2768) ≈ 33K comparisons Time: 35 seconds ``` ### Why Hash + Length? Using 64-bit FNV-1a hash + string length as a composite key: - Collision probability: ~1 in 2⁶⁴ × typical_path_lengths - No string storage needed in index (just 12-16 bytes per entry) - Integer comparisons are faster than string comparisons - Verification on match handles the rare collision case </details> --- _AI-assisted development. All changes tested on hardware._
This commit is contained in:
Daniel Chelling
Uri Tauber
parent
11d30e7ca8
commit
b64f2a941d
@ -230,6 +230,8 @@ bool Epub::load(const bool buildIfMissing) {
|
||||
Serial.printf("[%lu] [EBP] Cache not found, building spine/TOC cache\n", millis());
|
||||
setupCacheDir();
|
||||
|
||||
const uint32_t indexingStart = millis();
|
||||
|
||||
// Begin building cache - stream entries to disk immediately
|
||||
if (!bookMetadataCache->beginWrite()) {
|
||||
Serial.printf("[%lu] [EBP] Could not begin writing cache\n", millis());
|
||||
@ -237,6 +239,7 @@ bool Epub::load(const bool buildIfMissing) {
|
||||
}
|
||||
|
||||
// OPF Pass
|
||||
const uint32_t opfStart = millis();
|
||||
BookMetadataCache::BookMetadata bookMetadata;
|
||||
if (!bookMetadataCache->beginContentOpfPass()) {
|
||||
Serial.printf("[%lu] [EBP] Could not begin writing content.opf pass\n", millis());
|
||||
@ -250,8 +253,10 @@ bool Epub::load(const bool buildIfMissing) {
|
||||
Serial.printf("[%lu] [EBP] Could not end writing content.opf pass\n", millis());
|
||||
return false;
|
||||
}
|
||||
Serial.printf("[%lu] [EBP] OPF pass completed in %lu ms\n", millis(), millis() - opfStart);
|
||||
|
||||
// TOC Pass - try EPUB 3 nav first, fall back to NCX
|
||||
const uint32_t tocStart = millis();
|
||||
if (!bookMetadataCache->beginTocPass()) {
|
||||
Serial.printf("[%lu] [EBP] Could not begin writing toc pass\n", millis());
|
||||
return false;
|
||||
@ -280,6 +285,7 @@ bool Epub::load(const bool buildIfMissing) {
|
||||
Serial.printf("[%lu] [EBP] Could not end writing toc pass\n", millis());
|
||||
return false;
|
||||
}
|
||||
Serial.printf("[%lu] [EBP] TOC pass completed in %lu ms\n", millis(), millis() - tocStart);
|
||||
|
||||
// Close the cache files
|
||||
if (!bookMetadataCache->endWrite()) {
|
||||
@ -288,10 +294,13 @@ bool Epub::load(const bool buildIfMissing) {
|
||||
}
|
||||
|
||||
// Build final book.bin
|
||||
const uint32_t buildStart = millis();
|
||||
if (!bookMetadataCache->buildBookBin(filepath, bookMetadata)) {
|
||||
Serial.printf("[%lu] [EBP] Could not update mappings and sizes\n", millis());
|
||||
return false;
|
||||
}
|
||||
Serial.printf("[%lu] [EBP] buildBookBin completed in %lu ms\n", millis(), millis() - buildStart);
|
||||
Serial.printf("[%lu] [EBP] Total indexing completed in %lu ms\n", millis(), millis() - indexingStart);
|
||||
|
||||
if (!bookMetadataCache->cleanupTmpFiles()) {
|
||||
Serial.printf("[%lu] [EBP] Could not cleanup tmp files - ignoring\n", millis());
|
||||
|
||||
@ -40,7 +40,6 @@ bool BookMetadataCache::endContentOpfPass() {
|
||||
bool BookMetadataCache::beginTocPass() {
|
||||
Serial.printf("[%lu] [BMC] Beginning toc pass\n", millis());
|
||||
|
||||
// Open spine file for reading
|
||||
if (!SdMan.openFileForRead("BMC", cachePath + tmpSpineBinFile, spineFile)) {
|
||||
return false;
|
||||
}
|
||||
@ -48,12 +47,41 @@ bool BookMetadataCache::beginTocPass() {
|
||||
spineFile.close();
|
||||
return false;
|
||||
}
|
||||
|
||||
if (spineCount >= LARGE_SPINE_THRESHOLD) {
|
||||
spineHrefIndex.clear();
|
||||
spineHrefIndex.reserve(spineCount);
|
||||
spineFile.seek(0);
|
||||
for (int i = 0; i < spineCount; i++) {
|
||||
auto entry = readSpineEntry(spineFile);
|
||||
SpineHrefIndexEntry idx;
|
||||
idx.hrefHash = fnvHash64(entry.href);
|
||||
idx.hrefLen = static_cast<uint16_t>(entry.href.size());
|
||||
idx.spineIndex = static_cast<int16_t>(i);
|
||||
spineHrefIndex.push_back(idx);
|
||||
}
|
||||
std::sort(spineHrefIndex.begin(), spineHrefIndex.end(),
|
||||
[](const SpineHrefIndexEntry& a, const SpineHrefIndexEntry& b) {
|
||||
return a.hrefHash < b.hrefHash || (a.hrefHash == b.hrefHash && a.hrefLen < b.hrefLen);
|
||||
});
|
||||
spineFile.seek(0);
|
||||
useSpineHrefIndex = true;
|
||||
Serial.printf("[%lu] [BMC] Using fast index for %d spine items\n", millis(), spineCount);
|
||||
} else {
|
||||
useSpineHrefIndex = false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
bool BookMetadataCache::endTocPass() {
|
||||
tocFile.close();
|
||||
spineFile.close();
|
||||
|
||||
spineHrefIndex.clear();
|
||||
spineHrefIndex.shrink_to_fit();
|
||||
useSpineHrefIndex = false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -124,6 +152,18 @@ bool BookMetadataCache::buildBookBin(const std::string& epubPath, const BookMeta
|
||||
// LUTs complete
|
||||
// Loop through spines from spine file matching up TOC indexes, calculating cumulative size and writing to book.bin
|
||||
|
||||
// Build spineIndex->tocIndex mapping in one pass (O(n) instead of O(n*m))
|
||||
std::vector<int16_t> spineToTocIndex(spineCount, -1);
|
||||
tocFile.seek(0);
|
||||
for (int j = 0; j < tocCount; j++) {
|
||||
auto tocEntry = readTocEntry(tocFile);
|
||||
if (tocEntry.spineIndex >= 0 && tocEntry.spineIndex < spineCount) {
|
||||
if (spineToTocIndex[tocEntry.spineIndex] == -1) {
|
||||
spineToTocIndex[tocEntry.spineIndex] = static_cast<int16_t>(j);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
ZipFile zip(epubPath);
|
||||
// Pre-open zip file to speed up size calculations
|
||||
if (!zip.open()) {
|
||||
@ -133,31 +173,56 @@ bool BookMetadataCache::buildBookBin(const std::string& epubPath, const BookMeta
|
||||
tocFile.close();
|
||||
return false;
|
||||
}
|
||||
// TODO: For large ZIPs loading the all localHeaderOffsets will crash.
|
||||
// However not having them loaded is extremely slow. Need a better solution here.
|
||||
// Perhaps only a cache of spine items or a better way to speedup lookups?
|
||||
if (!zip.loadAllFileStatSlims()) {
|
||||
Serial.printf("[%lu] [BMC] Could not load zip local header offsets for size calculations\n", millis());
|
||||
bookFile.close();
|
||||
spineFile.close();
|
||||
tocFile.close();
|
||||
zip.close();
|
||||
return false;
|
||||
// NOTE: We intentionally skip calling loadAllFileStatSlims() here.
|
||||
// For large EPUBs (2000+ chapters), pre-loading all ZIP central directory entries
|
||||
// into memory causes OOM crashes on ESP32-C3's limited ~380KB RAM.
|
||||
// Instead, for large books we use a one-pass batch lookup that scans the ZIP
|
||||
// central directory once and matches against spine targets using hash comparison.
|
||||
// This is O(n*log(m)) instead of O(n*m) while avoiding memory exhaustion.
|
||||
// See: https://github.com/crosspoint-reader/crosspoint-reader/issues/134
|
||||
|
||||
std::vector<uint32_t> spineSizes;
|
||||
bool useBatchSizes = false;
|
||||
|
||||
if (spineCount >= LARGE_SPINE_THRESHOLD) {
|
||||
Serial.printf("[%lu] [BMC] Using batch size lookup for %d spine items\n", millis(), spineCount);
|
||||
|
||||
std::vector<ZipFile::SizeTarget> targets;
|
||||
targets.reserve(spineCount);
|
||||
|
||||
spineFile.seek(0);
|
||||
for (int i = 0; i < spineCount; i++) {
|
||||
auto entry = readSpineEntry(spineFile);
|
||||
std::string path = FsHelpers::normalisePath(entry.href);
|
||||
|
||||
ZipFile::SizeTarget t;
|
||||
t.hash = ZipFile::fnvHash64(path.c_str(), path.size());
|
||||
t.len = static_cast<uint16_t>(path.size());
|
||||
t.index = static_cast<uint16_t>(i);
|
||||
targets.push_back(t);
|
||||
}
|
||||
|
||||
std::sort(targets.begin(), targets.end(), [](const ZipFile::SizeTarget& a, const ZipFile::SizeTarget& b) {
|
||||
return a.hash < b.hash || (a.hash == b.hash && a.len < b.len);
|
||||
});
|
||||
|
||||
spineSizes.resize(spineCount, 0);
|
||||
int matched = zip.fillUncompressedSizes(targets, spineSizes);
|
||||
Serial.printf("[%lu] [BMC] Batch lookup matched %d/%d spine items\n", millis(), matched, spineCount);
|
||||
|
||||
targets.clear();
|
||||
targets.shrink_to_fit();
|
||||
|
||||
useBatchSizes = true;
|
||||
}
|
||||
|
||||
uint32_t cumSize = 0;
|
||||
spineFile.seek(0);
|
||||
int lastSpineTocIndex = -1;
|
||||
for (int i = 0; i < spineCount; i++) {
|
||||
auto spineEntry = readSpineEntry(spineFile);
|
||||
|
||||
tocFile.seek(0);
|
||||
for (int j = 0; j < tocCount; j++) {
|
||||
auto tocEntry = readTocEntry(tocFile);
|
||||
if (tocEntry.spineIndex == i) {
|
||||
spineEntry.tocIndex = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
spineEntry.tocIndex = spineToTocIndex[i];
|
||||
|
||||
// Not a huge deal if we don't fine a TOC entry for the spine entry, this is expected behaviour for EPUBs
|
||||
// Logging here is for debugging
|
||||
@ -169,16 +234,25 @@ bool BookMetadataCache::buildBookBin(const std::string& epubPath, const BookMeta
|
||||
}
|
||||
lastSpineTocIndex = spineEntry.tocIndex;
|
||||
|
||||
// Calculate size for cumulative size
|
||||
size_t itemSize = 0;
|
||||
const std::string path = FsHelpers::normalisePath(spineEntry.href);
|
||||
if (zip.getInflatedFileSize(path.c_str(), &itemSize)) {
|
||||
cumSize += itemSize;
|
||||
spineEntry.cumulativeSize = cumSize;
|
||||
if (useBatchSizes) {
|
||||
itemSize = spineSizes[i];
|
||||
if (itemSize == 0) {
|
||||
const std::string path = FsHelpers::normalisePath(spineEntry.href);
|
||||
if (!zip.getInflatedFileSize(path.c_str(), &itemSize)) {
|
||||
Serial.printf("[%lu] [BMC] Warning: Could not get size for spine item: %s\n", millis(), path.c_str());
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Serial.printf("[%lu] [BMC] Warning: Could not get size for spine item: %s\n", millis(), path.c_str());
|
||||
const std::string path = FsHelpers::normalisePath(spineEntry.href);
|
||||
if (!zip.getInflatedFileSize(path.c_str(), &itemSize)) {
|
||||
Serial.printf("[%lu] [BMC] Warning: Could not get size for spine item: %s\n", millis(), path.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
cumSize += itemSize;
|
||||
spineEntry.cumulativeSize = cumSize;
|
||||
|
||||
// Write out spine data to book.bin
|
||||
writeSpineEntry(bookFile, spineEntry);
|
||||
}
|
||||
@ -248,21 +322,38 @@ void BookMetadataCache::createTocEntry(const std::string& title, const std::stri
|
||||
return;
|
||||
}
|
||||
|
||||
int spineIndex = -1;
|
||||
// find spine index
|
||||
// TODO: This lookup is slow as need to scan through all items each time. We can't hold it all in memory due to size.
|
||||
// But perhaps we can load just the hrefs in a vector/list to do an index lookup?
|
||||
spineFile.seek(0);
|
||||
for (int i = 0; i < spineCount; i++) {
|
||||
auto spineEntry = readSpineEntry(spineFile);
|
||||
if (spineEntry.href == href) {
|
||||
spineIndex = i;
|
||||
int16_t spineIndex = -1;
|
||||
|
||||
if (useSpineHrefIndex) {
|
||||
uint64_t targetHash = fnvHash64(href);
|
||||
uint16_t targetLen = static_cast<uint16_t>(href.size());
|
||||
|
||||
auto it =
|
||||
std::lower_bound(spineHrefIndex.begin(), spineHrefIndex.end(), SpineHrefIndexEntry{targetHash, targetLen, 0},
|
||||
[](const SpineHrefIndexEntry& a, const SpineHrefIndexEntry& b) {
|
||||
return a.hrefHash < b.hrefHash || (a.hrefHash == b.hrefHash && a.hrefLen < b.hrefLen);
|
||||
});
|
||||
|
||||
while (it != spineHrefIndex.end() && it->hrefHash == targetHash && it->hrefLen == targetLen) {
|
||||
spineIndex = it->spineIndex;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (spineIndex == -1) {
|
||||
Serial.printf("[%lu] [BMC] addTocEntry: Could not find spine item for TOC href %s\n", millis(), href.c_str());
|
||||
if (spineIndex == -1) {
|
||||
Serial.printf("[%lu] [BMC] createTocEntry: Could not find spine item for TOC href %s\n", millis(), href.c_str());
|
||||
}
|
||||
} else {
|
||||
spineFile.seek(0);
|
||||
for (int i = 0; i < spineCount; i++) {
|
||||
auto spineEntry = readSpineEntry(spineFile);
|
||||
if (spineEntry.href == href) {
|
||||
spineIndex = static_cast<int16_t>(i);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (spineIndex == -1) {
|
||||
Serial.printf("[%lu] [BMC] createTocEntry: Could not find spine item for TOC href %s\n", millis(), href.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
const TocEntry entry(title, href, anchor, level, spineIndex);
|
||||
|
||||
@ -2,7 +2,9 @@
|
||||
|
||||
#include <SDCardManager.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
class BookMetadataCache {
|
||||
public:
|
||||
@ -53,6 +55,27 @@ class BookMetadataCache {
|
||||
FsFile spineFile;
|
||||
FsFile tocFile;
|
||||
|
||||
// Index for fast href→spineIndex lookup (used only for large EPUBs)
|
||||
struct SpineHrefIndexEntry {
|
||||
uint64_t hrefHash; // FNV-1a 64-bit hash
|
||||
uint16_t hrefLen; // length for collision reduction
|
||||
int16_t spineIndex;
|
||||
};
|
||||
std::vector<SpineHrefIndexEntry> spineHrefIndex;
|
||||
bool useSpineHrefIndex = false;
|
||||
|
||||
static constexpr uint16_t LARGE_SPINE_THRESHOLD = 400;
|
||||
|
||||
// FNV-1a 64-bit hash function
|
||||
static uint64_t fnvHash64(const std::string& s) {
|
||||
uint64_t hash = 14695981039346656037ull;
|
||||
for (char c : s) {
|
||||
hash ^= static_cast<uint8_t>(c);
|
||||
hash *= 1099511628211ull;
|
||||
}
|
||||
return hash;
|
||||
}
|
||||
|
||||
uint32_t writeSpineEntry(FsFile& file, const SpineEntry& entry) const;
|
||||
uint32_t writeTocEntry(FsFile& file, const TocEntry& entry) const;
|
||||
SpineEntry readSpineEntry(FsFile& file) const;
|
||||
|
||||
@ -38,6 +38,9 @@ ContentOpfParser::~ContentOpfParser() {
|
||||
if (SdMan.exists((cachePath + itemCacheFile).c_str())) {
|
||||
SdMan.remove((cachePath + itemCacheFile).c_str());
|
||||
}
|
||||
itemIndex.clear();
|
||||
itemIndex.shrink_to_fit();
|
||||
useItemIndex = false;
|
||||
}
|
||||
|
||||
size_t ContentOpfParser::write(const uint8_t data) { return write(&data, 1); }
|
||||
@ -129,6 +132,15 @@ void XMLCALL ContentOpfParser::startElement(void* userData, const XML_Char* name
|
||||
"[%lu] [COF] Couldn't open temp items file for reading. This is probably going to be a fatal error.\n",
|
||||
millis());
|
||||
}
|
||||
|
||||
// Sort item index for binary search if we have enough items
|
||||
if (self->itemIndex.size() >= LARGE_SPINE_THRESHOLD) {
|
||||
std::sort(self->itemIndex.begin(), self->itemIndex.end(), [](const ItemIndexEntry& a, const ItemIndexEntry& b) {
|
||||
return a.idHash < b.idHash || (a.idHash == b.idHash && a.idLen < b.idLen);
|
||||
});
|
||||
self->useItemIndex = true;
|
||||
Serial.printf("[%lu] [COF] Using fast index for %zu manifest items\n", millis(), self->itemIndex.size());
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@ -180,6 +192,15 @@ void XMLCALL ContentOpfParser::startElement(void* userData, const XML_Char* name
|
||||
}
|
||||
}
|
||||
|
||||
// Record index entry for fast lookup later
|
||||
if (self->tempItemStore) {
|
||||
ItemIndexEntry entry;
|
||||
entry.idHash = fnvHash(itemId);
|
||||
entry.idLen = static_cast<uint16_t>(itemId.size());
|
||||
entry.fileOffset = static_cast<uint32_t>(self->tempItemStore.position());
|
||||
self->itemIndex.push_back(entry);
|
||||
}
|
||||
|
||||
// Write items down to SD card
|
||||
serialization::writeString(self->tempItemStore, itemId);
|
||||
serialization::writeString(self->tempItemStore, href);
|
||||
@ -215,19 +236,50 @@ void XMLCALL ContentOpfParser::startElement(void* userData, const XML_Char* name
|
||||
for (int i = 0; atts[i]; i += 2) {
|
||||
if (strcmp(atts[i], "idref") == 0) {
|
||||
const std::string idref = atts[i + 1];
|
||||
// Resolve the idref to href using items map
|
||||
// TODO: This lookup is slow as need to scan through all items each time.
|
||||
// It can take up to 200ms per item when getting to 1500 items.
|
||||
self->tempItemStore.seek(0);
|
||||
std::string itemId;
|
||||
std::string href;
|
||||
while (self->tempItemStore.available()) {
|
||||
serialization::readString(self->tempItemStore, itemId);
|
||||
serialization::readString(self->tempItemStore, href);
|
||||
if (itemId == idref) {
|
||||
self->cache->createSpineEntry(href);
|
||||
break;
|
||||
bool found = false;
|
||||
|
||||
if (self->useItemIndex) {
|
||||
// Fast path: binary search
|
||||
uint32_t targetHash = fnvHash(idref);
|
||||
uint16_t targetLen = static_cast<uint16_t>(idref.size());
|
||||
|
||||
auto it = std::lower_bound(self->itemIndex.begin(), self->itemIndex.end(),
|
||||
ItemIndexEntry{targetHash, targetLen, 0},
|
||||
[](const ItemIndexEntry& a, const ItemIndexEntry& b) {
|
||||
return a.idHash < b.idHash || (a.idHash == b.idHash && a.idLen < b.idLen);
|
||||
});
|
||||
|
||||
// Check for match (may need to check a few due to hash collisions)
|
||||
while (it != self->itemIndex.end() && it->idHash == targetHash) {
|
||||
self->tempItemStore.seek(it->fileOffset);
|
||||
std::string itemId;
|
||||
serialization::readString(self->tempItemStore, itemId);
|
||||
if (itemId == idref) {
|
||||
serialization::readString(self->tempItemStore, href);
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
++it;
|
||||
}
|
||||
} else {
|
||||
// Slow path: linear scan (for small manifests, keeps original behavior)
|
||||
// TODO: This lookup is slow as need to scan through all items each time.
|
||||
// It can take up to 200ms per item when getting to 1500 items.
|
||||
self->tempItemStore.seek(0);
|
||||
std::string itemId;
|
||||
while (self->tempItemStore.available()) {
|
||||
serialization::readString(self->tempItemStore, itemId);
|
||||
serialization::readString(self->tempItemStore, href);
|
||||
if (itemId == idref) {
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (found && self->cache) {
|
||||
self->cache->createSpineEntry(href);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,6 +1,9 @@
|
||||
#pragma once
|
||||
#include <Print.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <vector>
|
||||
|
||||
#include "Epub.h"
|
||||
#include "expat.h"
|
||||
|
||||
@ -28,6 +31,27 @@ class ContentOpfParser final : public Print {
|
||||
FsFile tempItemStore;
|
||||
std::string coverItemId;
|
||||
|
||||
// Index for fast idref→href lookup (used only for large EPUBs)
|
||||
struct ItemIndexEntry {
|
||||
uint32_t idHash; // FNV-1a hash of itemId
|
||||
uint16_t idLen; // length for collision reduction
|
||||
uint32_t fileOffset; // offset in .items.bin
|
||||
};
|
||||
std::vector<ItemIndexEntry> itemIndex;
|
||||
bool useItemIndex = false;
|
||||
|
||||
static constexpr uint16_t LARGE_SPINE_THRESHOLD = 400;
|
||||
|
||||
// FNV-1a hash function
|
||||
static uint32_t fnvHash(const std::string& s) {
|
||||
uint32_t hash = 2166136261u;
|
||||
for (char c : s) {
|
||||
hash ^= static_cast<uint8_t>(c);
|
||||
hash *= 16777619u;
|
||||
}
|
||||
return hash;
|
||||
}
|
||||
|
||||
static void startElement(void* userData, const XML_Char* name, const XML_Char** atts);
|
||||
static void characterData(void* userData, const XML_Char* s, int len);
|
||||
static void endElement(void* userData, const XML_Char* name);
|
||||
|
||||
@ -4,6 +4,8 @@
|
||||
#include <SDCardManager.h>
|
||||
#include <miniz.h>
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
bool inflateOneShot(const uint8_t* inputBuf, const size_t deflatedSize, uint8_t* outputBuf, const size_t inflatedSize) {
|
||||
// Setup inflator
|
||||
const auto inflator = static_cast<tinfl_decompressor*>(malloc(sizeof(tinfl_decompressor)));
|
||||
@ -74,6 +76,10 @@ bool ZipFile::loadAllFileStatSlims() {
|
||||
file.seekCur(m + k);
|
||||
}
|
||||
|
||||
// Set cursor to start of central directory for sequential access
|
||||
lastCentralDirPos = zipDetails.centralDirOffset;
|
||||
lastCentralDirPosValid = true;
|
||||
|
||||
if (!wasOpen) {
|
||||
close();
|
||||
}
|
||||
@ -102,15 +108,35 @@ bool ZipFile::loadFileStatSlim(const char* filename, FileStatSlim* fileStat) {
|
||||
return false;
|
||||
}
|
||||
|
||||
file.seek(zipDetails.centralDirOffset);
|
||||
// Phase 1: Try scanning from cursor position first
|
||||
uint32_t startPos = lastCentralDirPosValid ? lastCentralDirPos : zipDetails.centralDirOffset;
|
||||
uint32_t wrapPos = zipDetails.centralDirOffset;
|
||||
bool wrapped = false;
|
||||
bool found = false;
|
||||
|
||||
file.seek(startPos);
|
||||
|
||||
uint32_t sig;
|
||||
char itemName[256];
|
||||
bool found = false;
|
||||
|
||||
while (file.available()) {
|
||||
file.read(&sig, 4);
|
||||
if (sig != 0x02014b50) break; // End of list
|
||||
while (true) {
|
||||
uint32_t entryStart = file.position();
|
||||
|
||||
if (file.read(&sig, 4) != 4 || sig != 0x02014b50) {
|
||||
// End of central directory
|
||||
if (!wrapped && lastCentralDirPosValid && startPos != zipDetails.centralDirOffset) {
|
||||
// Wrap around to beginning
|
||||
file.seek(zipDetails.centralDirOffset);
|
||||
wrapped = true;
|
||||
continue;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
// If we've wrapped and reached our start position, stop
|
||||
if (wrapped && entryStart >= startPos) {
|
||||
break;
|
||||
}
|
||||
|
||||
file.seekCur(6);
|
||||
file.read(&fileStat->method, 2);
|
||||
@ -123,15 +149,25 @@ bool ZipFile::loadFileStatSlim(const char* filename, FileStatSlim* fileStat) {
|
||||
file.read(&k, 2);
|
||||
file.seekCur(8);
|
||||
file.read(&fileStat->localHeaderOffset, 4);
|
||||
file.read(itemName, nameLen);
|
||||
itemName[nameLen] = '\0';
|
||||
|
||||
if (strcmp(itemName, filename) == 0) {
|
||||
found = true;
|
||||
break;
|
||||
if (nameLen < 256) {
|
||||
file.read(itemName, nameLen);
|
||||
itemName[nameLen] = '\0';
|
||||
|
||||
if (strcmp(itemName, filename) == 0) {
|
||||
// Found it! Update cursor to next entry
|
||||
file.seekCur(m + k);
|
||||
lastCentralDirPos = file.position();
|
||||
lastCentralDirPosValid = true;
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
// Name too long, skip it
|
||||
file.seekCur(nameLen);
|
||||
}
|
||||
|
||||
// Skip the rest of this entry (extra field + comment)
|
||||
// Skip extra field + comment
|
||||
file.seekCur(m + k);
|
||||
}
|
||||
|
||||
@ -253,6 +289,8 @@ bool ZipFile::close() {
|
||||
if (file) {
|
||||
file.close();
|
||||
}
|
||||
lastCentralDirPos = 0;
|
||||
lastCentralDirPosValid = false;
|
||||
return true;
|
||||
}
|
||||
|
||||
@ -266,6 +304,80 @@ bool ZipFile::getInflatedFileSize(const char* filename, size_t* size) {
|
||||
return true;
|
||||
}
|
||||
|
||||
int ZipFile::fillUncompressedSizes(std::vector<SizeTarget>& targets, std::vector<uint32_t>& sizes) {
|
||||
if (targets.empty()) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
const bool wasOpen = isOpen();
|
||||
if (!wasOpen && !open()) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!loadZipDetails()) {
|
||||
if (!wasOpen) {
|
||||
close();
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
file.seek(zipDetails.centralDirOffset);
|
||||
|
||||
int matched = 0;
|
||||
uint32_t sig;
|
||||
char itemName[256];
|
||||
|
||||
while (file.available()) {
|
||||
file.read(&sig, 4);
|
||||
if (sig != 0x02014b50) break;
|
||||
|
||||
file.seekCur(6);
|
||||
uint16_t method;
|
||||
file.read(&method, 2);
|
||||
file.seekCur(8);
|
||||
uint32_t compressedSize, uncompressedSize;
|
||||
file.read(&compressedSize, 4);
|
||||
file.read(&uncompressedSize, 4);
|
||||
uint16_t nameLen, m, k;
|
||||
file.read(&nameLen, 2);
|
||||
file.read(&m, 2);
|
||||
file.read(&k, 2);
|
||||
file.seekCur(8);
|
||||
uint32_t localHeaderOffset;
|
||||
file.read(&localHeaderOffset, 4);
|
||||
|
||||
if (nameLen < 256) {
|
||||
file.read(itemName, nameLen);
|
||||
itemName[nameLen] = '\0';
|
||||
|
||||
uint64_t hash = fnvHash64(itemName, nameLen);
|
||||
SizeTarget key = {hash, nameLen, 0};
|
||||
|
||||
auto it = std::lower_bound(targets.begin(), targets.end(), key, [](const SizeTarget& a, const SizeTarget& b) {
|
||||
return a.hash < b.hash || (a.hash == b.hash && a.len < b.len);
|
||||
});
|
||||
|
||||
while (it != targets.end() && it->hash == hash && it->len == nameLen) {
|
||||
if (it->index < sizes.size()) {
|
||||
sizes[it->index] = uncompressedSize;
|
||||
matched++;
|
||||
}
|
||||
++it;
|
||||
}
|
||||
} else {
|
||||
file.seekCur(nameLen);
|
||||
}
|
||||
|
||||
file.seekCur(m + k);
|
||||
}
|
||||
|
||||
if (!wasOpen) {
|
||||
close();
|
||||
}
|
||||
|
||||
return matched;
|
||||
}
|
||||
|
||||
uint8_t* ZipFile::readFileToMemory(const char* filename, size_t* size, const bool trailingNullByte) {
|
||||
const bool wasOpen = isOpen();
|
||||
if (!wasOpen && !open()) {
|
||||
|
||||
@ -3,6 +3,7 @@
|
||||
|
||||
#include <string>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
class ZipFile {
|
||||
public:
|
||||
@ -19,12 +20,33 @@ class ZipFile {
|
||||
bool isSet;
|
||||
};
|
||||
|
||||
// Target for batch uncompressed size lookup (sorted by hash, then len)
|
||||
struct SizeTarget {
|
||||
uint64_t hash; // FNV-1a 64-bit hash of normalized path
|
||||
uint16_t len; // Length of path for collision reduction
|
||||
uint16_t index; // Caller's index (e.g. spine index)
|
||||
};
|
||||
|
||||
// FNV-1a 64-bit hash computed from char buffer (no std::string allocation)
|
||||
static uint64_t fnvHash64(const char* s, size_t len) {
|
||||
uint64_t hash = 14695981039346656037ull;
|
||||
for (size_t i = 0; i < len; i++) {
|
||||
hash ^= static_cast<uint8_t>(s[i]);
|
||||
hash *= 1099511628211ull;
|
||||
}
|
||||
return hash;
|
||||
}
|
||||
|
||||
private:
|
||||
const std::string& filePath;
|
||||
FsFile file;
|
||||
ZipDetails zipDetails = {0, 0, false};
|
||||
std::unordered_map<std::string, FileStatSlim> fileStatSlimCache;
|
||||
|
||||
// Cursor for sequential central-dir scanning optimization
|
||||
uint32_t lastCentralDirPos = 0;
|
||||
bool lastCentralDirPosValid = false;
|
||||
|
||||
bool loadFileStatSlim(const char* filename, FileStatSlim* fileStat);
|
||||
long getDataOffset(const FileStatSlim& fileStat);
|
||||
bool loadZipDetails();
|
||||
@ -39,6 +61,10 @@ class ZipFile {
|
||||
bool close();
|
||||
bool loadAllFileStatSlims();
|
||||
bool getInflatedFileSize(const char* filename, size_t* size);
|
||||
// Batch lookup: scan ZIP central dir once and fill sizes for matching targets.
|
||||
// targets must be sorted by (hash, len). sizes[target.index] receives uncompressedSize.
|
||||
// Returns number of targets matched.
|
||||
int fillUncompressedSizes(std::vector<SizeTarget>& targets, std::vector<uint32_t>& sizes);
|
||||
// Due to the memory required to run each of these, it is recommended to not preopen the zip file for multiple
|
||||
// These functions will open and close the zip as needed
|
||||
uint8_t* readFileToMemory(const char* filename, size_t* size = nullptr, bool trailingNullByte = false);
|
||||
|
||||
Loading…
Reference in New Issue
Block a user