0001048: The new filesystem is extremely slow

View Revisions: Issue #1048		[ All Revisions ] [ Back to Issue ]
Summary	0001048: The new filesystem is extremely slow

Revision	2012-08-07 03:45 by Tow
Description	On my working copy debug mode build needs about 40 seconds to initialize: - CResourceHandler::initialize() takes 12 s. - CResourceHandler::loadFileSystem("ALL/config/filesystem.json") takes 28s. - CResourceHandler::loadModsFilesystems() takes about 0.1 s. I have about 6400+ files in 340 folders in working location. That's much but loading should not take that long. The extremely check-heavy debug VC implementation of STL surely adds to it (release mode build needs about 4 s.) but it can't be helped. 40 s. startup makes my development efforts very miserable. On my H3 installation VCMI (release mode) startup time is about triple of what 0.89 needed (~3s instead of ~1s). That's also bad. Therefore some optimizations are needed ASAP. Some ideas: * initialLoader can collect only DIR and ARCHIVE entries * it seems that same locations are scanned multiple times (LOCAL/Data is the same as LOCAL/Data on Win, right?) * don't collect entries of type OTHER — they're not used anyway * get rid of initialLoader altogether — filesystem.json is assumed to be in de-facto hardcoded location anyway and that ALL/LOCAL/GLOBAL substitutions should be easy to handle by hand. We should not scan the whole DATA_DIR, just go for the locations listed in filesystem.json. If you any further info, let me know. I can try looking for bottlenecks with profiler but I guess some "obvious" optimizations should be done first.

Revision	2012-08-07 03:44 by Tow
Description	On my working copy debug mode build needs about 40 seconds to initialize: - CResourceHandler::initialize() takes 12 s. - CResourceHandler::loadFileSystem("ALL/config/filesystem.json") takes 28s. - CResourceHandler::loadModsFilesystems() takes about 0.1 s. I have about 6400+ files in 340 folders in working location. That's much but loading should not take that long. The extremely check-heavy debug VC implementation of STL surely adds to it (release mode build needs 0000007:0000004 s.) but it can't be helped. 40 s. startup makes my development efforts very miserable. On my H3 installation VCMI (release mode) startup time is about triple of what 0.89 needed (~3s instead of ~1s). That's also bad. Therefore some optimizations are needed ASAP. Some ideas: * initialLoader can collect only DIR and ARCHIVE entries * it seems that same locations are scanned multiple times (LOCAL/Data is the same as LOCAL/Data on Win, right?) * don't collect entries of type OTHER — they're not used anyway * get rid of initialLoader altogether — filesystem.json is assumed to be in de-facto hardcoded location anyway and that ALL/LOCAL/GLOBAL substitutions should be easy to handle by hand. We should not scan the whole DATA_DIR, just go for the locations listed in filesystem.json. If you any further info, let me know. I can try looking for bottlenecks with profiler but I guess some "obvious" optimizations should be done first.

Site | Forums | Wiki | Slack | GitHub