Skip to content
Snippets Groups Projects
Select Git revision
  • master
1 result

lmdb_index.py

Blame
    • ph's avatar
      b719a75a
      * added crude transaction support · b719a75a
      ph authored
      * added various corner case queries
      * abstract methods now raise correct NotImplementedError
      * added zlib compression of __data__
      * added support fort returning iterators, along with keeping relevant
        cursors around
      * limited sort test (heapsort stop after skip+limit) - python implementation too slow
      * in-memory dict index sort attempt (max 2 seconds difference to lmdb index)
      * pandas.Index sorted set attempt, however necessary set order brings too
        much data into memory sets and defeats sorted set benefit
      * walk-sort - do not use real sort, but walk the _whole_ reverse index for
        sorted column and allow only for result set values to go through
        walking the whole index is much faster than sort for big result sets,
        while negligible in impact to small queries!
      * preread_all possibility to load all result sets to memory for profiling
      * query range attempt with python lambda stop condition, surprisingly fast
      * two pass range - first takewhile over keys counts results, second islice
        takes counted number of values
      b719a75a
      History
      * added crude transaction support
      ph authored
      * added various corner case queries
      * abstract methods now raise correct NotImplementedError
      * added zlib compression of __data__
      * added support fort returning iterators, along with keeping relevant
        cursors around
      * limited sort test (heapsort stop after skip+limit) - python implementation too slow
      * in-memory dict index sort attempt (max 2 seconds difference to lmdb index)
      * pandas.Index sorted set attempt, however necessary set order brings too
        much data into memory sets and defeats sorted set benefit
      * walk-sort - do not use real sort, but walk the _whole_ reverse index for
        sorted column and allow only for result set values to go through
        walking the whole index is much faster than sort for big result sets,
        while negligible in impact to small queries!
      * preread_all possibility to load all result sets to memory for profiling
      * query range attempt with python lambda stop condition, surprisingly fast
      * two pass range - first takewhile over keys counts results, second islice
        takes counted number of values
    lmdb_index.py 5.52 KiB