Skip to content
Snippets Groups Projects
  1. Jul 28, 2021
    • Yorhel's avatar
    • Yorhel's avatar
      Fix Dir.fmtPath() when given the root dir · ba14c093
      Yorhel authored
      ba14c093
    • Yorhel's avatar
      Implement a more efficient hard link counting approach · 0d314ca0
      Yorhel authored
      As aluded to in the previous commit. This approach keeps track of hard
      links information much the same way as ncdu 1.16, with the main
      difference being that the actual /counting/ of hard link sizes is
      deferred until the scan is complete, thus allowing the use of a more
      efficient algorithm and amortizing the counting costs.
      
      As an additional benefit, the links listing in the information window
      now doesn't need a full scan through the in-memory tree anymore.
      
      A few memory usage benchmarks:
      
                    1.16  2.0-beta1  this commit
      root:          429        162          164
      backup:       3969       1686         1601
      many links:    155        194          106
      many links2*:  155        602          106
      
      (I'm surprised my backup dir had enough hard links for this to be an
      improvement)
      (* this is the same as the "many links" benchmarks, but with a few
      parent directories added to increase the tree depth. 2.0-beta1 doesn't
      like that at all)
      
      Performance-wise, refresh and delete operations can still be improved a
      bit.
      0d314ca0
  2. Jul 26, 2021
    • Yorhel's avatar
      Add parent node pointers to Dir struct + remove Parents abstraction · 36bc405a
      Yorhel authored
      While this simplifies the code a bit, it's a regression in the sense
      that it increases memory use.
      
      This commit is yak shaving for another hard link counting approach I'd
      like to try out, which should be a *LOT* less memory hungry compared to
      the current approach. Even though it does, indeed, add an extra cost of
      these parent node pointers.
      36bc405a
  3. Jul 18, 2021
  4. Jul 16, 2021
    • Yorhel's avatar
      Store Ext before Entry · e9c8d12c
      Yorhel authored
      Which is slightly simpler and should provide a minor performance
      improvement.
      e9c8d12c
  5. Jul 13, 2021
    • Yorhel's avatar
      Implement directory refresh · 6c2ab500
      Yorhel authored
      This complicated the scan code more than I had anticipated and has a
      few inherent bugs with respect to calculating shared hardlink sizes.
      
      Still, the merge approach avoids creating a full copy of the subtree, so
      that's another memory usage related win compared to the C version.
      On the other hand, it does leak memory if nodes can't be reused.
      
      Not quite as well tested as I should have, so I'm sure there's bugs.
      6c2ab500
  6. Jul 06, 2021
    • Yorhel's avatar
      Add link path listing to information window · ff3e3bcc
      Yorhel authored
      Two differences compared to the C version:
      - You can now select individual paths in the listing, pressing enter
        will open the selected path in the browser window.
      - Creating this listing is much slower and requires, in the worst case,
        a full traversal through the in-memory tree. I've tested this without
        the same-dev and shared-parent optimizations (i.e. worst case) on an
        import with 30M files and performance was still quite acceptable - the
        listing completed in a second - so I didn't bother adding a loading
        indicator. On slower systems and even larger trees this may be a
        little annoying, though.
      
      (also, calling nonl() apparently breaks detection of the return key,
      neither \n nor KEY_ENTER are emitted for some reason)
      ff3e3bcc
  7. Jun 11, 2021
    • Yorhel's avatar
      Add item info window · 618972b8
      Yorhel authored
      Doesn't display the item's path anymore (seems rather redundant) but
      adds a few more other fields.
      618972b8
  8. Jun 07, 2021
    • Yorhel's avatar
      Update for Zig 0.8 HashMap changes · 40f9dff5
      Yorhel authored
      I had used them as a HashSet with mutable keys already in order to avoid
      padding problems. This is not always necessary anymore now that Zig's
      new HashMap uses separate arrays for keys and values, but I still need
      the HashSet trick for the link_count nodes table, as the key itself
      would otherwise have padding.
      40f9dff5
  9. Jun 01, 2021
    • Yorhel's avatar
      Support hard link counts when importing old ncdu dumps · e6b2cff3
      Yorhel authored
      Under the assumption that there are no external references to files
      mentioned in the dump, i.e. a file's nlink count matches the number of
      times the file occurs in the dump.
      
      This machinery could also be used for regular scans, when you want to
      scan an individual directory without caring about external hard links.
      Maybe that should be the default, even? Not sure...
      e6b2cff3
  10. May 29, 2021
    • Yorhel's avatar
      Improved error reporting + minor cleanup · 59ef5fd2
      Yorhel authored
      59ef5fd2
    • Yorhel's avatar
      Handle allocation failures · 23903088
      Yorhel authored
      In a similar way to the C version of ncdu: by wrapping malloc(). It's
      simpler to handle allocation failures at the source to allow for easy
      retries, pushing the retries up the stack will complicate code somewhat
      more. Likewise, this is a best-effort approach to handling OOM,
      allocation failures in ncurses aren't handled and display glitches may
      occur when we get an OOM inside a drawing function.
      
      This is a somewhat un-Zig-like way of handling errors and adds
      scary-looking 'catch unreachable's all over the code, but that's okay.
      23903088
    • Yorhel's avatar
      Implement JSON file import · c077c5be
      Yorhel authored
      Performance is looking great, but the code is rather ugly and
      potentially buggy. Also doesn't handle hard links without an "nlink"
      field yet.
      
      Error handling of the import code is different from what I've been doing
      until now. That's intentional, I'll change error handling of other
      pieces to call ui.die() directly rather than propagating error enums.
      The approach is less testable but conceptually simpler, it's perfectly
      fine for a tiny application like ncdu.
      c077c5be
  11. May 24, 2021
  12. May 23, 2021
    • Yorhel's avatar
      Implement all existing browsing display options + some fixes · 7b3ebf92
      Yorhel authored
      I plan to add more display options, but ran out of keys to bind.
      Probably going for a quick-select menu thingy so that we can keep the
      old key bindings for people accustomed to it.
      
      The graph width algorithm is slightly different, but I think this one's
      a minor improvement.
      7b3ebf92
  13. May 06, 2021
    • Yorhel's avatar
      More UI stuff + shave off 16 bytes from model.Dir · 27cb599e
      Yorhel authored
      I initially wanted to keep a directory's block count and size as a
      separate field so that exporting an in-memory tree to a JSON dump would
      be easier to do, but that doesn't seem like a common operation to
      optimize for. We'll probably need the algorithms to subtract sub-items
      from directory counts anyway, so such an export can still be
      implemented, albeit slower.
      27cb599e
  14. May 01, 2021
  15. Apr 30, 2021
  16. Apr 29, 2021
    • Yorhel's avatar
      WIP: Experimenting with a rewrite to Zig & a new data model · 0783d357
      Yorhel authored
      The new data model is supposed to solve a few problems with ncdu 1.x's
      'struct dir':
      - Reduce memory overhead,
      - Fix extremely slow counting of hard links in some scenarios
        (issue #121)
      - Add support for counting 'shared' data with other directories
        (issue #36)
      
      Quick memory usage comparison of my root directory with ~3.5 million
      files (normal / extended mode):
      
        ncdu 1.15.1:     379M / 451M
        new (unaligned): 145M / 178M
        new (aligned):   155M / 200M
      
      There's still a /lot/ of to-do's left before this is usable, however,
      and there's a bunch of issues I haven't really decided on yet, such as
      which TUI library to use.
      
      Backporting this data model to the C version of ncdu is also possible,
      but somewhat painful. Let's first see how far I get with Zig.
      0783d357
Loading