Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add memory allocation tracking #69

Merged
merged 1 commit into from
Apr 10, 2024

Conversation

bens
Copy link
Collaborator

@bens bens commented Mar 18, 2024

Adds a custom Allocator wrapper which keeps track of the allocated memory, then record the maximum allocation readings in the Runner alongside timing readings and carry them into the Result. I've added equivalent statistics for memory use as for timings, but maybe we only need something simpler?

Current output:

benchmark              runs     total time     time/run (avg ± σ)     (min ... max)                p75        p99        p995
-----------------------------------------------------------------------------------------------------------------------------
My Benchmark           10       1.118ms        111.879us ± 14.24us    (98.058us ... 136.68us)      125.995us  136.68us   136.68us
My Benchmark [MEMORY]                          2.000KiB ± 0B          (2.000KiB ... 2.000KiB)      2KiB       2KiB       2KiB

@bens bens requested a review from hendriknielaender March 18, 2024 03:02
@bens bens self-assigned this Mar 18, 2024
@bens bens added the enhancement New feature or request label Mar 18, 2024
@bens bens linked an issue Mar 18, 2024 that may be closed by this pull request
@bens bens marked this pull request as draft March 19, 2024 06:37
@bens bens force-pushed the memory-tracking branch 3 times, most recently from d4b8c21 to b3b2b91 Compare March 21, 2024 02:21
@bens bens marked this pull request as ready for review March 22, 2024 05:36
@bens bens force-pushed the memory-tracking branch from 9dad995 to 244c3ab Compare March 22, 2024 05:38
@hendriknielaender
Copy link
Owner

Man really cool stuff that you recently provided to zbench 🙌
I will have a look into it - when i tried a POC for this one, i was trying to go a similar route as andrew in https://github.com/andrewrk/poop.

@bens
Copy link
Collaborator Author

bens commented Mar 22, 2024

Man really cool stuff that you recently provided to zbench 🙌 I will have a look into it - when i tried a POC for this one, i was trying to go a similar route as andrew in https://github.com/andrewrk/poop.

Cheers, and thank you for the valuable reviews, hopefully the code is solid ;)

Yeah, trying out the hardware performance counters would be interesting. I guessed that using an allocator wrapper would be very lightweight and trivially cross platform; poop is linux only currently though there are probably methods for other OSes too. FWIW, from benchmarking on my laptop with and without allocation tracking I can't detect any measurable overhead.

util/runner.zig Outdated
.state = .{ .running = .{
.iterations_count = iterations,
.iterations_remaining = iterations,
.timings_ns = try allocator.alloc(u64, iterations),
.allocation_maxes = if (track_allocations) try allocator.alloc(usize, iterations) else null,
Copy link
Owner

@hendriknielaender hendriknielaender Mar 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: What do you think of encapsulating the tracking logic within a dedicated struct?
It could encapsulate the tracking logic in a single place, making it easier to manage and modify.

Something like this?

const AllocationTracker = struct {
    maxes: []usize,
    counts: []usize,

    pub fn init(allocator: *std.mem.Allocator, iterations: usize, track: bool) !?AllocationTracker {
        if (!track) return null;
        return AllocationTracker{
            .maxes = try allocator.alloc(usize, iterations),
            .counts = try allocator.alloc(usize, iterations),
        };
    }

    pub fn update(self: *AllocationTracker, index: usize, max: ?usize, count: ?usize) void {
        if (max) |m| self.maxes[index] = m;
        if (count) |c| self.counts[index] = c;
    }

    pub fn deinit(self: *AllocationTracker, allocator: *std.mem.Allocator) void {
        allocator.free(self.maxes);
        allocator.free(self.counts);
    }
};

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea I think, then the maxes and counts are always null or not null in synch.

zbench.zig Outdated
@@ -68,13 +73,27 @@ const Definition = struct {
if (self.config.hooks.before_each) |hook| hook();
defer if (self.config.hooks.after_each) |hook| hook();

var tracking = TrackingAllocator.init(allocator);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one i haven't really verified, but maybe avoid creating initialization of TrackingAllocator when self.config.track_allocations is false.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I tried to do initially, but I was struggling to factor out choosing which allocator while not duplicating the rest of the function in an if. If the TrackingAllocator is defined on the stack inside a sub block it'll be freed outside of that sub block, so I couldn't do something like const alloc = if (self.config.track_allocations) TrackingAllocator.init(allocator).allocator() else allocator;. I can try other ways but always creating the tracking allocator was a simple method.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay, understood. No need to change it, but a small comment would be helpful.

@hendriknielaender
Copy link
Owner

from my side it looks good, should we merge it @bens?

@bens
Copy link
Collaborator Author

bens commented Mar 27, 2024

from my side it looks good, should we merge it @bens?

There are still a few things to take care of.

@bens bens force-pushed the memory-tracking branch from c290d45 to 790e7e3 Compare April 10, 2024 11:36
Adds a custom Allocator wrapper which keeps track of the allocated memory, then
record the maximum allocation readings in the Runner alongside timing readings
and carry them into the Result.
@bens bens force-pushed the memory-tracking branch from 790e7e3 to 213f4af Compare April 10, 2024 11:46
@bens
Copy link
Collaborator Author

bens commented Apr 10, 2024

from my side it looks good, should we merge it @bens?

Ok, I'm happy with it now. Thanks @hendriknielaender :)

@hendriknielaender hendriknielaender merged commit 926eebc into hendriknielaender:main Apr 10, 2024
5 checks passed
@bens bens deleted the memory-tracking branch April 10, 2024 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Memory Consumption Calculation to Benchmark Function
2 participants