[Melbourne-pm] Still performing well..

Sam Watkins sam at nipl.net
Tue May 18 20:52:25 PDT 2010

On Wed, May 19, 2010 at 12:26:58PM +1000, Daniel Pittman wrote:
> [2]  ...unless the C version cheats and skips all the memory management
>      overhead, by not freeing anything 'til the end of the process or
>      something exciting like that. :)

My test program keeps all the data in memory, maybe this is different from what
you have in mind.

If I wanted to read each record then discard it, I could use resizable buffers
to hold each record or field, and avoid (or 99.9% avoid) having to call malloc
or free during the loop.

For a C version that keeps all data in memory, it could allocate slabs of
memory, there's no need to allocate each string separately.

That said, my test program I mentioned earlier does call strdup (malloc) for
each string it keeps.  I don't think it calls free at all, at the moment.

If an app wants to keep everything in memory, I would be inclined to allocate
space for the entire file, read it all in, break it up and index it.  That
would require only one or two alloc/free pairs; this (or mmap private) is how I
would write a C program if I was optimizing it for speed, and this is how I
allocate memory in the next version of my brace translator, which is very fast.

The only problem with doing this is that allocating large chunks can supposedly
lead to memory fragmentation, or rather memory fragmentation can lead to malloc
failures for large chunks.  In practise I have never observed this problem,
it might affect small embedded systems, but such systems probably don't have
the resources to load the whole file at once anyway, and should use a streaming
model.  It is also less of a problem with computers / operation systems, having
virtual memory, such as every PC and my phone.

I regard fragmentation / large blocks problem as a problem with the malloc
model of allocation: it should be possible for the system to automatically
relocate objects if needed.  This would entail using indexes and relative
pointers, I believe it would not be a significant performance hit for most
apps, given that the allocator can only move stuff when you invoke it.


More information about the Melbourne-pm mailing list