Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Bumber]

I ended up going with:

Code: [Select]

...
current_line += input[iter];
if (current_line.length() > max_line_length)
{
size_t i = current_line.length(); //index to split at
while (--i > 0 && current_line[i] != ' '); //find start of current word

if (i == 0) //need to push at least one char
i = max_line_length; //word will be split at last char

//push current_line before i as cstring to out_vector
cstring_ptr = new char[i + 1];
strncpy(cstring_ptr, current_line.c_str(), i);
cstring_ptr[i] = '\0';
out_vector.push_back(cstring_ptr);

if (current_line[i] == ' ')
i++; //don't need to keep space
current_line.erase(0, i); //erase pushed substring
}
...
Which seems to work well, and is close enough interpretation to the original. (I ended up ditching "ignore_space", so it's not going to remove surplus spaces like Toady's code does. Might adjust that later.)

[WealthyRadish]

It shouldn't ever happen here due to the max_line_length check, but bear in mind that something like "while (--i > 0)" will underflow if "i" starts at zero and is an unsigned type like size_t.  Since it's an std::string, you can avoid writing the loop altogether:

Code: [Select]

size_t space_idx = current_line.find_last_of(' ');
if (space_idx == std::string::npos) {
// char not found
}
else {
// char found
}


[da_nang]

So... Log4Shell is wreaking havoc.

Sysadmins, hold the line! Programmers, update your shit!

Programming just got serious.

[Telgin]

I did a pass over our stuff today to make sure we weren't accidentally vulnerable.  As far as I can tell none of the libraries we've installed even make reference to it, and we aren't using it directly.

I sure hope we don't get a zero day exploit like this that does affect something we're using though.  I've been trying for over a month to get integrated into our new overlords' change management system and they really just don't seem interested in helping us deploy changes.

[McTraveller]

Wow have all the programmers really gone silent for this long?

I have a question related to a pet project on which I'm working: do modern OSs provide any guarantee - or explicitly say they provide no guarantees - on the maximum difference between two pointers obtained by malloc() or any other allocation routine within a single process?

Specifically, am I safe to assume that if I am running a program, are all allocations going to be within 48 bits of each other?  I'd really like to be able to pack a 16 bit data field and a pointer into 64 bits, instead of having to use 128 bits to store the 16 bits plus a 64 bit pointer. I suppose I could try aligned_alloc() and force all allocations to be on a 64kB boundary, but I don't know how happy OSs would be at fragmenting the allocation space that way.

Basically I want to use that 48 bits as a delta from some base address, where I can be fairly certain any allocation within the program will be within 48 bits of that base address.  But I don't know if things like ASLR make malloc() and its relatives span the full 64-bit virtual address space for applications...

[da_nang]

A cursory google search suggests alignment is architecture and implementation dependent, so I doubt the OS will provide any guarantees.

On an x86-64 architecture, depending on compiler, you'll usually get either 4-byte-aligned or 8-byte-aligned.

So a 64-bit pointer + 16-bit field could be packed [8|2] for 12 bytes or 16 bytes, if in a struct. If it's two separate malloc calls, then I wouldn't make any assumptions. Usually more memory is allocated than needed, but who knows how the OS, CPU, and MMU handle it all. It could be either of the byte-aligned, or it could be something entirely different from legacy OS code.

[McTraveller]

A cursory google search suggests alignment is architecture and implementation dependent, so I doubt the OS will provide any guarantees.

On an x86-64 architecture, depending on compiler, you'll usually get either 4-byte-aligned or 8-byte-aligned.

So a 64-bit pointer + 16-bit field could be packed [8|2] for 12 bytes or 16 bytes, if in a struct. If it's two separate malloc calls, then I wouldn't make any assumptions. Usually more memory is allocated than needed, but who knows how the OS, CPU, and MMU handle it all. It could be either of the byte-aligned, or it could be something entirely different from legacy OS code.

The rules for malloc state the pointer you get may be aligned to the size of the element you are being allocated if that is smaller than the size of a pointer, or the alignment of a pointer, but that's it. So if you malloc(1) you can get a pointer where the 1s bit is significant.  But if you malloc(87), you are probably going to get a pointer with the three low bits set to zero.

POSIX has aligned_alloc() which lets you specify an alignment bigger than 8, but it has to be a power of 2; you can't ever ask for an alignment of, say, 12 bytes.

I really am trying to not waste space; I don't want to use two 64-bit words to store a pointer plus my 16 bits of "useful" information; this means I'm storing 16 bytes for every 10 bytes of useful info, which is 60% overhead - this adds up in the large dataset I'm working with.

So if I can demonstrate the difference between two malloc() in a single program instance is less than 2^48 (technically 2^55, since I can aligned_alloc() on 128 bytes and not have to worry about the low 7 bits of the pointers), then I can use an arbitrary malloc() as a "base address" and then every other pointer I can treat as an offset from that base address - allowing me to use 6 bytes for the pointer and reduce my memory requirements by 60%.

I could manually create a base address and create my own indirection table, but again because of large data sets I'd rather not do that because of the performance hits associated with too many indirections in likely different areas of memory which will not be cache-local.

[WealthyRadish]

The delta solution seems essentially equivalent to packing in the upper 16 bits, discussed at length here:

https://stackoverflow.com/questions/16198700/using-the-extra-16-bits-in-64-bit-pointers

[Telgin]

It sounds like you may have already considered this, but you can write your own memory allocator.  If you know how big your memory requirements are ahead of time, you can preallocate a large chunk of memory and parcel it out yourself as needed.  That should guarantee that all of the memory is contiguous and within a 48-bit range, unless of course you somehow needed more memory than that.

Of course, writing the allocator is an exercise left to the reader and may not be remotely worth the trouble.  I know games routinely do this for things like object pools where they know they'll never need more than, say, 100 Foos.  They can just allocate space for 100 of them and use the buffer as a ring buffer, flag objects as alive or dead, or whatever.  No extra indirection table needed, but it does mean you need to be able to know your maximum memory usage ahead of time.

[McTraveller]

So firstly, as the stack overflow says, those "unused" address bits aren't really unused. Especially if virtual addresses are being used; those upper bits are indeed significant.  You can safely use the low bits (by masking them out) because often the low 2 or 3 bits of a pointer are meaningless for aligned pointers.

I have thought about doing a big mmap() and doing my own full memory allocation - in fact I do have a memory allocation subsystem "on top of" malloc; maybe I should just indeed do a single malloc/mmap and work within that...

[WealthyRadish]

Right, but if you treat those 16 bits as being arbitrarily significant then a 48-bit delta wouldn't suffice either, and if you more narrowly focus on e.g. the x86-64 canonical form you'll likely end up resorting to the same "sign extension" to restore the pointer in either approach (so they're basically equivalent).

It's obviously brittle long-long term, but unlike previous eras where these same techniques were used and required extensive rewrites I think this will be longer-lived; I bet most compilers will have flags to explicitly request only 48/47-bit address space from the OS, if they don't already.


For anyone interested in custom allocators in C++, I found this cppcon talk helpful:

https://www.youtube.com/watch?v=LIb3L4vKZ7U

Using custom allocators is much less of a nightmare to use when they are composed like this, such as seamlessly including a "backup" allocator when they run out of memory (i.e. fall back on malloc) or being able to easily insert a debugging layer capable of checking for leaks and corruption.

[McTraveller]

Huh - I'm digging more into that stack overflow response, and it seems to not match with what my system actually does.  I don't like coding empirically - I wish things matched documented.

Specifically it says that x86_64, even virtual memory addresses, should be "canonical" which means copy bit 47 into bits 63-48.  But when I do malloc() on my machine, the pointers are all of the form:

0x7fea nnnn nnnn nnnn

So this is NOT a sign extension, since all the bits are not the same; I'd expect them to be all 1 (start with 0xffff) or all 0 (start with 0x0000).  So I'm not sure what's going on with my machine / OS.

EDIT: hey I can't count (coupled with %p formatting strips leading zeros)!  Turns out all my pointers are of the form 0x0000 7fea nnnn nnnn.

So they ARE canonical.

EDIT2:  So now I have to decide if I want to write nonportable code, or do something like put my own allocator on top of the standard ones.  (Also note: I'm a C purist, none of this ++ or # stuff  )

[WealthyRadish]

Does it do it while in both your compiler's debug and release mode?

Edit: also, 64 bit mode ofc

[Folly]

[Telgin]

Modern C++ is almost as impenetrable to me, so I can't even complain.