Valgrind Warning: Client Switching Stacks?

If you've been doing any C or C++ development for some time, you might have heard of Valgrind. It's undoubtedly an excellent tool, and it helped me track down hard-to-find bugs on many occasions. However, its output is sometimes a bit difficult to grasp. So I made a little toy program to shed some light on one of the more cryptic warnings. The example I give here might not be the only possible cause of the warning, but it is a common one. This is how the warning looks like:

Warning: client switching stacks? SP change: 0x1fff000940 --> 0x1ffe800940
         to suppress, use: --max-stackframe=10022912 or greater

With this warning, the interesting part is, in fact, the two values of the stack pointer that is printed out. If you do the math, you'll find out that in my example here, the pointers are 8MiB apart from each other. This might, indeed, be problematic.

Why? When your process runs, the stack has a fixed size. And when you run out of space on the stack, the stack will overflow, and your program will crash. Do you remember those days when you were writing your first recursive functions? That kind of crash.

What's the size of the stack, then? If you're running Linux, you can check the stack size using ulimit:

$ ulimit -s
9788

The output is in KiB. Here, ulimit tells me that I have, give or take, 9.3 MiB of stack space. In this contrived example, my program somehow, mysteriously, moved the SP 8 MiB backward. Or, forward? See Eli Bendersky's article on the topic if you'd like to clear out the confusion (hint: the stack grows towards lower addresses on x86). The program doesn't crash just yet, but note that if it moves the SP a bit further, all bets are off. What exactly happened, then? Let's take a look at the code:

#include <iostream>
#include <cstdlib>

void crashme(size_t n_KiB) {
  char big[n_KiB*1024];
  big[0] = 0xFF;
}

int main() {
  crashme(8192);

  std::cout << "It works!" << std::endl;
}

Now it should be obvious. The program does not switch stacks per se. It's the big array that's allocated inside crashme that triggered the warning!

The size of the array is a parameter of the function. If this is raising your eyebrows, be advised that this is, in fact, perfectly fine in C++ compiled with GCC. It's a great feature in that you can have variable length arrays without dynamic allocation. You need to be aware of the limitations, though. If someone calls this with a value equal to ulimit -s or greater, your program will crash. If someone calls it from a place where a more significant portion of the stack has been consumed for other things, things can go pear-shaped even with a lot lower values.

How do you fix it, then? The question to ask is: Is the performance gain from the automatic allocation worth the risk? I'd argue that most of the time, it isn't. If that's the case, a straightforward fix is to allocate the array in question dynamically, either with new [] or by changing it to a vector.

Hopefully, this post will help you understand and fix the warning the next time you see it. As you've seen, it might be pointing to an issue with your code that can lead to a stack overflow down the line.

---

Cover photo by Ryan Fields on Unsplash