Dakotacon 2015 RE Challenge C2 Writeup
The Dakotacon C2 challenge was moderately difficult in that it actually required you to do a little programming work – most of the other C-based challenges in this CTF allowed you to set a breakpoint at a smart place (e.g. right before
strcmp()), and view the key in-memory.
Instead of comparing your input with the actual key, this binary encodes your input and compares it with an encoded version of the key.
Here is the encoding algorithm, in IDA-assembly, with my psuedocode added as comment on the right:
So now we need to figure out the algorithmic inverse. It’s not difficult – all we need to do is change the direction of the rotates from left to right. Let’s do it in psuedocode real quick:
Let’s turn that psuedocode into some actual C code:
Now all we need is the encoded key from the program, so we can run it through that decode loop. One way to get that encoded key is simply to set a breakpoint on the
strncmp() at address
0x01231178, which is the check for the correct key. The first argument pushed onto the stack is a pointer to the encoded key – you can verify this by looking a few dozen instructions earlier, where you’ll see the arguments being set up/pushed onto the stack. If you’re following along in IDA, you should see something like this:
Note the highlighted bytes in the lower-left-hand panel. That’s the encoded key, and what we want… those bytes were pushed onto the stack earlier in the program.
If we copy/paste those into our program, and we can take a shot at decoding them!
Here’s the complete decoder I wrote to solve this challenge.