Pozzo & Lucky Busted. The Tales of a Mathematician / SOC analyst… (Part-3)

Before we even begin:

pip install entropy     # contains a C implementation of the 
                        # Shannon Entropy algorithm for byte strings
pip install fitter      # Compares datasets with known distributions
pip install pandas      # and returns similarity ratios (fitter requirement)

pip install matplotlib  # Makes python laugh in Matlab's face...



"Requirement already satisfied (use --upgrade to upgrade): [...]"(2016, pip)

Windows 7 Ultimate 64 bit VM installed. Firewall is Down –I repeatFirewall is Down.

Get Set,

This Article/Post is the last of a series about steganography in IP/TCP headers and a Remote Code Execution tool that uses this technique. It will demonstrate ways of busting traffic created from this tool and maybe other similarly functioning tools. The whole background story can (and should) be read at:



The entropy discussion

The whole concept with randomness in Pozzo & Lucky was to trick Security Devices on thinking that the “dirty” packets were OS (or nmap) created packets and ultimately to avoid any kind of cryptanalysis. But this didn’t work so well (at least for all used TCP/IP fields).

The Good News

The good news is that the IP identification field is effectively random in OS SYN packets.

Specifically my scripts got me something like this:

Entropy for /dev/random is 0.836880 for 152 bytes
Entropy for Pozzo packets ID field is 0.836477 for 152 bytes
Entropy for Lucky packets ID field is 0.849231 for 152 bytes

Entropy for /dev/random is 0.922406 for 304 bytes
Entropy for Pozzo packets SEQ field is 0.919504 for 304 bytes
Entropy for Lucky packets SEQ field is 0.906196 for 304 bytes

for the Pozzo & Lucky command:

head /etc/shadow

To get root user’s password hash and 9 more lines. The entropy is pretty damn high for a data transfer, which means that the Rotating Encryption Scheme (explained in part-2) is working flawlessly!

The -not so- Good News

But what about a real nmap on a Windows machine (most common case)?


got us this…

Entropy for /dev/random is 0.987001 for 1982 bytes
Entropy for SYN packets ID field is 0.989606 for 2150 bytes
Entropy for RST packets ID field is 0.755026 for 1982 bytes

Entropy for /dev/random is 0.994242 for 3964 bytes
Entropy for SYN packets SEQ field is 0.272816 for 4300 bytes
Entropy for RST packets SEQ field is 0.000000 for 3964 bytes

That’s a ZERO there. Those damn RSTs always have a Sequence Number of a literal 0 in port scans. Specifically (RFC 793, page 65):

    If the state is CLOSED [...] then
      [...]  The acknowledgment and sequence field values are selected
      to make the reset sequence acceptable to the TCP that sent
      the offending segment.

      If the ACK bit is off, sequence number zero is used,   # This is a 
        <SEQ=0><ACK=SEG.SEQ+SEG.LEN><CTL=RST,ACK>            # port scan reply

      If the ACK bit is on,

This sent me flying. I don’t know how I ignored it the first time I read the RFC Bible. The impact of this is simply: No Stego in Sequence Field of packets sent from the Lucky side. And that means responses will have a bandwidth of 1 byte (+1 for the opcode) coming from the IP Identification Field – not feasible.

Blind Remote Code Execution should still work as intended though

Generally the SYN packets from Pozzo seem to blend in well with the OS packets, as the OS is using randomness in the ID field. (The nmap command -without flags- doesn’t use spoofed packets, it uses the connect() System Call from the OS API. That’s why it doesn’t need root too).

A histogram equals a thousand words after all:

Remember, the Possible values are 0-65535(2^16)

The RST packets though do not blend so very well…


This slideshow requires JavaScript.

And that’s because the OSes use Sequential IDs as responses to port scans instead of random values. The Lucky RSTs have really small bins all over the histograms while the OS RSTs are all interpreted as single high bins (as their values are very close).

And let’s conclude the IP Identification field analysis with some notes. In the RFC that updates (almost redefines the ID field – RFC 6864) there is a concern about Covert Channels (Security Considerations, p16) and also the idea for middle devices (routers, firewalls) to rewrite the ID field of passing packets to improve untrackability and prevent OS fingerprinting (keep this in mind for a while). Generally this RFC is the big “Ooops!!!” on the single line definition of the ID field back in IP protocol’s first definition (RFC 791).


The Sequence Field in TCP

Here we are really losing the battle…

Let’s look at the histograms of the SYN packet ISN values :
(max value 2^32 = 4.294.967.296 ~ 4,294 * 10^-9)


It is clear that nmap is failing us… It uses the same Sequence Number to knock all ports… And that is weird too. Because the “nmap -sS” is a root process and could lazily craft the same packet all over (without changing the ISN, only the Destination Port), but why the OS powered nmap? Isn’t the OS stack responsible for changing the ISN value?

A closer look at the Source Port:SYN_SPORT_Linux.png

So nmap uses the same Source Port to knock all target ports, that’s why it uses the same ISN all over. So Pozzo & Lucky can’t resemble nmap. It’s final

But what about the netcat? Leaving Pozzo & Lucky alone with netcat in a histogram shows this:SYN_SEQ_netcat.png

And yes, netcat uses multiple Source Ports, as well.SYN_SPORT_netcat.png

Here you can see that Pozzo & Lucky, uses only a range of high ports (The OTP Scheme is responsible for that), while netcat is a lot more random. While this is a recognizable pattern for sure, I don’t believe that it is a Covert Channel evidence.


Those RSTs…

The Pozzo & Lucky RSTs are so scattered that can’t be seen.

This is clearly the TCP protocol violation mentioned earlier. The RSTs Sequence Number is always ZERO unless Pozzo & Lucky hacked you. This can’t be hidden. This flawed us. At least NextGen Firewalls should catch this… – or #not.


There is more!

Protocol field values and violations aren’t the only deviations from port scans or normal protocol usage. And while other aspects can’t be proofs of Covert Channeling they can reveal patterns and indications that will need extra analysis. And extra analysis will eventually mean “let’s look at that box a little bit, what was its hostname again?“, and then it’s Game Over. Lucky is a process, a memory dump (even if hidden correctly) will reveal it.


And its Time, time, time…

The timeframes between packets from the same source always reveal patterns too. Masscan, zmap, nmap and netcat are all port scanners. But they are really different in this aspect.

This is a great histogram:RST_TIME_Linux.png

Pretty self explanatory! Netcat is slower. Maybe because it switches Source Ports all the time! But what about Pozzo & Lucky?

Pozzo & Lucky by design has a lot of overhead before sending and after receiving a packet. It calculates 4 SHA512 hashes for every packet, XORs and deChunks while making dictionary lookups for the Opcodes… If netcat needs an average 0.0005 sec between packets, Pozzo & Lucky needs 0.1-0.2 sec. That’s several orders of magnitude higher.



Hands on now! What About Firewalls?

Let’s now actually check the traces! Talked too much, did too little. I hate that. Let’s get our hands dirty! Let’s establish a very sneaky backdoor using Pozzo & Lucky.

The Test Setup

The Actors

I got 2 lil’ machines. Two Ubuntu 12.04.5 32-bit VMs, not fully upgraded (who has time for upgrades), that are gonna run the experiment. One will be Pozzo and the other will be Lucky. They will be at the 2 sides of a VM Firewall.

The Cop

Who else? pfSense with Suricata plugin (IDS/IPS) will be the testbench. Suricata will have all free rules enabled in full log mode. We are gonna see everything! Snort’s registered user rules won’t be absent too!

 The (Test) Case

Pozzo host will generate an SSH key pair, will use Pozzo & Lucky to send the public key to the other side (maybe in /root/.ssh/authorized_keys) and login to Lucky Host as root interactively.

The Outcome…

Suricata and pfSense logs for both interfaces.

What you will see:

At first I turn on the syslog listener and pipe it to a file (syslog.log). Then I watch the file for changes (the sed and grep panic is just used for formatting).

I then show that a spoofed scapy RST packet raises 2 alerts, 1 from Suricata IDS and 1 from the Firewall itself. I do this to demonstrate that the logging is working as expected.

Then I start Lucky and Pozzo (the order doesn’t matter). I create the .ssh/ directory at  /root/ and append my SSH Public key in the authorized_keys file. After that I log in with SSH but unfortunately the program crashed and you can’t see that.

What you saw:
 (Left+Up-Pozzo, Left+Down-scapy, Right+Up-Lucky, Right+Center-Syslogs from pfSense, Right+Down-netcat syslog listener)

As you can see there are NO LOGS. Low rate port scans DO NOT GET LOGGED without special Firewall rules. And even getting port scan logs is useless. TCP anomaly logs (from Suricata’s encoder.alerts) could be better. None saw the Remote Command Execution we had to that Right Box… Round 1 belongs to Pozzo & Lucky!


The Plot Twist…

pfSense can dodge Pozzo & Lucky shell in 2-3 mouse clicks, without even knowing it…

I was always wondering who the hell has actually read the protocol RFCs. Well I got my answer today! Firewall developers read them. And read them good!

In this video pfSense totally KO’s the Stego between the 2 hosts by altering the random bits in the header by itself. This is not a new idea either!
Here goes (RFC 6864, p16 – also referenced elsewhere):

[...] (IP Identification) field can more easily be used as a covert channel.
For some atomic datagrams it is now possible, and may be desirable,
to rewrite the IPv4 ID field to avoid its use as such a channel.

And pfSense got the second round…

Is there a third round?

No, there isn’t. Both opponent aren’t ready for this… pfSense lacks protocol sanitizers as it seems (ignoring a TCP violation is a good reason to think of this), while Pozzo & Lucky isn’t smart enough to work when its bandwidth is trimmed. It just jams, losing you the root shell you painfully earned.
Maybe someday…



Final thoughts on the project

A crazy ride in TCP/IP Protocols and their implementations!

Learned about the Linux Weak Host Network Stack (RFC1122, p63), the Kernel Stack Override issue (“which packet leaves first, kernel’s or scapy’s?“) and other similar frustrating network issues…
I learned all of them the hard way and “lost” hours on trying to work around them. Made me better. And older…


Da Code…

When the tool becomes stable enough I will upload most of it to my github page. I will omit the parts of Command Execution while keeping the data transfer methods intact. Anyone familiar with Python can put back the Command Execution features, but I don’t want to distribute them personally. People sometimes get irresponsible if there are no logs of their actions.
I will also upload the analysis scripts I created for this Article. All histograms were created with a single button push and I find that valuable enough to share! They need a bit more polishing and they’ll be ready!

Edit 15/7/17: I didn’t manage to tidy the code up to the point to not be ashamed of it so I didn’t upload it. Instead I am currently developing a whole new framework, that models all kinds of shells, and let’s a security tool developer to create his/her own Post-Exploitation tool. This framework is public at time of writing, and it can be found in this post (repo included). An (lot cleaner) implementation of Pozzo & Lucky will emerge from this eventually. I apologize for the false-promise.


Next Project…

Maybe the world lacks a fully customizable Network Intrusion Detection System (IDS)… Or I will stop coding to actually sit for the SPSE course. Coding doesn’t leave enough time to code sometimes! Strange but true…




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s