Following on from Part 1 , we will further enhance our script to trace incoming connections through the kernel to the application.
Before doing that we will tweak the script to provide a CSV output. We do this using a BEGIN probe to print the header and then update the probes. We will also split the probes. Where there is a match, multiple probes are triggered in the order listed in the script. We can therefore say we are only interested in a specific port and then do the heavy lifting later on.
It is also important that you clear variables once you finish with them so as not to fill up the kernel buffers (you will loose traces if you don't; given enough probes firing).
#!/usr/sbin/dtrace -Cs
#pragma D option quiet
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/ip.h>
BEGIN
{
printf("TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax\n");
}
fbt:ip:tcp_conn_request:entry
{
self->tcpq = *(tcp_t**)((char*)arg0+0x28);
self->lport = ntohs(*(uint16_t*)((char*)self->tcpq->tcp_connp+0x10a));
}
fbt:ip:tcp_conn_request:entry
/self->lport == $1/
{
this->srcAddr = 0;
this->srcPort = 0;
printf("%d,%Y,%s,0x%08x,%d,%d,%d,%d,%d\n",
timestamp, walltimestamp, "syn",
this->srcAddr, this->srcPort,
self->lport,
self->tcpq->tcp_conn_req_cnt_q0,
self->tcpq->tcp_conn_req_cnt_q,
self->tcpq->tcp_conn_req_max
);
}
fbt:ip:tcp_conn_request:return
/self->lport/
{
self->tcpq = 0;
self->lport = 0;
}
Whilst in tcp_conn_request() we don't actually have a connection from the client; the kernel is about to build one.
It is the message in the call that is actually the incoming packet to process. IP has dealt with it; now it is TCP's turn. Remember that the tcpq we have used so far is the listener's connection (the one that is in the listen state), not the eager's (new connection); so looking at that is not going to be productive.
We could probe another function or we could decode the message; tcp_accept_comm() is called by tcp_conn_request() to accept the connection providing a few things are satisfied such as Qmax not been reached. If we are only interested in connections that aren't dropped due to, e.g. Q==Qmax, then this would be a good place. However, if we also want to capture failed connections, we need to do the processing ourselves.
So, let's decode the message :D
Looking at the source for tcp_conn_request() we can see that arg1 ( mp , the mblk_t ) is the message. Not surprising as this is a STREAMS module. Further down we can see where the data is and the format of the data. i.e. we have a pointer to the IP header here.
ipha = (struct ip *)mp->b_rptr;
As mblk_t is a standard structure we can easily map this within our DTrace script. We can therefore change the setting of srcAddr to the following:
this->mblk = (mblk_t *)arg1;
this->iphdr = (struct ip*)this->mblk->b_rptr;
this->srcAddr = this->iphdr->ip_src.s_addr;
If we then run the script and connect to the server running on 172.16.170.133 from client 172.16.170.1 we get the following:
root@sol10-u9-t4# ./tcpq_ex7.d 22
TS,Date,Action,Source IP,Source Port,Dest Port,Q0,Q,Qmax
180548302609318,2016 Aug 11 14:37:02,syn, 0x85aa10ac ,0,22,0,0,8
As you can see, the hex address is that of the destination and not the source. Given that an IP header is a standard format (defined in rfc791 ) this is a bit surprising. So, what is going on?
A Slight Detour
The IP header is defined in /usr/include/netinet/ip.h . It is as expected. Whilst I won't go into all the steps to find the cause, here is the quick version.
The DTrace compiler uses the C pre-processor, but compiles it's own code for the kernel to use. DTrace should be able to import C headers to allow you to use names, etc, but clearly something has gone wrong.
One test I did is to create my own header using the constituent parts and a key tweak; one struct doesn't use fields (bitmasks) but both are the same size.
#ifndef MYIP_H
#define MYIP_H
#include <sys/types.h>
typedef struct {
uchar_t c1;
uchar_t c2,c3,c4;
ushort_t s1,s2,s3,s4;
uint32_t src,dst;
} myip_t;
typedef struct {
uchar_t