高性能TCPServer

最近两天正在学习TCPServer，在网上看到比较好的两篇文章，在这里记录下来。

• High Performance .NET Socket Server Using Async Winsock（原文地址）

Introduction

Sockets are the foundation of all network based programming. Every web server, ftp, chat, or media server is built upon the sockets API core functions. In apast example, I showed you how simple it is to build a basic client to server interaction using the sockets API in .NET. This example will show you how to scale the sockets API so that hundreds, or even thousands of clients can use your server at the same time. This example introduces the idea of asynchronous sockets, which were added to the Winsock framework to promote scalability in large applications. Async sockets use multiple application threads to simultaneously read and write to many connection sockets.

Building the Sample

This is a console application written in C# and .NET. This sample should be compiled in Visual Studio 2012 or above. You can run more than one instance of this application on a single machine, or across multiple machinees on a network, which in fact you will want to do to test the scalability of the application. Only one server should be instantiated within the application per port on a machine, however.

Description

This is quite a large example, and contains a few bits of complex code. However, I have tried to document the code heavily inside of the source. So you may want to open the source code in Visual Studio 2012 or above and follow along in there. I am going to describe the entire application here, focusing on the interesting parts of async sockets programming. To understand the complete flow of the example, you should examine the source or step through it, starting in the Main() function of the application.

Beginning at the top of the program, we have created a simple stack class,OSAsyncEventStack,that will contain a reusable pool ofSocketAsyncEventArgs that we will use in our server to handle client connections. You can think of SocketAsyncEventArgs as representing an async socket connection, complete with its underlying plumbing and sockets. In this example, each client that connects to the server will consume oneSocketAsyncEventArgs instance, and to promote scalability and performance, we are going to reuse a collection of these from the stack. The stack has only two methods, Pop() and Push(). Those of you who remember assembler code will recognize these as gettting and putting an instance on the stack, respectively.

the next class in the example,OSUserToken, is where we will actually do our socket processing on the server. In other words, when a client is connected and the server has accepted the connection, the UserToken will be instantiated to read the data off of the client connection and could actually write an acknowledgment back to the client, although in this example that code is only partially implemented and is commented out. Later on, I will show you how the OSUserToken is tied to the actual client connection socket at run-time. For now, let's just look at the two functions inside of the user token that do actual work:

C#

//Thismethodgetsthedataoutofthereadsocketandaddsittotheaccumulatorstringbuilder
publicboolReadSocketData(SocketAsyncEventArgsreadSocket)
{
Int32bytecount=readSocket.BytesTransferred;

if((totalbytecount+bytecount)>stringbuilder.Capacity)
{
LastError="ReceiveBuffercannotholdtheentiremessageforthisconnection.";
returnfalse;
}
else
{
stringbuilder.Append(Encoding.ASCII.GetString(readSocket.Buffer,readSocket.Offset,bytecount));
totalbytecount+=bytecount;
returntrue;
}
}

The ReadSocketData() function reads the data on the client connection socket (called a read socket). When we instantiate the UserToken class, we specify how big a client message can be in total. This function checks to see whether our buffer (in this case a StringBuilder) can hold any more read data, and if it can we append the data in the read socket to the string. To understand this function, it is important to know that a client can send many packets of information to a server through a connection, and each one will generate a read socket on that connection. So we may need to call this function many timed for a single client connection, in order to receive the entire message from the client.

When we have finished receiving data from the client, we probably want to do something useful with it. The ProcessData() function is where we do that.

C#

//Dosomethingwiththereceiveddata,thenresetthetokenforusebyanotherconnection.
//Thisiscalledwhenallofthedatahavebeenreceivedforareadsocket.
publicvoidProcessData(SocketAsyncEventArgsargs)
{
//Getthelastmessagereceivedfromtheclient,whichhasbeenstoredinthestringbuilder.
Stringreceived=stringbuilder.ToString();

//TODOUsemessagereceivedtoperformaspecificoperation.
Console.WriteLine("Received:\"{0}\".Theserverhasread{1}bytes.",received,received.Length);

//TODO:Loadupasendbuffertosendanackbacktothecallingclient
//Byte[]sendBuffer=Encoding.ASCII.GetBytes(received);
//args.SetBuffer(sendBuffer,0,sendBuffer.Length);

//ClearStringBuffer,soitcanreceivemoredatafromtheclient.
stringbuilder.Length=0;
totalbytecount=0;
}

In our example, we are going to echo to the console whatever message was received from the client. We could also send something back to the client at this point, but we are not going to do that in our example, so that code is commented out. When we are done with the processing, we clear out the string builder because each user token will be contained within an AsyncEventArgs instance on the reusable stack that we established earlier, so we may want to use the user token over and over again.

Now we move on to theOSCoreclass. This example contains both client and server code. As we saw in my earlier example, Winsock clients and servers share a lot of common code. This class contains the code that is common for our client and server classes.

Some things that are contained in this class that you might want to change are the default server name, port, and buffer size for the application. In a real application, you would probably put these in a configuration file and read them in at start up.

The two core functions in this class are the CreateIPEndPoint() function, which takes a server name and port number and converts it into a structure that the socket needs to be instantiated. It looks like this:

C#

//AnIPEndPointcontainsalloftheinformationaboutaserverorclient
//machinethatasocketneeds.Herewecreateonefrominformation
//thatwesendinasparameters
publicIPEndPointCreateIPEndPoint(stringservername,intportnumber)
{
try
{
//WegettheIPaddressandstufffromDNS(DomainNameServices)
//IthinkyoucanalsopassinanIPaddress,butIwouldnotbecause
//thatwouldnotbeextensibletoIPV6later
IPHostEntryhostInfo=Dns.GetHostEntry(servername);
IPAddressserverAddr=hostInfo.AddressList[0];
returnnewIPEndPoint(serverAddr,portnumber);
}
catch(Exceptionex)
{
exceptionthrown=true;
lasterror=ex.ToString();
returnnull;
}
}

The other important function takes the IPEndPoint and uses it to create a connection socket, either for the client or the server. It is called CreateSocket(). Think of a connection socket as the plug that connects the client or server application to the network. Each side of the connection needs a socket to communicate over the network. Only one line of code is needed to create a Winsock connection socket. It looks like this:

C#

connectionsocket=newSocket(connectionendpoint.AddressFamily,SocketType.Stream,ProtocolType.Tcp);

CreateSocket() does some other things to allow flexibility in the socket creation, but they are pretty self-explanatory so I won't go into them here.

Now we come to the meet of the example, theOSServerclass. This class creates and manages the socket server. It is derived from OSCore, as you might expect. At the top of the class, we do a couple of things:

C#

//Welimitthisserverclientconnectionsfortestpurposes
protectedconstintDEFAULT_MAX_CONNECTIONS=4;

//WeuseaMutextoblockthelistenerthreadsothatlimitedclientconnectionsareactive
//ontheserver.Ifyoustoptheserver,themutexisreleased.
privatestaticMutexmutex;

//Hereiswherewetrackthenumberofclientconnections
protectedintnumconnections;

//Hereiswherewetrackthetotalbytesreadbytheserver
protectedinttotalbytesread;

//Hereisourstackofavailableacceptsockets
protectedOSAsyncEventStacksocketpool;

Notice that our server is limited to 4 client connections. You can change this if you like. If you test the server and open up 5 copies of the application as clients, you will see that only 4 can connect. The 5th client is unable to send messages to the server. Also notice that we name our socketpool object here of our type OSAsyncEventStack.

In our default constructor, we set up the things we need to make the server work:

C#

//Defaultconstructor
publicOSServer()
{
exceptionthrown=false;

//Firstwesetupourmutex
mutex=newMutex();
numconnections=0;

//Thenwecreateourstackofreadsockets
socketpool=newOSAsyncEventStack(DEFAULT_MAX_CONNECTIONS);

//Nowwecreateenoughreadsocketstoservicethemaximumnumberofclients
//thatwewillallowontheserver
//WealsoassigntheeventhandlerforIOCompletedtoeachsocketaswecreateit
//andsetupitsbuffertotherightsize.
//Thenwepushitontoourstacktowaitforaclientconnection
for(Int32i=0;i<DEFAULT_MAX_CONNECTIONS;i++)
{
SocketAsyncEventArgsitem=newSocketAsyncEventArgs();
item.Completed+=newEventHandler<SocketAsyncEventArgs>(OnIOCompleted);
item.SetBuffer(newByte[DEFAULT_BUFFER_SIZE],0,DEFAULT_BUFFER_SIZE);
socketpool.Push(item);
}
}

We will use a Mutex to control thread safety in the application. We will look at that more later. Then we create our socket pool, and fill it with SocketAsyncEventArg objects, as many as we can have client connections (DEFAULT_MAX_CONNECTIONS). Notice within this loop that we also attach the OnIOCompleted function to the Completed event in each SocketAsyncEventArgs item in our pool. This will cause that event to fire and run that function anytime a socket finishes an IO (Input/Output) operation, in other words a read or write to the socket.

To start the server, we call the Start() function.

C#

//Wecallthismethodoncetostarttheserverifitisnotstarted
publicboolStart(stringcmdstring)
{
exceptionthrown=false;

//Firstcreateagenericsocket
if(CreateSocket(cmdstring))
{
try
{
//NowmakeitalistenersocketattheIPaddressandportthatwespecified
connectionsocket.Bind(connectionendpoint);

//Nowstartlisteningonthelistenersocketandwaitforasynchronousclientconnections
connectionsocket.Listen(DEFAULT_MAX_CONNECTIONS);
StartAcceptAsync(null);
mutex.WaitOne();
returntrue;
}
catch(Exceptionex)
{
exceptionthrown=true;
lasterror=ex.ToString();
returnfalse;
}
}
else
{
lasterror="UnknownErrorinServerStart.";
returnfalse;
}
}

In the Start() function, we create a listener socket using the generic OSCore CreateSocket() function. Then we Bind that socket to connection endpoint. This is like plugging the socket into the network at a specific plug point, which is an IP address and Port Number combination. When this is done, we start to Listen() on the endpoint for clients to connect. Now the async magic happens. We call StartAsyncAccept() one time, and block the current thread so that the server just hangs open, listening for connections. The server will not finish executing the Start() method (the part after the WaitOne call) until the Mutex is released somewhere else in the program, which is the Stop() function.

The StartAcceptAsync() function has some tricky business going on in it. Let's take a look.

C#

//Thismethodimplementstheasynchronousloopofevents
//thatacceptsincomingclientconnections
publicvoidStartAcceptAsync(SocketAsyncEventArgsacceptEventArg)
{
//Ifthereisnotanacceptsocket,createit
//Ifthereis,reuseit
if(acceptEventArg==null)
{
acceptEventArg=newSocketAsyncEventArgs();
acceptEventArg.Completed+=newEventHandler<SocketAsyncEventArgs>(OnAcceptCompleted);
}
else
{
acceptEventArg.AcceptSocket=null;
}

//thiswillreturntrueifthereisaconnection
//waitingtobeprocessed(IOPending)
boolacceptpending=connectionsocket.AcceptAsync(acceptEventArg);

//Ifnot,wecangoaheadandprocesstheaccept.
//Otherwise,theCompletedeventwetackedontotheacceptsocketwilldoitwhenitcompletes
if(!acceptpending)
{
//Processtheacceptevent
ProcessAccept(acceptEventArg);
}
}

In this function, we create an accept socket in the form of a SocketAsyncEventArgs object, and we assign a Complete event to it. Then the call to connectionsocket.AcceptAsync() will either return true or false. This function is accepting a client request. If it fully accepts the client request in the time it takes to execute the call (synchronously) then it returns true. If not, it returns false. In either case, ProcessAccept() is going to get called. The difference is that if the accept is synchronous (right away), then we directly invoke ProcessAccept(). Otherwise, the Completed event that was assigned to the accept socket further up in this function will fire when the accept is completed later, and the function OnAcceptCompleted() will invoke ProcessAccept(). In either case, ProcessAccept() will get called eventually when a client socket connection is accepted onto the accept socket. This makes the server highly scalable and performant because we can have lots of client connections being accepted even on a slow network or server, and we know that eventually they will get done, within reason.

Once we accept a connection, we process it with ProcessAccept().

C#

//Thismethodisusedtoprocesstheacceptsocketconnection
privatevoidProcessAccept(SocketAsyncEventArgsAsyncEventArgs)
{
//Firstwegettheacceptsocketfromthepassedinarguments
Socketacceptsocket=AsyncEventArgs.AcceptSocket;

//Iftheacceptsocketisconnectedtoaclientwewillprocessit
//otherwisenothinghappens
if(acceptsocket.Connected)
{
try
{
//Gogetareadsocketoutofthereadsocketstack
SocketAsyncEventArgsreadsocket=socketpool.Pop();

//Ifwegetasocket,useit,otherwiseallthesocketsinthestackareusedup
//andwecan'tacceptanymoreconnectionsuntilonefreesup
if(readsocket!=null)
{
//Createouruserobjectandputtheacceptsocketintoittouselater
readsocket.UserToken=newOSUserToken(acceptsocket,DEFAULT_BUFFER_SIZE);

//Wearenotusingthisrightnow,butitisusefulforcountingconnections
Interlocked.Increment(refnumconnections);

//Startareceiverequestandimmediatelychecktoseeifthereceiveisalreadycomplete
//OtherwiseOnIOCompletedwillgetcalledwhenthereceiveiscomplete
boolIOPending=acceptsocket.ReceiveAsync(readsocket);
if(!IOPending)
{
ProcessReceive(readsocket);
}
}
else
{
acceptsocket.Close();
Console.WriteLine("Clientconnectionrefusedbecausethemaximumnumberofclientconnectionsallowedontheserverhasbeenreached.");
varex=newException("Nomoreconnectionscanbeacceptedontheserver.");
throwex;
}
}
catch(Exceptionex)
{
exceptionthrown=true;
lasterror=ex.ToString();
}

//Starttheprocessagaintowaitforthenextconnection
StartAcceptAsync(AsyncEventArgs);
}
}

In this function, we check to make sure we still have a connected client on the accept socket, and if we do we Pop the next available read socket off of our read socket stack, provided they are all not used up by active client connections. Then we create the user token within our read socket so that we can process the read socket when we get something into it from the client. Then we do the same sort of thing that we did with the accept socket. We perform a ReceiveAsync() on the acceptsocket, which will either return right away or will complete sometime later. In either case, we will perform ProcessReceive() when the ReceiveAsync() completes. Notice at the bottom of this function that we call StartAcceptAsync() again, sort of recursively. This sets up an endless loop of accepting client connections and then processing the read events on those connections until someone stops the server.

When the receive is complete, the ProcessReceive() function looks at the read socket and processes like this:

C#

//Thismethodprocessesthereadsocketonceithasatransaction
privatevoidProcessReceive(SocketAsyncEventArgsreadSocket)
{
//ifBytesTransferredis0,thentheremoteendclosedtheconnection
if(readSocket.BytesTransferred>0)
{
//SocketError.Successindicatesthatthelastoperationontheunderlyingsocketsucceeded
if(readSocket.SocketError==SocketError.Success)
{
OSUserTokentoken=readSocket.UserTokenasOSUserToken;
if(token.ReadSocketData(readSocket))
{
Socketreadsocket=token.OwnerSocket;

//Ifthereadsocketisempty,wecandosomethingwiththedatathatweaccumulated
//fromallofthepreviousreadrequestsonthissocket
if(readsocket.Available==0)
{
token.ProcessData(readSocket);
}

//Startanotherreceiverequestandimmediatelychecktoseeifthereceiveisalreadycomplete
//OtherwiseOnIOCompletedwillgetcalledwhenthereceiveiscomplete
//Wearebasicallycallingthissamemethodrecursivelyuntilthereisnomoredata
//onthereadsocket
boolIOPending=readsocket.ReceiveAsync(readSocket);
if(!IOPending)
{
ProcessReceive(readSocket);
}
}
else
{
Console.WriteLine(token.LastError);
CloseReadSocket(readSocket);
}

}
else
{
ProcessError(readSocket);
}
}
else
{
CloseReadSocket(readSocket);
}
}

First we make sure that some data got transferred on the read socket, by looking at the BytesTransferred property of the read socket. If something is there, we call the ReadSocketData() method on the user token, which is part of (inside of) the read socket. If no more data is waiting to be processed for this request on this read socket (if readsocket.Available == 0), then we go ahead and process the data in our token. Then we call the ProcessReceive() function recursively, so that we can receive the next message being sent from the client. This will go on as long as data is being received for this read socket, after which we will close the socket and put it back on the socket stack, like this:

C#

//Thismethodclosesthereadsocketandgetsridofourusertokenassociatedwithit
privatevoidCloseReadSocket(OSUserTokentoken,SocketAsyncEventArgsreadSocket)
{
token.Dispose();

//Decrementthecounterkeepingtrackofthetotalnumberofclientsconnectedtotheserver.
Interlocked.Decrement(refnumconnections);

//Putthereadsocketbackinthestacktobeusedagain
socketpool.Push(readSocket);
}

That about sums up the server class.

TheOSClientclass is a basic socket client, of which there are countless examples on the Internet, so I will not go through it in detail here. Suffice to say that it creates a connection socket, connects to the server, and is able to send text data over that connection. We reuse OSCore in this client to minimize redundant code.

TheOSUtilclass just contains the parsing functions for the commands that are valid on the client and server. If you start the application and type help, you will see that the valid commands are:

startserver <port> = Start the OS Server (Limit 1 per box)

connect <server> <port> = Connect the client to the OS Server

disconnect = Disconnect from the OS Server

send <message> = Send a message to the OS Server

exit = Stop the server and quit the program

Note that in startserver, the port is optional. if you don't put one, the default port will be used, which in my example is 804. As the command help says, you should only start one server on a given port. The server always starts on localhost, which is IP 127.0.0.1

The program is a simple console application that allows the user to start a server, connect a client to it, and send messages to the server from the client. You can start the application as many times as you like on your desktop, or you can even start it on another machine on your network. After you have started the program, you should run the command startserver on one of the instances of the application. Do this only once on a port on each box that you are using.

Once you have a server running, any of the application instances can connect to it by calling connect. You must supply a server name and port to this command. If you started the server with default options, you can simply type "connect localhost 804". This will connect you to your local server. Note that there is no guarantee that your connection can actually be used to talk to the server. It is an idiosyncrasy of Winsock that the server in most instances will accept the socket connection from the client, even if it cannot service the connection (for instance because too many clients are connected). You will not know that your client connection is good until you try to send information to the server. You can work around this shortcoming by programming connection acknowledgment code from the server to the client, but that is beyond the scope of this example.

Once you are connected to a server, use the send command to send text to the server. The server will echo the text that is sent to acknowledge that it received the command and text.

Here is a screenshot of the server in action, with a local client connected to it:

Some interesting experiments that you can perform with the code include:

1) Trying to connect too many clients

2) Reducing the buffer size on the server and trying to send a message that is too big.

3) Trying to connect to a server that is not running.

This example should get you well on the way to building your own server for the network. Who knows what you can do with it? The possibilities are endless.

• C# SocketAsyncEventArgs High Performance Socket Code（原文地址）

Introduction

Microsoft created theSocketAsyncEventArgsclass to help you write scalable, high performance socket server code.SocketAsyncEventArgsuses I/O Completion Ports via the asynchronous methods in the .NETSocketclass. A proven way to write scalable, high performance socket code for TCP/IP in Windows can be seen in this article onI/O Completion Ports. And here's another link to a Microsoft page onI/O Completion Ports.SocketAsyncEventArgshelps us to access a socket with advantages like working asynchronously, raising theCompletedevent, setting buffer space, object pooling, having a state object, accessing the socket through a property, etc., while having the performance characteristics ofI/O completion ports. Very nice.

The purpose of this article is to help you understand the fundamentals of using theSocketAsyncEventArgsclass.

Background

You may have started your research into this topic at Microsoft's main page for theSocketAsyncEventArgsclass, as I did. The example code on that page got me started. But it was also confusing. Some of the problems that I noticed with their example code were:

1. It seems that Microsoft removed example code about theUserToken property. TheUserTokenis really important, because if you have to post multiple receive operations to receive a message, then you will need a place to store data between operations. And the same is true for send operations.
2. Some of the method names in the example code were a bit confusing, as were some of the variable names.
3. Their reason for using a Semaphore was not explained really.
4. The example in theBufferManagercode on Microsoft's page for theSetBuffermethod shows how to build the buffer. While their code for building theBufferManagerwas mostly good, the way that they dealt with theSetBuffermethod in theirProcessReceivemethod in the example code for theSocketAsyncEventArgsclass will pretty much work in only the narrowest of examples. If you send a 10 bytestring, and then a 20 bytestring, it won't work. Their code sets the buffer to be whatever size you send on the first message from the client. So after the first message, it would just send back the first 10 bytes. So, we need a better example of how to get the data from the buffer after a receive operation completes, use the data, put data in the buffer for a send operation, and resize the buffer before and after a send operation. Also, in theSocketListenerconstructor where theBufferManageris created, Microsoft's example included the variableopsToPreAllocin the calculation fortotalBytes, but not ofbufferSize. That's a mistake which leaves half of the total buffer space being unused.
5. In the explanation of their example code, they said: "For example, if a server application needs to have 15 socket accept operations outstanding at all times to support incoming client connection rates, it can allocate 15 reusableSocketAsyncEventArgsobjects for that purpose." But then their example only included reusableSocketAsyncEventArgsobjects for receive/send, not for accept. TheSocketAsyncEventArgsobject for the accept operation would wait until the receive/send finished to do another accept op. Instead, we can use a pool, as they mentioned in their explanation, and post accept operations faster.

After Microsoft invested the resources to create theSocketAsyncEventArgsclass, it is surprising that they did not invest the resources to give good understandable example code and explanation to help us learn how to use it. This article is designed to fill in that void, because the class really is well done, and very helpful after you understand it.

The code in this article was developed on Visual Studio 2008 using .NET 3.5. This article assumes some knowledge of delegates and event handling in Windows.

Regarding theSocketAsyncEventArgsclass, Microsoft's website says it requires "Platforms: Windows 7, Windows Vista, Windows XP SP3, Windows Server 2008, Windows Server 2003. (The) .NET Framework Supported in: 4, 3.5 SP1, 3.0 SP1, 2.0 SP1. (The) .NET Framework Client Profile Supported in: 4, 3.5 SP1."

TCP Socket Basics

If you have experience with socket server code, you can skip this section.

A socket is like a reference or "handle" to aportwhich allows us to access data sent to that port, which is a reserved space in memory. We will be accessing network data through a socket which "listens" on aTCP port. For those new to socket programming, there are four main steps in using a socket server with TCP. (It's often described as six parts, but I like to put the first three together into one.)

1. Listenfor connection requests on the server

   In order to listen, you need to:

   1. create a socket
   2. bind that socket to a port
   3. listen with that socket

   The client must do its part too. A client (not a server) can initiate a connection request by using theConnectorConnectAsyncmethod. The client machine's Windows TCP/IP subsystem willautomaticallyassign an outgoing port to the socket on the client machine. It will contact the server by sending a SYN packet which is addressed to the socket server's IP address and port number. The client doesnotlisten for incoming connections. It always initiates connections and the server responds. After a client initiates a connection on the server's listening socket, the Windows TCP/IP subsystem of the server will respond with SYN, ACK. Then the client machine's Windows TCP/IP subsystem responds back with an ACK packet. When the ACK is received by the server, the "handshake" is complete, and the connection is established. Windows will handle this TCP/IP protocol stuff for you. In other words, SYN, ACK, PSH, packets, and similar parts of TCP/IP protocol donothave to be coded by you. (Smile here.)

   The server's listening socket can maintain a queue of connection requests waiting to be accepted. This queue is called the "backlog". The listening socket passes the connection info to another socket via an "accept" operation, and then gets the next incoming connection in the backlog queue, or if there is none, waits till there is a new connection request from a client.

2. Acceptconnection requests

   In order to have multiple connections on the same port, the server's listening socket must pass off the connection info to another socket, which accepts it. The accepting socket isnotbound to the port. You post an accept operation to pass the connection from the listening socket to the accepting socket. The accept operation can be posted before the incoming connection is established, so that the listening socket immediately passes off the new connection info to the accepting socket. The client doesnotneed to perform an accept operation.

3. Receive/Sendvia the connection

   After the accept operation has completed, you can now receive or send data with that connection. (The sameSocketAsyncEventArgsobject that did the accept operation could also do the receiving or sending, if we post a receive or send on it and have buffer space for it.) In the design of the code below, theSocketAsyncEventArgswhich did the accept operation passes the connection info over to anotherSocketAsyncEventArgsobject to do receiving/sending. "Receive" is also known as "read". "Send" is also referred to as "write". (We could also split the receiving and sending into two separateSocketAsyncEventArgsobjects, if we wish. But that is more difficult.)

4. Closethe connection

   Either client or server can initiate an operation to close the connection. Usually, the client would initiate that. Again, the lower level TCP/IP of the disconnect is handled by the Windows Operating System. The connection can be closed using theClosemethod, which destroys theSocketand cleans up its managed and unmanaged resources.

So, those are the four main steps in using a socket server with TCP. There are a few more things that you must understand about TCP, in order to be able to write code that uses it.

With TCP, there is no guarantee that one send operation on the client will be equal to one receive operation on the server. One send operation on the client might be equal to one, two, or more receive operations on the server. And the same is true going back to the client from the server. This peculiarity can be due to buffer size, network lag, and the way that the Operating System handles TCP to improve performance. So you must have some way of determining where a TCP message begins and/or ends. Three possible ways to handle TCP messages are:

1. Prefix every message with an integer that tells the length of the message.
2. All messages be fixed length. And both client and server must know the length before the message is sent.
3. Append every message with a delimiter to show where it ends. And both client and server must know what the delimiter is before the message is sent.

Also, your communications protocol should include whether there will be a response (send operation) from the server back to the client after each received message or not. Will that response be after one complete received TCP message, or can it be after more than one message? If it is after one message, the code is simpler probably.

Okay, so let's think about the possible situations that might occur with the data that the server receives in one receive operation:

1. On the first receive op, receive less bytes than the length of the prefix.
2. After having the received part of the prefix on a previous receive op or ops, then receive another part of the prefix, but not all of it.
3. After having received part of the prefix on a previous receive op or ops, then receive the rest of the prefix, but nothing more.
4. After having received part of the prefix on a previous receive op or ops, we then receive the rest of it, plus part of the message.
5. After having received part of the prefix on a previous receive op or ops, we then receive the rest of it, plus all of the message.
6. Receive exactly the number of bytes that are in the prefix, but nothing more.
7. After having received exactly the number of bytes that are in the prefix on a previous receive op or ops, we then receive part of the message.
8. After having received exactly the number of bytes that are in the prefix on a previous receive op or ops, we then receive all of the message.
9. Receive the number of bytes for the prefix plus part of the message, butnotall of the message.
10. After having received the prefix and part of the message on a previous receive op or ops, we then receive another part of the message, but not all of it.
11. After having received the prefix and part of the message on a previous receive op or ops, we then receive all the rest of the message.
12. Receive the number of bytes for the prefix plus all of the message on the first receive op.

    The last one is actually the most common thing that will happen. But all of the above things can happen and do happen. If both client and server have buffer sizes larger than the messages, then the situations above may not happen when running both the client and the server on the same machine, or even on a LAN. But TCP is more unpredictable over the Internet where the data passes through multiple machines. So your code needs to allow for all of those possibilities.

Intro to the code

Accept operations. In this app, the socket which does the accept operation can be accessed through aSocketAsyncEventArgsobject, in itsAcceptSocketproperty. On Microsoft'sAcceptSocketpage, it says, "If not supplied (set tonull) before calling theSocket.AcceptAsyncmethod, a new socket will be created automatically." That's what we will do in the code below. A newSocketobject will be created for every new connection by theSocket.AcceptAsyncmethod. According to theSocket.AcceptAsyncpage, the "new socket is constructed with the sameAddressFamily,SocketType, andProtocolTypeas the current socket", which is the listening socket. I have found that in .NET 3.5theSocket.AcceptAsyncmethodalso copies the settings forLingerStateandNoDelay, even thoughtheSocket.AcceptAsyncpage does not state it.Not sure about other versions of .NET.

We can have a pool of theseSocketAsyncEventArgsobjects to deal with accept operations. In this pool, you do not need one object for each connection the server is maintaining, because after the accept operation completes, a reference to the socket is handed off to anotherSocketAsyncEventArgsobject pretty fast. It doesnotseem to help to put a lot ofSocketAsyncEventArgsobjects in the pool for accept operations. Again, repeating for clarity, the socket which does the accept operation can be accessed through theSocketAsyncEventArgs.AcceptSocketproperty of theSocketAsyncEventArgsobjects that come out of the pool of theSocketAsyncEventArgsobjects that we create for accept operations. After we pass a reference to the socket object from theSocketAsyncEventArgsobject that does accept ops to aSocketAsyncEventArgsobject that does send/receive operations, then you will access the socket through theSocketAsyncEventArgs.AcceptSocketproperty of theSocketAsyncEventArgsobjects that does send/receive operations.

In .NET, instead of creating a new socket object for every accept op, there is the option of having a pool ofSocketobjects and reusing sockets. A socket pool can yield a performance increase on aserverin a situation where there are many clients connecting and disconnecting very quickly. (Don't try to reuse a socket on aclientunless you will be connecting to a different server endpoint when you reuse it.) Use theDisconnectorDisconnectAsyncmethod with appropriate options, instead of theClosemethod, if you use a socket pool.

Receive/Send operations. In this app, the receive and send operations are handled viaSocketAsyncEventArgsobjects that come out of a pool ofSocketAsyncEventArgsobjects that we create forreceive/sendoperations. This isnotthe same pool as we just examined regarding accept operations. To improve performance, we have a pool of these objects which do receive and send operations. The number ofSocketAsyncEventArgsobjects in the pool for receive/send operations should probably be at leastequal to the maximum number of concurrent connections allowed.

What is our communication protocol in this code?

1. One message from client will correspond with one message from the server.
2. After a connection is made, the client will send a message first, and then post a receive op to wait for the response from the server. And for each message that the client sends, the server will respond with a message to the client. Then, it will post another receive op and wait for the next message from the client. In our code, the server will make a few changes to the data before responding to the client, so that we do more than just echo data sent by the client. That approach should help you see what happens with the buffers.
3. We will prefix every message with an integer that tells the length of the message.

Note: The code below isnotsufficient to handle the situation where one computer sends multiple messages to the other before the second one responds. That's more complex, and having code for that would pull our thoughts away from the primary purpose of this article and code.

Buffer: Buffers in TCP are unmanaged, that is, not controlled by the .NET Framework, but by the Windows system. So the buffer gets "pinned" to one place in memory, thereby causing memory fragmentation, since the .NET Garbage Collector will not be able to collect that space. This situation is improved by putting all the buffers together in one block of memory, and just reusing that same space over and over.Pay special attention to the code related to buffers, as buffer-related stuff seems to be an area where people have more difficulty.

In this code, I use separate buffer space for send and receive. You could just reuse the same space for both send and receive, thereby using only half as much memory. It isnotnecessarily best to separate the two. I just did it to help you think about buffers. (If you reuse the same space, then you can get rid ofbufferOffsetReceiveandbufferOffsetSendinDataHoldingUserToken. And instead, just use theOffsetproperty in theSocketAsyncEventArgsobject. That's what theOffsetproperty is there for.)

The theoretical maximum size for the buffer block is 2.147 GB, since it uses an integer data type. And you would probably really want less than 500 MB, if on 32 bit Windows. This limitation should only be relevant if you use a large buffer size and/or have a large number of simultaneous connections. For example, if you use a buffer size of 50,000 bytes, and have a separate buffer for send and receive, then that is 100,000 bytes per connection. 2.147 GB divided by 100,000 bytes = 21,470, which would be the maximum number of connections that could use this buffer block with this buffer size and design.

General: I use explanatory comments in the code below, to make them easily understandable whether viewed on screen or a printed page. Sometimes in code comments, I abbreviate "SocketAsyncEventArgs" as "SAEA".

Collapse|Copy Code

class Program
{
    //This variable determines the number of
    //SocketAsyncEventArg objects put in the pool of objects for receive/send.
    //The value of this variable also affects the Semaphore.
    //This app uses a Semaphore to ensure that the max # of connections
    //value does not get exceeded.
    //Max # of connections to a socket can be limited by the Windows Operating System
    //also.
    public const Int32 maxNumberOfConnections = 3000;

    //If this port # will not work for you, it's okay to change it.
    public const Int32 port = 4444;

    //You would want a buffer size larger than 25 probably, unless you know the
    //data will almost always be less than 25. It is just 25 in our test app.
    public const Int32 testBufferSize = 25;

    //This is the maximum number of asynchronous accept operations that can be
    //posted simultaneously. This determines the size of the pool of
    //SocketAsyncEventArgs objects that do accept operations. Note that this
    //is NOT the same as the maximum # of connections.
    public const Int32 maxSimultaneousAcceptOps = 10;

    //The size of the queue of incoming connections for the listen socket.
    public const Int32 backlog = 100;

    //For the BufferManager
    public const Int32 opsToPreAlloc = 2;    // 1 for receive, 1 for send

    //allows excess SAEA objects in pool.
    public const Int32 excessSaeaObjectsInPool = 1;

    //This number must be the same as the value on the client.
    //Tells what size the message prefix will be. Don't change this unless
    //you change the code, because 4 is the length of 32 bit integer, which
    //is what we are using as prefix.
    public const Int32 receivePrefixLength = 4;
    public const Int32 sendPrefixLength = 4;

    public static Int32 mainTransMissionId = 10000;
    public static Int32 startingTid; //
    public static Int32 mainSessionId = 1000000000;

    public static List<dataholder /> listOfDataHolders;

    // To keep a record of maximum number of simultaneous connections
    // that occur while the server is running. This can be limited by operating
    // system and hardware. It will not be higher than the value that you set
    // for maxNumberOfConnections.
    public static Int32 maxSimultaneousClientsThatWereConnected = 0;

    static void Main(String[] args)
    {
        try
        {
            // Get endpoint for the listener.
            IPEndPoint localEndPoint = new IPEndPoint(IPAddress.Any, port);

            WriteInfoToConsole(localEndPoint);

            //This object holds a lot of settings that we pass from Main method
            //to the SocketListener. In a real app, you might want to read
            //these settings from a database or windows registry settings that
            //you would create.
            SocketListenerSettings theSocketListenerSettings =
	            new SocketListenerSettings(maxNumberOfConnections,
	            excessSaeaObjectsInPool, backlog, maxSimultaneousAcceptOps,
	            receivePrefixLength, testBufferSize, sendPrefixLength, opsToPreAlloc,
	            localEndPoint);

            //instantiate the SocketListener.
            SocketListener socketListener = new SocketListener(theSocketListenerSettings);
        }
        catch (Exception ex)
        {
            Console.WriteLine("Error: " + ex.Message);
        }
}

The primary class isSocketListener.

Collapse|Copy Code

class SocketListener
{
    //Buffers for sockets are unmanaged by .NET.
    //So memory used for buffers gets "pinned", which makes the
    //.NET garbage collector work around it, fragmenting the memory.
    //Circumvent this problem by putting all buffers together
    //in one block in memory. Then we will assign a part of that space
    //to each SocketAsyncEventArgs object, and
    //reuse that buffer space each time we reuse the SocketAsyncEventArgs object.
    //Create a large reusable set of buffers for all socket operations.
    BufferManager theBufferManager;

    // the socket used to listen for incoming connection requests
    Socket listenSocket;

    //A Semaphore has two parameters, the initial number of available slots
    // and the maximum number of slots. We'll make them the same.
    //This Semaphore is used to keep from going over max connection #.
    //(It is not about controlling threading really here.)
    Semaphore theMaxConnectionsEnforcer;

    //an object that we pass in and which has all the settings the listener needs
    SocketListenerSettings socketListenerSettings;

    PrefixHandler prefixHandler;
    MessageHandler messageHandler;

    // pool of reusable SocketAsyncEventArgs objects for accept operations
    SocketAsyncEventArgsPool poolOfAcceptEventArgs;
    // pool of reusable SocketAsyncEventArgs objects for
    //receive and send socket operations
    SocketAsyncEventArgsPool poolOfRecSendEventArgs;

    //_______________________________________________________________________________
    // Constructor.
    public SocketListener(SocketListenerSettings theSocketListenerSettings)
    {
        this.socketListenerSettings = theSocketListenerSettings;
        this.prefixHandler = new PrefixHandler();
        this.messageHandler = new MessageHandler();

        //Allocate memory for buffers. We are using a separate buffer space for
        //receive and send, instead of sharing the buffer space, like the Microsoft
        //example does.
        this.theBufferManager = new BufferManager(this.socketListenerSettings.BufferSize
        	* this.socketListenerSettings.NumberOfSaeaForRecSend
        	* this.socketListenerSettings.OpsToPreAllocate,
        		this.socketListenerSettings.BufferSize
        	* this.socketListenerSettings.OpsToPreAllocate);

        this.poolOfRecSendEventArgs = new
        	SocketAsyncEventArgsPool(this.socketListenerSettings.NumberOfSaeaForRecSend);

        this.poolOfAcceptEventArgs = new
        	SocketAsyncEventArgsPool(this.socketListenerSettings.MaxAcceptOps);

        // Create connections count enforcer
        this.theMaxConnectionsEnforcer = new
        	Semaphore(this.socketListenerSettings.MaxConnections,
        	this.socketListenerSettings.MaxConnections);

        //Microsoft's example called these from Main method, which you
        //can easily do if you wish.
        Init();
        StartListen();
    }

    //____________________________________________________________________________
    // initializes the server by preallocating reusable buffers and
    // context objects (SocketAsyncEventArgs objects).
    //It is NOT mandatory that you preallocate them or reuse them. But, but it is
    //done this way to illustrate how the API can
    // easily be used to create reusable objects to increase server performance.

    internal void Init()
    {
        // Allocate one large byte buffer block, which all I/O operations will
        //use a piece of. This guards against memory fragmentation.
        this.theBufferManager.InitBuffer();

        // preallocate pool of SocketAsyncEventArgs objects for accept operations
        for (Int32 i = 0; i < this.socketListenerSettings.MaxAcceptOps; i++)
        {
            // add SocketAsyncEventArg to the pool
            this.poolOfAcceptEventArgs.Push(
                CreateNewSaeaForAccept(poolOfAcceptEventArgs));
        }

        //The pool that we built ABOVE is for SocketAsyncEventArgs objects that do
        // accept operations.
        //Now we will build a separate pool for SAEAs objects
        //that do receive/send operations. One reason to separate them is that accept
        //operations do NOT need a buffer, but receive/send operations do.
        //ReceiveAsync and SendAsync require
        //a parameter for buffer size in SocketAsyncEventArgs.Buffer.
        // So, create pool of SAEA objects for receive/send operations.
        SocketAsyncEventArgs eventArgObjectForPool;

        Int32 tokenId;

        for (Int32 i = 0; i < this.socketListenerSettings.NumberOfSaeaForRecSend; i++)
        {
            //Allocate the SocketAsyncEventArgs object for this loop,
            //to go in its place in the stack which will be the pool
            //for receive/send operation context objects.
            eventArgObjectForPool = new SocketAsyncEventArgs();

            // assign a byte buffer from the buffer block to
            //this particular SocketAsyncEventArg object
            this.theBufferManager.SetBuffer(eventArgObjectForPool);

            tokenId = poolOfRecSendEventArgs.AssignTokenId() + 1000000;

            //Attach the SocketAsyncEventArgs object
            //to its event handler. Since this SocketAsyncEventArgs object is
            //used for both receive and send operations, whenever either of those
            //completes, the IO_Completed method will be called.
            eventArgObjectForPool.Completed += new
            		EventHandler<socketasynceventargs />(IO_Completed);

            //We can store data in the UserToken property of SAEA object.
            DataHoldingUserToken theTempReceiveSendUserToken = new
            	DataHoldingUserToken(eventArgObjectForPool, eventArgObjectForPool.Offset,
            	eventArgObjectForPool.Offset + this.socketListenerSettings.BufferSize,
            	this.socketListenerSettings.ReceivePrefixLength,
            	this.socketListenerSettings.SendPrefixLength, tokenId);

            //We'll have an object that we call DataHolder, that we can remove from
            //the UserToken when we are finished with it. So, we can hang on to the
            //DataHolder, pass it to an app, serialize it, or whatever.
            theTempReceiveSendUserToken.CreateNewDataHolder();

            eventArgObjectForPool.UserToken = theTempReceiveSendUserToken;

            // add this SocketAsyncEventArg object to the pool.
            this.poolOfRecSendEventArgs.Push(eventArgObjectForPool);
    }

    //____________________________________________________________________________
    // This method is called when we need to create a new SAEA object to do
    //accept operations. The reason to put it in a separate method is so that
    //we can easily add more objects to the pool if we need to.
    //You can do that if you do NOT use a buffer in the SAEA object that does
    //the accept operations.
    internal SocketAsyncEventArgs CreateNewSaeaForAccept(SocketAsyncEventArgsPool pool)
    {
        //Allocate the SocketAsyncEventArgs object.
        SocketAsyncEventArgs acceptEventArg = new SocketAsyncEventArgs();

        //SocketAsyncEventArgs.Completed is an event, (the only event,)
        //declared in the SocketAsyncEventArgs class.
        //See http://msdn.microsoft.com/en-us/library/
        //       system.net.sockets.socketasynceventargs.completed.aspx.
        //An event handler should be attached to the event within
        //a SocketAsyncEventArgs instance when an asynchronous socket
        //operation is initiated, otherwise the application will not be able
        //to determine when the operation completes.
        //Attach the event handler, which causes the calling of the
        //AcceptEventArg_Completed object when the accept op completes.
        acceptEventArg.Completed +=
                new EventHandler<SocketAsyncEventArgs>(AcceptEventArg_Completed);

        AcceptOpUserToken theAcceptOpToken = new
        			AcceptOpUserToken(pool.AssignTokenId() + 10000);

        acceptEventArg.UserToken = theAcceptOpToken;

        return acceptEventArg;

        // accept operations do NOT need a buffer.
        //You can see that is true by looking at the
        //methods in the .NET Socket class on the Microsoft website. AcceptAsync does
        //not require a parameter for buffer size.
    }

    //____________________________________________________________________________
    // This method starts the socket server such that it is listening for
    // incoming connection requests.
    internal void StartListen()
    {
        // create the socket which listens for incoming connections
        listenSocket = new
                Socket(this.socketListenerSettings.LocalEndPoint.AddressFamily,
                SocketType.Stream, ProtocolType.Tcp);

        //bind it to the port
        listenSocket.Bind(this.socketListenerSettings.LocalEndPoint);

        // Start the listener with a backlog of however many connections.
        //"backlog" means pending connections.
        //The backlog number is the number of clients that can wait for a
        //SocketAsyncEventArg object that will do an accept operation.
        //The listening socket keeps the backlog as a queue. The backlog allows
        //for a certain # of excess clients waiting to be connected.
        //If the backlog is maxed out, then the client will receive an error when
        //trying to connect.
        //max # for backlog can be limited by the operating system.
        listenSocket.Listen(this.socketListenerSettings.Backlog);

        //Server is listening now****

        // Calls the method which will post accepts on the listening socket.
        //This call just occurs one time from this StartListen method.
        //After that the StartAccept method will be called in a loop.
        StartAccept();
    }

    //____________________________________________________________________________
    // Begins an operation to accept a connection request from the client
    internal void StartAccept()
    {
        //Get a SocketAsyncEventArgs object to accept the connection.
        SocketAsyncEventArgs acceptEventArg;
        //Get it from the pool if there is more than one in the pool.
        //We could use zero as bottom, but one is a little safer.
        if (this.poolOfAcceptEventArgs.Count > 1)
        {
            try
            {
                acceptEventArg = this.poolOfAcceptEventArgs.Pop();
            }
            //or make a new one.
            catch
            {
                acceptEventArg = CreateNewSaeaForAccept(poolOfAcceptEventArgs);
            }
        }
        //or make a new one.
        else
        {
            acceptEventArg = CreateNewSaeaForAccept(poolOfAcceptEventArgs);
        }

        // Semaphore class is used to control access to a resource or pool of
        // resources. Enter the semaphore by calling the WaitOne method, which is
        // inherited from the WaitHandle class, and release the semaphore
        // by calling the Release method. This is a mechanism to prevent exceeding
        // the max # of connections we specified. We'll do this before
        // doing AcceptAsync. If maxConnections value has been reached,
        // then the thread will pause here until the Semaphore gets released,
        // which happens in the CloseClientSocket method.
        this.theMaxConnectionsEnforcer.WaitOne();

        // Socket.AcceptAsync begins asynchronous operation to accept the connection.
        // Note the listening socket will pass info to the SocketAsyncEventArgs
        // object that has the Socket that does the accept operation.
        // If you do not create a Socket object and put it in the SAEA object
        // before calling AcceptAsync and use the AcceptSocket property to get it,
        // then a new Socket object will be created for you by .NET.
        bool willRaiseEvent = listenSocket.AcceptAsync(acceptEventArg);

        // Socket.AcceptAsync returns true if the I/O operation is pending, i.e. is
        // working asynchronously. The
        // SocketAsyncEventArgs.Completed event on the acceptEventArg parameter
        // will be raised upon completion of accept op.
        // AcceptAsync will call the AcceptEventArg_Completed
        // method when it completes, because when we created this SocketAsyncEventArgs
        // object before putting it in the pool, we set the event handler to do it.
        // AcceptAsync returns false if the I/O operation completed synchronously.
        // The SocketAsyncEventArgs.Completed event on the acceptEventArg parameter
        // will NOT be raised when AcceptAsync returns false.
        if (!willRaiseEvent)
        {
            // The code in this if (!willRaiseEvent) statement only runs
            // when the operation was completed synchronously. It is needed because
            // when Socket.AcceptAsync returns false,
            // it does NOT raise the SocketAsyncEventArgs.Completed event.
            // And we need to call ProcessAccept and pass it the SAEA object.
            // This is only when a new connection is being accepted.
            // Probably only relevant in the case of a socket error.
            ProcessAccept(acceptEventArg);
        }
    }

    //____________________________________________________________________________
    // This method is the callback method associated with Socket.AcceptAsync
    // operations and is invoked when an async accept operation completes.
    //This is only when a new connection is being accepted.
    //Notice that Socket.AcceptAsync is returning a value of true, and
    //raising the Completed event when the AcceptAsync method completes.
    private void AcceptEventArg_Completed(object sender, SocketAsyncEventArgs e)
    {
        //Any code that you put in this method will NOT be called if
        //the operation completes synchronously, which will probably happen when
        //there is some kind of socket error. It might be better to put the code
        //in the ProcessAccept method.
        ProcessAccept(e);
    }

    //____________________________________________________________________________
    //The e parameter passed from the AcceptEventArg_Completed method
    //represents the SocketAsyncEventArgs object that did
    //the accept operation. in this method we'll do the handoff from it to the
    //SocketAsyncEventArgs object that will do receive/send.
    private void ProcessAccept(SocketAsyncEventArgs acceptEventArgs)
    {
        // This is when there was an error with the accept op. That should NOT
        // be happening often. It could indicate that there is a problem with
        // that socket. If there is a problem, then we would have an infinite
        // loop here, if we tried to reuse that same socket.
        if (acceptEventArgs.SocketError != SocketError.Success)
        {
            // Loop back to post another accept op. Notice that we are NOT
            // passing the SAEA object here.
            LoopToStartAccept();

            AcceptOpUserToken theAcceptOpToken = 
		(AcceptOpUserToken)acceptEventArgs.UserToken;

            //Let's destroy this socket, since it could be bad.
            HandleBadAccept(acceptEventArgs);

            //Jump out of the method.
            return;
        }

        //Now that the accept operation completed, we can start another
        //accept operation, which will do the same. Notice that we are NOT
        //passing the SAEA object here.
        LoopToStartAccept();

        // Get a SocketAsyncEventArgs object from the pool of receive/send op
        //SocketAsyncEventArgs objects
        SocketAsyncEventArgs receiveSendEventArgs = this.poolOfRecSendEventArgs.Pop();

        //Create sessionId in UserToken.
        ((DataHoldingUserToken)receiveSendEventArgs.UserToken).CreateSessionId();

        //A new socket was created by the AcceptAsync method. The
        //SocketAsyncEventArgs object which did the accept operation has that
        //socket info in its AcceptSocket property. Now we will give
        //a reference for that socket to the SocketAsyncEventArgs
        //object which will do receive/send.
        receiveSendEventArgs.AcceptSocket = acceptEventArgs.AcceptSocket;

        //We have handed off the connection info from the
        //accepting socket to the receiving socket. So, now we can
        //put the SocketAsyncEventArgs object that did the accept operation
        //back in the pool for them. But first we will clear
        //the socket info from that object, so it will be
        //ready for a new socket when it comes out of the pool.
        acceptEventArgs.AcceptSocket = null;
        this.poolOfAcceptEventArgs.Push(acceptEventArgs);

        StartReceive(receiveSendEventArgs);
    }

    //____________________________________________________________________________
    //LoopToStartAccept method just sends us back to the beginning of the
    //StartAccept method, to start the next accept operation on the next
    //connection request that this listening socket will pass of to an
    //accepting socket. We do NOT actually need this method. You could
    //just call StartAccept() in ProcessAccept() where we called LoopToStartAccept().
    //This method is just here to help you visualize the program flow.
    private void LoopToStartAccept()
    {
        StartAccept();
    }

    //____________________________________________________________________________
    // Set the receive buffer and post a receive op.
    private void StartReceive(SocketAsyncEventArgs receiveSendEventArgs)
    {
        //Set the buffer for the receive operation.
        receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetReceive,
             this.socketListenerSettings.BufferSize);

        // Post async receive operation on the socket.
        bool willRaiseEvent =
             receiveSendEventArgs.AcceptSocket.ReceiveAsync(receiveSendEventArgs);

        //Socket.ReceiveAsync returns true if the I/O operation is pending. The
        //SocketAsyncEventArgs.Completed event on the e parameter will be raised
        //upon completion of the operation. So, true will cause the IO_Completed
        //method to be called when the receive operation completes.
        //That's because of the event handler we created when building
        //the pool of SocketAsyncEventArgs objects that perform receive/send.
        //It was the line that said
        //eventArgObjectForPool.Completed +=
        //     new EventHandler<SocketAsyncEventArgs>(IO_Completed);

        //Socket.ReceiveAsync returns false if I/O operation completed synchronously.
        //In that case, the SocketAsyncEventArgs.Completed event on the e parameter
        //will not be raised and the e object passed as a parameter may be
        //examined immediately after the method call
        //returns to retrieve the result of the operation.
        // It may be false in the case of a socket error.
        if (!willRaiseEvent)
        {
            //If the op completed synchronously, we need to call ProcessReceive
            //method directly. This will probably be used rarely, as you will
            //see in testing.
            ProcessReceive(receiveSendEventArgs);
        }
    }

    //____________________________________________________________________________
    // This method is called whenever a receive or send operation completes.
    // Here "e" represents the SocketAsyncEventArgs object associated
    //with the completed receive or send operation
    void IO_Completed(object sender, SocketAsyncEventArgs e)
    {
        //Any code that you put in this method will NOT be called if
        //the operation completes synchronously, which will probably happen when
        //there is some kind of socket error.

        // determine which type of operation just
        // completed and call the associated handler
        switch (e.LastOperation)
        {
            case SocketAsyncOperation.Receive:
                ProcessReceive(e);
                break;

            case SocketAsyncOperation.Send:
                ProcessSend(e);
                break;

            default:
                //This exception will occur if you code the Completed event of some
                //operation to come to this method, by mistake.
                throw new ArgumentException("The last operation completed on
                       the socket was not a receive or send");
        }
    }

    //____________________________________________________________________________
    // This method is invoked by the IO_Completed method
    // when an asynchronous receive operation completes.
    // If the remote host closed the connection, then the socket is closed.
    // Otherwise, we process the received data. And if a complete message was
    // received, then we do some additional processing, to
    // respond to the client.
    private void ProcessReceive(SocketAsyncEventArgs receiveSendEventArgs)
    {
        DataHoldingUserToken receiveSendToken =
                     (DataHoldingUserToken)receiveSendEventArgs.UserToken;

        // If there was a socket error, close the connection. This is NOT a normal
        // situation, if you get an error here.
        // In the Microsoft example code they had this error situation handled
        // at the end of ProcessReceive. Putting it here improves readability
        // by reducing nesting some.
        if (receiveSendEventArgs.SocketError != SocketError.Success)
        {
            receiveSendToken.Reset();
            CloseClientSocket(receiveSendEventArgs);

            //Jump out of the ProcessReceive method.
            return;
        }

        // If no data was received, close the connection. This is a NORMAL
        // situation that shows when the client has finished sending data.
        if (receiveSendEventArgs.BytesTransferred == 0)
        {
            receiveSendToken.Reset();
            CloseClientSocket(receiveSendEventArgs);
            return;
        }

        //The BytesTransferred property tells us how many bytes
        //we need to process.
        Int32 remainingBytesToProcess = receiveSendEventArgs.BytesTransferred;

        //If we have not got all of the prefix already,
        //then we need to work on it here.
        if (receiveSendToken.receivedPrefixBytesDoneCount <
                           this.socketListenerSettings.ReceivePrefixLength)
        {
            remainingBytesToProcess = prefixHandler.HandlePrefix(receiveSendEventArgs,
                      receiveSendToken, remainingBytesToProcess);

            if (remainingBytesToProcess == 0)
            {
                // We need to do another receive op, since we do not have
                // the message yet, but remainingBytesToProcess == 0.
                StartReceive(receiveSendEventArgs);
                //Jump out of the method.
                return;
            }
        }

        // If we have processed the prefix, we can work on the message now.
        // We'll arrive here when we have received enough bytes to read
        // the first byte after the prefix.
        bool incomingTcpMessageIsReady = messageHandler
                  .HandleMessage(receiveSendEventArgs,
                  receiveSendToken, remainingBytesToProcess);

        if (incomingTcpMessageIsReady == true)
        {
            // Pass the DataHolder object to the Mediator here. The data in
            // this DataHolder can be used for all kinds of things that an
            // intelligent and creative person like you might think of.
            receiveSendToken.theMediator.HandleData(receiveSendToken.theDataHolder);

            // Create a new DataHolder for next message.
            receiveSendToken.CreateNewDataHolder();

            //Reset the variables in the UserToken, to be ready for the
            //next message that will be received on the socket in this
            //SAEA object.
            receiveSendToken.Reset();

            receiveSendToken.theMediator.PrepareOutgoingData();
            StartSend(receiveSendToken.theMediator.GiveBack());
        }
        else
        {
            // Since we have NOT gotten enough bytes for the whole message,
            // we need to do another receive op. Reset some variables first.

            // All of the data that we receive in the next receive op will be
            // message. None of it will be prefix. So, we need to move the
            // receiveSendToken.receiveMessageOffset to the beginning of the
            // receive buffer space for this SAEA.
            receiveSendToken.receiveMessageOffset = receiveSendToken.bufferOffsetReceive;

            // Do NOT reset receiveSendToken.receivedPrefixBytesDoneCount here.
            // Just reset recPrefixBytesDoneThisOp.
            receiveSendToken.recPrefixBytesDoneThisOp = 0;

            // Since we have not gotten enough bytes for the whole message,
            // we need to do another receive op.
            StartReceive(receiveSendEventArgs);
        }
    }

    //____________________________________________________________________________
    //Post a send op.
    private void StartSend(SocketAsyncEventArgs receiveSendEventArgs)
    {
        DataHoldingUserToken receiveSendToken =
                       (DataHoldingUserToken)receiveSendEventArgs.UserToken;

        //Set the buffer. You can see on Microsoft's page at
        //http://msdn.microsoft.com/en-us/library/
        //         system.net.sockets.socketasynceventargs.setbuffer.aspx
        //that there are two overloads. One of the overloads has 3 parameters.
        //When setting the buffer, you need 3 parameters the first time you set it,
        //which we did in the Init method. The first of the three parameters
        //tells what byte array to use as the buffer. After we tell what byte array
        //to use we do not need to use the overload with 3 parameters any more.
        //(That is the whole reason for using the buffer block. You keep the same
        //byte array as buffer always, and keep it all in one block.)
        //Now we use the overload with two parameters. We tell
        // (1) the offset and
        // (2) the number of bytes to use, starting at the offset.

        //The number of bytes to send depends on whether the message is larger than
        //the buffer or not. If it is larger than the buffer, then we will have
        //to post more than one send operation. If it is less than or equal to the
        //size of the send buffer, then we can accomplish it in one send op.
        if (receiveSendToken.sendBytesRemainingCount
                       <= this.socketListenerSettings.BufferSize)
        {
            receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetSend,
                       receiveSendToken.sendBytesRemainingCount);
            //Copy the bytes to the buffer associated with this SAEA object.
            Buffer.BlockCopy(receiveSendToken.dataToSend,
                       receiveSendToken.bytesSentAlreadyCount,
                  receiveSendEventArgs.Buffer, receiveSendToken.bufferOffsetSend,
                  receiveSendToken.sendBytesRemainingCount);
        }
        else
        {
            //We cannot try to set the buffer any larger than its size.
            //So since receiveSendToken.sendBytesRemainingCount > BufferSize, we just
            //set it to the maximum size, to send the most data possible.
            receiveSendEventArgs.SetBuffer(receiveSendToken.bufferOffsetSend,
                        this.socketListenerSettings.BufferSize);
            //Copy the bytes to the buffer associated with this SAEA object.
            Buffer.BlockCopy(receiveSendToken.dataToSend,
                       receiveSendToken.bytesSentAlreadyCount,
                  receiveSendEventArgs.Buffer, receiveSendToken.bufferOffsetSend,
                  this.socketListenerSettings.BufferSize);

            //We'll change the value of sendUserToken.sendBytesRemainingCount
            //in the ProcessSend method.
        }

        //post asynchronous send operation
        bool willRaiseEvent =
             receiveSendEventArgs.AcceptSocket.SendAsync(receiveSendEventArgs);

        if (!willRaiseEvent)
        {
            ProcessSend(receiveSendEventArgs);
        }
    }

    //____________________________________________________________________________
    // This method is called by I/O Completed() when an asynchronous send completes.
    // If all of the data has been sent, then this method calls StartReceive
    //to start another receive op on the socket to read any additional
    // data sent from the client. If all of the data has NOT been sent, then it
    //calls StartSend to send more data.
    private void ProcessSend(SocketAsyncEventArgs receiveSendEventArgs)
    {
        DataHoldingUserToken receiveSendToken =
                       (DataHoldingUserToken)receiveSendEventArgs.UserToken;

        receiveSendToken.sendBytesRemainingCount =
                         receiveSendToken.sendBytesRemainingCount
                       - receiveSendEventArgs.BytesTransferred;
        receiveSendToken.bytesSentAlreadyCount +=
                       receiveSendEventArgs.BytesTransferred;

        if (receiveSendEventArgs.SocketError == SocketError.Success)
        {
            if (receiveSendToken.sendBytesRemainingCount == 0)
            {
                StartReceive(receiveSendEventArgs);
            }
            else
            {
                //If some of the bytes in the message have NOT been sent,
                //then we will need to post another send operation.
                //So let's loop back to StartSend().
                StartSend(receiveSendEventArgs);
            }
        }
        else
        {
            //If we are in this else-statement, there was a socket error.
            //In this example we'll just close the socket if there was a socket error
            //when receiving data from the client.
            receiveSendToken.Reset();
            CloseClientSocket(receiveSendEventArgs);
        }
   }

    //_______________________________________________________________________
    // Does the normal destroying of sockets after
    // we finish receiving and sending on a connection.
    private void CloseClientSocket(SocketAsyncEventArgs e)
    {
        var receiveSendToken = (e.UserToken as DataHoldingUserToken);

        // do a shutdown before you close the socket
        try
        {
            e.AcceptSocket.Shutdown(SocketShutdown.Both);
        }
        // throws if socket was already closed
        catch (Exception)
        {
        }

        //This method closes the socket and releases all resources, both
        //managed and unmanaged. It internally calls Dispose.
        e.AcceptSocket.Close();

        //Make sure the new DataHolder has been created for the next connection.
        //If it has, then dataMessageReceived should be null.
        if (receiveSendToken.theDataHolder.dataMessageReceived != null)
        {
            receiveSendToken.CreateNewDataHolder();
        }

        // Put the SocketAsyncEventArg back into the pool,
        // to be used by another client. This
        this.poolOfRecSendEventArgs.Push(e);

        // decrement the counter keeping track of the total number of clients
        //connected to the server, for testing
        Interlocked.Decrement(ref this.numberOfAcceptedSockets);

        //Release Semaphore so that its connection counter will be decremented.
        //This must be done AFTER putting the SocketAsyncEventArg back into the pool,
        //or you can run into problems.
        this.theMaxConnectionsEnforcer.Release();
    }

    //____________________________________________________________________________
    private void HandleBadAccept(SocketAsyncEventArgs acceptEventArgs)
    {
        var acceptOpToken = (acceptEventArgs.UserToken as AcceptOpUserToken);

        //This method closes the socket and releases all resources, both
        //managed and unmanaged. It internally calls Dispose.
        acceptEventArgs.AcceptSocket.Close();

        //Put the SAEA back in the pool.
        poolOfAcceptEventArgs.Push(acceptEventArgs);
    }
}

class PrefixHandler
{
    public Int32 HandlePrefix(SocketAsyncEventArgs e,
           DataHoldingUserToken receiveSendToken,
           Int32 remainingBytesToProcess)
    {
        //receivedPrefixBytesDoneCount tells us how many prefix bytes were
        //processed during previous receive ops which contained data for
        //this message. Usually there will NOT have been any previous
        //receive ops here. So in that case,
        //receiveSendToken.receivedPrefixBytesDoneCount would equal 0.
        //Create a byte array to put the new prefix in, if we have not
        //already done it in a previous loop.
        if (receiveSendToken.receivedPrefixBytesDoneCount == 0)
        {
            receiveSendToken.byteArrayForPrefix = new
                             Byte[receiveSendToken.receivePrefixLength];
        }

        //If this next if-statement is true, then we have received at
        //least enough bytes to have the prefix. So we can determine the
        //length of the message that we are working on.
        if (remainingBytesToProcess >= receiveSendToken.receivePrefixLength
                                - receiveSendToken.receivedPrefixBytesDoneCount)
        {
            //Now copy that many bytes to byteArrayForPrefix.
            //We can use the variable receiveMessageOffset as our main
            //index to show which index to get data from in the TCP
            //buffer.
            Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset
                      - receiveSendToken.receivePrefixLength
                      + receiveSendToken.receivedPrefixBytesDoneCount,
                receiveSendToken.byteArrayForPrefix,
                receiveSendToken.receivedPrefixBytesDoneCount,
                receiveSendToken.receivePrefixLength
                      - receiveSendToken.receivedPrefixBytesDoneCount);

            remainingBytesToProcess = remainingBytesToProcess
                      - receiveSendToken.receivePrefixLength
                      + receiveSendToken.receivedPrefixBytesDoneCount;

            receiveSendToken.recPrefixBytesDoneThisOp =
                receiveSendToken.receivePrefixLength
                      - receiveSendToken.receivedPrefixBytesDoneCount;

            receiveSendToken.receivedPrefixBytesDoneCount =
                receiveSendToken.receivePrefixLength;

            receiveSendToken.lengthOfCurrentIncomingMessage =
                BitConverter.ToInt32(receiveSendToken.byteArrayForPrefix, 0);

            return remainingBytesToProcess;
        }

        //This next else-statement deals with the situation
        //where we have some bytes
        //of this prefix in this receive operation, but not all.
        else
        {
            //Write the bytes to the array where we are putting the
            //prefix data, to save for the next loop.
            Buffer.BlockCopy(e.Buffer, receiveSendToken.receiveMessageOffset
                        - receiveSendToken.receivePrefixLength
                        + receiveSendToken.receivedPrefixBytesDoneCount,
                    receiveSendToken.byteArrayForPrefix,
                    receiveSendToken.receivedPrefixBytesDoneCount,
                    remainingBytesToProcess);

            receiveSendToken.recPrefixBytesDoneThisOp = remainingBytesToProcess;
            receiveSendToken.receivedPrefixBytesDoneCount += remainingBytesToProcess;
            remainingBytesToProcess = 0;
        }

        // This section is needed when we have received
        // an amount of data exactly equal to the amount needed for the prefix,
        // but no more. And also needed with the situation where we have received
        // less than the amount of data needed for prefix.
        if (remainingBytesToProcess == 0)
        {
            receiveSendToken.receiveMessageOffset =
            	receiveSendToken.receiveMessageOffset - 
		receiveSendToken.recPrefixBytesDoneThisOp;
            receiveSendToken.recPrefixBytesDoneThisOp = 0;
        }
        return remainingBytesToProcess;
    }
}

class MessageHandler
{
    public bool HandleMessage(SocketAsyncEventArgs receiveSendEventArgs,
                DataHoldingUserToken receiveSendToken,
                Int32 remainingBytesToProcess)
    {
        bool incomingTcpMessageIsReady = false;

        //Create the array where we'll store the complete message,
        //if it has not been created on a previous receive op.
        if (receiveSendToken.receivedMessageBytesDoneCount == 0)
        {
            receiveSendToken.theDataHolder.dataMessageReceived =
                   new Byte[receiveSendToken.lengthOfCurrentIncomingMessage];
        }

        // Remember there is a receiveSendToken.receivedPrefixBytesDoneCount
        // variable, which allowed us to handle the prefix even when it
        // requires multiple receive ops. In the same way, we have a
        // receiveSendToken.receivedMessageBytesDoneCount variable, which
        // helps us handle message data, whether it requires one receive
        // operation or many.
        if (remainingBytesToProcess + receiveSendToken.receivedMessageBytesDoneCount
                   == receiveSendToken.lengthOfCurrentIncomingMessage)
        {
            // If we are inside this if-statement, then we got
            // the end of the message. In other words,
            // the total number of bytes we received for this message matched the
            // message length value that we got from the prefix.

            // Write/append the bytes received to the byte array in the
            // DataHolder object that we are using to store our data.
            Buffer.BlockCopy(receiveSendEventArgs.Buffer,
                receiveSendToken.receiveMessageOffset,
                receiveSendToken.theDataHolder.dataMessageReceived,
                receiveSendToken.receivedMessageBytesDoneCount,
                remainingBytesToProcess);

            incomingTcpMessageIsReady = true;
        }
        else
        {
            // If we are inside this else-statement, then that means that we
            // need another receive op. We still haven't got the whole message,
            // even though we have examined all the data that was received.
            // Not a problem. In SocketListener.ProcessReceive we will just call
            // StartReceive to do another receive op to receive more data.

            Buffer.BlockCopy(receiveSendEventArgs.Buffer,
                    receiveSendToken.receiveMessageOffset,
                    receiveSendToken.theDataHolder.dataMessageReceived,
                    receiveSendToken.receivedMessageBytesDoneCount,
                    remainingBytesToProcess);

            receiveSendToken.receiveMessageOffset =
                    receiveSendToken.receiveMessageOffset -
                    receiveSendToken.recPrefixBytesDoneThisOp;

            receiveSendToken.receivedMessageBytesDoneCount += remainingBytesToProcess;
        }
        return incomingTcpMessageIsReady;
    }
}

class BufferManager
{
    // This class creates a single large buffer which can be divided up
    // and assigned to SocketAsyncEventArgs objects for use with each
    // socket I/O operation.
    // This enables buffers to be easily reused and guards against
    // fragmenting heap memory.
    //
    //This buffer is a byte array which the Windows TCP buffer can copy its data to.

    // the total number of bytes controlled by the buffer pool
    Int32 totalBytesInBufferBlock;

    // Byte array maintained by the Buffer Manager.
    byte[] bufferBlock;
    Stack<int> freeIndexPool;
    Int32 currentIndex;
    Int32 bufferBytesAllocatedForEachSaea;

    public BufferManager(Int32 totalBytes, Int32 totalBufferBytesInEachSaeaObject)
    {
        totalBytesInBufferBlock = totalBytes;
        this.currentIndex = 0;
        this.bufferBytesAllocatedForEachSaea = totalBufferBytesInEachSaeaObject;
        this.freeIndexPool = new Stack<int>();
    }

    // Allocates buffer space used by the buffer pool
    internal void InitBuffer()
    {
        // Create one large buffer block.
        this.bufferBlock = new byte[totalBytesInBufferBlock];
    }

    // Divide that one large buffer block out to each SocketAsyncEventArg object.
    // Assign a buffer space from the buffer block to the
    // specified SocketAsyncEventArgs object.
    //
    // returns true if the buffer was successfully set, else false
    internal bool SetBuffer(SocketAsyncEventArgs args)
    {
        if (this.freeIndexPool.Count > 0)
        {
            //This if-statement is only true if you have called the FreeBuffer
            //method previously, which would put an offset for a buffer space
            //back into this stack.
            args.SetBuffer(this.bufferBlock, this.freeIndexPool.Pop(),
                       this.bufferBytesAllocatedForEachSaea);
        }
        else
        {
            //Inside this else-statement is the code that is used to set the
            //buffer for each SAEA object when the pool of SAEA objects is built
            //in the Init method.
            if ((totalBytesInBufferBlock - this.bufferBytesAllocatedForEachSaea) <
                       this.currentIndex)
            {
                return false;
            }
            args.SetBuffer(this.bufferBlock, this.currentIndex,
                             this.bufferBytesAllocatedForEachSaea);
            this.currentIndex += this.bufferBytesAllocatedForEachSaea;
        }
        return true;
    }

    // Removes the buffer from a SocketAsyncEventArg object. This frees the
    // buffer back to the buffer pool. Try NOT to use the FreeBuffer method,
    // unless you need to destroy the SAEA object, or maybe in the case
    // of some exception handling. Instead, on the server
    // keep the same buffer space assigned to one SAEA object for the duration of
    // this app's running.
    internal void FreeBuffer(SocketAsyncEventArgs args)
    {
        this.freeIndexPool.Push(args.Offset);
        args.SetBuffer(null, 0, 0);
    }
}

class DataHoldingUserToken
{
    internal Mediator theMediator;
    internal DataHolder theDataHolder;
    internal readonly Int32 bufferOffsetReceive;
    internal readonly Int32 permanentReceiveMessageOffset;
    internal readonly Int32 bufferOffsetSend;
    private Int32 idOfThisObject;

    internal Int32 lengthOfCurrentIncomingMessage;

    //receiveMessageOffset is used to mark the byte position where the message
    //begins in the receive buffer. This value can sometimes be out of
    //bounds for the data stream just received. But, if it is out of bounds, the
    //code will not access it.
    internal Int32 receiveMessageOffset;
    internal Byte[] byteArrayForPrefix;
    internal readonly Int32 receivePrefixLength;
    internal Int32 receivedPrefixBytesDoneCount = 0;
    internal Int32 receivedMessageBytesDoneCount = 0;
    //This variable will be needed to calculate the value of the
    //receiveMessageOffset variable in one situation. Notice that the
    //name is similar but the usage is different from the variable
    //receiveSendToken.receivePrefixBytesDone.
    internal Int32 recPrefixBytesDoneThisOp = 0;

    internal Int32 sendBytesRemainingCount;
    internal readonly Int32 sendPrefixLength;
    internal Byte[] dataToSend;
    internal Int32 bytesSentAlreadyCount;

    //The session ID correlates with all the data sent in a connected session.
    //It is different from the transmission ID in the DataHolder, which relates
    //to one TCP message. A connected session could have many messages, if you
    //set up your app to allow it.
    private Int32 sessionId;

    public DataHoldingUserToken(SocketAsyncEventArgs e, Int32 rOffset, Int32 sOffset,
           Int32 receivePrefixLength, Int32 sendPrefixLength, Int32 identifier)
    {
        this.idOfThisObject = identifier;

        //Create a Mediator that has a reference to the SAEA object.
        this.theMediator = new Mediator(e);
        this.bufferOffsetReceive = rOffset;
        this.bufferOffsetSend = sOffset;
        this.receivePrefixLength = receivePrefixLength;
        this.sendPrefixLength = sendPrefixLength;
        this.receiveMessageOffset = rOffset + receivePrefixLength;
        this.permanentReceiveMessageOffset = this.receiveMessageOffset;
    }

    //Let's use an ID for this object during testing, just so we can see what
    //is happening better if we want to.
    public Int32 TokenId
    {
        get
        {
            return this.idOfThisObject;
        }
    }

    internal void CreateNewDataHolder()
    {
        theDataHolder = new DataHolder();
    }

    //Used to create sessionId variable in DataHoldingUserToken.
    //Called in ProcessAccept().
    internal void CreateSessionId()
    {
        sessionId = Interlocked.Increment(ref Program.mainSessionId);
    }

    public Int32 SessionId
    {
        get
        {
            return this.sessionId;
        }
    }

    public void Reset()
    {
        this.receivedPrefixBytesDoneCount = 0;
        this.receivedMessageBytesDoneCount = 0;
        this.recPrefixBytesDoneThisOp = 0;
        this.receiveMessageOffset = this.permanentReceiveMessageOffset;
    }
}

class Mediator
{
    private IncomingDataPreparer theIncomingDataPreparer;
    private OutgoingDataPreparer theOutgoingDataPreparer;
    private DataHolder theDataHolder;
    private SocketAsyncEventArgs saeaObject;

    public Mediator(SocketAsyncEventArgs e)
    {
        this.saeaObject = e;
        this.theIncomingDataPreparer = new IncomingDataPreparer(saeaObject);
        this.theOutgoingDataPreparer = new OutgoingDataPreparer();
    }

    internal void HandleData(DataHolder incomingDataHolder)
    {
        theDataHolder = theIncomingDataPreparer.HandleReceivedData
                       (incomingDataHolder, this.saeaObject);
    }

    internal void PrepareOutgoingData()
    {
        theOutgoingDataPreparer.PrepareOutgoingData(saeaObject, theDataHolder);
    }

    internal SocketAsyncEventArgs GiveBack()
    {
        return saeaObject;
    }
}

class IncomingDataPreparer
{
    private DataHolder theDataHolder;
    private SocketAsyncEventArgs theSaeaObject;

    public IncomingDataPreparer(SocketAsyncEventArgs e)
    {
        this.theSaeaObject = e;
    }

    private Int32 ReceivedTransMissionIdGetter()
    {
        Int32 receivedTransMissionId =
              Interlocked.Increment(ref Program.mainTransMissionId);
        return receivedTransMissionId;
    }

    private EndPoint GetRemoteEndpoint()
    {
        return this.theSaeaObject.AcceptSocket.RemoteEndPoint;
    }

    internal DataHolder HandleReceivedData(DataHolder incomingDataHolder,
                        SocketAsyncEventArgs theSaeaObject)
    {
        DataHoldingUserToken receiveToken =
                       (DataHoldingUserToken)theSaeaObject.UserToken;
        theDataHolder = incomingDataHolder;
        theDataHolder.sessionId = receiveToken.SessionId;
        theDataHolder.receivedTransMissionId =
                      this.ReceivedTransMissionIdGetter();
        theDataHolder.remoteEndpoint = this.GetRemoteEndpoint();
        this.AddDataHolder();
        return theDataHolder;
    }

    private void AddDataHolder()
    {
        lock (Program.lockerForList)
        {
            Program.listOfDataHolders.Add(theDataHolder);
        }
    }
}

class OutgoingDataPreparer
{
    private DataHolder theDataHolder;

    internal void PrepareOutgoingData(SocketAsyncEventArgs e,
                       DataHolder handledDataHolder)
    {
        DataHoldingUserToken theUserToken = (DataHoldingUserToken)e.UserToken;
        theDataHolder = handledDataHolder;

        //In this example code, we will send back the receivedTransMissionId,
        // followed by the
        //message that the client sent to the server. And we must
        //prefix it with the length of the message. So we put 3
        //things into the array.
        // 1) prefix,
        // 2) receivedTransMissionId,
        // 3) the message that we received from the client, which
        // we stored in our DataHolder until we needed it.
        //That is our communication protocol. The client must know the protocol.

        //Convert the receivedTransMissionId to byte array.
        Byte[] idByteArray = BitConverter.GetBytes
                       (theDataHolder.receivedTransMissionId);

        //Determine the length of all the data that we will send back.
        Int32 lengthOfCurrentOutgoingMessage = idByteArray.Length
                       + theDataHolder.dataMessageReceived.Length;

        //So, now we convert the length integer into a byte array.
        //Aren't byte arrays wonderful? Maybe you'll dream about byte arrays tonight!
        Byte[] arrayOfBytesInPrefix = BitConverter.GetBytes
                       (lengthOfCurrentOutgoingMessage);

        //Create the byte array to send.
        theUserToken.dataToSend = new Byte[theUserToken.sendPrefixLength
                       + lengthOfCurrentOutgoingMessage];

        //Now copy the 3 things to the theUserToken.dataToSend.
        Buffer.BlockCopy(arrayOfBytesInPrefix, 0, theUserToken.dataToSend,
                       0, theUserToken.sendPrefixLength);
        Buffer.BlockCopy(idByteArray, 0, theUserToken.dataToSend,
                       theUserToken.sendPrefixLength, idByteArray.Length);
        //The message that the client sent is already in a byte array, in DataHolder.
        Buffer.BlockCopy(theDataHolder.dataMessageReceived, 0,
               theUserToken.dataToSend, theUserToken.sendPrefixLength
               + idByteArray.Length, theDataHolder.dataMessageReceived.Length);

        theUserToken.sendBytesRemainingCount =
                theUserToken.sendPrefixLength + lengthOfCurrentOutgoingMessage;
        theUserToken.bytesSentAlreadyCount = 0;
    }
}

class DataHolder
{
    //Remember, if a socket uses a byte array for its buffer, that byte array is
    //unmanaged in .NET and can cause memory fragmentation. So, first write to the
    //buffer block used by the SAEA object. Then, you can copy that data to another
    //byte array, if you need to keep it or work on it, and want to be able to put
    //the SAEA object back in the pool quickly, or continue with the data
    //transmission quickly.
    //DataHolder has this byte array to which you can copy the data.
    internal Byte[] dataMessageReceived;

    internal Int32 receivedTransMissionId;

    internal Int32 sessionId;

    //for testing. With a packet analyzer this can help you see specific connections.
    internal EndPoint remoteEndpoint;
}

internal sealed class SocketAsyncEventArgsPool
{
    //just for assigning an ID so we can watch our objects while testing.
    private Int32 nextTokenId = 0;

    // Pool of reusable SocketAsyncEventArgs objects.
    Stack<socketasynceventargs /> pool;

    // initializes the object pool to the specified size.
    // "capacity" = Maximum number of SocketAsyncEventArgs objects
    internal SocketAsyncEventArgsPool(Int32 capacity)
    {
        this.pool = new Stack<socketasynceventargs />(capacity);
    }

    // The number of SocketAsyncEventArgs instances in the pool.
    internal Int32 Count
    {
        get { return this.pool.Count; }
    }

    internal Int32 AssignTokenId()
    {
        Int32 tokenId = Interlocked.Increment(ref nextTokenId);
        return tokenId;
    }

    // Removes a SocketAsyncEventArgs instance from the pool.
    // returns SocketAsyncEventArgs removed from the pool.
    internal SocketAsyncEventArgs Pop()
    {
        lock (this.pool)
        {
            return this.pool.Pop();
        }
    }

    // Add a SocketAsyncEventArg instance to the pool.
    // "item" = SocketAsyncEventArgs instance to add to the pool.
    internal void Push(SocketAsyncEventArgs item)
    {
        if (item == null)
        {
            throw new ArgumentNullException("Items added to a
            		SocketAsyncEventArgsPool cannot be null");
        }
        lock (this.pool)
        {
            this.pool.Push(item);
        }
    }
}

class SocketListenerSettings
{
    // the maximum number of connections the sample is designed to handle simultaneously
    private Int32 maxConnections;

    // this variable allows us to create some extra SAEA objects for the pool,
    // if we wish.
    private Int32 numberOfSaeaForRecSend;

    // max # of pending connections the listener can hold in queue
    private Int32 backlog;

    // tells us how many objects to put in pool for accept operations
    private Int32 maxSimultaneousAcceptOps;

    // buffer size to use for each socket receive operation
    private Int32 receiveBufferSize;

    // length of message prefix for receive ops
    private Int32 receivePrefixLength;

    // length of message prefix for send ops
    private Int32 sendPrefixLength;

    // See comments in buffer manager.
    private Int32 opsToPreAllocate;

    // Endpoint for the listener.
    private IPEndPoint localEndPoint;

    public SocketListenerSettings(Int32 maxConnections,
    Int32 excessSaeaObjectsInPool, Int32 backlog, Int32 maxSimultaneousAcceptOps,
    Int32 receivePrefixLength, Int32 receiveBufferSize, Int32 sendPrefixLength,
    Int32 opsToPreAlloc, IPEndPoint theLocalEndPoint)
    {
        this.maxConnections = maxConnections;
        this.numberOfSaeaForRecSend = maxConnections + excessSaeaObjectsInPool;
        this.backlog = backlog;
        this.maxSimultaneousAcceptOps = maxSimultaneousAcceptOps;
        this.receivePrefixLength = receivePrefixLength;
        this.receiveBufferSize = receiveBufferSize;
        this.sendPrefixLength = sendPrefixLength;
        this.opsToPreAllocate = opsToPreAlloc;
        this.localEndPoint = theLocalEndPoint;
    }

    public Int32 MaxConnections
    {
        get
        {
            return this.maxConnections;
        }
    }
    public Int32 NumberOfSaeaForRecSend
    {
        get
        {
            return this.numberOfSaeaForRecSend;
        }
    }
    public Int32 Backlog
    {
        get
        {
            return this.backlog;
        }
    }
    public Int32 MaxAcceptOps
    {
        get
        {
            return this.maxSimultaneousAcceptOps;
        }
    }
    public Int32 ReceivePrefixLength
    {
        get
        {
            return this.receivePrefixLength;
        }
    }
    public Int32 BufferSize
    {
        get
        {
            return this.receiveBufferSize;
        }
    }
    public Int32 SendPrefixLength
    {
        get
        {
            return this.sendPrefixLength;
        }
    }
    public Int32 OpsToPreAllocate
    {
        get
        {
            return this.opsToPreAllocate;
        }
    }
    public IPEndPoint LocalEndPoint
    {
        get
        {
            return this.localEndPoint;
        }
    }
}

The Server App

After downloading the zip file that contains the code, save it to disk. In order not to have problems using it in Visual Studio, before extracting it, right-click on the saved zip file and choose Properties, and then Unblock, then OK. Then extract it. If you do not do that, you may get security warning errors from Visual Studio.

Before running the server code the first time, you may want to change the folder where the logs are written. The default path isc:\LogForSaeaTest\, which will be created at server startup, if it does not exist. If you do not want to use the default folder, change the path in theTestFileWriterclass before running the app the first time. (TheTestFileWritercode is not included in the article above, but is in the source code.) For the most part, I havenotset the server application up so that theSocketListenerSettingsand other variables can be controlled from the Console. You'll need to change the source code and rebuild to make most changes during testing.

It's much better to run the client on one machine and server on another. If you try to run the client and server on the same machine, try to use the computer name first as the value of the "host" variable on the client. If that does not work, try "localhost" as the value of the "host" variable on the client.

When trying to connect the client to the server, if you get a "connection actively refused" message, check to see if you have a firewall that is blocking the transmissions on the incoming port of the server. You might have to allow incoming transmissions on that port on your local network. And if you have a firewall that blocks outgoing transmission on the client, then you would need to change settings for that too.

The Client App

A lot of the code in the client app is very similar to the server app. The server code is fully commented. So I did not always fully comment the client code. If in the client you find code that you do not understand and it is not commented on, then check similar portions of the server app for code comments.

The client app isnota normal client, but an app designed to test the server. It is set up to deliver as many connections as you want to throw at the server. The client app is set up to build all the messages for all of those connections before the test is started. It sends a different message each time for each client. And all the messages are put in memory before the test starts, so message creation won't be a drag on the client app during the test. If you choose to have 3000 connections sending 50000 messages per connection, then that is 150 million messages. That is too many for memory probably. If you want to do a long test like that, then in the client app, changerunLongTesttotrue. In that case, instead of sending a separate message for each message from each client, it will send the same array of messages over and over for each client. That way, the messages can fit in memory. (If you are doing a long test like that, also setrunLongTesttotrueon the server. That will keep the server app from writing the received data to a dictionary, which displays data at the end of the test. Otherwise, you'll run the server out of memory probably.)

The client app is set up so that you can do the following from the Console:

1. Put in the method for finding the network address of the host machine, either machine name or IP address,
2. Put in the correct string for the host machine name or IP address, depending on what method you chose for getting the network address of the host,
3. Type in the port number of the host app, or accept the default,
4. Specify a folder name for the log file to be written to, or accept the default,
5. Specify the buffer size, or accept the previous value,
6. Specify the number of client connections to attempt, or accept the previous value,
7. Indicate the number of TCP messages to send per connection, or accept the previous value.

In the downloadable source code, there is plenty of capability to visualize what is happening by writing to a log and the Console. The things that you can visualize are:

1. Program flow, as it moves from method to method,
2. Connects and disconnects,
3. The data which was sent from client to server and server to client,
4. Threads, only in the server app (Save the thread watching for last)

Simple Process to Understand the Code Well

First, if you have software firewalls on client and/or server, open those firewalls to allow the apps to transmit to port 4444 on the server, or whatever port you use. 4444 is the default port.

Start the server app, and make sure you see "Server is listening" in the console. Then start the client app. You'll be asked to specify the host in the console. Use the machine name, unless the server has a fixed IP address, in which case you can just use the IP address. For the other items it will ask you, hopefully you can just accept the defaults. When it displays "Press Enter to begin socket test", press Enter. It should finish quickly. Then close both client and server, to make the logs finish writing. You just sent one message from one client connection to the server, and a response message from the server back to that client connection. (If you had a problem, see the stuff about firewalls above.)

Now look at the log files from both the server and client. (It's easier if you print them.) Compare them to the code, and think about it a long time. You are looking at one message sent from one connection, and the response to that message. You should be able to see and understand the program flow very well from it.

(Tools that you may find helpful. When testing any network application, you will learn much more if you use the great free programWireshark, or something like it, to see everything that is happening on the network. For reading very large text files which can be generated by the logger when running long tests, tryCream for Vim.)

Now go through that same process for each of the tests below. Start the server, and then the client. The client will ask you about buffer size, number of connections, and number of messages. You'll make some selections for those things on the client. When you change the number of connections, you are changing the number of simulated users. One connection is like one client user running on one machine on the Internet, or a LAN. Run the test. And then after each test, close the client and server apps to write the logs.Look at the logs and make sure you understand what is happening after each test. Then go to the next test in the list.

• Test 1.buffer sizestays same (25),number of connections= 2,number of messages= 2.
• Test 2.buffer sizestays same (25),number of connections= 3,number of messages= 5.
• Test 3.buffer size= 5 on client,number of connections= 3,number of messages= 5. (In this test, the buffer is smaller than the message with its prefix. So the client will require multiple send ops to send one message. And the client will require multiple receive ops to receive one message. You can see that in the client log. What happens in the server logs can vary, since one send op from client doesnotnecessarily correlate with one receive op on the server, due to the way that TCP works.)
• Test 4.buffer size= 3 on client,number of connections= 3,number of messages= 5. (In this test, the buffer is smaller than even the prefix by itself. Multiple send ops will be required.)
• Test 5. Putbuffer sizeback to 25 on client,number of connections= 3,number of messages= 5. Now on the server, in theProgram.cscode, changetestBufferSizefrom 25 to 5. Build the server app. Run the test. You'll see that multiple receive ops are required to receive a message from the client. And multiple send ops are required to send from the server back to the client.
• Test 6. On the server, puttestBufferSizeback to 25, changewatchThreadsfromfalsetotrue, and build. Leave settings the same as they were on the client, and run the test again. Now, after you close the server app, the server log file will show info on threads. In Notepad, or whatever program you use to look at the logs, search for the phrase "New managed thread". You'll see that phrase every time a new thread starts. Usually, there are only 2-4 managed threads running for the app. The managed thread numbers and socket handle identifiers are displayed throughout the log file now.
• Test 7. On the server, changewatchThreadstofalse,watchProgramFlowtofalse,maxNumberOfConnectionsto 10,000, and build. On the client, changewatchProgramFlowtofalse, and build. In the client console, at startup, makenumber of connections= 1000,number of messages= 500. Run the test. (If this crashes your client app, changerunLongTesttotrueon the client, and build it again. If you changerunLongTesttotrueon the client, there will not be a huge array of messages created on the client before starting the test. Instead, just one array of messages will be created, and it will be sent over and over, the same array being used for all of the connections. WhenrunLongTestisfalseon the client, every connection has its own unique array of messages.)
• Test 8. To run bigger tests, I suggest you changerunLongTesttotrueon the server also. Try settingnumber of connections= 8000,number of messages= 500000 on the client, and see if it crashes your server app. On the client, there is atickDelayBeforeNextConnvariable which is set to delay the next connection by 50000 ticks (5 milliseconds). You can play around with that some. If you send too many new connection requests at once, you'll overwhelm the server. It's kind of fun to do once or twice. How many connections can the server handle? It depends on your hardware, configuration and version of Windows. In some testing I did, when running the server on an older single processor Dell desktop with Windows XP Pro 32 bit on a wired local area network with 100 MB NICs, it could handle 2000 connections sending/receiving messages continually with no problems. It usually does well with 3000 connections. And 4000 connections may cause problems sometimes. Keep it mind that it is possible to overwhelm a machine that is running the client app, if you open thousands of connections, because you are opening a port for every connection. If the server is under heavy load, it might reject some new connections. But the client app is able to retry a rejected connection until the connection is successful.

History

• December 13, 2010: Version 1.3 of code. Changed client to use a blockingStack<T>to handle the initiation of new connections in a more controlled way. Previously, when the server was overloaded and rejecting a lot of connections, the connection retries were not handled very well by the client app. AddedShutdown()beforeDisconnectAsync()in client. Changed client so that it is closing a socket properly after finishing with it. In server app in v1.2, the code to handle bad accept socket was inCloseClientSocket. Now it is handled inHandleBadAccept. That change eliminates the checking fornullafter the cast of the user token in theCloseClientSocketmethod that was in v1.2.
• October 12, 2010: Version 1.2 of code. Fixed bug inProcessAcceptwhere, in the case of socket error, we needed toLoopToStartAcceptand get rid of the bad socket. Fixed bug inCloseClientSocketwhen we tried to destroy the socket in an SAEA for accept ops, because the SAEA's user token could not be cast toDataHoldingUserTokentype. Fixed bug in client code -- when therunlongtestvariable istruethe client was still trying to save received data in a List<T>, which would eventually cause it to run out of memory and crash in a test with a huge number of messages. Improved some log writing code. Cleaned up the client test app code, which I had just sort of thrown together without worrying about readability. Moved a couple of lines fromSocketListener.ProcessReceivetoPrefixHandler. Improved error handling in the client app. In the server, some code was moved from theMainmethod into other methods, just for readability.
• September 8, 2010: Version 1.1 of code. TheProcessReceivemethod was too big previously. I moved part of it into thePrefixHandlerandMessageHandlerclasses. Moved receive buffer setting fromProcessSend()toStartReceive(). Fixed two bugs. One was when the number of bytes received exactly equalled the prefix length. The other was noticed by user mixal11. I had failed to reset some of the Token members in the case of abnormal disconnect. Thanks to mixal11. Made some minor changes to the article.
• August 14, 2010: Fixed a few typographical errors in the article. Changed some formatting in the article, and moved one paragraph for clarity. Reworded a few things.

源码1

源码2