A348 Transport Control Protocol
powered by FreeFind
Modified: 

Overview

Normally, when one thinks of Web clients, one thinks of a browser. But suppose that your program needed information stored on a Web server, say the price of shipping a package by UPS to some zip code. To implement a server application, you could visit the UPS Web site with a browser client and cut and paste all the pricing information to all zip codes into your program and look it up from your own database of UPS prices. Not a very viable solution if UPS changes the rates often. A better solution is to have the server program connect to the UPS Web site and get the information directly on its own. However, this approach requires that the program behave like a Web client, able to connect to a server, request and download data. The following notes explore implementing our own specialized clients and servers for gathering data from other servers.

TCP

The basis for nearly all Web clients and servers is the reliable communications protocol of the Internet, TCP. TCP is a protocol or set of rules that define how programs can communicate over a network. Being a reliable, connection-oriented protocol allows programs to assume that what is sent will be received in the same order sent, the connection behaves like a pipe, what goes in comes out unchanged. TCP supports the client/server model where the client program initiates requests and the server responds to requests.

To better understand how TCP operates, consider when a Web browser connects to a Web server and requests certain files to be sent back. The steps are:
 

Client

0. Initialize TCP communication socket
1. Request connection to server on port 80 
3. Request a file.
5. Receive file.
7. Close connection

 Server

0. Initialize TCP communication socket on port 80
2. Accept connection request on port 80
4. Send file.
6. Close connection.

Perl - A Perl implementation of both the client and server is below, though a browser client can use the Perl server, or the Perl client can use a Web server. The client attempts to connect on port 80, the same port used by Web servers. Note that the server must be started before the client. If a server on the localhost accepts the connection request, the Perl client sends  "GET / HTTP/1.0\n\n" to the server, requesting the default Web page to be returned and printed by the client. The Perl server listens on port 80, the same port used by Web servers. When a browser client connects to the Perl server and sends anything, the Perl server will send "<h1>Hello World</h1> back to the client. Obviously the Perl client and server leave much to be desired but do follow a small part of the full HTTPvprotocol by which the Web client and server communicate.

Exercise 0 - Client/Server Test

Mimic the actions of a browser using TELNET.

Web servers and clients adhere to the HTTP protocol for communication. Web servers generally listen for HTTP connections on the well-known port 80 and clients connect to port 80. A computer named localhost with SMTP (Simple Mail Transport Protocol) will listen on port 25 for SMTP connections, and TELNET on port 20. A client connecting to the SMTP server would connect to localhost and port 25.

A Web server running on localhost waits for a client to connect on port 80. A client connects to localhost and port 80 to communicate using HTTP. Browsers by default use port 80 but can use other connection ports (e.g. http://www.ius.edu:6666 would use port 6666 instead of 80, although no server may be listening to port 6666).  Other clients applications can also connect to port 80. For example TELNET normally uses port 20 but can connect to port 80. Since TELNET sends whatever you type to the server and displays whatever the server send back, it can be used as a crude browser.

A browser sends HTTP to the server and the server sends back HTTP and some content such as HTML. The browser then renders the HTML so the browser user sees other than raw HTML. Using TELNET as a browser you need to type the HTTP commands to send to the server and will get back the server response. To test, enter the following two lines at the command prompt. Line 1 runs TELNET to connect to the localhost port 80, the port on which the Web server listens. Line 2 (which doesn't display in the Windows 2000 telnet) is the HTTP sent to the server requesting the default Web page to be returned. Line 3 is the blank line at the end of a HTTP command. The remainder is the server response.
 
Using TELNET as a Web Client
  1. telnet localhost 80         
  2. GET  /  HTTP/1.0
Note that Windows 2000 telnet does not do a local echo so you can't see what you type. Turn on local echo in telnet by:
  • Ctrl ] keys
  • set local_echo
  • Enter key

HTTP/1.1 302 Object moved
Server: Microsoft-IIS/5.0
Date: Mon, 07 May 2001 18:01:26 GMT
Location: localstart.asp
Connection: Keep-Alive
Content-Length: 135
Content-Type: text/html
Set-Cookie: ASPSESSIONIDGGQGGKNK=HGMLHBHDNOMPHODCJIAAJBID; path=/
Cache-control: private

<head><title>Object moved</title></head>
<body>
<h1>Object Moved</h1>
This object may be found <a HREF="localstart.asp">here</a>.
</body>

Connection to host lost.
 

  1. Repeat the above example by typing in lines 1-3. Line 3 is a required blank line.
  2. Use a different Web server addresses such as www.ius.edu, some of the default Web pages returned may be quite large.
  3. Use a browser to locate a Web page on a server off campus. Then use TELNET to retrieve the page. You'll need the server address and the HTTP GET information for the Web page file.
  4. Run the Client.pl program below, it should connect to the localhost server and display the HTML for the default page since it sends a GET / HTTP/1.0 to the server.
  5. To display another file, say /Vnursery/VNursery.Asp from the server, change the line:
      6. print F "GET / HTTP/1.0\n\n";            to         print F "GET /Vnursery/VNursery.Asp HTTP/1.0\n\n";
  6. Change the host address from localhost to another site with a Web server in the following Client.pl line:
      7. $sin = sockaddr_in(80, inet_aton("localhost"));
  7. Copy and paste the Server.pl program.
  8. Change the port 80 to port 800. Run the Server.pl program and from a browser connect to location localhost:800. The Server.pl program will display whatever the browser sends to it and will send <h1>Hello World</h1> back to the browser.
Further information - See the ActivePerl documentation under IO/Socket.
 
Perl Simple Client and Server
Client.pl - Use: perl Client.pl

use Socket;
  $proto = getprotobyname('tcp');
  socket(F, PF_INET, SOCK_STREAM, $proto); # TCP socket

 $sin = sockaddr_in(80, inet_aton("localhost")); 
                                                 # localhost port 80 

 connect(F, $sin);                        # Wait for connection 
  select(F);    $| = 1;                    # Don't buffer I/O
  select(STDOUT);                        # Set default print

  print F "GET / HTTP/1.0\n\n";        # Request Web page
 while(<F>) { print;  }                   # While connection open
                                                 # Print what server sends


Output

<h1>Hello World</h1> 

 

 Server.pl - Use: perl Server.pl

use Socket;
  $proto = getprotobyname('tcp');
  socket(F, PF_INET, SOCK_STREAM, $proto); # TCP socket
  $sin = sockaddr_in(80, INADDR_ANY);           # Use port 80
  bind(F, $sin);                                   # Bind F to port 80
  listen(F,1);                                       # Listen for 1 connection 

   accept(FH, F);                                 # Wait for connection
   select(FH);   $|=1;                           # Don't buffer I/O
   select(STDOUT);                              # Set default print 

   do { $buffer=<FH>;                           # Read one client line
         print $buffer;                              # Print to STDOUT
   } until $buffer eq "\n";                        # Empty client line?

   print FH "<h1>Hello World</h1>\n";       # Send to client

 close(FH);                                          # Close connection


Output

GET / HTTP/1.0
A More Realistic Server  - The previous server didn't do much, ignoring any file requests from the client. The following server will return requested files back to the client. To test:
  1. Copy and paste.
  2. Run the HttpServer.pl program. Note that it uses port 80 which conflicts with any Web server.
  3. Suppose that at D:\A348\HTML\Notes.htm there is an HTML file. In the browser, enter:
      4. localhost/A348/HTML/Notes.htm
HttpServer.pl
use Socket; 
use IO::File; 

$proto = getprotobyname('tcp');
socket(F, PF_INET, SOCK_STREAM, $proto);        # TCP socket
$sin = sockaddr_in(80, INADDR_ANY);             # Use port 80
bind(F, $sin);                                  # Bind F to port 80
listen(F,1);                                    # Listen for 1 connection 

accept(FH, F);                                  # Wait for connection
select(FH);   $|=1;                             # Don't buffer I/O
select(STDOUT);                                 # Set default print 

$line = <FH>;                                   # Read first client line
($get, $line) = split(/GET /, $line);           # Locate GET
($fname, $line) = split(/ /, $line);            # File name

$theFile = new IO::File $fname, "r";            # Open, assume on same drive as server

if (defined $theFile) { 
        print FH <$theFile>;                    # Print file to client
        undef $theFile;        
} 
else {
        print FH "<H1>$fname not found!</H1>";
}

close(FH); 

Exercise 1 - Extending the Server

  1. Why does the broswer hang with the HttpServer on a refresh?
  2. Modify the server so that it continues to serve files.
  3. Can more that one client hit the server at the same time? What general modifications would be needed?
Visual Basic - A VB implementation of the client and server is below, conceptually similar to the Perl above. Visual Basic implements connection services (Transport Control Protocol) using a tool named Winsock, Windows Sockets, which has the same capabilities as Perl since both use the TCP protocol. One key distinction is that VB uses graphical tools, the tool for Winsock appears as . Since it is not normally on the toolbar, it must be placed there by:
  1. Click on Project menu and select Components.
  2. Select the Microsoft Winsock Component.
  3. Drop the Winsock component on the form.
  4. Change the Winsock component name from Winsock1 to httpClient.
Visual Basic Client and Server
Client

Object Design view

Private Sub Form_Load()
 ' The name of the Winsock control is httpClient. To specify a remote
 ' host, use either the IP address (ex: "121.111.1.1") or name.
    httpClient.RemoteHost = "localhost"
    httpClient.RemotePort = 80
    If (httpClient.State <> sckClosed) Then
        httpClient.Close
    End If

    httpClient.Connect          ' Connect to localhost:80
End Sub

Private Sub cmdSend_Click()     ' Connect and send data to server
    httpClient.SendData "GET / HTTP/1.0" + 
           Chr$(10) + Chr$(13) + Chr$(10) + Chr$(13)
End Sub

Private Sub httpClient_DataArrival(ByVal bytesTotal As Long)
' Display whatever received
    Dim strData As String
    httpClient.GetData strData
    txtOutput.Print strData
End Sub

Private Sub httpClient_Close()  ' When server closes connection
    If (httpClient.State <> sckClosed) Then
        httpClient.Close
    End If
    txtOutput.Print "Closed Connection"
End Sub


Output - What the server sent.
Server

Object Design view

Private Sub Form_Load()
    ' Winsock control is httpServer. Listen for connection on port 80
    httpServer.LocalPort = 80
    httpServer.Listen
End Sub

Private Sub httpServer_Close()
    httpServer.Close
End Sub

Private Sub httpServer_ConnectionRequest(ByVal requestID As Long)
    ' Check if the control's State is closed. If not,
    ' close the connection before accepting the new connection.
    If httpServer.State <> sckClosed Then
        httpServer.Close
    End If

    httpServer.Accept requestID        ' Accept connection localhost:80
End Sub

Private Sub httpServer_DataArrival(ByVal bytesTotal As Long)
    ' http received data
    Dim strData As String

    httpServer.GetData strData
    txtOutput.Print strData

    httpServer.SendData "<h1>Hello World</h1>" + Chr$(10) + Chr$(13)
End Sub

Private Sub httpServer_SendComplete()
    httpServer.Close
End Sub


Output - What the client sent.

Internet Transfer Control - An even simpler way to communicate, using HTTP, FTP, or presumably other protocols, is the Microsoft Internet Transfer Control. It can be added to the VB toolbar in much the same way as the Winsock tool. Its main advantage is that it handles everything for a client connection that sends one request to a server and returns the results, you need only supply the URL to the server. To duplicate the above, using this control is very simple.
 
Visual Basic Client using Internet Transfer Control
Object Design View


Private Sub Form_Load()
Text1.Text = Inet1.OpenURL("http://localhost")
End Sub		 Text returned from server


What was sent to server

GET / HTTP/1.1
Accept: image/gif,image/x-xbitmap,image/jpeg,image/pjpeg,*/*
User-Agent: Microsoft URL Control - 6.00.8169
Host: localhost
Connection: Keep-Alive
Cookie: ASPSESSIONIDFFFEDJBB=BJHJJNDAJCAADCIEOLIBEIHG
Further information- See the Visual Interdev help on Internet, Common Gateway Interface (CGI).

Email

One common client function is to send email. The method given here is intended to explain how an email client interacts with a SMTP (Simple Mail Transport Protocol) server at its most basic level. Higher level packages for sending email are widely available for Perl/VB/etc. and are recommended for real applications.

The following illustrates the use of TELNET as a client email program and the client/server interaction. The client first connects to the email server mail-server.ucs.indiana.edu and begins sending requests and the server responds. The email server does not know or care where the requests originate, only that a connection is made on port 25 and the requests follow the SMTP standards. The server responds to each client request with a reponse number that indicates whether the request was fulfilled or not. The client/server interaction illustrates what the client sends (underlined) to the server and receives from the server.
 
TELNET as an Email Client
telnet mail-server.ucs.indiana.edu 25

220 fins.uits.indiana.edu ESMTP Sendmail 8.9.3/8.9.3/1.1IUPO; 
       Thu, 13 Apr 2000 13:30:23 -0500 (EST)
HELO ius.edu
250 mask.uits.indiana.edu Hello [149.160.21.92], pleased to meet you
MAIL FROM: <dubay@whitehouse.gov>
250 <dubay@whitehouse.gov>... Sender ok
RCPT TO: <rwisman@ius.edu>
250 <rwisman@ius.edu>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself

How is the class going?
.

250 KAA16194 Message accepted for delivery
QUIT
221 mask.uits.indiana.edu closing connection

Mail Sent!


Mail received

Subject: 
   Date:  Thu, 13 Apr 2000 10:36:39 -0500 (EST)
   From: dubay@whitehouse.gov
   To:     undisclosed-recipients:;

How is the class going?

The following shows the client email program at left and the client/server interaction at right. The client first connects to the email server mail-server.ucs.indiana.edu and begins sending requests and the server responds. The email server does not know or care where the requests originate, only that the requests follow the SMTP standards. The server responds to each client request with a reponse number that indicates whether the request was fulfilled or not. The client/sever interaction at right illustrates what the client sends (underlined) to the server and receives from the server.

You might notice that the actual email client implementation is very similar to the above Web client implementation. A connection to the server,  (on port 25, the standard SMTP email port) is first established, then the client and server communicate following some predefined protocol, in this case SMTP.

Further information - See the ActivePerl documentation under IO/Socket.
 
Email client and client/server interaction
Email client - Use: perl email.pl

use Socket;

  $proto = getprotobyname('tcp');
  socket(F, PF_INET, SOCK_STREAM, $proto);
  $sin = sockaddr_in(25, inet_aton("mail-server.ucs.indiana.edu"));
  connect(F, $sin);
  select(F);
  $| = 1;
  select(STDOUT);

  $_ = <F>;    print;                     # Read and print server response
  if (/^220/) {                               # Did server repond with 220?
    print F "HELO ius.edu\n";        # Send HELO to server
    $_ = <F>; print;
    if (/^250/) {
       print F "MAIL FROM: <jholly\@ius.edu>\n";
       $_ = <F>; print;
       if (/^250/) {
          print F "RCPT TO: <rwisman\@ius.edu>\n";
          $_ = <F>; print;
          if (/^250/) {
             print F "DATA\n";
             $_ = <F>; print;
             if (/^354/) {
                do { $to = <STDIN>;
                        print F $to;
                } until $to eq ".\n";
                $_ = <F>; print;
                if (/^250/) {
                   print F "QUIT\n";
                   $_ = <F>; print;
                   if (/^221/) { print "Mail Sent!\n";    }
  } } } }}}
Email client/server interaction (client sending to server is italics)

perl email.pl

220 fins.uits.indiana.edu ESMTP Sendmail 8.9.3/8.9.3/1.1IUPO; 
       Thu, 13 Apr 2000 13:30:23 -0500 (EST)
HELO ius.edu
250 mask.uits.indiana.edu Hello [149.160.21.92], pleased to meet you
MAIL FROM: <jholly@ius.edu>
250 <jholly@ius.edu>... Sender ok
RCPT TO: <rwisman@ius.edu>
250 <rwisman@ius.edu>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself

How is the class going?
.

250 KAA16194 Message accepted for delivery
QUIT
221 mask.uits.indiana.edu closing connection

Mail Sent!


Mail received

Subject: 
   Date:  Thu, 13 Apr 2000 10:36:39 -0500 (EST)
   From: jholly@ius.edu
   To:     undisclosed-recipients:;

How is the class going?

ASP Email - Email can easily be sent from a NT server running SMTP or Microsoft Exchange email server (not generally available under Window 95/98) using Collaboration Data Objects. It provides an easy means of integrating sending email into other applications. The same message as above can be sent out using:
 

Email.asp in VBScript
<% 
 Dim objNewMail

 Set objNewMail = Server.CreateObject("CDONTS.NewMail")

 objNewMail.To = "rwisman@ius.edu"
 objNewMail.From = "jholly@ius.edu"

 objNewMail.Body = "How is the class going?"

 objNewMail.Send() 
%>

Exercise 2 - Email

  1. Following the TELNET example above, send yourself and the instructor at rwisman@ius.edu some email from a famous person.

Gathering Data from a Server

Suppose that an application needed data available on another server, perhaps it needed to compute shipping costs based on UPS rates. The data might be available but buried in an HTML document normally viewed from a browser or if we're incrediably lucky, returned in some standard data representation form (maybe XML). Basically we want our server to act licke a client and get data from another server. There are several approaches, if we know that in the future we will need some data (daily stock quotes, shipping rates, etc.) we can write a client program in any language that connects to the server, extracts the necessary data, and possibly builds a database for easy retrieval later. If we need instanteous data (stock data right now!) our server running an ASP or CGI would itself act as a client to another server, retrieving and processing the data. This makes things a little more complicated because not all scripting languages (VBScript and JScript) directly support Internet socket programming (the problem is that you can't drop the Winsock control on a VBScript, one must write a DLL that encapsulates the Winsock object). There are other ActiveX Internet controls that can be used but Microsoft has issued security warnings about using them in ASP scripts.

To get an idea of what needs to be done we'll write our own simple client, starting off in PerlScript since it is easy and nearly the same as we've already examined in Perl. When the client runs on the server, a connection is made, the client sends GET / HTTP/1.0 to the localhost, the text returned is then scanned for the word Date: and when found, the remainder of the line is written to the browser client.

Further information - See the ActivePerl documentation under IO/Socket.
 

PerlScript Simple ASP Http Client to Return Localhost Server Date/Time
PerlScript ASP

<%@ LANGUAGE = PerlScript %>
<%
use Socket; 
        $proto = getprotobyname('tcp'); 
        socket(F, PF_INET, 
           SOCK_STREAM, $proto);                                   # TCP socket 

        $sin = sockaddr_in(80, inet_aton("localhost"));        # localhost server port 80 

        connect(F, $sin);                                               # Wait for connection 
        select(F);    $| = 1;                                           # Don't buffer I/O 

        print F "GET / HTTP/1.0\n\n";                                # Request Web page 
        while(<F>) {                                                      # Process what is returned
                     ($before, $date) = split(/Date:/, $_);        # Locate "Date" line
                     if ($date ne "") {
                        $Response->write("<H1>$date</H1>");   # Print date returned 
          } 
 }
%>
Some of what server returns to PerlScript

HTTP/1.1 302 Object moved 
Server: Microsoft-IIS/4.0 
Date: Thu, 20 Apr 2000 13:44:52 GMT 
Location: /IISSamples/Default/welcome.htm 
Content-Type: text/html 
Set-Cookie: ASPSESSIONIDFFFEOOFF=KNDBILHDDEEGLLGGLICEGKGC; path=/ 
Cache-control: private 
What PerlScript ASP writes to browser client
<H1>Thu, 20 Apr 2000 13:44:52 GMT</H1>

Exercise 3 - Simple Data Gathering

  1. Copy and paste the above PerScript into a virtual drive as Date.Asp then execute from a browser.
  2. Modify the PerlScript to return the server version.
HTTP and URL's - The low-level TCP protocol functions provide complete control but require some effort to use. A simpler method is to use higher level protocol functions for HTTP, FTP, etc. that handle the low-level TCP. The following uses the class Win32::Internet that includes the FetchURL method to connect to a URL and return any results to a string. The following example uses the Yahoo! financial server to retrieve stock information on Microsoft symbol MSFT. Yahoo! returns a large amount of HTML that must be scanned for the character strings >MSFT< that indicate that the stock quote follows. One fundamental weakness of this method is its dependency on Yahoo! not to change the format of the HTML file returned. Another serious concern is that our server is occupied waiting for the Yahoo! server to respond, creating potential for poor response time and poor utilization of server resources.

Further information -  See the ActivePerl documentation under Win32/Internet.
 

PerlScript ASP to connect to Yahoo! and retrieve Stock Quote for Microsoft
PerlScript ASP

<%@ LANGUAGE = PerlScript %>
<%
use Win32::Internet; 
  $INET = new Win32::Internet();
  $Response->write("<H1>Connecting to yahoo. </H2><br><H2>Please wait.</H2><br>");
  $file = $INET->FetchURL("http://finance.yahoo.com/q?s=msft&d=v1&o=t"); 

  if ($file eq "") {             // Empty file indicates connection timed out
     $Response->write("<H2>Connection failed. Try again later.</H2><br>"); 
     return;
  } 
  ($before, $after) = split(/>MSFT</, $file);                              # Locate ">MFST<"
  if ($after ne "") { 
     ($before, $after) = split(/<b>/, $after);                              # Locate <b>
     ($stockquote, $after) = split(/<\/b>/, $after);                     # Locate </b>
     if ($stockquote ne "") {                                                   # <b>$stockquote</b>
         $Response->write("<br><H2>Microsoft MSFT is $stockquote</H2><br>");
     } 
  }
  $INET->Close();
%>
Some of what Yahoo! returned with stock quote

Symbol      Last Trade              Change             Volume   More Info 
<a href="/q?s=MSFT&d=t">MSFT</a>    10:04AM     <b>78 11/16</b>          0      0.00%  5,920,300 
What PerlScript ASP writes to browser client

<br><H2>Microsoft MSFT is 78 11/16</H2><br>

Exercise 4 - Gathering Data from HTML Documents

  1. From a browser enter the URL http://finance.yahoo.com/q?s=msft&d=v1&o=t and examine the source returned.
  2. Copy and paste the PerlScript ASP above and execute from a browser.
  3. Modify the PerlScript to display the price of another stock such as IBM or GM.
  4. Modify the PerlScript to display useful information from another server.
POSTing - Many servers support data retrieval services where one server communicates with another to retrieve some data. The request may be sent as a URL with data added in the manner of the GET method or as a stream in the manner of the POST method. The URL or GET method is simple to implement since the data is simply appended to the server URL, the POST method allows more data to be sent in the request but is a more complex request.

The example below uses the POST method for the request since that is what the server expects. In HTML a POST request is easily made by use in a form of the method=POST, the browser handles formatting the POST for the request. Since we're not using a browser, we're responsible for formatting the POST request, as given in the example below. The POST request to a UPS shipping price server would be similar to (it would actually be one long line or stream of data):
 

POST /using/services/rave/qcost_dss.cgi HTTP/1.0
Content-type:application/x-www-form-urlencoded
Content-length:272

AppVersion=1.2&AcceptUPSLicenseAgreement=YES&ResponseType=application/x-ups-rss&ActionCode=3
&ServiceLevelCode=1DA&RateChart=Regular+Daily+Pickup&ShipperPostalCode=47165&ConsigneePostalCode=47150
&ConsigneeCountry=US&PackageActualWeight=17&ResidentialInd=1&PackagingType=00 
 

Embedded in the POST data stream is zip code and other data used by the UPS server to compute shipping price. The following Perl example uses the UPS site to get the cost of shipping a 17 pound package from Pekin 47165 to New Albany, Indiana 47150. The price looks to be $27.50, the %10 usually indicates that a new line character (hexadecimal 10) has been inserted. Full information of this service is available at the UPS site. It is up to the requesting application to parse out the needed information and handle failures such as the UPS site is down or busy.  Parsing the hexadecimal codes that get returned is not trivial but can done using standard methods.
 
Perl using TCP to the UPS server to gather Shipping Rate information
Perl

use Socket;

  $proto = getprotobyname('tcp');
  socket(F, PF_INET, SOCK_STREAM, $proto);
  $sin = sockaddr_in(80, inet_aton("www.ups.com"));
  connect(F, $sin) || print "Failed";
  select(F);
  $| = 1;
  select(STDOUT);
  print F
    "POST /using/services/rave/qcost_dss.cgi HTTP/1.0\n".
    "Content-type:application/x-www-form-urlencoded\n" .
    "Content-length:272\n" .
    "\n" .
    "AppVersion=1.2&AcceptUPSLicenseAgreement=YES&" .
    "ResponseType=application/x-ups-rss&ActionCode" .
    "=3&ServiceLevelCode=1DA&RateChart=Regular+Dai" .
    "ly+Pickup&ShipperPostalCode=47165&ConsigneePo" .
    "stalCode=47150&ConsigneeCountry=US&PackageAct" .
    "ualWeight=17&ResidentialInd=1&PackagingType=0" .
    "0 \n";
 while(<F>) { print;  }
 
 
 
 

 

 UPS server response

HTTP/1.1 200 OK
Server: Netscape-Enterprise/3.6 SP3
Date: Tue, 25 Apr 2000 15:41:35 GMT
Content-type: multipart/mixed;boundary=UPSBOUNDARY

--UPSBOUNDARY
Content-type: text/html
Content-length: 30

<HTML>
<BODY>
</BODY>
</HTML>
 

--UPSBOUNDARY
Content-type: application/x-ups-rss
Content-length: 85

UPSOnLine%1.2%0000%0000Success%3%1DA
%47165%US%47150%US%102%17%27.50%0.00%27.50
%103000

--UPSBOUNDARY--

Exercise 5 - Gathering Data from Data Servers

  1. Copy and paste the Perl program above as UPS.pl. Modify the request for a different zip code and weight.
  2. Execute at the command line by: perl UPS.pl
  3. Locate the price in the resulting output.
Example - A more realistic example of  data gathering from other sites again uses the UPS site to retrieve selected data on shipping costs. The key points of note are: The data sent to the UPS server requested the package be picked up, the weight was 20 pounds, it was shipped from Pekin to New Albany using 10:30 A.M. guaranteed Next Day Air delivery.
 
HTML and CGI using UPS Site to Compute Shipping Cost 
Form with shipping parameters



UPSRates.htm 

<HTML> 
<TITLE>UPS Shipping Rate</TITLE>
<Script Language = "JavaScript" > 

 function isNumeric(string) {
  if (string.length == 0) return false;
  var digits = "0123456789";
  for (i=0; i<string.length; i++) {
   if(digits.indexOf(string.charAt(i),0) == -1)
    return false;
  }
  return true;
 }

 function validZip(value) {
  return (value.length == 5 && isNumeric(value));
 }

 function check(form) {
  if (!validZip(form.ToZip.value)) {
   alert("To zip invalid " + form.ToZip.value);
   return false;
  }
  if (!validZip(form.FromZip.value)) {
   alert("From zip invalid " + form.FromZip.value);
   return false;
  }
  form.submit();
  return true;
 } 
</Script>
<BODY BGCOLOR="white">
  <H1>UPS Package Rate</H1><br>
  <form name=Shipping action="GetRate.pl" 
            method="POST">
   <b>From Zip Code: </b>
        <input type='text' name='FromZip' size=5 ><br>
   <b>  To Zip Code: </b>
       <input type='text' name='ToZip' size=5 ><br>
   <b>Pounds</b><BR>
   <SELECT name="pounds" size=4>
    <OPTION>1</OPTION>
    <OPTION>5</OPTION>
    <OPTION>10</OPTION>
    <OPTION>15</OPTION>
    <OPTION>20</OPTION>
   </SELECT><br>
   <INPUT type=button 
           onclick="check(this.form)" value="Get Rate">
  </form>
</BODY>
</HTML>

 Page constructed from UPS shipping data



GetRate.pl 
use Socket; 
use CGI qw/:standard/;            # load standard CGI routines 

 $query = new CGI;                # GET and POST data
 print header;                    # create the HTTP header

 $FromZip = $query->param("FromZip");  
 $ToZip = $query->param("ToZip");      
 $pounds = $query->param("pounds");     

 $proto = getprotobyname('tcp'); 
 socket(F, PF_INET, SOCK_STREAM, $proto); 
 $sin = sockaddr_in(80, inet_aton("www.ups.com")); 
 connect(F, $sin) || print "Failed";    # Make connection 
 select(F); 
 $| = 1; 
 select(STDOUT); 

 print "<u>Connected to UPS</u><br>\n"; 

 #   Print the following to the www.ups.com Web server. A period 
 #   is concatenation in Perl 
 print F  
        "POST /using/services/rave/qcost_dss.cgi HTTP/1.0\n" . 
        "Content-type:application/x-www-form-urlencoded\n" . 
        "Content-length:272\n" . 
        "\n" . 
        "AppVersion=1.2&AcceptUPSLicenseAgreement=YES&" . 
        "ResponseType=application/x-ups-rss&ActionCode" . 
        "=3&ServiceLevelCode=1DA&RateChart=Regular+Dai" . 
        "ly+Pickup&ShipperPostalCode=" . $FromZip . "&ConsigneePo" . 
        "stalCode=" . $ToZip . "&ConsigneeCountry=US&PackageAct" . 
        "ualWeight=" . $pounds . "&ResidentialInd=1&PackagingType=0" . 
                                                "0 \n"; 
 #  Process the UPS response 
 while(<F>) {  
        ($before, $after) = split(/UPSOnLine/, $_); 
        ($before, $after) = split(/Success%/,$after);  
        ($a,$b,$c,$d,$e,$f,$g,$h,$Price) = split(/%/,$after); 
        if ($Price ne "") {  
                print << "End";
                        <H2>UPS Shipping Rate</h2> 
                        From: $FromZip <br>  
                        To: $ToZip <br>  
                        Weight: $pounds pounds<br>  
                        Price: $Price <br> 
End
        } 
 } 
 

Exercise 6 - Using Gathered Data

  1. Copy and paste the HTML and Perl code.
  2. Test that it works. Enter an invalid zip code.
  3. This was executed in Summer 2000, has the shipping price gone up?
  4. What problems are likely to develop over time in maintaining such a system? Suggestions for improvement?