| << Prev | - Up - | Next >> |
The examples given in this chapter are functors. As we've stated before, in Chapter 2, the main programming interface of the HttpClient package is provided by the specialised classes found within HttpClient module, namely CgiGET, CgiPOST, CgiISINDEX, UrlGET and UrlHEAD. In order to use them, we have to import 'x-ozlib://mesaros/net/HTTPClient.ozf' functor in our examples, since it includes all the functors of this package in a prelinked fashion.
This short example shows how a document located at Url can be downloaded using HttpClient.getUrl class. The first step is the initialisation of the class, using its init method. Note that the provided input parameters are unbound. In this case they will be given default values. The second step is calling the getService method, thus invoking the service. We don't want to reach the output parameters so we don't bound them.
The content of the downloaded document will be put into the file having the given Url based name, "index.html" in our case.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Simple use of HttpClient.urlGET class %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
functor
import
System
Application
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
proc {GetDoc Url}
HCObj
in
HCObj={New HTTPClient.urlGET init(_ _)}
try
{HCObj getService(Url _ _)}
catch E then
{System.show E}
end
end
in
{GetDoc "http://www.info.ucl.ac.be/index.html"}
{Application.exit 0}
end
HttpClient.getHEAD class can be used in the same manner. In this case the content of the document will not be fetched, but its HTTP response headers will be returned.
This example shows how HttpClient.cgiGET class can be used to build CGI queries (e.g. to Yahoo web search engine). The parameters of the query is given by CgiParams which is a list of "name"#"value" strings pairs. In our very case, we want to retrieve a document containing a list of keyword "travel" related resources. Since the name of the file the retrieved document should be put is provided, it will be used. toFile feature is true by default.
The size of bytes of the requested document OutPrms.sizeRead read from the socket is displayed. Since it is a string, the name of the peer HttpPrms.server is displayed as an atom, so we can easily see it. Note the use of HttpReqPrms record in order to specify the languages we accept from the peer.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Simple use of HttpClient.cgiGET class %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
functor
import
System
Application
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
Url = "search.yahoo.com/bin/search"
CgiParams = ["p"#"travel"]
OutPrms
HRepPrms
HCObj
in
HCObj={New HTTPClient.cgiGET init(inPrms(file:"ysearch")
httpReqPrms(accept_language:"fr-BE, en"))}
try
{HCObj getService(Url CgiParams OutPrms HRepPrms)}
catch E then
{System.show E}
end
{System.show sizeread#OutPrms.sizeRead}
{System.show http_server#{VirtualString.toAtom HRepPrms.server}}
{Application.exit 0}
end
HttpClient.cgiPOST and HttpClient.cgiISINDEX classes can be used in the same manner. In the latter case CgiParams has to be a string.
Among the examples presented in this chapter, this one involves the use of the grater number of service's methods, namely bytesRead, stopTemp and startTemp. Thus, in order to have access to these methods, while the document is being fetched, we will separate the processes by using the threads.
By calling GetRate procedure, it will display the transmission rate, the time it was computed and the instantaneous number of bytes read, by reading the rateStrm stream and calling bytesRead method, respectively. The transmission rates are computed as an average over 350 ms time interval (as indicated by tInterval). After a period of 850 ms the transmission will be temporarily stopped and continued after other 3000 ms period.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Simple use of the bytesRead, stopTemp and %%
%% startTemp methods of HttpClient.urlGET class %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
functor
import
System
Application
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
Url = "http://rfc.fh-koeln.de/rfc/html/rfc0959.html"
OutPrms
proc {UseIt}
HCObj
ServiceEnd
proc {GetRate Stream}
case Stream of nil then skip
[] X|Xr then
if X\=nil then
{System.show X#{HCObj bytesRead($)}}
{GetRate Xr}
end
end
end
in
HCObj={New HTTPClient.urlGET init(inPrms(toFile:false tInterval:350) _)}
thread
try
{HCObj getService(Url OutPrms _)}
catch E then
{System.show E}
finally
ServiceEnd=unit
end
end
thread {GetRate OutPrms.rateStrm} end
{Delay 850}
{HCObj stopTemp}
{Delay 3000}
{HCObj startTemp}
{Wait ServiceEnd}
end
in
{UseIt}
{Application.exit 0}
end
In order to emphasise the advantages of having access to the transfer rate of the document being downloaded, we've conceived a small example. Given a list Urls of different Url addresses of the same document, a threshold rate RateTh and a time period TimePeriod, it will try to fetch the first document in the list whose transfer rate is grater or equal with RateTh. TimePeriod represents the time period the rate transfer is allowed to be below RateTh. If the rate transfer rests below RateTh a certain period grater than TimePeriod, the current download is terminated and the next address in the Urls list will be processed, by using GetDoc.
functor
import
System
Application
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
<GetDoc: get a document considering the transfer rate>
Urls = ["http://www.ring.gr.jp/pub/doc/RFC/rfc2616.txt"
"http://sunsite.doc.ic.ac.uk/rfc/rfc2616.txt"]
TimePeriod = 1000 %% milliseconds
RateTh = 25.0 %% KB/s
in
{List.dropWhile Urls GetDoc _}
{Application.exit 0}
end
GetDoc is the function used for downloading a document considering its transfer rate. It contains two other procedures, namely GetRate and WaitFor and returns Continue as being false if the current service has fulfilled the imposed conditions or true otherwise.
fun {GetDoc Url}
<GetRate: get and evaluate the transfer rate>
<WaitFor: wait a period before closing the service>
HCObj
OutPrms
TimeCounter = {Cell.new 0}
Continue
in
{System.show trying#{VirtualString.toAtom Url}}
<HttpService: call getURL service>
Continue
end
A simple way to call an HttpClient service in parallel with a procedure, namely GateRate, which treats one of its output, namely rateStrm, is shown below.
HCObj={New HTTPClient.urlGET init(_ _)}
thread {GetRate OutPrms.rateStrm _ _} end
try
{HCObj getService(Url OutPrms _)}
catch _ then
Continue = true
end
The transfer rate evaluation is done by GetRate procedure. It compares the instantaneous rate X.1 with RateTh. There are two main situations, the first one: the instantaneous rate is grater or equal with RateTh, and the second one: the instantaneous rate is below RateTh. Since the rate occurrence over rateStrm is aleatoric, a time counter TimeCounter and WaitFor procedure are used.
proc {GetRate Stream PrevTime PLock} LLock in
case Stream of nil then
PLock=unit
Continue=true
[] X|Xr then
if X\=nil then Tp in
PLock=unit
if X.1'<'RateTh then Dif in
if {IsDet PrevTime} then
Dif = {Cell.access TimeCounter}+(X.2-PrevTime)
else Dif = 0 end
{Cell.assign TimeCounter Dif}
Tp = TimePeriod
else
{Cell.assign TimeCounter 0}
Tp = 2*TimePeriod
end
{WaitFor Tp-{Cell.access TimeCounter} LLock}
{GetRate Xr X.2 LLock}
else
PLock=unit
if {IsFree Continue} then Continue=false end
end
end
end
WaitFor procedure is called at each rate occurrence over rateStrm and is used for terminating the current service if after a given Td time period the current rate is still below RateTh.
proc {WaitFor Td WLock}
thread
{Delay Td}
if {IsFree WLock} then
Continue=true
{HCObj closeAll(false)}
end
end
end
This small example shows how two services can be started in parallel in order to download the same document having different locations Url1 and Url2. This will try to fetch the documents while each service runs within its own thread. When one of the services completes, it closes the other one and the application exits. Since the two names of the files the document will be saved in will be the same, we set overwr to false. Then, by calling closeAll method with the parameter set to false, the corresponding file will be removed. Hence, only the file corresponding to the first successful service will remain on the HDD.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Parallel document retrieval %%
%% using HttpClient.urlGET class %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
functor
import
System
Application
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
Url1 = "http://www.ring.gr.jp/pub/doc/RFC/rfc2616.txt"
Url2 = "http://sunsite.doc.ic.ac.uk/rfc/rfc2616.txt"
proc {UseIt}
HCObj1 HCObj2
OutPrms1 OutPrms2
ServiceEnd1 ServiceEnd2
in
HCObj1={New HTTPClient.urlGET init(inPrms(overwr:false) _)}
HCObj2={New HTTPClient.urlGET init(inPrms(overwr:false) _)}
thread
try
{HCObj1 getService(Url1 OutPrms1 _)}
{System.show thread1#finished}
{HCObj2 closeAll(false)}
ServiceEnd1=unit
catch E then
{System.show E}
ServiceEnd1=unit
end
end
thread
try
{HCObj2 getService(Url2 OutPrms2 _)}
{System.show thread2#finished}
{HCObj1 closeAll(false)}
ServiceEnd2=ServiceEnd1
catch E then
{System.show E}
ServiceEnd2=ServiceEnd1
end
end
{Wait ServiceEnd2}
end
in
{UseIt}
{Application.exit 0}
end
As we've stated in Chapter 2, an HTTP connection implies a socket one, and thus the use of connect method of Oz Open.socket class. This method blocks the entire Oz system until it succeeds, namely until a socket connection is established. Sometimes, this inconvenient becomes really unbearable i.e. when the network is slow, or the peer is off, or does not exist. One solution to solve this inconvenience is to wrap it into a functor and allocate a new process (virtual site) for it. This is done by using the Remote module.
The example shown below will try to do something, Loop, within the current process and download a document in an other one, at the same time. Hence, by using virtual sites we can have parallel jobs running while others blocks for some reasons. The two processes start at the same time but the latter one will block immediately until it succeeds. After succeeded, its transfer rate will be fetched by using the fetchRate method.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% Avoiding system blocking %%
%% by using Virtual Sites %%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
functor
import
Application
System
Remote
define
Funct = functor
import
System(show:Show)
HTTPClient at 'x-ozlib://mesaros/net/HTTPClient.ozf'
define
proc {FetchRate}
if {IsFree ServiceEnd} then
{Delay 300}
{Show rate#{HCObj fetchRate($)}.1}
{FetchRate}
end
end
proc {UseIt}
HCObj={New HTTPClient.urlGET init(_ _)}
thread {FetchRate} end
try
{HCObj getService(Url _ _)}
catch E then
{Show E}
finally
ServiceEnd = unit
end
end
HCObj
ServiceEnd
Url="http://elle.c5.utcluj.ro"
in
{UseIt}
end
proc {Loop N}
if N>0 then
{Delay 100}
{System.show looping#N}
{Loop N-1}
end
end
thread {Loop 3000} end
R={New Remote.manager init}
{R apply(Funct _)}
{Application.exit 0}
end
| << Prev | - Up - | Next >> |