(IJCNS) International Journal of Computer and Network Security, 179

Vol. 2, No. 10, 2010

Evaluation and improvement of resource retrieving

method, using reinforcement learning in Grid
Network
Erfan Shams1, Abolfazl Toroghi Haghighat2 and Ehsan Ghamari3
1
Islamic Azad University Qazvin Branch, School of Computer Engineering,
Qazvin, Iran
Eshams1364@yahoo.com
2
Islamic Azad University Qazvin Branch, School of Computer Engineering,
Qazvin, Iran
at_haghighat@yahoo.com
3
Islamic Azad University Qazvin Branch, School of Computer Engineering,
Qazvin, Iran
ehsan.ghamari@gmail.com

Abstract: Network calculations make virtual organization able
to share distributed resources in geographical view, to achieve
common goals this method lacks a place, a central controlling
and trusted relation generally , in order to solve the gride
problem. It is necessary to find the most suitable resource in the
shortest time. This purpose as applied in part of solving process.
All devoted approaches of information retrieving try to serve all
requests optimally in the shortest time, but they are unable to
match with or be flexible to grid network changes. Therefore,
the flexibility in retrieving and resource allocation are
necessary. In the presented paper, a few part is inserted into
protocol in order to manage decision controlling based on
reinforcement learning, that using learning patterns in grid
network, space recognition , factor recognition, a number of
factors and obtaining resource information in grid network ,this
part performs retrieving operation and resources allocation
more optical than other methods.
Keywords: resource retrieving, grid , reinforcement learning.
1. Introduction
Network computation is a computation model that huge
computations can be processed through them, using
computation power of many netted computes in addition to
keeping them as a unique virtual computer in view. In other
words, rid is able to solve enormous computational
problems, using computational power of several separated
computers which are mostly connected through network
(internet) [1, 3]. One of the most important current issues of
computer network is the distributed network with grid
architecture. Regarding of computer applications
development as well as hardware rapid, advancement ,
creating integrated systems ( from free heterogeneous
resources in order to multi purpose process along with
supporting of maximum- efficiency resources).
As many as some of the investigate and computational
application these networks are for bottleneck problem
solving and in order to provide users dedicating optimal
resources to execute these processes is significant issue in
the field of distributed networks with grid architecture.
In this paper, we will consider a part, called decision
controlling management , based on reinforcement learning
in resource management unit in grids, which makes us
know better about the resources by passing of time and also
makes the resources map onto the request optimally and
rapidly.
We will utilize reinforcement learning since this method
matches grid network. Reinforcement learning is the online
learning and also is applied in environment which is
reportedly visible .this method is independent of long
information mass for instructor (such as neural network and
genetic – based method). But in our framework, regarding
rewards which are achieved in the of path, we obtain
recognition or generally rewards based on traversed steps
repeating this approach. Our resource (node) recognition
will be stronger; therefore we can understand the network
better.
Following, we introduce a new approach which is proved by
comparison and testing, to show important and efficiency of
our work to the previous studies.
2. Introduction of information retrieving in
grid network
Serving request is performed in two manners: 1.real-time
and 2. Non-real-time method. First, serving is rapidly
performed to set out appropriate resources as soon as
receiving request from broker resource which has mentioned
conditions, send a message to broker, then broker selects the
best resource based on different factors such as distance and
send the request to the resource for serving. Obviously, this
method makes the network traffic heavier. In second,
information of resources is available at request reporting
time the discovered resources are saved by brokers in
resource management part. Optimal resources are searched
180
and selected as soon as request report and then the request
will be sent for selected resources. According to this way,
network traffic is significantly decreased and therefore the
requests are responded more rapid.
2.1 Breadth First Search (BFS)
BFS is one the simplest and most functional searching
method with easy process. In this method, every node what
has request, sends the request to all its neighbors. The node
also searches its local information in order to find perfect
answer. These mentioned steps will be done for every node
which receives request in the case of finding necessary
resourced , a message asks the resources through the nearest
node which has replied to its request [8]. Making heavy
traffic is the disadvantage of this method. Assume that, we
have ‘n’ given nodes so that every node has m neighbors. In
the first step, the broker sends a message to or responding m
nodes .after passing of some steps, several messages are
sent. This phenomenon makes the network occupied.
2.2 Random Breadth First Search (RBFS)
This method is similar to the previous one with this
different that corresponding node doesn’t send the message
for all its neighbors in every step, but it sends the inquiry to
the part of neighbors also this method has some
disadvantages and advantages decreasing network traffic
and rapid operating of the search are advantages. Since
nodes haven’t any information about neighbors that the
message is sent for them. Therefore there is no wasting time
to verify and decide in other words. Every node selects
randomly some of the neighbors and sends the message to
them. Neighbor random selection is the significant
disadvantage of this method because the dead parts of
network which are weakly connected to the network, almost
never are made inquiry.
2.3 Random Breadth First Search with RND-Step
actually , this is improvement of previous ones in this
method, we start the search, using ‘n’ brokers instead of
only one broker(‘n’ is depended on given steps) and then
every node of n node searches the releasing resources.
Disadvantages of this method are like RBFS deliberations.
In other words, because of being random performance in
searching step, optimal results will not e obtained. On other
hand, searching through couple of paths (and linear
increasing of number of neighbors of all nodes) makes the
efficiency higher.
2.4 Searching with keeping information
In spite of 3 mentioned methods, this method responds the
request in non-real status. There are some methods which
consider the status of neighbors and responses to the
requests, including directed breadth first search (DBFS) and
hashing method the efficiency of those methods are higher
then random based methods. In detail, these methods
decrease significantly network traffic and mount of inquires
and therefore are able to rapidly the required resources to in-
time responses of requests [10, 12].the notable difference of
(IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 10, 2010
our presented method with mentioned ones are applying of
reinforcement learning which makes suitable recognition by
passing of time in to allocate optimally resources.
3. Reinforcement learning
Reinforcement learning [13] generally is the art of finding
strategy to improve the status to achieve the certain goal,
regarding of environment recognition, relation results
behind environment and benefits-damages of performing
several task simply, Reinforcement learning is the learning
through environmental relation to achieve specified goal
decision maker and the person who learn, are called
“Agents”. The thing what agent make relation with it,
named environment (in fact, every foreign thing of agent is
involved). This relation continuously, is occurred in this
manner that, agent makes decision and then operates an
action accordingly. After that, the environment will respond
with granting reward. Lastly, the agent will be transferred to
new state.
In detail, agent and environment have relation sequencly
through time step t=1,2,3,…. . in every step, for example in
step t, the agent receives a new state from environment. In
this paper, we suppose that the whole space of grid is s. st Є
S where S is the possible state set of allocating environment
resources. at is the possible task set of agent whom does
them in state st. in the new step, environment grants reward
R in time t + 1, so that r(t+1) Є R. based on its previous
task, the agent will be transferred to the new state S(t+1). To
mathematics, a policy is a mapping. For instance
Π :S× A 
→[0,1] is a policy. It means, a number in
[0,1] is appropriated for any pair (action, state) like (s,a)
which are belong to Π : S × A . This is shown by
Π ( S × A) as follow:
Pr{at = a}{st = s} = Π (s, a )
The set of states, actions and sequenced rewards is
considered in reinforcement learning as follow:
Figure 1. The set of stats, actions and sequenced rewards.
The value of s is a function of state value:
Calculating of real value of the whole state is known as
policy evaluation providing of following of policy Π and it
is necessary for perfect learning the value which can be
considered for action-state (s,a) is:
Example: when we are going to learn reinforcement
learning system appropriated for the separated and best
resources to serve requests. According to the mount and
time of processing to allocate resources, we initial this unit
+1 : appropriate, -1 “ inappropriate , 0: middle. It is notable
that rewarding should be granted so that agent van satisfy
us, maximizing reward and also it shouldn’t learn, how to
 (IJCNS) International Journal of Computer and Network Security, 181
satisfy o instance, in example 2 , score +1 ill be granted
when the best resource is selected for processing.
4. Proposed approach to improve retrieving
resources
in this paper, it is considered that, firstly there is no exact
information based on grid network environment (about
entity nodes) but generally, allocating resources will be
more optimal, using patterns learning in grid network and
recognition spaces. Agent st and number of agent a as well
as gaining resource information in grid network. There are
3 general decisional states to allocate resources for agent
including:
• The resource has suitable processing power and
execution memory as well as suitable bandwidth.
• The resource has not suitable processing power and
execution memory but there is suitable bandwidth( this is
notable that 2 components of resources are not
appropriately recognized to process information for
making network process rate stable and also it is not
recorded at RLDU).
• The resource owns suitable power of processor and
executable memory but the appropriated bandwidth is
not suitable (it is notable that the resource is recognized
proper for small processing and the resource allocation
unit appropriates the small processing for this unit) thus
s or state space is S={Good Node, Bad Node, Normal
Node } and selection or tasks apace is : A={Specialty
Process For This Process, Not Specialty Process} . To
optimize resource allocation in a grid network, select
resource issue has effect on efficiency of thought of
network and resource managing select resource has 3
above general stats to select the best resource in state
space s. in reinforcement learning, the agent goal is
considered in the form of reward signal which is
percepred from the environment. In every time step,this
reward is considered as simple number rt Є R. In simple
statement, is that the agent goal is maximizing of total of
these rewards. Reward maximizing should be done in
long term. this doesn’t mean maximizing reward in
every step, to use this method in resource allocation, it is
supposed that a unit is added to the network layout in
grid network and rewards/punishes to resource allocation
unit based on 2 factors as follow: 1- Appropriated
processing time for resource 2- Recorded information of
resource in workload management. Fig.2 sows the
implementation of 2 new proposed added units to the
grid layout network.
In this fig, two different unit are added to the grid protocol
that these two units make decision for resource allocation to
process.
Reinforcement learning decision unit (RLDU):
Vol. 2, No. 10, 2010
Figure 2. Adding reward and punishment units to the grid
model
This term is modeled reinforcement learning core based on
marcov model and dynamic programming which
dynamically make relation with 3 other layers (scheduler-
broker- Gram). This unit decides +according to the two time
factors which are for a processing in previous mapping and
recorded information in above unit such as processing rate
and data transfer bandwidth to appropriate this resource.
RLDU announces proposed resources (the output of RLDU)
to the grid resource manager-Gram. Reword/punishment
(RE/PE) unit grants score +1 t suitable resource allocation
for the processing and otherwise grants score -1 to RLDU.
4.1 Resource classification
Information is saved on RLDUU based on every mapping
for resource, but information included processor power,
bandwidth, mount of score based on reward/punishment unit
and 3 above condition will be recorded in RLDU before
allocating resource the same as in search time of
information and resource comparison. If there is no suitable
resource for processing the request will be queued until
finding suitable resource .the main reason of this strategy is
to optimize and profit resource allocation , since if the
request is delivered to the appropriate resource, it will not
probably be completed on certain time. The resource
allocation operation and planning must be repeated this is a
disadvantage.
In this method, this score will be granted to the resources
nor request, hence the core will not be granted to the
resources of RLDU, until performing resource allocation
and request planning we can utilize a selection function for
choosing a batch of resources to refer to the resources in
order to avoid increasing information volume in RLDU. For
example, we separate 20 resources which are matched the
request from bank and then we start searching for finding
optimal resource until the search time is decreased. Then
resource will be classified in 3 batches based on
classification space defined above, by passing of few times
that the information is collected over the network. Dynamic
property of grid is notable, since the resource of every
system gets off inadvertently.
In this case, the corresponding node must e deleted from
classification, the score -2 will be granted by RE/PE unit,
because of non-responding of RLDU. This process makes
the resource delete however the information of resources
will be updated permanently by broker, since if the deleted
182
resource comes back to the network, it will be classified.
The RE/PE unit grants score -2 whenever resource
allocation and planning are completed, as well as the
request is sent to the resource. But the corresponding
resource will be deleted from network before delivering the
request.
5. Experimental results and valuations
The simulation results show that proposed approach is
matched with distributed network along with grid
architecture and also efficiency stable.
As compare with other methods of searching and allocation,
out approach has high efficiency and optimality. In addition
to that, because of using intelligent factor which make
relation with other units, proposed approach is matched
with grid dynamic heterogeneous network. Using limitation-
maker search function, we are able to decrease search rate of
recorder information in RLDU. In order to perform this
decreasing, we can consider a search strategy applying
different method in order to select resource. We can
consider optimality of search time and selects resource
simultaneously.
In Figure 3 shows a number of search methods in search
space, using proposed reinforcement learning which is
presented in model and also compared with others.
Figure 3. Optimization of resource allocation based on
reinforcement learning.
As shown, search time as compared with used methods is
decreased. This improvement is evaluated in view of time
and selecting optimal resource for request. In fig 4, search
methods are compared with proposed approach in
distributed grid network .as result, my work is optimal in
view of search time and cost. It means we can optimize the
result about 15% which are significant in time and cost.
Figure 4. Compartment of search method with presented
approach
(IJCNS) International Journal of Computer and Network Security,
Vol. 2, No. 10, 2010
6. Conclusion
In this paper, we try to propose an approach to retrieve
resource and verify its efficiency in order to evaluate
resource retrieving methods in grid network. Generally we
come to conclusion that:
• The efficiency of presented work will be more optimal by
passing of time and resource recognition.
• In the case of changing network significantly and high
importing and exporting of network, the efficiency of our
work will be decreased.
In future work, we are going to improve the efficiency of our
work and presenting some search methods for resource in
network. This work uses genetic algorithm and
reinforcement learning at the same time.
References
[1] B. Jacob, L. Ferreira, N. Bieberstein, C. Glizean, J.
Girars, R. Strachowski, and S. Yu, Enabling,
Applications for Grid Computing with Globus, IBM
Red Book Series, 3P, 2003.
[2] I. Foster, C. Kesselman, and S. Tuecke, ”The Anatomy
of Grid, Enabling Scalable Vitrual Organizations”,
International Journal of High Performance Computing
Applications, XXX (3), pp. 200-222, 2001.
[3] A. Abbas, “Grid Computing, A Practical Guide to
Technology and Applications”, Charles Rivar Media,
2004.
[4] M. D. Dikaiakos,” Grid benchmarking: vision,
challenges, and current status”, Concurrency and
Computation: Practice & Experience, 19 (1), pp. 89-
105, Jan 2007.
[5] ”How does the Grid work?”, Available:
http://gridcafe.web.cern.ch/gridcafe/openday/How-
works.html.
[6] R. Buyya, S. Venugopal, “A Gentle Introduction to Grid
Computing and Technologies”, CSI Communications,
Computer, Science of India,July 2005.
[7] “what is the difference between the internet, the web
and he Grid?”, Available: http://gridcafe.web.cern.ch/
gridcafe/openday/web-grid.html.
[8] “Job life-cycle”, Available: http://gridcafe.web.cern.ch/
gridcafe/openday/New-JobCycle.html.
[9] David Johnson, “Desktop Grids”, Entropia.
[10] Rajkumar Buyya, “Grid Technologies and Resource
Management Systems”, A thesis submitted in
fulfillment of the requirements for the Degree of Doctor
of Philosophy, School of Computer Science and
Software Engineering, Monash University, Melbourne,
Australia, April,12, 2002.
[11] “Sun Powers the Grid, An Overview of Grid
Computing”, Sun Microsystems, 2001.
[12] I. Foster, “What is the Grid? A Three Point Checklist”,
Argonne National Laboratory & University of Chicago,
July 20, 2002.
[13] "Reinforcement Learning: An Introduction", Richard
S.Sutton and Andrew G.Barto, Cambridge, MA, 1998,
MIT Press.