Future Generation Computer Systems 99 (2019) 500–507

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

A smart agriculture IoT system based on deep reinforcement learning

∗
Fanyu Bu a , , Xin Wang b
a
College of Computer and Information Management, Inner Mongolia University of Finance and Economics, Hohhot, China
b
Center of Information and Network Technology, Inner Mongolia Agricultural University, Hohhot, China

highlights

• We design a smart agriculture IoT system based on an edge-cloud computing.

• We present several representative deep reinforcement learning models.
• We discuss the possible challenges and applications of deep reinforcement learning in smart agriculture.

article info a b s t r a c t

Article history: Smart agriculture systems based on Internet of Things are the most promising to increase food
Received 15 March 2019 production and reduce the consumption of resources like fresh water. In this study, we present a
Received in revised form 1 April 2019 smart agriculture IoT system based on deep reinforcement learning which includes four layers, namely
Accepted 17 April 2019
agricultural data collection layer, edge computing layer, agricultural data transmission layer, and cloud
Available online 4 May 2019
computing layer. The presented system integrates some advanced information techniques, especially
Keywords: artificial intelligence and cloud computing, with agricultural production to increase food production.
Deep reinforcement learning Specially, the most advanced artificial intelligence model, deep reinforcement learning is combined
Smart agriculture IoT in the cloud layer to make immediate smart decisions such as determining the amount of water
Edge computing needed to be irrigated for improving crop growth environment. We present several representative deep
Cloud computing reinforcement learning models with their broad applications. Finally, we talk about the open challenges
and the potential applications of deep reinforcement learning in smart agriculture IoT systems.
© 2019 Elsevier B.V. All rights reserved.

1. Introduction systems. IoT is mainly used to automatically collect agricultural

Agriculture is one of the biggest concerns to all mankind since
most of food is produced by agriculture. At present, many people
still suffer from hunger due to lack of food in some countries,
especially in Africa. Specially, hunger caused the chronic un-
dernourishment of more than 800 million people in the world
in 2016. More notably, more than 10 million people died from
hunger per year. Obviously, increasing food production is un-
doubtedly an effective means of eradicating hunger and poverty.
However, the agriculture is far from modernization in most of
developing countries, leading to low food production.
Recently, smart agriculture has been proposed to promote
the modernization of agriculture for increasing food production
greatly [1,2]. Specially, smart agriculture systems introduce many
advanced computer and information technologies such as Inter-
net of Things, artificial intelligence, and cloud computing into the
agricultural production. Internet of Things (IoT) and artificial in-
telligence are two core techniques for building smart agriculture
∗ Corresponding author.
E-mail address: bufanyu@imufe.edu.cn (F. Bu).
data and to transmit the collected data to the data centers [3,4]
while artificial intelligence techniques such as artificial neural
networks and clustering are used to analyze the agricultural
data for smart decision-making. An example is to decide the
amount of water needed to be irrigated by analyzing the collected
agricultural environment data. In detail, once the requirement of
irrigation is detected when the environment is lack of water, the
action will be immediately taken to offer the appropriate amount
of water. As a recently emerged advanced artificial intelligence
technique, deep reinforcement learning (DRL) was presented for
smart decision-making several years ago and it has enjoyed a
great success in several domains such as computer game, medical
diagnosis and energy management with the deep combination of
deep learning models and reinforcement learning strategies [5,6].
The representative examples of deep reinforcement learning are
deep Q-network, multi-agent deep reinforcement learning, and
deep successor network. Therefore, it is a promising model used
in building smart agriculture systems. Cloud computing is an-
other crucial computer technique for building smart agriculture
systems since it provides a powerful computing infrastructure
for data analytics and deep learning models. Generally, large

https://doi.org/10.1016/j.future.2019.04.041
0167-739X/© 2019 Elsevier B.V. All rights reserved.
 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507 501

amounts of agricultural data are collected from agriculture pro-

duction and they must be processed and analyzed at a high speed.
Cloud computing has demonstrated its strong power in the area
of big data computing [7]. For example, cloud computing has been
successfully used to improve the efficiency for deep computation
models and big data processing [8,9].
In this paper, we design a smart agriculture IoT system that
is made of four layers from bottom to up including agricultural
data collection layer, edge computing layer, data transmission
layer, and cloud computing layer. As a crucial component, deep
reinforcement learning is deployed in the cloud layer for making
immediate smart decisions. Specially, we present several repre-
sentative DRL models that are potential to be used in building
smart agriculture systems. This work is expected to promote the
development of smart agriculture and furthermore contribute to
increasing food production.
The paper has three contributions listed below.

• We design a smart agriculture IoT system based on an

edge-cloud computing.
• We present several representative deep reinforcement
learning models.
• We discuss the possible challenges and applications of deep
reinforcement learning in smart agriculture.

In the following parts, the smart agriculture IoT system is

presented in Section 2 and representative deep reinforcement
learning models are described in Section 3. Section 4 discusses
the potential applications and challenges of deep reinforcement
learning models in building smart agriculture systems.
Fig. 1. Architecture of smart agriculture IoT system.

2. Smart agriculture IoT system

3. Deep reinforcement learning

This section presents the smart agriculture IoT system. In
particular, the framework of the smart agriculture IoT system is
illustrated in Fig. 1.
In the smart agriculture IoT system, the bottom layer is aimed
at agriculture data collection. In this layer, various sensors are
deployed to collected the environmental parameters that are
important for crop growth. Representative environmental param-
eters include the air temperature and humidity, carbon dioxide
concentration, soil moisture and temperature, and light intensity,
etc. Specially, these parameters are collected every five minutes.
Before the collected parameters are transmitted to the data
center in the cloud for analytics, they are preprocessed in the
edge computing layer. In particular, two preprocessing tasks are
carried out in the edge layer. The objective of the first task is to
monitor the state of each sensors and this objective is accom-
plished by detecting the incomplete data or the outliers from the
collected parameters. The objective of the other task is to greatly
reduce the amount of data and this objective is accomplished
by compressing the collected parameters. For example, if the air
temperature is kept at 25 degrees in ten sampling periods, each
with five minutes, only one value namely 25 will be transmitted
to the data center. This way the bandwidth can be saved by
reducing the amount of data transferred.
The collected data is analyzed by various data mining algo-
rithms and artificial intelligence models in the cloud layer. In a
smart agriculture IoT system, an important task is to take a series
of measures to adjust the environment to adapt to crop growth.
This task is done by using deep reinforcement learning to make
smart decisions in our system.
Deep reinforcement learning, a novel model that combines
deep learning such as stacked auto-encoders and convolutional
neural networks and reinforcement learning such as Q-leaning,
was initially presented by DeepMind to optimize a policy for
complex tasks [10]. In detail, deep learning aims to learn ab-
stract feature representations for high-dimensional raw state in-
put from the environment while reinforcement learning methods
are used to find optimal policies that enable the agent to obtain
the maximum cumulative reward [11–13]. In 2015, DeepMind,
the artificial intelligence research group in Google, published
their research results on deep reinforcement learning in Nature
magazine, implying that deep reinforcement learning has made
an exciting advance [10]. Specially, DeepMind developed a game
program, AlphaGo, based on deep reinforcement learning. In the
five-game Go match between AlphaGo and Lee Sedol who is
a South Korean professional Go player of 9-dan rank, AlphaGo
defeats Lee Sedol with 4 : 1, indicating that deep reinforcement
learning model has reached the human level in complex chess
games. Besides, deep reinforcement learning has also achieved
an excellent performance in computer vision, robot control, and
language modeling.
One of the commonly used reinforcement learning approach
is policy gradient. In particular, policy gradient methods find
the optimal policy directly by parameterizing a policy. They first
estimate the gradients of the objective with regard to the pa-
rameters, and then use the gradient descent strategy to update
the parameters for the optimal policy. Different from the value-
based methods that can only be used for deterministic policies,
policy gradient methods can obtain the optimal policy for both
deterministic policies and stochastic policies.
502 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507

A parameterized policy π is defined as a probability distri- where Q̂tπ denotes an estimation, such as Rt , of Qtπ . If f satisfies
bution from the state space s ∈ S to the action space a ∈ A, Eq. (9) and compatible property simultaneously, i.e.,
i.e., at ∼ π (s, a; θ ) at the time step t, where θ ∈ Rl for l << |S | ∂ fw (s, a) ∂π (s, a) 1
denotes a parameter vector for the policy π . = , (10)
It is assumed that the policy π (s, a; θ ) is differential to the ∂w ∂θ π (s, a)
parameters θ . The objective can be defined as: the gradient can be obtained via:
∂ J(θ ) ∑ π ∑ ∂π
∫ ∫
J(θ ) = dπ (s) π (s, a; θ )r(s, a)dsda, (1) = d (s) fw (s, a). (11)
s A ∂θ s a
∂θ
where dπ (s) = limt →∞ P(st = s|s0 , π ) denotes the state dis-
More recently, deterministic policy gradient algorithms have
tribution under the policy π . Policy gradient methods obtain
been proposed [17]. Deterministic policy gradient algorithms per-
the optimal policy πθ by using the gradient descent strategy to
∂ J(θ ) form better than stochastic policy gradient algorithms in high-
update the parameters θ for ∂θ → 0. dimensional action space, so they have attracted much interests
Since π (s, a; θ ) cannot be represented explicitly, the gradient in recent years. Other policy gradient algorithms, including reg-
∂Jθ
∂θ
can only be estimated approximately. An approximation for ularized policy gradients and batch policy gradient, have been
∂Jθ
the gradient ∂θ was proposed for any Markov Decision Process presented in the past few years for reinforcement learning.
as [14]: Another idea of the policy gradient methods is to increase
∂ J θ ∑ π ∑ ∂π (s, a) π the occurrence probability of the good actions. Based on this
= d (s) Q (s, a), (2) idea, advantage functions can be used to evaluate an action in
∂θ s a
∂θ
reinforcement learning. Specially, the policy gradient g with the
where Q π (s, a) denotes the return of the policy π (s, a; θ ). Gener- advantage function is constructed as:
ally, the analytic expression for Q π (s, a) cannot also be obtained T −1
directly, so some policy gradient methods were presented to
∑
g = ∇θ Ât log π (st , at ; θ ), (12)
estimate Q π (s, a).
t =0
The earliest policy gradient ∑∞algorithm, called REINFORCE [15],
k=1 γ r(st +k , at +k ) as the approx-
k−1
used the actual return, R = where Ât denotes an estimator for the advantage function of the
imation of Q π (st , at ) at the time step t, action–state pair (st , at ) that is basically defined as:
∂ J(θ ) ∂π (st , at ) 1 γ
Ât = rt + γ rt +1 + · · · − V (st ),
∝ Rt . (3) (13)
∂θ ∂θ π (st , at )
where V (st ) denotes a baseline associate with the current tra-
The infinite-horizontal policy-gradient (GPOMDP) was pro- γ
jectory. Ât > 0 will increase the probability that the action is
posed to obtain a biased approximation of the gradient of the
selected.
average reward in partially observable Markov Decision Processes
Some deep reinforcement learning models based on policy
(POMDPs) [16]. Specially, GPOMDP estimates the policy gradient
gradient have been presented in the past few years. Basic deep
as:
policy gradient methods use a deep learning model to param-
∂ J(θ ) ∂π (s, a) π 1 eterize a policy and then find the optimal policy by utilizing
≈ Q (st , at ) , (4)
∂θ ∂θ π (s, a) policy gradient methods to optimize the parameters [18]. These
where Q π (st , at ) is estimated by the return of the current state– methods require N sample trajectories {τi }Ni=1 to update the policy
action pair: gradient in each iteration. However, it is difficult to obtain a
large number of training samples online in many complex tasks,
L−t
leading to the local optima solution. To tackle this problem, the
Q π (st , at ) =
∑
γ k−1 r(st +k , at +k ). (5)
actor–critic model has been introduced into the deep policy gra-
k=1
dient method. The actor–critic model combines the value function
GPOMDP updates the eligible trace et and the gradient ∆t as: and the policy gradient for challenging tasks in the traditional
∂π (st , at ) 1 reinforcement learning [19]. Specially, it is composed of two
et +1 = γ et + . (6) independent components, i.e., actor that is used to select an
∂θ π (st , at )
action, and critic that estimates the value function and calcu-
∆t +1 = ∆t + r(st +1 , at +1 )et +1 . (7) lates the temporal difference to evaluate the selected action. The
∂ J(θ ) ∂ J(θ ) architecture of the actor–critic model is shown in Fig. 2.
After L iterations, the gradient ∂θ
is estimated as ∂θ
= 1
L
∆L .
Furthermore, In the actor–critic model, the agent selects an action depend-
ing on the actor’s policy. The critic receives the immediate reward
L
∑ by interacting with the environment under the selected action to
∆L = r(st , at )et = update the value function and calculates the error of temporal
t =1
(8) difference accordingly. The actor updates the policy depending
L L−t
∑ ∂π (st , at ) 1 ∑ on the error of temporal difference to increase the occurrence
[ γ k−1 r(st +k , at +k )]. probability of the good actions and to decrease the occurrence
∂θ π (st , at )
t =0 k=1 probability of the bad actions. Given the parameterized policy
Although GPOMDP extended REINFORCE to infinite-horizon π (s; θ ) with the parameters θ ∈ Rn and the value function V (s; ϕ )
Markov Decision Process, it still has a low convergence. with the parameters ϕ ∈ Rn , the error of temporal difference from
Sutton et al. introduced the function approximation into policy the state st to the state st +1 is defined as:
gradient (PGFA) for reinforcement learning with continuous state
space [14]. Assume that Q π is approximated by a function fw : δTD,t = rt +1 + γ V (st +1 ; ϕt ) − V (st ; ϕt ). (14)
S × A → R with the parameters w satisfying the following Accordingly, the policy parameters and the value parameters
property: are updated via:
2
(Q̂tπ − f (st , at )) , ∂π (st ; θt )
∑
w = arg min (9)
w θt +1 = θt + αA,t [ut − π (st ; θt )]δTD,t . (15)
t ∂θ
 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507 503

To alleviate the problem of instability when combining tradi-

tional policy gradient methods and deep neural networks, deep
policy gradient methods use experience replay to select train-
ing samples. However, experience replay requires large memory
and computation during interaction with environment and it
requires off-policy learning algorithms which could only update
from training samples produced by an older policy. Aiming at
these problems, Mnih et al. presented a lightweight framework
for deep reinforcement learning based on asynchronous gradi-
ent descent [23]. This framework uses asynchronous gradient
descent to optimize the parameters of deep learning model con-
trollers and combines many reinforcement learning models. Spe-
cially, asynchronous advantage actor–critic (A3C) performs best
for control tasks in the continuous action space.
In the traditional deep reinforcement learning methods, the
Fig. 2. The architecture of the actor–critic model.
agent can only solve one single task after it is trained every time.
However, in some complex settings, the agent should be able to
∂ V (st ; ϕt ) accomplish multiple tasks simultaneously, that is, multi-task and
ϕt +1 = ϕt + αC ,t δTD,t . (16) transfer reinforcement learning [24]. Specially, an agent that is
∂ϕ
trained to accomplish multiple tasks should have an ability to
Lillicrap et al. presented a deep deterministic policy gradient generalize its knowledge among the tasks, and further to transfer
algorithm (DDPG) based on the actor–critic model for deep re- its knowledge to new tasks. Using multi-task and transfer learn-
inforcement learning in the continuous action space [20]. DDPG ing, the agent could speed up learning effectively, and therefore
uses a deep neural network with θ µ to represent the determinis- reduce training time.
tic policy π (s, a; θ µ ) and uses a deep neural network with θ Q to Parisotto et al. proposed a deep multi-task and transfer Pol-
represent the value function Q (s, a; θ Q ). Besides, DDPG defines icy Gradient Method based on actor-mimic (AM-DMTRL) [25].
the objective function as the expected discounted cumulative AM-DMTRL first designs a actor-mimic method to train a single
reward: multi-task policy network which is able to accomplish multiple
given source tasks S1 , S2 , . . . , SN . The basic idea of actor-mimic is
J(θ µ ) = Eθ µ [r1 + γ r2 + γ 2 r3 + · · ·]. (17)
to force the student network to mimic the expert network for
Furthermore, DDPG uses the gradient descent strategy to opti- making decisions at each state. To achieve a single multi-task
mize the objective function. The gradient of the objective function policy network, this algorithm constructs the deep Q-networks,
with regard to θ µ equals the expected gradient of the value E1 , E2 . . . , EN , for S1 , S2 , . . . , SN , respectively, as the expert net-
function with regard to θ µ : works. Specially, this algorithm transforms every expert deep
Q-network into a policy network:
∂ J(θ µ ) ∂ Q (s, a; θ Q )
µ
= Es [ ]. (18) τ −1 QEi (s,a)
∂θ ∂θ µ e
πEi (a|s) = ∑ , (21)
Depending on the deterministic policy a = π (s; θ µ ), e
τ −1 QEi (s,a′ )
a′ ∈AE
i
∂ J(θ µ ) ∂ Q (s, a; θ Q ) ∂π (s; θ µ )
= E s [ ]. (19) where τ denotes a temperature parameter and AEi denotes the
∂θ µ ∂a ∂θ µ action space of Ei . For each state s of the task Si , the policy ob-
The critic network is updated by using the method of the deep jective function is defined as the cross-entropy between πEi (a|s)
Q-network updating the value network: and the multi-task policy πAMN (a|s; θ ) parameterized by θ :
∂ L(θ Q ) ∂ Q (s, a; θ Q )
∑
= E ′ ∼D [(y − Q (s, a; θ ))
Q
], (20) Lipolicy (θ ) = πEi (a|s) log πAMN (a|s; θ ). (22)
s, a ,s
∂θ Q ∂θ Q a∈AE
i

where y = r + γ Q (s′ , π (s′ ; θ̂ µ ); θ̂ Q ) with θ̂ µ and θ̂ Q denoting
the parameters of the target policy network and the target value
network, respectively. DDPG selects samples from D in the replay
memory, and transfers the gradient of the Q value function with
regard to the actions from the critic network to the actor network.
The actor updates the parameters of the policy network according
to Eq. (20).
However, DDPG is only suitable for training deterministic poli-
cies. To tackle this problem, Heess et al. proposed a unified
framework to learn optimal policies for continuous control tasks,
called stochastic value gradient (SVG) [21]. Specially, they extend
the deterministic value gradient methods to the optimization
of stochastic policies by using ‘‘re-parameterization’’ that is a
mathematical tool used for generative models. Balduzzi and Ghi-
fary presented a deep reinforcement learning method based on
compatible function approximation [22]. In this method, they
proposed a deviator actor–critic model consisting of three neural
networks used to estimate the value function, the gradient, and
the actor’s policy, respectively.
To get the guidance from the expert networks further, a fea-
ture regression network fi (hAMN (s)) is defined to estimate the
feature values hEi (s) from hANM (s), where hEi (s) and hANM (s) denote
the activations in the final hidden layer of the multi-task network
and the ith expert network for the state s, respectively. Specially,
fi is trained by minimizing the following objective function:
LiFR (θ, θfi ) = (fi (hAMN (s; θ ); θfi ) − hEi (s))2 , (23)
where θ and θfi denote the parameters of the multi-task network
and the ith expert network, respectively. Training this objec-
tive function forces the multi-task network to calculate features
which could estimate the expert’s features. After training the
objective function, the feature information of the ith expert is
included into the multi-task network.
Furthermore, the objective function of actor-mimic is defined
as the combination of the policy regression function and the
feature regression function:
LiAM = Lipolicy (θ ) + LiFR (θ, θfi ). (24)
504 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507

for a state s:

π (l) (s, a) := hlL (s). (26)

Furthermore, an action is selected from this distribution and

yields the subsequent state.
Another multi-task reinforcement learning was presented
based on a hierarchical Bayesian approach. Specially, a hierar-
chical Bayesian mixture framework is used to model a Markov
Decision Process. When encountering a new Markov Decision
Process, they use the learned distribution as the prior knowledge
for model-based Bayesian learning to speed up the adaptation
to new environments. Li et al. proposed a multi-task reinforce-
ment learning for partially observable stochastic environments
by designing a regionalized policy representation to describe the
behavior of the agent in different tasks [27].
Recently, a deep multi-task and transfer reinforcement learn-
ing, called policy distillation, was presented which determines
the Q-value regression function based on the error between the
student network and the expert network. Specially, policy distilla-
tion transfers knowledge from an expert network T to a student
network S. Given the training dataset DT = {(si , qi )}Ni=0 , where
Fig. 3. Progressive neural network with three columns. si and qi denote an observation sequence and an unnormalized
Q-value vector, respectively, generated by the expert network T ,
three methods can be used for policy distillation from T to S.
Finally, the multi-task network can be obtained using the In the first method, only the action with the biggest Q-value
gradient descent strategy to train this objective. ai,best = arg max(qi ) is transferred from T to S. In this case, the
Rusu et al. proposed a method of progressive neural networks parameters of S are trained using a negative log likelihood loss to
for deep multi-task and transfer policy gradient, which is effective estimate the same action:
for continual learning [26]. Progressive neural networks achieve |D |
∑
two goals simultaneously, i.e., avoiding catastrophic forgetting L(DT , θs ) = − log P(ai = ai,best |xi , θs ). (27)
previously acquired knowledge and transferring prior knowledge i=1
to new tasks, by lateral connections to learned features. Specially,
The second method uses the mean squared error loss to train
forgetting is avoided by instantiating a column for each task
the parameters of S by transferring all Q-values from T to S:
and transferring is achieved by lateral connections to previously
learned features. Fig. 3 shows a progressive neural network with |D|
∑
three columns. L(DT , θs ) = ∥qTi − qSi ∥22 , (28)
In the first column, a deep learning model with L layers, the i=1
(1)
ith layer having an activation output hi ∈ Rni with ni denoting
where qT and qS denote the Q-value vectors of the expert network
the number of units, is trained for the first task. Specially, θ (1) is
and the student network, respectively. This method preserves all
used to represent the parameters of this deep neural network. In
the action-values into the trained student network.
order to train the second task, a new deep neural network with
The last method utilizes the Kullback–Leibler divergence to
parameters θ (2) is constructed in the second column with random
transfer the Q-values from T to S:
initialization. During training the second neural network, the
(2) qTi
parameters θ (1) are fixed and the ith layer hi obtains input from |D|
qTi soft max( )
τ
∑
(2) (1)
hi−1 and hi−1 via lateral connections. This manner generalizes to L(DT , θs ) = soft max( ) ln , (29)
τ soft max(qSi )
K tasks: i=1

where τ denotes a temperature parameter, and softmax is a mul-

(k)
∑
hi = f (Wi(k) h(k)
i−1 +
(k:j) (j)
Ui hi−1 ) , (25)
tiple regression function which can transform a D-dimensional
j<k
vector to another D-dimensional vector where each element is in
(0, 1) and the sum of all elements is 1. Increasing τ can transfer
(k)
where Wi denotes the weight matrix of the ith layer of the
(k:j) more secondary knowledge from T to S.
kth deep neural network, Ui denotes the lateral connection
from the i − 1th layer of the jth deep neural network to the ith Experiments conducted in [28] demonstrated that the second
layer of the kth deep neural network. f is an activation function method performed worst while the third method performed best.
f (x) = max(0, x). Rusu et al. showed two policy distillations, i.e., single-task
When solving a sequence of tasks, progressive neural net- policy distillation and multi-task policy distillation, as shown in
works achieves transfer learning by connecting the previously Figs. 4 and 5 [28], respectively.
learned features to the subsequent deep neural networks and Multi-task policy distillation uses N separately-trained deep
extracting valuable features. Furthermore, when constructing a Q-network single-task experts to generate inputs, task ids and
new deep neural network to solve a new task, the parameters of targets, and stores them into separate memory buffers. Id is used
other deep neural networks are preserved, which can catastrophic to identify each expert network which has a distinct output layer
forgetting. since different tasks usually have different action spaces. The stu-
To apply progressive neural networks to reinforcement learn- dent network learns from the N expert networks sequentially by
ing, each deep neural network is trained to solve a Markov selecting some samples from each replay memory every episode.
Decision Process. Specially, the lth deep neural network deter- In some real dynamic settings, the tasks of reinforcement
mines a policy π (k) (s, a) which produces probabilities over actions learning are notoriously complex, which requires multiple agents
 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507
Fig. 4. Single-task policy distillation.
Fig. 5. Multi-task policy distillation.
to accomplish together. For example, in the game Atari 2600
where multiple agents participate at the same time, only learning
policies for a single agent cannot achieve the best results. In
this case, policies learning for multiple agents making decisions
should be solved, which involves the problems of cooperation,
communication and competition between agents. While multi-
agent systems enhance the learning ability to solve complex
tasks, some novel challenges about multi-agent deep reinforce-
ment learning arise. For example, convergence and consistency of
reinforcement learning methods cannot be guaranteed in multi-
agent settings. Besides, each agent requires to follow other agents
since its action value relies on others’ actions.
Tampuu et al. generalized the deep Q-network to multi-agent
environments to explore how multiple agents to cooperate and
compete in the video game Pong [29]. This work constructs a
multi-agent deep reinforcement learning system by assigning
an independent deep Q-network for each agent for distributed
learning, which can reduce the learning difficulty and computa-
tional complexity. Various kinds of collective behavior of multi-
ple agents are investigated by adjusting the rewarding schemes,
i.e., changing the reward an agent can receive during playing in
the game Pong.
In the first case where the agent who wins obtains a positive
reward of +1 and the agent who fails obtains a negative reward
of −1, a fully competitive policy can be obtained.
In the second setting, both of the agents whenever they win
or fail are penalized with a reward of −1, resulting in a fully
cooperative policy.
When the winner obtains a reward r ∈ [−1, 1] and the
loser is penalized with a reward of −1, increasing r results in
a policy towards competition while reducing r leads to a more
cooperative policy.
In the fully competitive policy, each agent receives an im-
mediate reward either +1 for winning or −1 for losing. In this
case, each agent tries to learn for winning a match as soon
as possible, so the players become increasingly professional as
training continues.
In the fully cooperative policy, both agents obtain an imme-
diate punishment whenever they win or fail, motivating them to
505
keep the game going as long as possible. In this case, the agent
aims to keep the match for a long time by learning policies.
The multi-agent deep reinforcement learning assumes that
all the agents can fully observe the state as input, however,
this assumption is not valid in some cases. In the partially ob-
servable settings, each agent must communicate with others to
learn policies for maximum returns, which poses a challenge on
deep reinforcement learning. The deep Q-network for a single
agent to solve the partially observable Markov Decision Process
has been successfully investigated, namely deep recurrent Q-
network. The deep recurrent Q-network (DRQN) estimates Q (o, a)
with a recurrent neural network with the Q function represented
by Q (ot , ht −1 , a; θ ), where ot denotes the partial observation at
the time t, ht −1 denotes the hidden state of the long short-term
memory network, a denotes the taken action and θi denotes the
parameters of the network in the ith iteration. The recurrent
neural network for estimating Q (o, a) is able to aggregate ob-
servations over time. Therefore, the outputs of DRQN include Qt
and ht . So the most straightforward method for the multi-agent
environment is to incorporate the deep recurrent Q-network with
independent Q-learning. However, this method performs poor in
most cases.
Therefore, Foerster et al. presented deep distributed recurrent
Q-networks (DDRQN) to solve communication-based cooperative
tasks for multiple agents by making three major changes to the
above method [30].
First, at each time step, the previous last-action is fed into
the input of the agent to enable the agent to estimate action-
observation histories.
The second modification is inter-agent weight sharing. Each
agent is still trained depending on its observations. Meanwhile,
the number of parameters required to be learned is reduced
significantly via weight sharing, resulting in a great speedup for
learning.
Finally, DDRQN does not use experience replay longer.
Based on the three changes, the Q-function of DDRQN has a
t , ht −1 , m, at −1 , at ; θi ) where m denotes the index
m m
form of Q (om m
m
of the mth agent. at −1 denotes a portion of the history and
am
t denotes the taken action according to the estimator of the
Q-network.
In order to enhance the ability to solve the hierarchical tasks,
the agent is required to be equipped with both sensing function
and memory function. Therefore, the deep reinforcement learn-
ing with memory networks has been studied based on the fact
that neural networks with external memory have made a great
progress. For example, Graves et al. presented a neural turing
machine that can be trained using gradient descent [31]. The
neural turing machine is able to achieves some simple memory
and inferring functions including copying, sorting and associative
recall. Afterwards, Sukhbaatar et al. presented a memory network
that is trained end-to-end, for question-answering and language
modeling [32].
More recently, Oh et al. presented a memory-based deep rein-
forcement learning architecture by adding a network with exter-
nal memory to the traditional deep reinforcement learning mod-
els [33]. Specially, they constructed three memory-based deep
reinforcement learning models, i.e., memory Q-network (MQN),
recurrent memory Q-network (RMQN), and feedback recurrent
memory Q-network (FRMQN), as shown in Fig. 6.
Recent years have witnessed a great advance in deep rein-
forcement learning. Deep reinforcement learning combines deep
learning models for learning abstract feature representations
from high-dimensional raw state input and reinforcement learn-
ing methods that enable the agent to learn an optimal policy for
maximizing its cumulative reward. Deep reinforcement learning
has been used in solving many challenging tasks. One most
506 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507
Fig. 6. Different memory-based deep Q-network architectures.
representative application of deep reinforcement learning is high-
dimensional robot control. For example, Zhang et al. applied deep
reinforcement learning with internal memory to the complex
robotic manipulators [34] while Finn et al. focused on torque
control of high-dimensional robotic systems by combining deep
reinforcement learning and inverse optimal control [35]. Another
representative application of deep reinforcement learning is in
computer vision, including video prediction in computer games
and visual navigation [36]. In addition, Zhang et al. [37] applied
deep reinforcement learning for energy management. Besides,
deep reinforcement learning has shown super performance in
natural language processing [38]. For instance, Li et al. applied
deep policy gradient to dialog generation by modeling reward
sequences depending on informatively, coherence and ease of an-
swering [39]. Satijia et al. [40] achieves a simultaneous machine
translation system by integrating deep reinforcement learning
with neural machine translation. Other application examples of
deep reinforcement learning include parameters optimization
and game theory with deep learning models and reinforcement
learning strategies [41,42].
4. Conclusion
In this paper, we presented a smart agriculture system based
on deep reinforcement learning. Specially, the deep reinforce-
ment learning models are used to make smart decisions to adjust
the environment to adapt to the crow growth. We presented
recently developed representative deep reinforcement learning
models and algorithms.
Although deep reinforcement learning has shown a great
progress in model design and training algorithms, it cannot
achieve the human-level performance in adaptation to dynamic
environments and solving complex tasks. In the future, efforts
can be made on deep reinforcement learning to improve its
performance in the following directions. The first direction is
to design the incremental models to speed up the training for
deep reinforcement learning in dynamic environments for smart
agriculture systems. Integrating different memory units such as
long short-term memory and neural turing machine to deep
reinforcement learning to improve its performance for active rea-
soning and cognition. Another direction is to apply more effective
transfer learning methods to deep reinforcement learning for the
settings which are short of training data. Cloud computing should
be used to improve the training efficiency of large-scale deep
reinforcement learning for complex tasks. Finally, improving the
versatility of deep reinforcement learning by combining multi-
task learning and deep computation is also an important topic
for smart agriculture in the future.
Acknowledgments
This paper is supported by the National Natural Science Foun-
dation of China (Grants No. 61762068 No. 61702309), Inner
Mongolia Talent Development Funded Project, China, CERNET
Innovation Project, China (No. NGII20161209), Natural Science
Foundation of Inner Mongolia Autonomous Region of China No.
2017MS0610, Program for Young Talents of Science and Technol-
ogy in Universities of Inner Mongolia Autonomous Region, China
(No. NJYT-18-A13), Inner Mongolia Key Laboratory of economic
data analysis and mining, China.
Conflict of interest
The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared
to influence the work reported in this paper.
References
[1] N. Gondchawar, R.S. Kawitkar, IoT based smart agriculture, Int. J. Adv. Res.
Comput. Commun. Eng. 5 (6) (2016) 838–842.
[2] M. Roopaei, P. Rad, K.R. Choo, Cloud of things in smart agriculture:
Intelligent irrigation monitoring by thermal imaging, IEEE Cloud Comput.
4 (1) (2017) 10–15.
[3] Q. Zhang, L.T. Yang, Z. Chen, P. Li, F. Bu, An adaptive droupout deep
computation model for industrial IoT big data learning with crowdsourcing
to cloud computing, IEEE Trans. Ind. Inf. (2018) http://dx.doi.org/10.1109/
TII.2018.2791424.
[4] J. Gubbi, R. Buyya, S. Marusic, M. Palaniswami, Internet of things (IoT): A
vision, architectural elements, and future directions, Future Gener. Comput.
Syst. 29 (7) (2013) 1645–1660.
[5] H. Huang, M. Lin, Q. Zhang, Double-Q learning-based DVFS for multi-core
real-time systems, in: Proceedings of IEEE International Conference on
Green Computing and Communications, 2017.
[6] Z. Liu, C. Yao, H. Yu, T. Wu, Deep reinforcement learning with its
application for lung cancer detection in medical internet of things, Future
Gener. Comput. Syst. 97 (2019) 1–9.
[7] Q. Zhang, C. Bai, Z. Chen, P. Li, H. Yu, S. Wang, H. Gao, Deep learning
models for diagnosing spleen and stomach diseases in smart chinese
medicine with cloud computing, Concurr. Comput.: Pract. Exper. (2019)
http://dx.doi.org/10.1002/cpe.5252.
[8] Q. Zhang, L.T. Yang, Z. Chen, Privacy preserving deep computation model
on cloud for big data feature learning, IEEE Trans. Comput. 65 (5) (2016)
1351–1362.
[9] Q. Zhang, H. Zhong, L.T. Yang, Z. Chen, F. Bu, PPHOCFS: Privacy preserving
high-order CFS algorithm on the cloud for clustering multimedia data, ACM
Trans. Multimedia Comput. Commun. Appl. 12 (4s) (2016) 66.
[10] V. Mnih, K. Kavukcuoglu, D. Silver, A.A. Rusu, J. Veness, M.G. Bellemare,
A. Graves, M. Riedmiller, A.K. Fidjeland, G. Ostrovski, S. Peterson, C.
Beasttie, A. Sadik, L. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg,
D. Hassabis, Human-level control through deep reinforcement learning,
Nature 518 (7540) (2015) 529–533.
[11] Q. Zhang, L.T. Yang, Z. Chen, P. Li, Incremental deep computation model
for wireless big data feature learning, IEEE Trans. Big Data (2019) http:
//dx.doi.org/10.1109/TBDATA.2019.2903092.
[12] M.L. Littman, Reinforcement learning improves behaviour from evaluative
feedback, Nature 521 (7553) (2015) 445–451.
[13] Q. Zhang, C. Bai, Z. Chen, P. Li, S. Wang, H. Gao, Smart chinese medicine
for hypertension treatment with a deep learning model, J. Netw. Comput.
Appl. 129 (2019) 1–8.
[14] R.S. Sutton, D. Mcallester, S. Singh, Y. Mansour, Policy gradient methods
for reinforcement learning with function approximation, in: Proceedings of
Advances on Neural Information Processing Systems, 2000, pp. 1057–1063.
[15] R.J. Willaims, Simple statistical gradient-following algorithms for connec-
tionist reinforcement learning, Mach. Learn. 8 (1992) 229–256.
[16] J. Baxter, P.L. Bartlett, Infinite-horizon policy-gradient estimation, J.
Artificial Intelligence Res. 15 (2001) 319–350.
[17] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, M. Riedmiller, Determin-
istic policy gradient algorithms, in: Proceedings of International Conference
on Machine Learning, 2014, pp. 387–395.
[18] S. Levine, C. Finn, T. Darrell, P. Abbeel, End-to-end training of deep
visuomotor policies, J. Mach. Learn. Res. 17 (2016) 1–40.
[19] S. Bhatnagar, R.S. Sutton, M. Ghavamzadeh, M. Lee, Incremental natural
actor-critic algorithms, in: Proceedings of Advances in Neural Information
Processing Systems, 2008, pp. 105–112.
[20] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D.
Wierstra, Continuous Control with Deep Reinforcement Learning, (2015),
arXiv:1509.02971.
[21] N. Heess, G. Wayne, D. Silver, T. Lillicrap, Y. Tassa, T. Erez, Learning
continuous control policies by stochastic value gradients, in: Proceedings of
Advances in Neural Information Processing Systems, 2015, pp. 2944–2952.
 F. Bu and X. Wang / Future Generation Computer Systems 99 (2019) 500–507
[22] D. Balduzzi, M. Ghifary, Compatible Value Gradients for Reinforcement
Learning of Continuous Deep Policies, (2015), arXiv:1509.03005.
[23] V. Mnih, A.P. Badia, M. Mirza, A. Graves, T. Harley, T.P. Lillicrap, D. Silver,
K. Kavukcuoglu, Asynchronous methods for deep reinforcement learning,
in: Proceedings of International Conference on Machine Learning, 2016,
pp. 1928–1937.
[24] D. Calandriello, A. lazaric, M. Restelli, Sparse multi-task reinforcement
learning, in: Proceedings of Advances in Neural Information Processing
Systems, 2014, pp. 819–827.
[25] E. Parisotto, J. Ba, R. Salakhutdinov, Actor-mimic deep multitask and trans-
fer reinforcement learning, in: Proceedings of International Conference on
Learning Representations, 2016, pp. 156–171.
[26] A.A. Rusu, N.C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K.
Kavukcuoglu, R. Pascanu, R. Hadsell, Progressive Neural Networks, (2016),
arXiv:1606.04671.
[27] H. Li, X. Liao, L. Carin, Multi-task reinforcement learning in par-
tially observable stochastic environment, J. Mach. Learn. Res. 10 (2009)
1131–1186.
[28] A.A. Rusu, S.G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R.
Pascanu, V. Mnih, K. Kavukcuoglu, R. Hadsell, Policy Distrillation, (2015),
arXiv:1511.06295.
[29] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J.
Aru, R. Vicente, Multiagent Cooperation and Competition with Deep
Reinforcement Learning, (2015), arXiv:1511.08779.
[30] J.N. Foerster, Y.M. Assael, N. de Freitas, S. Whiteson, Learning to Communi-
cate to Solve Riddles with Deep Distributed Recurrent Q-Networks, (2016),
arXiv:1602.02672.
[31] A. Graves, G. Wayne, I. Danihelka, Neural Turing Machines, (2014), arXiv:
1410.5401.
[32] S. Sukhbaatar, A. Szlam, J. Weston, R. Fergus, End-to-end memory net-
works, in: Proceedins of Advances in Neural Information Processing
Systems, 2015, pp. 2440–2448.
[33] J. Oh, V. Chockalingam, S. Singh, H. Lee, Control of memory, active percep-
tion, and action in minecraft, in: Proceedings of International Conference
on Machine Learning, 2016, pp. 2790–2799.
[34] M. Zhang, Z. McCarthy, C. Finn, S. Levine, P. Abbeel, Learning deep neural
network policies with continuous memory states, in: Proceedings of IEEE
International Conference on Robotics and Automation, 2016, pp. 520–527.
507
[35] C. Finn, S. Levine, P. Abbeel, Guided cost learning: Deep inverse optimal
control via policy optimization, in: Proceedings of International Conference
on Machine Learning, 2016, pp. 49–58.
[36] Y. Rao, J. Lu, J. Zhou, Attention-aware deep reinforcement learning for
video face recognition, in: Proceedings of IEEE International Conference
on Computer Vision, 2017, pp. 3951–3960.
[37] Q. Zhang, M. Lin, L.T. Yang, Z. Chen, S.U. Khan, P. Li, A double deep
Q-learning model for energy-efficient edge scheduling, IEEE Trans. Serv.
Comput. (2018) http://dx.doi.org/10.1109/TSC.2018.2867482.
[38] A. Das, S. Kottur, J.M.F. Moura, S. Lee, D. Batra, Learning cooperative visual
dialog agents with deep reinforcement learning, in: Proceedings of IEEE
International Conference on Computer Vision, 2017, pp. 2970–2979.
[39] J. Li, W. Monroe, A. Ritter, M. Galley, J. Gao, D. Jurafsky, Deep reinforcement
learning for dialogue generation, in: Proceedings of the Conference on
Empirical Methods in Natural Language Processing, 2016, pp. 1192–1202.
[40] H. Satija, M. Mcgill, J. Pineau, Simultaneous machine translation using deep
reinforcement learning, in: Proceedings of the Workshops of International
Conference on Machine Learning, 2016, pp. 110–119.
[41] Q. Zhang, L.T. Yang, Z. Chen, P. Li, Dependable deep computation model
for feature learning on big data in cyber-physical systems, ACM Trans.
Cyber-Phys. Syst. 3 (1) (2018) 11.
[42] Q. Zhang, L.T. Yang, Z. Yan, Z. Chen, P. Li, An efficient deep learning model
to predict cloud workload for industry informatics, IEEE Trans. Ind. Inf. 14
(7) (2018) 3170–3178.
Fanyu Bu received the BSc degree in computer science
from Inner Mongolia Agricultural University, Hohhot,
China, in 2003, and the MSc degree in computer appli-
cation from Inner Mongolia University, Hohhot, China,
in 2009. He got the Ph.D degree in computer applica-
tion technology from Dalian University of Technology,
Dalian, China, in 2018. He is currently an assistant
professor at Department of Computer and Information
Management at Inner Mongolia University of Finance
and Economics, China. His research interests include
Big Data, Smart agriculture and Internet of Things.