Definition of Markov Analysis

Markov Analysis is a method used to forecast the value of a variable whose future value is
influenced only by its current position or state, not by any prior activity that led the variable to its
current position or state. In essence, it forecasts the activity of a random variable based solely
upon the current circumstances surrounding the random variable.

The technique is named after Russian mathematician Andrei Andreyevich Markov, who
pioneered the study of stochastic processes, which are processes that involve the operation of
chance. He first used the process to predict the behavior of gas particles trapped in an enclosed
container. The Markov Analysis process is a method for forecasting random variables, and is
often used for predicting behaviors and decisions within large groups of people.

Breaking-Down Markov Analysis

The Markov Analysis process involves defining the likelihood of a future action given the
current state of a variable. Once the probabilities of future actions at each state are defined, a
decision tree can be drawn and the probability of a result can be calculated given the current state
of a variable. Markov Analysis has a number of applications in the business world. It is often
used to predict the number of defective pieces that will come off of an assembly line given the
operating status of the machines on the line.

It can also be used to predict the proportion of a company's accounts receivables that will
become bad debts. Certain stock price and option price forecasting methods also incorporate
Markov Analysis. Lastly, companies often use it to forecast future brand loyalty of current
customers and the outcome of these consumer decisions on a company's market share.

https://www.investopedia.com/terms/m/markov-analysis.asp
Introduction to Markov Chains

Markov Chains are actually extremely intuitive. Formally, they are examples of Stochastic
Processes, or random variables that evolve over time. You can begin to visualize a Markov
Chain as a random process bouncing between different states. Here is a basic but classic
example of what a Markov chain can actually look like:

It’s probably not hard to tell what’s going on here, even without thorough explanation. This
Markov Chain is describing the random process of weather over multiple days. In this simple
example, we’re assuming that it can either be sunny or rainy in a single day. If today is sunny,
then tomorrow there is a .2 probability that it is rainy, and a .8 probability that it is sunny
again (shown here by the arrows leaving from the ‘Sunny’ circle). If it’s rainy, then tomorrow
there is a .3 probability that it rainy again on the next day and a .7 probability that it is sunny
on the next day. You can envision a particle bouncing around this chain, moving once each
day according to the probabilities drawn here.

Makes sense, right? Let’s talk about some simple terminology. We’re going to call the circles
- here we have one that says ‘Sunny’ and one that says ‘Rainy’ - nodes (also often
called states), and the arrows connecting them edges. Each edge has a probability associated
with it; as specified above, this is the probability of ‘following’ that edge if you start at that
node.
Note that the probabilities of all the edges leaving out of a specific node must sum to 1. For
example, if it’s Sunny, we have a .8 probability and a .2 probability of it being Sunny and
Rainy tomorrow, respectively, and .8+.2=1. This makes sense; if we start at a node, we have
to go somewhere on the next step, so the probabilities of all of the edges leaving the nodes
must add up to 1. If you had a node where the edge probabilities summed to .9, for example,
this wouldn’t make much sense. Where would the particle move 10% of the time?
The next step is to define the Transition Matrix, which we usually notate as Q. This is simply
the matrix that contains the probabilities of, you guessed it, transferring between nodes. In
this case, the transition matrix looks like:

These are also intuitive to read: the {ith,jth} entry (the value in the ith row and jth column)
marks the probability of going from State i to State j. In this case, we named the states instead
of numbering them; here, then, we associate the first row and first column with the ‘Sunny’
state, and the second row and column with the ‘Rainy’ state. For example, the probability of
going from Sunny to Rainy is coded here in the first row and second column (.2).

It’s important to note, again, that the rows of these transition matrices must sum to 1. That’s
because from a certain state, you must go somewhere. Again, if these probabilities added up
to something like .6, you wouldn’t know where the state traveled to 40% of the time. These
transition matrices will also always be square (i.e., same number of rows as columns) since
we want to keep track of the probability of going from every state to every other state, and
they will always have the same number of rows (and same number of columns) as the number
of states in the chain.
The Markov Property

All of this is well and good, but we still haven’t gotten to what really makes a Markov
Chain Markov. Formally, a Markov Chain must have the ‘Markov Property’. This is somewhat
of a subtle characteristic, and it’s important to understand before we dive deeper into Markov
Chains. Take this Markov Chain, for example, where the states are labeled more generically as 1,
2 and 3.

Take a second to convince yourself that if you sum transition probabilities associated with the
edges leading out of any particular node, you get 1. See if you can write out the transition
matrix.

Now, the essence of the Markov Property is that the future only depends on the immediate
past. That is, you only have to condition on one previous step to get all of the relevant
information about predicting the future; conditioning on steps further in the past gives no
valuable information.

Think about this concept in the context of this specific chain. Say we arbitrarily start at node
2; as the chain is defined above, we have a .1 probability of going to State 1. Now imagine
that we ‘run’ the chain for t periods (allow the ‘random particle’ to bounce around in the
chain t times), and say that each period the particle happens to take the ‘self-referencing’ loop
back to State 2 (the probability of this happening would be .9t, since each step has
probability .9 and we use the multiplication rule, but we’re not worried about that at the

moment). So, given that we’re at State 2 at time t, what’s the probability that we’re in State 1
at time t+1? Well, it’s intuitively still .1, just like it was the first time we started at State 2,
just like, in fact, it is every single time we are in State 2.

The point is, it doesn’t matter how you got there (to State 2). If we’re thinking about the
probability of transitioning to State 1, it only matters that we’re in State 2 at the previous step.
We don’t have to look any further back in history, and considering the behavior of the chain
further in the past will not change the probability that we transition to State 1 in the next step.

Let’s pin this down with some notation. Let X be the stochastic process described by this
chain (remember, a stochastic process is just a random variable that evolves through
time). X can clearly take on the values 1, 2 or 3 (the different possible states). Let Xt be the
value of the process (what state we are in) at time t (after we let the ‘random particle’ move
states t times). Then:

The left hand side of this equation is giving the probability of being in State i at
period t+1, given that we were in State j at period t. In our case above, the probability of being
at State 1 at time t+1 given that we are in State 2 at time t is .1. Then, the right hand side of
the equation is the probability of being at State i at time t+1 given the entire past history of the
chain, or the location of the particle in every single State (here, the State j at time t, and all the
way down to x0, which is just arbitrary notation for some constant that X0 takes on at time 0).

The Markov Property states that these sides are equal; that is, knowing where you were in the
previous period makes the rest of the chain history irrelevant. If we know Xt = j, then the rest
of the history doesn’t add any information for predicting Xt+1; that’s why the two sides are
equal.
Let’s go through this with some thought examples. Imagine that you have a stochastic process
that models the cumulative sum of fair, six-sided die rolls, which we’ll notate as X. So, you
roll the die once and maybe get a 3; this means X1=3. You roll the die again and get 5; take
the sum of the two rolls and see that X2=5+3=8.

Could we write this as a Markov Chain? That is, does this satisfy the Markov Property? Let’s
think about this. Say that we are interested in the probability that X5 is equal to 15, and
imagine that we get to observe that X4 was equal to 12. Well, if we know the value of X4,
does knowing X3, X2 and X1 matter.

They don’t. It only matters that we got to (and are currently sitting at) the value 12; it does not
matter how we got there. The conditional probability (conditioning on X4=12) that X5 is 15
is 1/6, since this is the same probability as rolling a 3 on your next roll (which means going
from 12 to 15), regardless of how we got to X4=12. So, using this kind of ‘story’ or ‘heuristic’
proof, this process is Markovian.

https://bookdown.org/probability/beta/markov-chains.html