Equivalent States

Description

Returns a set of equivalent states and actions

Usage

equivalent_states(state)

equivalent_states_actions(state, action)

Arguments

Value

equivalent_states returns a list of state matrices

equivalent_states_actions returns a list of two lists: states, the set of equivalent states and actions, the set of equivalent actions

Hash Operations for Single State

Description

Hash Operations for Single State

Usage

haskey(x, ...)

## S3 method for class 'xhash'
x[state, ...]

## S3 replacement method for class 'xhash'
x[state, ...] <- value

## S3 method for class 'xhash'
haskey(x, state, ...)

Arguments

Value

• haskey returns a logical

• `[` returns a reference to the object

• `[<-` returns a value

Play Tic-Tac-Toe Game

Description

Start tic-tac-toe game on the console.

Usage

ttt(player1 = ttt_human(), player2 = ttt_human(), sleep = 0.5)

Arguments

Details

At default, the game is played between humans. Set player1 or player2 to ttt_ai() to play against an AI player. The strength of the AI can be adjusted by passing the level argument (0 (weekest) to 5 (strongest)) to the ttt_ai function.

To input your move, type the position like "a1". Only two-length string consisting of an alphabet and a digit is accepted. Type "exit" to finish the game.

You may set both player1 and player2 as AI players. In this case, the game transition is displayed on the console without human inputs. For conducting a large sized simulations of games between AIs, refer to ttt_simulate

See Also

ttt_ai, ttt_human, ttt_simulate

Examples

## Not run: 
ttt(ttt_human(), ttt_random())

## End(Not run)

Tic-Tac-Toe AI Player

Description

Create an AI tic-tac-toe game player

Usage

ttt_ai(name = "ttt AI", level = 0L)

ttt_random(name = "random AI")

Arguments

Details

level argument controls the strength of AI, from 0 (weekest) to 5 (strongest). ttt_random is an alias of ttt_ai(level = 0).

A ttt_ai object has the getmove function, which takes ttt_game object and returns a move considered as optimal. getmove function is designed to take a ttt_game object and returns a move using the policy function.

The object has the value and policy functions. The value function maps a game state to the evaluation from the first player's viewpoint. The policy function maps a game state to a set of optimal moves in light of the value evaluation. The functions have been trained through the Q-learning.

Value

ttt_ai object

Tic-Tac-Toe Game

Description

Object that encapsulates a tic-tac-toe game.

Usage

ttt_game()

Value

ttt_game object

Examples

x <- ttt_game()
x$play(3)
x$play(5)
x$show_board()

Human Tic-Tac-Toe Player

Description

Create an human tic-tac-toe player

Usage

ttt_human(name = "no name")

Arguments

Value

ttt_human object

Q-Learning for Training Tic-Tac-Toe AI

Description

Train a tic-tac-toe AI through Q-learning

Usage

ttt_qlearn(player, N = 1000L, epsilon = 0.1, alpha = 0.8, gamma = 0.99,
  simulate = TRUE, sim_every = 250L, N_sim = 1000L, verbose = TRUE)

Arguments

Details

This function implements Q-learning to train a tic-tac-toe AI player. It is designed to train one AI player, which plays against itself to update its value and policy functions.

The employed algorithm is Q-learning with epsilon greedy. For each state s, the player updates its value evaluation by

V(s) = (1-\alpha) V(s) + \alpha \gamma max_s' V(s')

if it is the first player's turn. If it is the other player's turn, replace max by min. Note that s' spans all possible states you can reach from s. The policy function is also updated analogously, that is, the set of actions to reach s' that maximizes V(s'). The parameter \alpha controls the learning rate, and gamma is the discount factor (earlier win is better than later).

Then the player chooses the next action by \epsilon-greedy method; Follow its policy with probability 1-\epsilon, and choose random action with probability \epsilon. \epsilon controls the ratio of explorative moves.

At the end of a game, the player sets the value of the final state either to 100 (if the first player wins), -100 (if the second player wins), or 0 (if draw).

This learning process is repeated for N training games. When simulate is set true, simulation is conducted after sim_every training games. This would be usefule for observing the progress of training. In general, as the AI gets smarter, the game tends to result in draw more.

See Sutton and Barto (1998) for more about the Q-learning.

Value

data.frame of simulation outcomes, if any

References

Sutton, Richard S and Barto, Andrew G. Reinforcement Learning: An Introduction. The MIT Press (1998)

Examples

p <- ttt_ai()
o <- ttt_qlearn(p, N = 200)

Simulate Tic-Tac-Toe Games between AIs

Description

Simulate Tic-Tac-Toe Games between AIs

Usage

ttt_simulate(player1, player2 = player1, N = 1000L, verbose = TRUE,
  showboard = FALSE, pauseif = integer(0))

Arguments

Value

integer vector of simulation outcomes

Examples

res <- ttt_simulate(ttt_ai(), ttt_ai())
prop.table(table(res))

Vectorized Hash Operations

Description

Vectorized Hash Operations

Usage

haskeys(x, ...)

setvalues(x, ...)

getvalues(x, ...)

## S3 method for class 'xhash'
getvalues(x, states, ...)

## S3 method for class 'xhash'
setvalues(x, states, values, ...)

## S3 method for class 'xhash'
haskeys(x, states, ...)

Arguments

Value

• haskeys returns a logical vector

• setvalues returns a reference to the object

• getvalues returns a list of values

Create Hash Table for Generic Keys

Description

Create Hash Table for Generic Keys

Usage

xhash(convfunc = function(state, ...) state, convfunc_vec = function(states,
  ...) unlist(Map(convfunc, states, ...)), default_value = NULL)

Arguments

Value

xhash object