Getting Started¶
You can interact with ColosseumRL environments in two ways. You can spawn a local environment and interact with it in a single process, or you can use the matchmaking API to play games between multiple processes or computers in a network.
Playing Games Locally¶
Local environments subclass the BaseEnvironment class, which you can instantiate and perform actions in on behalf of all players.
Below is a minimal working example to play a local game of Blokus in a single process:
# import BlokusEnvironment here
from colosseumrl.envs.blokus.BlokusEnvironment import BlokusEnvironment, display_board, start_gui, terminate_gui
import random
def rl_agent_choose_action(valid_actions, observation):
# dummy function, chooses a random action
return random.choice(valid_actions)
# Instantiate a blokus environment, you only need to do this once.
env = BlokusEnvironment()
# start a new game, we get back a state, and a list of currently acting players.
# In blokus, this current_players list always contains only one player
# since only one player ever is allowed to go at a time.
current_state, current_players = env.new_state()
start_gui()
display_board(current_state, current_players[0])
# Main game loop
while True:
# Get a list of valid actions. In this API, actions are pass around as strings.
# The action_to_string and string_to_action methods can be used to parse
#or create actions from components.
valid_actions = env.valid_actions(
state=current_state, player=current_players[0])
# States are only for the environment to use. We need to convert a state into
# an observation for an RL agent to look at it.
observation = env.state_to_observation(
state=current_state, player=current_players[0])
# Observations are always from a specific player's perspective.
# Each player views observations as if they are player 0 (out of 0,1,2,3)
# in the game.
# This way, you only ever have to train a agent to choose actions
# that make player 0 win the game.
# Our "RL agent" chooses and action base on the observation and valid actions
action = rl_agent_choose_action(valid_actions, observation)
# Here we advance the game by one step.
next_state, next_players, rewards, terminal, winners = env.next_state(
state=current_state,players=current_players,actions=[action])
# The BlokusEnv instance, env, doesn't keep track of the game's state. That's up to you to keep track of:
current_state = next_state
# next_players contains the next player who's turn it is now (again, it's a list with one player's number).
current_players = next_players
display_board(current_state, current_players[0])
if terminal:
# If the game is over, break out of the loop and print which player(s) won.
print("Winners: {}".format(winners))
break
terminate_gui()
Playing Games Online¶
Server¶
In order to play a game between agents on a network, you must first launch a server where the game will be hosted. Clients will be able to connect to this match server and play on its environment remotely. You can host a single episode of a multiplayer Tron game using the following script
python -m colosseumrl.match_server -e tron
where the -e flag specifies the environment to host. Run with the -h flag for more details. The server will automatically start and listen on all interfaced. Furthermore, for stability, the server has several timeouts built in. If not enough clients connect within 30 seconds, the server will timeout and close the game.
Client¶
In order to play on these remote servers, you must use a slightly different interface in the form of client environemnts.
These implement an OpenAI Gym like api to simplify communcation with a client. See colosseumrl.ClientEnvironment for more details on the specifics of this client environment. There is a basic, all-purpose client environment that will work with any server environment, but most games also feature a customized client environment that includes other helper functions. See colosseumrl.envs.TronGridClientEnvironment.
In order to create a client, you must create a function that receieve the client environment as the first parameter. Other parameters are optional and may be whatever you like. Afterwards, you may use colosseumrl.RLApp.create_rl_agent() in order to transform your function into a callable RLApp function. Aternatively, you may use colosseumrl.RLApp.launch_rl_agent() to create and launch the RLApp in one command. An example of a randomly moving tron client is provided.
from colosseumrl.RLApp import create_rl_agent
from colosseumrl.envs.tron import TronGridClientEnvironment
def tron_client(env: TronGridClientEnvironment, username: str):
logger.debug("Connecting to game server and waiting for game to start")
player_num = env.connect(username)
logger.debug("Player number: {}".format(player_num))
logger.debug("First observation: {}".format(env.wait_for_turn()))
logger.info("Game started...")
while True:
action = choice(env.valid_action())
new_obs, reward, terminal, winners = env.step(action)
logger.debug("Took step with action {}, got: {}".format(action, (new_obs, reward, terminal, winners)))
if terminal:
logger.info("Game is over. Players {} won".format(winners))
logger.info("Final observation: {}".format(new_obs))
break
agent = create_rl_agent(agent_fn=tron_client,
host="localhost",
port=7777,
auth_key='',
client_environment=TronGridClientEnvironment,
server_environment=TronGridClientEnvironment,
time_out=10)
agent(username="tester3000")
Matchmaking¶
A much more reliable way to start an online game is to use the matchmaking system. This is a secondary server that will remain open until you manually close it. The matchmaking server is responsible for waiting for enough clients to connect, launching a game server for the clients, ensuring that only the assigned clients actually connect to it, and clearning up the game server when finished or in case of a crash.
Another feature of the matchmaking system is a consistent database that stores all players to ever join the server and their ranking. We use a primative password system just to prevent accidentally overriding some elses ranking. We use the trueskill python library for keeping track of N-player free-for-all ranking in a fair and consistent way. All ranking information is stored in an sqlite file in the working directory.
We use GRPC for the low level communcation with the matchmaking system before transitioning to the full game protocol once the game servers have been created.
You can launch a matchmaking server with the following module script.
python -m colosseumrl.matchmaking_server
Again passing the the help -h flag will list all command line parameters. Note that, by default, the matchmaking server will only listed on localhost. If you want it to listen on all interfaces make sure to call it with the --hostname 0.0.0.0 option.
In order to connect to a matchmaking server, use colosseumrl.matchmaking.MatchmakingClient.request_game(), in order to recieve a game server and authentication key.
Example Matchmaking with Tron¶
In this example, we will create an example tron server and clients to demonstrate the process of launching matchmaking servers and clients to join them.
First we will launch a Tron matchmaking server with a board size of 50x50 that will listen on all hosts.
python -m colosseumrl.matchmaking_server -e tron -c 50 --hostname 0.0.0.0
Next, we need to create 4 clients that will connect to the matchmaking server and wait for a game to be created. In this case, we are running the agents on the same machine as the server.
python -m colosseumrl.examples.tron_client --host localhost
The code for the tron client is below.
import argparse
from random import choice, randint
from colosseumrl.matchmaking import request_game, GameResponse
from colosseumrl.RLApp import create_rl_agent
from colosseumrl.envs.tron import TronGridClientEnvironment
from colosseumrl.envs.tron import TronGridEnvironment
from colosseumrl.rl_logging import init_logging
logger = init_logging()
def tron_client(env: TronGridClientEnvironment, username: str):
""" Our client function for the random tron client.
Parameters
----------
env : TronGridClientEnvironment
The client environment that we will interact with for this agent.
username : str
Our desired username.
"""
# Connect to the game server and wait for the game to begin.
# We run env.connect once we have initialized ourselves and we are ready to join the game.
player_num = env.connect(username)
logger.debug("Player number: {}".format(player_num))
# Next we run env.wait_for_turn() to wait for our first real observation
env.wait_for_turn()
logger.info("Game started...")
# Keep executing moves until the game is over
terminal = False
while not terminal:
# See if there is a wall in front of us, if there is, then we will turn in a random direction.
n_loc, n_obj = env.next_location()
if n_obj == 0:
action = 'forward'
else:
action = choice(['left', 'right'])
# We use env.step in order to execute an action and wait until it is our turn again.
# This function will block while the action is executed and will return the next observation that belongs to us
new_obs, reward, terminal, winners = env.step(action)
logger.debug("Took step with action {}, got: {}".format(action, (new_obs, reward, terminal, winners)))
# Once the game is over, we print out the results and close the agent.
logger.info("Game is over. Players {} won".format(winners))
logger.info("Final observation: {}".format(new_obs))
if __name__ == '__main__':
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("--host", "-s", type=str, default="localhost",
help="Hostname of the matchmaking server.")
parser.add_argument("--port", "-p", type=int, default=50051,
help="Port the matchmaking server is running on.")
parser.add_argument("--username", "-u", type=str, default="",
help="Desired username to use for your connection. By default it will generate a random one.")
logger.debug("Connecting to matchmaking server. Waiting for a game to be created.")
args = parser.parse_args()
if args.username == "":
username = "Tester_{}".format(randint(0, 1000))
else:
username = args.username
# We use request game to connect to the matchmaking server and await a game assigment.
game: GameResponse = request_game(args.host, args.port, username)
logger.debug("Game has been created. Playing as {}".format(username))
logger.debug("Current Ranking: {}".format(game.ranking))
# Once we have been assigned a game server, we launch an RLApp agent and begin our computation
agent = create_rl_agent(agent_fn=tron_client,
host=game.host,
port=game.port,
auth_key=game.token,
client_environment=TronGridClientEnvironment,
server_environment=TronGridEnvironment)
agent(username)