literature

Chapter 5: Arch and Arg

Deviation Actions

jaqbenny's avatar
By
Published:
244 Views

Literature Text

Chapter 5: Architecture decisions

Now that I've got the basic design concept in my head I need to start really considering how to build a frame work that I can work with later. I'm personally found of the concept in agile of not making architecture decisions until the are nessisary, at least in this case. I'm not proficient in this type of system, there is no way I would be able to predict what all will be nessisary so I can't make a lot of architecture decisions, but the ones I am capable of doing now I will.

the first is I'm breaking from the original command structure that I came up with in chapter three and addopting a more traditional kind which will allow for more complex commands while also still being easier to implement. I'm going to use the argparse library from python 2.7x to parse my commands for me in a very easy and extensible way.

and really that's all I've got for now. I'm going to list here though the requirements for 0.2 version of the server.

Ceranubis Server 0.2:
adoption of argparse as the argument parser library.
formal introduction of the concept of projects:
be aware of a project
know it's program files
know it's data files
be able to send and recieve them


a notable thing that is lacking from this is the server's ability to have server side tasks to a project. most importantly is that of work items. I'm still going to implement that system, but I don't feel like it is nessisary for this version of the server. these two main items will take a fair amount of time to accomplish I'm certain.

argparse

I recycled only the first 25 lines of the previous server version. These are the lines that have the introductory tag, and prepare the socket etc. for now I am sticking with only 1 possible connection at a time. I will figure out more connections later.

For now though my chief concern is with the argparse library. when I first started looking to this i was hesitant as the library is used to parse commands from command line chiefly, but if you pass argument string to it then it will parse that string rather than sys.argv. which means I can indeed make use of this.

So the first step is to declare an argument parser object:
>>>parser = argparse.ArgumentParser(description='a distription goes here!')

with that done next you will add arguments to the parser:
>>> parser.add_argument('integers', metavar='N', type=int, nargs='+',
...                     help='an integer for the accumulator')
>>> parser.add_argument('--sum', dest='accumulate', action='store_const',
...                     const=sum, default=max,
...                     help='sum the integers (default: find the max)')


then finally you can parse your arguments:
>>> parser.parse_args(['--sum', '7', '-1', '42'])
Namespace(accumulate=<built-in function sum>, integers=[7, -1, 42])

The example doesn't have a variable to catch the return value of the parser. I would. lets say "foo". now to make use of the values in this namespace you simply treat it like an object: foo.integers returns [7, -1, 42] and accumulate takes an list of integers and returns the some of the list.

now, the important peice to understand, and the peice that will take more than 5 lines to do so is the arguments for add_argument. they are summarized by the documentaiton here like so:
    * name or flags - Either a name or a list of option strings, e.g. foo or -f, --foo
    * action - The basic type of action to be taken when this argument is encountered at the command-line.
    * nargs - The number of command-line arguments that should be consumed.
    * const - A constant value required by some action and nargs selections.
    * default - The value produced if the argument is absent from the command-line.
    * type - The type to which the command-line arg should be converted.
    * choices - A container of the allowable values for the argument.
    * required - Whether or not the command-line option may be omitted (optionals only).
    * help - A brief description of what the argument does.
    * metavar - A name for the argument in usage messages.
    * dest - The name of the attribute to be added to the object returned by parse_args().

and now I will give a little more information.

Name:

these are the flags of the arguments to be parsed. there are two kinds. the first is optional or non-positional arguments. these are the ones that are preceded by a - for a single letter or -- for a word. then there are non optional or positional arguments which have no flag before them, they are just simply a word. optional arguments may or may not be fulfilled by the command, the positional arguments however must be fulfilled by the command or they through a error.

optional flags have to be declared in the arguments when parsing also. meaning that if you have the arguments in the parser ['-f','-foo','bar'] and you passed 'fiddle sticks'.split() it will throw an error. (and if you are debuging in command line it will close your python session . >:( grrr!) also, positional arguments do not get called, they must always be satisfied and if they are not, an error is thrown. if you provide to many arguments an error is throw. this requires a bit of exaction. but this feature of hte arg parser is probably the most important. I could achieve all I need with just this, but lets see what else this class has up it's sleeve.

Action:

This is what the arg parser does witht the args you send to it, the default it to just simple store the argument in it's string form foo='bar' but there are other things that can be done. store_const for example can be used to call a positional argument with no argument. so with foo storing a const you could call '--foo' and it would return foo=<value of constant/> where that value can be anything you desire, technically even a function call. but if you make it a function call, say for example time.time(), when you add the arguement the function is run and it's return value is what is saved. e.g. time.time() will not run every time you throw that at the arg parser with the store_const function. but there are other ways to get this effect later.

Nargs

this one is simple. number of arguments it argument wants:
nargs=1: --foo bar
nargs=2: --foo bar val

const

This is the constant to be stored by the store_const action.

default

this is similar to const but the difference is that it is only given to the arguement should nothing have been given to the argument in the first place. if you pass it argparse.SUPPRESS it will keep the argument from throwing an error should there be no default value and no value passed to it.

type

the type that you want the argument to be returning, options are basically float, int, and file with string obviously being the default. if you say that a argument is a file it will immedeatly try to open that file with 'r'.

choices

restricted set of arguments. basically it makes it so you can only send certain commands to a argument so that you don't have to deal with edge cases.

required

turns an optional argument into a non-optional argument, but they are are still non-positional.

help

explaination of what an argument does.

metavar and dest

these attributes deal with the help print out. I'm not really that interested in such right now.

those are the attributes to the argument parser. now we can start working on some arguing! but one more little note: every time you call arg_parse all arguments are run and therefore the namespace that it returns will have every arguement. if the arguement did recieve anything and didn't cause an error it will just sit there with foo=None. the way around this if it is a problem is to set default to argparse.SUPPRESS. though you almost would rather not ahve it that way as you can't call ret.foo if you suppress it. if you call ret.foo with out supresssing it then you will get back None, but it won't throw and error.

now with the exploratory research done, i press on to developing how the system will work. now previously I had designed it so that you have command-args so i think I will stick with that. the first argument, a positional argument, will be the command to be called: getWorkItem, returnData etc. and than optional arguments will follow.

one command im particular i'm going to want is -l which will mean 'long'. as i'm not transfering more that 4 kb at a time and may very well need to make transfers into mb, the ability to send a communication that is longer than 4096b is nessisary.

to handle the arguements after all the data has been recieved I will use a dictionary. this will allow me to not have to bother with some sort of switching statement. when i get cmd and data back from the parser I can simply call dic[cmd](data) and it will run the command that has the name cmd is and give it the data that is in data that it needs. this also means all functions must be made to take only a string as their only argument.

now the next thing I need to do is th8ink about what commands to put in:

Enroll:

first command i can think of is 'enroll' this enrolls the client into the system if it doesn't have a config file.

Login:

this is a fairly obvious command. you need the ability to tell the server who you are whether you are an admin or a computer, the server won't know what to do with you untill this step happens.

Logout:

Again, fairly obvious, but very nessesary. it closes the connection between a client and the server.

Get Work Item:

As this system is build around the concept of work items it would be jolly good if the client machinces could ask for a work item now wouldn't it be?

Get Data:

again, data centered system. The clients are going to be requesting data left and right and this command is how they will do it.

Ret Data:

this is the mirrored function of Get Data, it returns data from the client to the server. these will be just as important as their sister.

Complete work item:

a flag sent back to the server to let it know that it has completed a work item. it sends the item's ID with the command. The reason the server can't assume is that the computer may fail and loose a work item, and i'm designing this system so that each client can take more than one work item at a time.

Status:

an optional command that will allow the server to know where a client sits on a certain task.

Admin commands:
these are command to be used only for teh command line interface to the server for administartion purposes:

Get enrollment:
returns the currently enrolled computers

get active:
returns computers that are giving regular status updates.

list projects:
list current projects in progress

shutdown:
stops the server for graceful shutdown

broadshutdown:
requests all client machines to shutdown their clients.

getWait:
returns the average amount of time spent waiting by all clients for the server to react.



The things I had to do to get the server are a little amusing. but all is well, the system runs just fine. you can now connect to the server and run any number of commands. it is very easy to add to these commands and I plan on having a system for custum extensions. but this is it. the 0.2 version of the server. Uh RAH!

Code:
##############################################################################
## Program: Ceranbuis Server
## Purpose: Distributed computing system
## Programmer: Toben "Littlefoot" Archer
## Version: 0.2
## Date: 12/3-7/2010
########################

import socket
import argparse

BUFFER_SIZE = 16

def getCmds():
return {'enroll':enroll,'login':login,'logout':logout,'gWU':getWorkItem,'rWU':completeWorkItem,'gData':getData,'rdata':returnData,'status':status,'gEn':getEnrollment,'gAc':getActive,'lsp':listProjects,'shutdown':shutdownSever,'broadshutdown':shutdonwClients,'gWait':getWait}

############################COMMAND DECLAATIONS!!!!####################################
def enroll(val):
pass
def login(val):
pass
def logout(val):
pass
def getWorkItem(val):
ret = ''
for i in val:
ret += i + ' '
return 'work item:' + ret
def completeWorkItem(val):
return 'THAAANK YOU!'
def getData(val):
return 'your data'
def returnData(val):
pass
def status(val):
pass

#### Admin commands
def getEnrollment(val):
pass
def getActive(val):
pass
def listProjects(val):
pass
def shutdownSever(val):
pass
def shutdonwClients(val):
pass
def getWait(val):
pass

##########################END COMMAND DECLAATIONS!!!!##################################

##return the argument parser for Ceranubis Server
def getParser():
##declare a new parser
parser = argparse.ArgumentParser('Ceranubis')

##add arguemnts to parser

##positional argument, the command to be called
parser.add_argument('cmd')#,choices=getCmds().keys())

##optional argument, conditional for a long argument
parser.add_argument('-l',action='store_true',default=False)

##optional argument, data
parser.add_argument('-d',nargs='*')

##return parser
return parser

####################REACTOR LOOP##########################
def reactor(conn,addr,parser):
##prints the screen that a connection was made and gives its address
print 'connection recieved at:', addr

##command dictionary
cdic = getCmds()

##logged in variable:
loggedIn = True
print 'starting reactor loop'
r = 1
##main reactor loop
while loggedIn:
print 'reactor cycle', r
r+=1
##waits for command
try:##safely waits for command
print 'waiting for command'
ret = conn.recv(BUFFER_SIZE)

#parses the command
print 'parsing command'
cmd = parser.parse_args(ret.split())

##pass the data to the data holder
print 'passing data to holder'
data = cmd.d

##checks to see if it is a long command
if cmd.l:
print 'it was a long command'
l = 1
while 1:##loops till the entire command is recieved
l += 1
print 'pulling more from long command, round:', l
ret = conn.recv(BUFFER_SIZE)##receives data
print ret
if len(ret)<BUFFER_SIZE: break##checks to see if that was the end
data[len(data)-1]+=ret##appends new data
print 'took:', l,' times to get all:', len(data), 'bytes'
print data
except Exception as e:##should something fail
print 'exception in the receval process:'
conn.sendall(e.__str__())
continue##just ignore it and continue

##executes command and gets result, command must be prepackaged
print 'running command'
val = cdic[cmd.cmd](data)

##if the command was to log out then the reactor needs to be interupted
if cmd.cmd == 'logout':
print 'logging out now'
loggedIn = False
break

##sends result back to client
print 'sending back to client'
conn.sendall(val)

##close out connection:
print 'closing connection'
conn.close()
return

##################END REACTOR LOOP########################

HOST = '' #to bind to all avaliable interfaces
PORT = 31415 #port number is pi, simple to remember.

##get socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind((HOST,PORT))

#threading comes later
s.listen(1)
print 'server up, listening...'

#breaks the tupple apart:
conn, addr = s.accept()

## starts reactor
reactor(conn,addr,getParser())
chapter 5 of my Dev log, focus on architecture and arg parsing.
© 2010 - 2024 jaqbenny
Comments0
Join the community to add your comment. Already a deviant? Log In