 |
Grid main functionality: manage services.
It contains list of servers, which provide services. When instance is published in the grid, corresponding servers object are created on each host and saved as list in the dictionary, with a key corresponding to their service_id.
Servers has part specific for communication method (at the creation stage), thus have to be defined in derived classes.
USAGE:
g = DerivedGrid(..)
g.copy_files(isd_src_path, ['*.pyc', '*.so'])
some_obj_instance = some_obj_class(..)
service_id = g.publish(some_obj_instance)
-> a) assigns some_service_id to this instance
b) creates servers that manage the execution of this instance
in remote hosts: servers[some_service_id] = [server1, server2, ...]
proxy = g.acquire_service(service_id) ## returns proxy object
proxy.set_parameters(p) # these functions will be performed sequentally
proxy.f() # and service won't be freed
proxy.g() # unless .release_service() is called
g.release_service(proxy) Inherits Thread.
Public Member Functions | |
| def | __init__ |
| not shared_temp_path: temp_path == None look for individual temp_paths in host list | |
| def | acquire_service |
| For a given service, it finds available server and returns corresponding proxy worker. | |
| def | add_server |
| (for internal use) | |
| def | add_time |
| def | copy_files |
| (for internal use) | |
| def | create_service_id |
| def | create_subroot |
| def | dp |
| def | find_server |
| (for internal use) | |
| def | initialise |
| Initialise the grid: Create temp paths if needed, either shared or remotely on each host Copy source files to this folder. | |
| def | ishalted |
| def | publish |
| Publish instance on all the hosts of the grid (specific for the communication method) | |
| def | release_service |
| Used to inform server/proxy that we are done using the service, and it should be released, when the last funciton call of the proxy is finished. | |
| def | set_debug |
| def | set_loader |
| def | terminate |
Public Attributes | |
| copied_files | |
| debug | |
| debug_out | |
| display | |
| hosts | |
| list of Host objects | |
| loader_filename | |
| others | |
| queues | |
| servers | |
| key = service_id self.servers[key] = list of servers providing this service self.queues[key] = a queue with available servers for the service | |
| shared_temp_path | |
| src_files | |
| temp_paths | |
| create temporary directories if needed | |
| verbose | |
| window_size | |
| X11_delay | |
| def IMP.isd.AbstractGrid.AbstractGrid.__init__ | ( | self, | |
| hosts, | |||
| src_path, | |||
| display, | |||
| X11_delay, | |||
| debug, | |||
| verbose, | |||
| shared_temp_path | |||
| ) |
not shared_temp_path: temp_path == None look for individual temp_paths in host list
| def IMP.isd.AbstractGrid.AbstractGrid.acquire_service | ( | self, | |
| service_id | |||
| ) |
For a given service, it finds available server and returns corresponding proxy worker.
| def IMP.isd.AbstractGrid.AbstractGrid.add_server | ( | self, | |
| server | |||
| ) |
(for internal use)
Registers the server within a grid
1) register itself (grid) in the server 2) proxy must have _selfrelease attribute to be able to release service (i.e. put the connected to it server in the .queues[service_id]), once the last function call before invoking .release_service() has finished calculations
self.queues: a dict[DerivedServer.service_id]=Queue(-1) containing the DerivedServers self.servers: dict[DerivedServer.service_id]=[list of DerivedServer instances]
| def IMP.isd.AbstractGrid.AbstractGrid.copy_files | ( | self, | |
| rootdir, | |||
| filelist, | |||
hosts = None, |
|||
exclude = ['cns/', |
|||
| svn | |||
| ) |
(for internal use)
Copies source files to a common folder so that they were accessible to the grid on remote hosts
g = DerivedGrid(...)
src_path = os.path.join(os.environ['ISD_ROOT'],'/src/py')
from compileall import compile_dir compile_dir(src_path, maxlevels=0, force=True, quiet=True)
g.copy_files(src_path, ['*.pyc','*.so'])
The method will replicate the directory structure of items in the filelist, e.g. will create data folder in the destination for
g.copy_files(src_path, ['data/*.txt'])
also, it will copy all subfolders matching the required name, i.e. it will copy all files called foo.txt from all existing subfolders it can find
g.copy_files(src_path, ['Isd/foo.txt']) will copy files like 'Isd/foo.txt', 'Isd/a/foo.txt', 'Isd/a/b/c/foo.txt' etc.
the exclude keyword excludes paths containing the given strings.
| def IMP.isd.AbstractGrid.AbstractGrid.find_server | ( | self, | |
| service_id | |||
| ) |
(for internal use)
Selects a server that is capable of executing the requested service. If they are all busy, waits until there will be one available.
| def IMP.isd.AbstractGrid.AbstractGrid.initialise | ( | self, | |
| src_path, | |||
| src_files | |||
| ) |
Initialise the grid: Create temp paths if needed, either shared or remotely on each host Copy source files to this folder.
| def IMP.isd.AbstractGrid.AbstractGrid.publish | ( | self, | |
| instance | |||
| ) |
Publish instance on all the hosts of the grid (specific for the communication method)
Returns: service_id
Internally:
It creates list of servers performing the job specified by this service_id
| def IMP.isd.AbstractGrid.AbstractGrid.release_service | ( | self, | |
| proxy | |||
| ) |
Used to inform server/proxy that we are done using the service, and it should be released, when the last funciton call of the proxy is finished.
However, it means that someone (the proxy or the grid) will have to call _release_service() itself and also restore ._selfrelease to the previos state (FALSE)
| IMP.isd.AbstractGrid.AbstractGrid.hosts |
list of Host objects
| IMP.isd.AbstractGrid.AbstractGrid.servers |
key = service_id self.servers[key] = list of servers providing this service self.queues[key] = a queue with available servers for the service
| IMP.isd.AbstractGrid.AbstractGrid.temp_paths |
create temporary directories if needed