Tuesday, December 24, 2013

List comprehension in Python.


List Comprehension in Pyhton
Python supports various compound data type and one of very important is list.List is similar to array in other languages but also provide more functionality.
A list comprehension provides a compact way creating a list from other list using some operation on the elements of list.It is derived from set notation.This topic is very important as per python based technical interviews are considered. I tried to cover this topic with some basic examples and some real world example.

Suppose we are having a list l and we want to create a new list from l with each element incremented by 1. Here is the simple way to do this.
>>> l=[1,2,3,4]
>>> k=[ele+1 for ele in l]
>>> print k
[2, 3, 4, 5]
>>> k
[2, 3, 4, 5]
>>> 

             
This is called as list comprehension.

Now lets try list comprehension with multidimensional list.


>>> l1=[[1,2,3], [3,4,5] ,[6,7,8]]
>>> col1=[ row[1] for row in l1]
>>> col1
[2, 4, 7]

Here l1 is 2 dimensional list.Each element of l1 is itself a list.So "for row in l1" will select each element of l1 and assign "row" as name to the element.Now each element is list itself. Here we extract second element of each list and append it to col1.

Now suppose we want to construct a new list from l1 so that we want to sum all the element of each sublist of l1 to from single element of new list.Here is the solution.

 
>>> lnew=[sum(row) for row in l1]

>>> lnew

[6, 12, 21]


Lets display this multidimensional list as single dimensional list.

>>> lsdim=[row for row1 in l1 for row in row1 ]

>>> print lsdim

[1, 2, 3, 3, 4, 5, 6, 7, 8] 



 Now suppose we want to display the elements of sublist if and only if they are even number.Here is the solution.
>>> leven=[row for row1 in l1 for row in row1 if row%2 == 0]

>>> print leven

[2, 4, 6, 8]




This are some simple programming tricks of list comprehension in python.

Lets see a simple real world example.

Suppose we want to display the files of root user in current working directory.

Here we have to use the glob module.We import the module and then get the list of all the files in current working directory using glob() function.We create a new list from this list of files using condition that the user id of file must be same as that of root i.e. 0. For getting the user id of file we used stat() function of os module.
>>> lofile=glob.glob('*.*')

>>> lofile

['java.zip', 'parchild.c~', 'inc.cpp', 'inc.cpp~', 'packetcapturing.tar.gz', 'ifstmt.c', 'ifstmt.cpp~', 'weather.cpp~', 'alk.cpp~', 'talk.c~', 'read.sh~', 'Python-3.3.2', 'weather.cpp', 'my.sh', 'inc.c~', 'my.c', 'myc.c', 'a.out', 'ifstmt.cpp', 'mozilla.pdf', 'inc.c', 'myclass.py', 'parchild.c', 'oomp1.cpp~']

>>> lnew=[f for f in lofile if os.stat(f).st_uid ==0 ]

>>> print lnew

['ifstmt.c', 'my.sh', 'my.c', 'myc.c', 'inc.c', 'myclass.py', 'parchild.c']






Saturday, May 18, 2013

How to configure fairshare in maui.


What is fairshare?
Fairshare allows historical resource utilization information to be incorporated in Job feasibility and priority decision.Job feasibility means whether to schedule job or not.

To start fairshare policy of maui we have to specify fairshare specific parameter in maui.cfg.Some of them are listed below.

FSINTERVAL  duration of each fairshare window
FSDEPTH       number of fairshare windows factored into current fairshare utilization
FSDECAY      decay factor applied to weighting the contribution of each fairshare window
FSPOLICY     metric to use when tracking fairshare usage

Scheduling cycle is divided into equal size window of size FSINTERVAL and number of windows are specified by FSDEPTH.

Now fairshare can be applied on credential basis by using credential specific parameter like USERCFG,GROUPCFG,CLASSCFG etc.

Fairshare affects maui's scheduling decision using 2 approach
1)Fairshare Target
2)Fairshare caps

Fairshare Target is responsible for job priority decision. And Fairshare cap is responsible for job feasibility decision. It is of two type absolute and relative.
Unfortunately no proper documentation is provided for capping policy.In moab when we want to specify capping we have to use FSCAP.In maui FSCAP is provided in parameter list but is not handled in source code.

To specify capping in maui we have to use FSTARGET. Thus suppose credential is that I want to constrained is User and Name of user is Shailesh.Suppose jobs submitted by shailesh must not consume more than 4000 proc sec.
Thus we can specify this in maui.cfg as given below
USERCFG[shailesh]  FSTARGET=4000^.

This is called as absolute capping.
In relative capping we consider percentage of total delivered cycles and can be specified by jst replacing ^ with %.