Physics Hacker

If you run a python script that imports GaudiPython (or any LHCb modules for that matter) inside gdb on any lxplus node, you will run into an error like this:

[lxplus223] ~ > gdb python
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"
   ...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) set args test.py
(gdb) run
Starting program:
   /afs/cern.ch/sw/lcg/external/Python/2.5.4p2/slc4_amd64_gcc34/bin/python test.py
[Thread debugging using libthread_db enabled]
[New Thread 182894205120 (LWP 11117)]
Traceback (most recent call last):
  File "test.py", line 1, in
    import GaudiPython
ImportError: No module named GaudiPython

Program exited with code 01.
(gdb)

This is due to our group-login script (including the “rc” one)
using LbLogin and that reshuffles the paths. When gdb starts
the process to be debugged and if a SHELL variable is present,
it is started with “$SHELL -c”.
When it is perfectly legal for bash (it only starts a new shell),
with (t)csh it starts a new *login* shell corrupting the already
setup paths (PATH, LD_LIBRARY_PATH, PYTHONPATH etc).

So in order to solve this you have to unset SHELL inside
gdb (maybe best to place that in .gdbinit):

unset environment SHELL

This is the first part of 2 minimal examples, on how to interact with the Facebook REST API. This part will be an example in python, the next part will do the exact same thing in erlang. (Note that there are python and erlang libs in various stages of completion, but my focus is on minimal examples and understanding here.)

The code below calls the function users.getinfo in the “call” method. The request send to the Facebook REST server has to be signed according to Facebook’s signing algorithm this is done in “get_rest_url”.

The result will be requested to be transmitted as JSON encoded text. I am using cjson to decode the result, but you could use any of the python JSON libs (Note that from python 2.6 onward there is a json lib in the standard lib).

__secret__     = 'a1b1c1e1d1f1......'
__api_key__    = 'a2b2c2e2d2f2......'
__restserver__ = 'api.facebook.com'
__url_start__  = '/restserver.php?'

import time
import hashlib
import httplib
import cjson

def get_rest_url(d):
  # this implements the recipe you can find on this page:
  # http://wiki.developers.facebook.com/index.php/How_Facebook_Authenticates_Your_Application
  l = d.items()
  l.sort()
  l = [str(x[0])+'='+str(x[1]) for x in l]
  sig = hashlib.md5(''.join(l) + __secret__).hexdigest()
  url_list = [ __url_start__ ]
  url_list.extend([x+'&' for x in l])
  url_list.append('sig='+sig)
  return ''.join(url_list)

def call(d):
  # make sure certain keys are set
  d.update({'api_key': __api_key__,   # my application key
            'call_id': time.time(),   # just any value that increases
            'format' : 'JSON'})       # request JSON formatted reply
  url = get_rest_url(d)
  connection = httplib.HTTPConnection(__restserver__)
  connection.request('GET',url)
  response = connection.getresponse()
  return cjson.decode(response.read())

def main():
  d = {'method':'users.getinfo',
       'v':'1.0',
       'uids':'1234567...',    #use ur own userid for testing
       'fields':'contact_email,first_name',
      }
  print call(d)
if __name__ == '__main__':
  main()

The following illustrates how to update an SVG embeded in XHTML when new information is available on the server (aka COMET style). The trick is to querry the Server in a “long poll” through an XMLHttpRequest and then wait until Server has the new info ready. In this Example the delayed info is a new time string from the server after a REST style querry to the exposed “sleep” function of the CherryPy server.

The resulting app will be served under http://localhost:8080/xhr
and should show a rotating clock with the time send from the server
like seen here:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cherrypy
import time

__author__ = 'Stephan Nies'

server = 'localhost'

class Comet(object):
  """CherryPy for Comet-style asynchronous communication through XMLHttpRequest"""

  @cherrypy.expose
  def xhr(self):
    xhtml="""<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
 "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="de">

  <script type="text/javascript">

    function sendToServer(url){
      try{
        var http = new XMLHttpRequest();
        var url = url;
        http.open("GET", url, false);
        http.send(null);
        return http.responseText;
      }
      catch(ex){
        throw ex;
      }
    }

    function time(sec){
      return sendToServer("http://%s:8080/sleep/"+sec);
    }

    function update(x) {
      if (x > 360) {
        x = 0;
      }
      x = x + 6;
      t = time(5);
      document.getElementById("txt").firstChild.textContent = t;
      document.getElementById("clock").setAttribute("transform", "rotate("+x+",200,200)");
      setTimeout("update("+x+")", 100);
    }
  </script>

  <head>
    <title>SVG clock with time by server side comet</title>
  </head>

  <body onload="update(0)">
    <svg width="600px" height="400px" version="1.1"
    xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
      <defs>
        <g id="MyClock">
          <rect x="0" y="0" rx="20" ry="20" width="250" height="100"
          style="fill:red;stroke:black;stroke-width:5;opacity:0.5"/>
          <text id="txt" x="15" y="56"
          style="font-family:Verdana;font-size:16px"> It's SVG! </text>
        </g>
      </defs>
      <use id="clock" x="50" y="150" xlink:href="#MyClock" />
    </svg>
  </body>
</html>
    """
    cherrypy.response.headers['Content-Type'] = 'application/xhtml+xml'
    return xhtml % server

  @cherrypy.expose()
  def sleep(self, sec):
    sec = float(sec)
    time.sleep(sec)
    return str(time.ctime(time.time()))

cherrypy.config.update({
    'log.screen':True,
    'tools.sessions.on': True,
    'checker.on':False
})
cherrypy.server.socket_host = server
cherrypy.tree.mount(Comet(), config=None)
cherrypy.engine.start()
cherrypy.engine.block()

Please note that this code uses only standard conform technologies like XHTML, JavaScript and SVG and therefore should work without plugins on any standard conform Browser (tested on Firefox 3.0.12 and Safari 4.0.2). In other words: This Code doesn’t work in any Internet Explorer available to date (<= IE8).

This time we do the same minimal histogram example, but using python. Again we assume that the user provides a file example.dat, that contains integer values in ascii format seperated by newlines.

call:

python historam.py example.dat

source for “histogram.py”:

import ROOT, sys

def readDataToHist(fileName):
  f = open(fileName);
  x_min = 0;
  x_max = 10;
  histo = ROOT.TH1I("h1","Histogram Title", 100,x_min,x_max);
  for line in f:
    raw = line[:-1]
    y = int(raw)
    histo.Fill(y);
  return histo;

def histogram(fileName):
  ROOT.gROOT.Reset()
  c1 = ROOT.TCanvas("c1","c1",800,800);
  histo = readDataToHist(fileName);
  c1.cd(1);
  histo.Draw();
  #c1.Update();
  c1.Print("histogram.pdf","Portrait pdf");

if __name__ == "__main__" :
  histogram(sys.argv[1])

To recreate L0 Banks for LHCb HLT to you can use this skript (stole most of it from Hans). It uses bankKiller and EventNodeKiller to clean the old L0Banks and recreates them. At the end it checks for all channel decisions involving muons or hadrons and writes out the events that had any such decision.

#  strip rawdata from a dst, and replace L0DU bank with proper bank
#  Checks for channeldesicions and writes only mu/had triggers

from Gaudi.Configuration import *

importOptions('$STDOPTS/LHCbApplication.opts')   # import old opts file
importOptions('$STDOPTS/DstDicts.opts')          # import old opts file
importOptions('$DDDBROOT/options/DC06.opts')     # import old opts file

# L0 configuration
importOptions('$L0MUONROOT/options/l0muonAlg.opts') # import old opts file
importOptions('$L0DUROOT/options/L0DUConfig.opts')  # import old opts file

#castor cards for mbias L0-stripped events from bookkeeping
#importOptions('/afs/cern.ch/user/d/dijkstra/python/l0-run/mb1-1.opts') # import old opts file
importOptions('$MOOREROOT/options/DC06_L0_v1_lumi2.opts')

# Gaudi.Configuration.Configurables is a ConfFacade - Object
# it is created upon first import of Gaudi.Configuration
# LHCbAlgs.LHCbAlgsConf.EventNodeKiller  /software/releases/LHCB/LHCB_v23r7/InstallArea/python/LHCbAlgs
# L0Calo.L0CaloConf.L0CaloAlg            /software/releases/LBCOM/LBCOM_v6r18/InstallArea/python/L0Calo
# L0Muon.L0MuonConf.L0MuonAlg            /software/releases/LBCOM/LBCOM_v6r18/InstallArea/python/L0Muon
# PuVeto.PuVetoConf.PuVetoAlg            /software/releases/LBCOM/LBCOM_v6r18/InstallArea/python/PuVeto
# L0DU.L0DUConf.L0DUAlg                  /software/releases/LBCOM/LBCOM_v6r18/InstallArea/python/L0DU
# DAQEvent.DAQEventConf.bankKiller       /software/releases/LHCB/LHCB_v23r7/InstallArea/python/DAQEvent
from Configurables import EventNodeKiller,L0CaloAlg,L0MuonAlg,PuVetoAlg,L0DUAlg,bankKiller

RemoveL0Banks = bankKiller('RemoveL0Banks')
RemoveL0Banks.BankTypes =["L0DU", "L0Calo", "L0Muon"]
L0 = GaudiSequencer('L0')

#-----------------------------------------------------
# Run the whole L0 sequence including the L0Processors
#-----------------------------------------------------
L0.Members += [ "bankKiller/RemoveL0Banks"
               ,"L0CaloAlg/L0Calo"
               ,"L0MuonAlg/L0Muon"
               ,"PuVetoAlg/L0PuVeto"
               ,"L0DUAlg/L0DU"
               ,"EventNodeKiller/EventNodeKiller"
]
L0DU = L0DUAlg('L0DU')
L0DU.StoreInBuffer   = True

rawwriter  = OutputStream('RawWriter', Preload = False,
  ItemList = ["/Event#1","/Event/DAQ#1","/Event/DAQ/RawEvent#1","/Event/DAQ/ODIN#1"],
  Output   = "DATAFILE='/local_home/snies/DC06_L0_v1_lumi2_MuonHadron.dst' TYP='POOL_ROOTTREE' OPT='REC' ")
appConf    = ApplicationMgr( OutputLevel = INFO, AppName = 'L0DUextract')
appConf.OutStream = [rawwriter]
appConf.ExtSvc += ['DataOnDemandSvc']
DataOnDemandSvc().Algorithms += ["DATA='/Event/Trig/L0/L0DUReport' TYPE='GaudiSequencer/L0'"]
#appConf.TopAlg    = ['GaudiSequencer/L0']

EventNodeKiller().Nodes    = ["Rec", "MC", "Raw", "Gen", "Link", "pSim" , "Prev", "PrevPrev", "Next"]
EventSelector().PrintFreq  = 50

import GaudiPython
#from gaudigadgets import *

appMgr = GaudiPython.AppMgr()
sel    = appMgr.evtsel()  #Event Selector (could be used to specify input files)
evt    = appMgr.evtsvc()  #Get event service for later use to access the transient event store

appMgr.algorithm('RawWriter').Enable = False  # stop automatic execution of RawWriter

nevent=0  # loop variable
l0acc=0   # number of L0 accepted events

appMgr.run(1) #step one event
while nevent < 100000:
  nevent = nevent + 1 # increase loop variable
  appMgr.run(1)       # read one event

  if evt['Rec/Header'] == None : break  # probably end of input

  # check L0
  L0dir = evt['Trig/L0/L0DUReport']  # get a container
  # get channel decisions
  Hadron = L0dir.channelDecisionByName('Hadron')
  Muon   = L0dir.channelDecisionByName('Muon')
  DiMuon = L0dir.channelDecisionByName('DiMuon')
  MuonNoGlob = L0dir.channelDecisionByName('MuonNoGlob')

  if Hadron or Muon or DiMuon or MuonNoGlob:
    l0acc = l0acc + 1        # count as L0 accepted event
    rc = appMgr.algorithm('RawWriter').execute() # output the L0 accepted event
    # rc probably a return code ? should be checked ?

print 'L0 accepted',l0acc,' in ',nevent,' events.'
print 'Read/Wrote ',nevent,l0acc

Since everybody and their dog is looking into erlang, i didn’ t want to miss the hipe and bought “Programming Erlang, Software for a Concurrent World” by Joe Armstrong.

So for a starter and since we really dig python here, i ported the C-code ports/example1.c on page 207 of the book to python.

#!/usr/bin/python2.4

# example1.py

import sys, struct

def twice(a):
  return 2*a

def sum(a,b):
  return a+b

while sys.stdin:
  try:
    len_raw  = sys.stdin.read(2)
    len      = struct.unpack('h',len_raw)[0]
    data_raw = sys.stdin.read(len)
    data     = struct.unpack(str(len)+'b',data_raw)
    fn       = data[0]
    x = 0
    if fn == 1:
      a = data[1]
      x = twice(a)
    elif fn == 2:
      a = data[1]
      b = data[2]
      x = sum(a,b)
    result = struct.pack('hb',1,x)
    sys.stdout.write(result)
    sys.stdout.flush()
  except:
    break

You can use it in combination with ports/example1.erl by Joe Armstrong,
just rename it to example1 and make it executeable.

Running larger sites demands for scripting node installation.

I like to use python for the task (surprise surprise). As it happens the mayor package managment tool for Red Hat, CentOS and Scientific Linux “yum” is itself written in python.

Since yum lacks documentation on how to use inside your own python code I have googled the web and found this nice page: Deciphering the Yum API.

For some reasons the code didn’t work for me on Scientific Linux 4 (Red Hat 3.4.6-9) which still uses python2.3. So I fiddled around a bit and got it working. Here are the updated examples.

Listing packages:

import yum

yb = yum.YumBase()
yb.doConfigSetup()
yb.doTsSetup()
yb.doRpmDBSetup()
for pkg in yb.rpmdb.getPkgList():
  print pkg

Searching packages:

In newer versions of yum there seems to be YumBase.searchGenerator, wich should be prefered for performance and memory footprint. However in the enterprise class distros i have to deal with, I could not use it. So I present a solution with YumBase.searchPackages.

import yum

yb = yum.YumBase()
yb.doConfigSetup()
yb.doRepoSetup()
yb.doSackSetup()
yb.doTsSetup()
yb.doRpmDBSetup()
fields = ['name']        # fields to look at
criteria = ["k3b"]       # strings to find in fields
matches = yb.searchPackages(fields, criteria)
for match in matches:
  print match

Ask if a certain package is installed:

import yum

yb = yum.YumBase()
yb.doConfigSetup()
yb.doTsSetup()
yb.doRpmDBSetup()
# prints 1 if installed else 0
print yb.rpmdb.installed('vim-enhanced')

Installing packages:

Install the editor joe. Note that we do not use the “standard” YumBase but the “command line” YumBaseCli. This class provides the installPkgs() function which magically sorts out which package we want by just saying “joe”. Note also that by using YumBaseCli in this way, we are bypassing some argument checks the command line yum would have done. To make sure yum doesn’t ask any questions we do “ybc.conf.setConfigOption(‘assumeyes’,True)”.
Beware of the consequences.

import sys

sys.path.append('/usr/share/yum-cli')

import cli

ybc = cli.YumBaseCli()
ybc.doConfigSetup()
ybc.doTsSetup()
ybc.doRpmDBSetup()
ybc.installPkgs(['joe'])
ybc.buildTransaction()
ybc.conf.setConfigOption('assumeyes',True)
ybc.doTransaction()