Physics Hacker

If you run a python script that imports GaudiPython (or any LHCb modules for that matter) inside gdb on any lxplus node, you will run into an error like this:

[lxplus223] ~ > gdb python
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"
   ...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) set args test.py
(gdb) run
Starting program:
   /afs/cern.ch/sw/lcg/external/Python/2.5.4p2/slc4_amd64_gcc34/bin/python test.py
[Thread debugging using libthread_db enabled]
[New Thread 182894205120 (LWP 11117)]
Traceback (most recent call last):
  File "test.py", line 1, in
    import GaudiPython
ImportError: No module named GaudiPython

Program exited with code 01.
(gdb)

This is due to our group-login script (including the “rc” one)
using LbLogin and that reshuffles the paths. When gdb starts
the process to be debugged and if a SHELL variable is present,
it is started with “$SHELL -c”.
When it is perfectly legal for bash (it only starts a new shell),
with (t)csh it starts a new *login* shell corrupting the already
setup paths (PATH, LD_LIBRARY_PATH, PYTHONPATH etc).

So in order to solve this you have to unset SHELL inside
gdb (maybe best to place that in .gdbinit):

unset environment SHELL

This is the second part of two minimal examples, on how to interact with the Facebook REST API. Last weeks part was in python, this time we do the exact same thing in erlang. (Note that there are python and erlang libs in various stages of completion, but my focus is on minimal examples and understanding here.)

The code below calls the function users.getinfo in the “call” method. The request send to the Facebook REST server has to be signed according to Facebook’s signing algorithm this is done in “get_rest_url”.

The result will be requested to be transmitted as JSON encoded text. I am using mochijson2 to decode the result.

-module(fb).
-export([call/1, test/0]).

-define(SECRET, "a1b1c1e1d1f1......"
-define(API_KEY, "a2b2c2e2d2f2......"
-define(REST_SERVER, "api.facebook.com").
-define(URL_START, "/restserver.php?").

get_rest_url(Args) ->
  % this implements the the facebook signing recipe
  % the link is in the text above
  SortedArgs = lists:sort(Args),
  Fun = fun({Key, Value}) -> Key ++ "=" ++ Value end,
  StringArgs = lists:map(Fun, SortedArgs),
  String = lists:flatten(StringArgs, ?SECRET),
  <<M:128>> = erlang:md5(String),
  Sig = lists:flatten(io_lib:format("~32.16.0b",[M])),  % string rep of hexdigest
  StringArgs2 = lists:map(fun(X) -> X ++ "&" end, StringArgs),
  lists:flatten(["http://", ?REST_SERVER, ?URL_START, StringArgs2, "sig=", Sig]).

call(Args) ->
  {Megaseconds,Seconds,Microseconds} = erlang:now(),
  CallId = integer_to_list(Megaseconds)
        ++ integer_to_list(Seconds)
        ++ integer_to_list(Microseconds),
  Args2 = lists:append(Args, [{"api_key", ?API_KEY},   % my application key
                             {"call_id", CallId},      % just any value that increases
                             {"format",  "JSON"}]),    % request JSON formatted reply
  Url = fb:get_rest_url(Args2),
  {ok, {{HttpVer, Code, Msg}, Headers, Body}} =
                    http:request(get, {Url, [{"User-Agent", "test "}]}, [], []),
  mochijson2:decode(Body).

test() ->
  call([{"method","users.getinfo"},
        {"v","1.0"},
        {"uids","1234567..."},
        {"fields","first_name"}]).

you would run the above like this:

erl -pa /some/path/to/mochiweb/ebin/
Erlang R13B04 (erts-5.7.5) [source] [smp:4:4] [rq:4] [async-threads:0] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> inets:start().
ok
2> c(fb).
./fb.erl:28: Warning: variable 'Code' is unused
./fb.erl:28: Warning: variable 'Headers' is unused
./fb.erl:28: Warning: variable 'HttpVer' is unused
./fb.erl:28: Warning: variable 'Msg' is unused
{ok,fb}
3> fb:test().
[{struct,[{<<"first_name">>,<<"Joe">>},
          {<<"uid">>,123456...}]}]
4>

This is the first part of 2 minimal examples, on how to interact with the Facebook REST API. This part will be an example in python, the next part will do the exact same thing in erlang. (Note that there are python and erlang libs in various stages of completion, but my focus is on minimal examples and understanding here.)

The code below calls the function users.getinfo in the “call” method. The request send to the Facebook REST server has to be signed according to Facebook’s signing algorithm this is done in “get_rest_url”.

The result will be requested to be transmitted as JSON encoded text. I am using cjson to decode the result, but you could use any of the python JSON libs (Note that from python 2.6 onward there is a json lib in the standard lib).

__secret__     = 'a1b1c1e1d1f1......'
__api_key__    = 'a2b2c2e2d2f2......'
__restserver__ = 'api.facebook.com'
__url_start__  = '/restserver.php?'

import time
import hashlib
import httplib
import cjson

def get_rest_url(d):
  # this implements the recipe you can find on this page:
  # http://wiki.developers.facebook.com/index.php/How_Facebook_Authenticates_Your_Application
  l = d.items()
  l.sort()
  l = [str(x[0])+'='+str(x[1]) for x in l]
  sig = hashlib.md5(''.join(l) + __secret__).hexdigest()
  url_list = [ __url_start__ ]
  url_list.extend([x+'&' for x in l])
  url_list.append('sig='+sig)
  return ''.join(url_list)

def call(d):
  # make sure certain keys are set
  d.update({'api_key': __api_key__,   # my application key
            'call_id': time.time(),   # just any value that increases
            'format' : 'JSON'})       # request JSON formatted reply
  url = get_rest_url(d)
  connection = httplib.HTTPConnection(__restserver__)
  connection.request('GET',url)
  response = connection.getresponse()
  return cjson.decode(response.read())

def main():
  d = {'method':'users.getinfo',
       'v':'1.0',
       'uids':'1234567...',    #use ur own userid for testing
       'fields':'contact_email,first_name',
      }
  print call(d)
if __name__ == '__main__':
  main()

Today i would like to introduce you to one of my favorite Podcast shows Chaosradio Express , which is a German Podcast on technology, software and webculture.

Some of the episodes i liked best are:

CRE 088 – Python und PyPy
CRE 086 – USB
CRE 082 – Erlang

But English speaking friends do not despair, there are also English episodes of the sister series Chaosradio Express International

All this is a shameless advertisement plug as demanded by Tim Pritlove the moderator of the show.

So this should be Nr. 44 of a linked chain of blog posts covering CRE.

== Previous Blog4CRE post (Nr.43 ) ========== Next Blog4CRE post (Nr.45) ==

The following illustrates how to update an SVG embeded in XHTML when new information is available on the server (aka COMET style). The trick is to querry the Server in a “long poll” through an XMLHttpRequest and then wait until Server has the new info ready. In this Example the delayed info is a new time string from the server after a REST style querry to the exposed “sleep” function of the CherryPy server.

The resulting app will be served under http://localhost:8080/xhr
and should show a rotating clock with the time send from the server
like seen here:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import cherrypy
import time

__author__ = 'Stephan Nies'

server = 'localhost'

class Comet(object):
  """CherryPy for Comet-style asynchronous communication through XMLHttpRequest"""

  @cherrypy.expose
  def xhr(self):
    xhtml="""<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
 "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="de">

  <script type="text/javascript">

    function sendToServer(url){
      try{
        var http = new XMLHttpRequest();
        var url = url;
        http.open("GET", url, false);
        http.send(null);
        return http.responseText;
      }
      catch(ex){
        throw ex;
      }
    }

    function time(sec){
      return sendToServer("http://%s:8080/sleep/"+sec);
    }

    function update(x) {
      if (x > 360) {
        x = 0;
      }
      x = x + 6;
      t = time(5);
      document.getElementById("txt").firstChild.textContent = t;
      document.getElementById("clock").setAttribute("transform", "rotate("+x+",200,200)");
      setTimeout("update("+x+")", 100);
    }
  </script>

  <head>
    <title>SVG clock with time by server side comet</title>
  </head>

  <body onload="update(0)">
    <svg width="600px" height="400px" version="1.1"
    xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
      <defs>
        <g id="MyClock">
          <rect x="0" y="0" rx="20" ry="20" width="250" height="100"
          style="fill:red;stroke:black;stroke-width:5;opacity:0.5"/>
          <text id="txt" x="15" y="56"
          style="font-family:Verdana;font-size:16px"> It's SVG! </text>
        </g>
      </defs>
      <use id="clock" x="50" y="150" xlink:href="#MyClock" />
    </svg>
  </body>
</html>
    """
    cherrypy.response.headers['Content-Type'] = 'application/xhtml+xml'
    return xhtml % server

  @cherrypy.expose()
  def sleep(self, sec):
    sec = float(sec)
    time.sleep(sec)
    return str(time.ctime(time.time()))

cherrypy.config.update({
    'log.screen':True,
    'tools.sessions.on': True,
    'checker.on':False
})
cherrypy.server.socket_host = server
cherrypy.tree.mount(Comet(), config=None)
cherrypy.engine.start()
cherrypy.engine.block()

Please note that this code uses only standard conform technologies like XHTML, JavaScript and SVG and therefore should work without plugins on any standard conform Browser (tested on Firefox 3.0.12 and Safari 4.0.2). In other words: This Code doesn’t work in any Internet Explorer available to date (<= IE8).

Here is a quick example of how one can create and fill a simple TTree in PyRoot. I find this especially usefull when the input data are in form of a flat text file as python can do string parsing very well.

This script creates a C-style struct and uses the addresses of the elements inside to this to fill the root tree.

#!/usr/bin/env python
from ROOT import TTree, TFile, AddressOf, gROOT

def main():
  # Make a tree
  f = TFile('myTest.root','RECREATE')
  t = TTree('MyTree','My test tree')

  # Create a struct
  gROOT.ProcessLine(\
    "struct MyStruct{\
      Int_t someInt;\
      Double_t someDouble;\
    };")
  from ROOT import MyStruct

  # Create branches in the tree
  s = MyStruct()
  t.Branch('rootInt',AddressOf(s,'someInt'),'someInt/I')
  t.Branch('rootDouble',AddressOf(s,'someDouble'),'someDouble/D')

  # Fill tree
  for i in range(100):
    s.someInt = i
    s.someDouble = i
    t.Fill()

  f.Write()
  f.Close()

if __name__=='__main__':main()

This time we do the same minimal histogram example, but using python. Again we assume that the user provides a file example.dat, that contains integer values in ascii format seperated by newlines.

call:

python historam.py example.dat

source for “histogram.py”:

import ROOT, sys

def readDataToHist(fileName):
  f = open(fileName);
  x_min = 0;
  x_max = 10;
  histo = ROOT.TH1I("h1","Histogram Title", 100,x_min,x_max);
  for line in f:
    raw = line[:-1]
    y = int(raw)
    histo.Fill(y);
  return histo;

def histogram(fileName):
  ROOT.gROOT.Reset()
  c1 = ROOT.TCanvas("c1","c1",800,800);
  histo = readDataToHist(fileName);
  c1.cd(1);
  histo.Draw();
  #c1.Update();
  c1.Print("histogram.pdf","Portrait pdf");

if __name__ == "__main__" :
  histogram(sys.argv[1])

We needed to create a histogram from some output of a python program. Therefore I wrote a minimal root program to read in the data from a file and save the histogram as a pdf.

The datafile should contain one entry per row. Save the following source code as e.g. histogram.c and then start root with

root -b -q 'histogram.C("filename")'

Here is the sourcecode:

#include <TCanvas.h>
#include <TH1.h>
#include <iostream.h>
#include <Riostream.h>
#include "Math/PdfFuncMathCore.h"

int main(){
  histogram("test")
}

void histogram(char* fileName) {

  gROOT->Reset();
  TCanvas *c1 = new TCanvas("c1","c1",800,800);
  TH1I *histo = readDataToHist(fileName);
  c1->cd(1);
  histo->Draw();
  c1->Update();
  c1->Print("histogram.pdf","Portrait pdf");
}

TH1I* readDataToHist(char* fileName)
{
  ifstream in;
  in.open(fileName);
  Double_t bin;
  Int_t nlines = 0;

  Int_t x_min = 0;
  Int_t x_max = 10;
  TH1I *histo = new TH1I("h1","Histogram Title", 100,x_min,x_max);

  while(1)
  {
    in >> bin;
    if (!in.good()) break;
    //cout << "bin: " << bin << '\n';
    histo->Fill(bin);
    nlines++;
  }
  in.close();
  return histo;
}

Here is a small python script that i found to come in handy at times. It just profiles the usage of virtual memory of a given command in linux (no profiling of child processes). I use this occasionally as a first test when i have the sneaking suspicion there might be a memory leak. Also works nice embedded in a test framework as a first warning of memory leaks when testing nightly builds.

#!/usr/bin/env python

import re, sys, time, popen2

usage = """
--- memprofile.py --------------------------
programm to monitor virtual memory
       usage under linux
--- usage:

    ./memprofile.py  

secs = time between samples
command = the command to profile

Feel free to use this in any kind of way
         Author: Stephan Nies - snies@cern.ch
---------------------------------------------
"""

def get_vm_size(pid):
  f = file('/proc/%s/stat' % str(pid))
  stat = f.read()
  f.close()
  stat_list = re.split(' ', stat)
  return int(stat_list[22])

if __name__ == '__main__':
  if len(sys.argv) < 2:
    print usage
    sys.exit(0)
  secs = sys.argv[1]
  cmd  = ' '
  for arg in sys.argv[2:]:
    cmd = cmd + ' ' + arg
  print "profiling:", cmd
  p = popen2.Popen4(cmd)
  f = file(str(p.pid)+'.log','w')

  while p.poll() == -1:
    vmsize = get_vm_size(p.pid)
    f.write('pid '+ str(p.pid) + ',  vmsize '+ str(vmsize) + ', time ' + str(time.time()) + '\n')
    time.sleep(1)
  print 'created logfile:', str(p.pid)+'.log'

UPDATE: Please note that the use of popen2.Popen4 may involve the creation of a shell. So the Pid might be one off. To avoid such problems use the subprocess module instead if you can.

I’m not a great fan of shell scripts, and have a hard time performing anything even slightly complex with them. Unfortunately sourcing shell scripts is something that is often hard to avoid. So here is a trick to use python for all the logic required and only using the shell script as a wrapper.

The shell script basically just executes a given python script, passing along all the arguments. One argument is reserved for identifying the type of script (sh or csh) calling the python script. Python outputs shell commands that are then evaluated in the shell by ‘eval’.

For bash:

#!/bin/bash
eval `$MYSCRIPTS/test.py sh $@`

For csh:

#!/bin/csh
eval `$MYSCRIPTS/test.py csh $argv`

The python script can then perform the complex stuff and print shell commands for the relevant shell type.

#!/usr/bin/env python
import sys

# Distinguish between sh and csh
shelltype = sys.argv[1]

# Get the rest of the arguments
arguments = sys.argv[2:]

# Generate some fancy commands for the shell
shellcommands = createfancycommands(shelltype, arguments)

# This is shown to the user
print >> sys.stderr, ‘Use stderr for user info:’

# This is evaluated by the shell
print shellcommands

And, voilà, the python script takes care of all you worries. Also the overhead involved in supporting both sh and csh is drastically reduced.