Thursday, September 22, 2011

Funny Incident @Work

Today, my manager asked me to send him my profile in one-page ppt. After I sent him the document he replied back saying 'You haven't included number of years of experience in it. Update it & send again' .
But I included that in document. So, to clarify, I call him.

Me: Hi <manager> , its regarding the profile. Years of experience, I have mentioned it in the document.
Manger: Where is it? I don't see it.
Me: Its in first sentence of document itself.
Manager: You mentioned it in letters. ah.. you should have mentioned it in digits.
Me: Okay.Anything else.
Manager: No, Nothing else. Just update it, change it to digit and resend me the document.

He wanted me to replace the word with a single digit and resend him the document. Amazing!!
Such incidents really provoke laughter and provide amusement at work.

Monday, September 5, 2011

Extracting movie title from torrent file name using Regular Expression

Movie files downloaded from torrent sites has file name which contains format types (like dvdrip, dvdscr, xvid etc), year, comments , user names and of course movie name. We want to extract movie name from this file names.

Input File namesOutput Achieved to zero

Nim's.Island[2008]DvDrip-aXXo.avinim's island



20000 Leagues Under The Sea.avi

Across The Universe.MoZinRaT CD1.aviacross the universe mozinrat

Adoration 2008 DvdRip ExtraScene RG.aviadoration

Amelie(English Dubbed).aviamelie






badder.santa.dvdrip.xvid-deity.avibadder santa

Balls of Fury[2007]DvDrip[Eng]-FXG.aviballs of fury

Bruno (2009) DVDRip-MAXSPEED

Defiance DvDSCR[2009] ( 10rating ).avidefiance

Down With Love (cute romantic comedy).avidown with love

Einstein.And.Eddington.2008.DVDRip.XviD.avieinstein and eddington


Based on the observation that

  • Most of the file name contains the format like 'dvdrip', 'xvid', 'brrip','dvdscr' or other words like 'CD1','(<year>)','[<year>]'  specified in the name and everything after any of this  words doesnot contains any useful data.

  • Sometimes extra information is added to file name inside bracket  like
    Defiance DvDSCR[2009] ( 10rating ).avi
    Down With Love (cute romantic comedy).avi
    so we can also ignore the part including and after brackets as movie names doesn't have brackets in them and have no useful information after it.

'(.*?)(dvdrip|xvid| cd[0-9]|dvdscr|brrip|divx|[\{\(\[]?[0-9]{4}).*'

This regular expression will find file names where we have dvdrip, brrip, xvid(we can specify any number of values here) or  year and finds first of any one of the appearing patterns because we have used lazy parsing here using .*?.  We then extract the first back referenced part \1.

Secondly, to remove the part within brackets we use

'(.*?)\(.*\)(.*)' regular expression.

Following code snippet gets the movie names(to an extent) from file name.

import re
fr = open('filenameslist.txt', 'r')
fw = open('movienames.txt', 'w')
for line in fr:
    text = line.strip()
text1 ='([^\\\]+)\.(avi|mkv|mpeg|mpg|mov|mp4)$', text)
if text1:
    text =
text = text.replace('.', ' ').lower()
text2 ='(.*?)(dvdrip|xvid| cd[0-9]|dvdscr|brrip|divx|[\{\(\[]?[0-9]{4}).*', text)
if text2:
    text =
text3 ='(.*?)\(.*\)(.*)', text)
if text3:
    text =
# print text
fw.write(text + '\n')

Output can be improved further observing things like we can replace characters like underscore with space, we can check for only four digits where next character is non word character ..

Sunday, September 4, 2011

Extracting IMDB data using Python

Sample code for extracting IMDB data using python BeautifulSoup package.

Requirement : To extract all the feature movie names and their ratings from IMDB database for a particular year.

Parameters used in query were identified using IMDB advanced search function. Start, count and year parameters were used in this case for querying.The url is queried for 100 records at a time since more than that is not allowed. After extracting movie names and rating for 100 records, the url is queried for next 100 records and so on.

Two files 'imdb_conf' and 'ratings' are created by the code. 'imdb_conf' file keeps track of the record number last read and 'ratings' file stores the movie name, rating and year.

Web scrapping the IMDB website for required data.

[sourcecode language="python"]
from BeautifulSoup import BeautifulSoup
import os
import re
import urllib2

def get_start_pos_yr(fimdb_config):
#for starting after last fetched record
#last line contains the last record fetched
nlines = fimdb_config.readlines()
startfrom = -1
year = None
if len(nlines) > 1:
if list_num:
startfrom = int(
return startfrom,year

def get_soup(url):
#get soup object for the url
page = urllib2.urlopen(url)
except urllib2.URLError, e:
print 'Failed to fetch ' + url
raise e

soup = BeautifulSoup(page)
except HTMLParser.HTMLParseError, e:
print 'Failed to parse ' + url
raise e
return soup

def get_ntotal(soup):
#fetch total number of records present for particular query
for div in soup.findAll('div', {'id':'left'}):
#print ivd.contents[0]
total_count ='[ ]+[0-9,]+',div.contents[0])
if total_count:',','').strip()
#print "total"+total_count
return total_count

def set_rating(soup,fwimdb_config,frating,year,startfrom):
cond = True
while cond:
for tr in soup.findAll('tr', {'class':re.compile('(odd|even)[ a-zA-Z]*')}): #each row
for td in tr.findAll('td', {'class':'title'} ):
for link in td.findAll('a',{'href':re.compile('/title/tt[^/]+/$')}):
movie_name=link.contents[0] #title name
for rating in td.findAll('div',{'class':'rating rating-list'}):
if rating.has_key('title'):
#print "hurray"
rt ='[0-9]+[^(]+',rating['title']) #rating
if rt:
#print movie_name+"\t"
if startfrom == 0:
startfrom = 101 #second run
startfrom = startfrom + 100

if startfrom >= int(total_res):
print str(startfrom)+" "+str(total_res)

def main():

frating = open("ratings","a") #ratings
startfrom,year = get_start_pos_yr(fwimdb_conf)

if startfrom == -1:
startfrom = 0
if year == None:
print startfrom



if __name__ == '__main__':

For demonstration purposes only. If you plan to use IMDB data beyond personal usage, you should contact IMDB Licensing department.