Had a bit of a play. This is a basic python script to scrape stuff from yell.com - edit - the forum editor strips all the tabs which makes it very hard to read
--------------------------------------------------------------------------------
import mechanize
import BeautifulSoup
def scrapePage(soup):
for item in soup.findAll('div', "vcard"): #, recursive, text, limit):
name = "-- none --" if item.find('span', "org") == None else item.find('span', "org").string
address = "-- none --" if item.find('span', "street-address") == None else item.find('span', "street-address").string
tel = "-- none --" if item.find('span', "tel") == None else item.find('span', "tel").string
print "%s | %s | %s" % (name, address, tel)
return
# open main YELL page
response = mechanize.urlopen("http://www.yell.com/")
forms = mechanize.ParseResponse(response, backwards_compat=False)
form = forms[0]
#
# hi, Ive got a minute before I have to get back to my friends but I need a quick opinion on something ...
# do you know who the builders are in swindon ?
#
# fill in the search form fields
kw_control = form.find_control("keywords")
kw_control.value = "builder"
lc_control = form.find_control("location")
lc_control.value = "swindon"
request = form.click()
response = mechanize.urlopen(request)
# first result page ...
print "================================================= ======================="
soup = BeautifulSoup.BeautifulSoup(response.read());
scrapePage(soup)
# all other results pages ...
nextText = soup.find(text='Next')
while nextText != None :
print "================================================= ======================="
nextLink = nextText.parent
response = mechanize.urlopen("http://www.yell.com/" + nextLink['href'])
soup = BeautifulSoup.BeautifulSoup(response.read());
scrapePage(soup)
nextText = soup.find(text='Next')
----------------------------------------------------------------------------------
which generates this output ....
----------------------------------------------------------------------------------
================================================== ======================
UKTC Need New Members Now! | -- none -- | -- none --
D R Smith Building | 329, Windrush, | 01793 762899
Free Quotes from Builders | -- none -- | -- none --
Needhams Commercial Building Contractors | Unit 8, Oak Tree Business Centre, Spitfire Way, | 01793 834237
A F G Civil Engineering & Construction | 36, Warneford Close, | 07984 009941
A D S BUILDING CONTRACTORS | 26, Longleaze, | 01793 355204
ADS | 26, Longleaze, | 01793 355204
DWR Design & Build | 2, Constable Rd, | 01793355098
At Home Maintenance | 5, Marsh Farm Lane, | 01793 698367
Swift Design & Build Ltd | 39, Capesthorne Drive, | 01793 685 890
Andrew J Parker | 1, Martens Road, | 07887 902741
Donnelly Construction | 12, Nash Close, | 01793 687106
ADVANCED DOMESTIC SOLUTIONS LTD | -- none -- | 01793 687737
Rigg Construction (Southern) Ltd | Lancaster House Lancaster Park, | 01225 705668
C White Building Services | 1, Fritterswell Cottages, | 01666 238138
Hall Construction Ltd | Units 2-4, Tannery Yard, | 01993 822110
P B Cutley Building Services | 2 Wiltshire Close, | 01488 684102
Cordier Projects Ltd | 48-52, Silver Street, | 01454 238811
================================================== ======================
Avtar Construction Ltd | 149, Marlborough Road, | 01793 644 657
Milbrooke Construction Ltd | 3,Charnwood Court, Newport St, | 01793 685041
Clements Building Contractors Ltd | 2, Colbert Park, | 01793 687412
C9 DESIGN & DEVELOPMENT | 61, Havisham Drive, Haydon End, | 01793 685051
Heath Design & Construction Ltd | 156, Cricklade Rd, | 01793 685246
COOK & MARSH BUILDING & MAINTENANCE | 37, Purley Rd, | 01793 790 186
T.L General Builder | 1 Manor Crescent, | 01793 542195
Cr8 Property Solutions | 41, Beverley, | 01793 688530
Bob Watts | 5, Speedwell Close, | 01793 329552
Old Town Developments | 44, High St, | 01793 687419
MOULDING | Rye Ville, | 01722 742228
================================================== ======================
Godwin Construction | 20, Shrivenham Road, | 01793 762021
A.A Tanner | 2, Burton Grove Cottage, | 01793 823002
Phoenix Extensions Ltd | 7 Anise Close, Swindon, | 01793 700454
Durnford, Roger Building | 6 Beranburh Field, | 01793 812275
Allbuild Building Ltd | 76, Elborough Road, | 01793 423966
Paul Cook Construction Ltd | 3, Beech Lea, | 01793 729449
Hillside Building Services | 9, Downs View Rd, | 07990 640808
JJ Plastering & Home Improvements | 49, Marlborough Road, | 07871 555670
SLM WALLING LTD | 51, Rowan Rd, | 07952 215437
Living Solutions Swindon Ltd | Old Shaw Lane, | 07961 960610
================================================== ======================
A Gizzi Builders | 63, Burford Avenue, | 07940 267323
STORM FACILITIES MANAGEMENT | 27-28 Commercial Road, | 07909 985767
Great Western Builders Ltd | 29, Watling Close, | 01793 323560
C.J Warren Carpentry & Builders | 86, Bowood Rd, | 01793 487830
Stratton Builders | 20, Marlowe Avenue, Parks, | 07775 424582
Baldwin Bros (Builders) Ltd | 16 Rowan House, Holbrook Way, | 01793 420931
Heath Design & Construction Ltd | 156, Cricklade Rd, | 01793 685246
Hillside Building Services | 9, Downs View Rd, | 07990 640808
Able Builders | 14, Egerton Close, | 01793 614099
Hillsmead Construction | 2, Cricklade Court, Cricklade St, | 0800 6785648
================================================== ======================
T & T Building Services | 77, Burford Avenue, | 07702 097784
DWR Design & Build | 2, Constable Rd, | 0 1793355098
JJW Construction Ltd | 62, Curtis St, | 01793 322740
Mytalas & Jones | 61, Upham Rd, | 01793 693816
Bond the Builder | 19, Hamworthy Rd, | 01793 541810
Howard Gordon Properties | 55, The Mall, | 01793 421474
ADB Building Services | 13, Maxwell St, | 07877 895013
BSL Building Contractors Ltd | 38, Radnor St, | 07795 055335
G Galati Builders | Graceland, 205, Whitworht Rd, | 01793 522369
M.H General Builders | 35, Kershaw Rd, Eldene, | 01793 495019
================================================== ======================
P S Building | 39, Marlborough Rd, | 01793 694668
JDB Services Group | A15, Station Industrial Estate, Sheppard St, | 01793 615349
Bushtonferris Ltd | 50, Westlecot Rd, | 07850 602674
PG Builders | 29a, Lytchett Way, | 01793 524702
At Home Maintenance | 5, Marsh Farm Lane, | 01793 698367
Deco-Build | 105, Albion St, | 01793 323331
Cobern Builders | 3, Okebourne Park, | 01793 692624
Kent Construction (Swindon) Ltd | Unit 84 Base Point Business Centre, Rivermead Drive, | 01793 613181
P C Builders | 29, Dores Rd, | 01793 726207
Okus Construction Ltd | 68, Okus Rd, | 07721 450085
================================================== ======================
R Powers Ltd | 68, Birch St, | 01793 511366
Hima Construction | 30, Farleigh Crescent, | 0845 8392332
Limewood Contracts Ltd | 141, Okus Rd, | 01793 617336
K W Interiors | 30, Darnley Close, | 07541 277667
L.F Chalk Building Services | 8, Blakesley Close, | 01793 434186
Milbrooke Construction Ltd | 3,Charnwood Court, Newport St, | 01793 685041
Brian V Woolford | 365, Ferndale Rd, | 01793 694494
R.S & S.L Hewlett Builders | 25, Ermin Street, | 01793 827155
At Home Maintenance | 5, Marsh Farm Lane, | 01793 698367
Terry Loveridge & Son General Builders | 1, Manor Crescent, | 07734 810206
================================================== ======================
D J Smith | 7, Balmoral Close, | 01793 535618
T & P Property Services Ltd | 30, Northern Rd, | 01793 336661
Elite-Construction | 76, Stratton Rd, | 07956 545570
South West Trades | 11, Bridport Rd, Parknorth, | 01793 329329
T.L General Builder | 1 Manor Crescent, | 01793 542195
Avtar Construction Ltd | 149, Marlborough Road, | 01793 644 657
Allbuild Building Ltd | 76, Elborough Road, | 01793 423966
Tme Builders | 57, Windflower Rd, | 01793 728902
Living Solutions Swindon Ltd | Old Shaw Lane, | 07961 960610
A T Kitchin | 28, Vicarage Rd, | 01793 615617
================================================== ======================
A T Kitchin | 28, Vicarage Rd, | 01793 615617
Conlon Plant Ltd | Unit 6, Kendrick Trading Estate, Galton Way, | 01793 644956
TRACE BUILDERS | 14, Moresby Close, | 07879 625051
Kellaway Building Supplies | Larch Close, Off Kembrey Street, | 01793 500210
Bob Watts | 5, Speedwell Close, | 01793 329552
Clements Building Contractors Ltd | 2, Colbert Park, | 01793 687412
B.R Matthews General Builder | 19, Taylor Crescent, Kingsdown Park, | 01793 825458
Why D.I.Y Ltd | Thorne Rd, Eldene, | 01793 345540
A Touch of Klass | 76, Okebourne Park, | 01793 610711
Premier Builders Swindon Ltd | 14a, Turnpike Rd, | 01793 704653
================================================== ======================
Raw Developments | 69, Nythe Rd, | 01793 728823
Andrew J Parker | 1, Martens Road, | 07887 902741
J's Property Services | 26, Meadowcroft, | 01793 828818
RH Improvements | 2, Larksfield, | 07816 917325
Dawson & Marsh Ltd | Unit C, Athena Business Centre, Moray Rd, Elgin Industrial Estate, | 0844 8003306
Donnelly Construction | 12, Nash Close, | 01793 687106
KW Builders | Flat 15, Charlton House, 7, Delft Crescent, | 07872 030987
R & B Developments | 54, Moredon Rd, | 07736 808581
B.A Platek | 57, Birchwood Rd, | 01793 828563
Fusco Builders Ltd | 8, Marlborough Rd, | 01793 850722
---------------------------------------------------------------------------------
Needs bit of work of course to turn it into a real application. Note that this was my first ever bit of coding in python, thought I would use this experiment to learn another language.