How to scrape a form out a website using beautifulSoup? - forms

Question

So far we have made this code to scrape the website: http://www.theft-alerts.com
In the website is a form and in that form a frmSFair. We need all the stolen artworks information. Can someone help?
If we scrape the form by:
import urllib2
from bs4 import BeautifulSoup
connection = urllib2.urlopen('http://www.theft-alerts.com')
soup = BeautifulSoup(connection, "html.parser")
form = soup.find_all(span="table")
for form in soup.form.stripped_strings:
print(str(form.encode('utf-7')))
Output:
Sign up for our newsletter
Add email address below
See a sample eSalvo
The code picks the table newsletter on the ride side of the website and we need the table in the middle. This information:
STOLEN : CHERUB IN MARBLE, PART OF A FOUNTAIN
Stolen from Canterbury, Kent, UK on 8 February 2016
Item : A copy of Verrocchio's cupid - winged cherub standing on one leg holding a dolphin - in white marble which formed the top part of a fountain. approximately 3 foot high. Item has discoloured due to weathering with some lichen growth.
Any info to : PC 12994 Canterbury. Tel [phone]
Messages : Send a message
Crime Ref : ZY - 4370 - 16
No of items stolen : 1
images:
Location : UK > Kent
Category : STATUARY
ID : 93578
User : 53329 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 1 month)
Date Created : 10 Feb 2016 14:36:23
Date Modified : 11 Feb 2016 16:40:06;

Answer 1

To get the text from each:
for sp in soup.select("table div.itemspacingmodified"):
for wd in sp.select("div.itemindentmodified"):
text = wd.text
if not text.startswith("Images :"):
print(text)
Which gives you:
STOLEN : CHERUB IN MARBLE, PART OF A FOUNTAIN
Stolen from Canterbury, Kent, UK on 8 February 2016
Item : A copy of Verrocchio's cupid - winged cherub standing on one leg holding a dolphin - in white marble which formed the top part of a fountain. approximately 3 foot high. Item has discoloured due to weathering with some lichen growth.
Any info to : PC 12994 Canterbury. Tel [phone]
Messages : Send a message
Crime Ref : ZY - 4370 - 16
No of items stolen : 1
Location : UK > KentCategory : STATUARYID : 93578User : 53329 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 1 month)Date Created : 10 Feb 2016 14:36:23Date Modified : 11 Feb 2016 16:40:06;
STOLEN : OVER 70 ANTIQUE YORK STONE PAVING SLABS
Stolen from Steyning on 30th October 2015
Item : Antique York Stone paving slabs stolen from historic landscaped garden overnight. Truck driven through electric gates to gain access.
Any info to : PCSO Stewart Metcalfe. Sussex Police mob. [phone]
Messages : Send a message
Web URL : https://stmarysbramber.co.uk
Crime Ref : 47150140173
No of items stolen : 70
Recovered Details : None
Location : UK > West SussexCategory : FLAGSTONES & FLOOR TILESID : 92311User : 52866 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 3 months)Date Created : 05 Nov 2015 12:04:50Date Modified : 05 Nov 2015 12:15:10;
STOLEN : GARDEN STATUE OF BOY
Stolen from Bridgnorth, Shropshire on 20th / 21st Aug 2015
Item : Small lead,(I think ! ),statue of a boy standing on a stone plinth
Any info to : West Mercia Police - Crime No 22FJ59981W15
Messages : Send a message
Crime Ref : 22FJ59981W15
No of items stolen : 2
Recovered Details : NA
Location : UK > ShropshireCategory : STATUARYID : 91278User : 52457 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 6 months)Date Reset : 10 Sep 2015 00:30:01Date Modified : 26 Aug 2015 15:07:31; EL from 26 Aug 2015 to 09 Sep 2015 26Aug15;EL History : 26 Aug 2015 10:52:49;
STOLEN : YORKSTONE FLAGSTONES
Stolen from Nr Sevenoaks, Kent, UK on 26 Aug 2015
Item : Flagstones from St Mary's Church, Sundridge, Sevenoaks, Kent TN14 6EA. 70 in total. They are old Yorkstone flagstones approx. 2" thick, the sizes are as follows:
24 x 3'x2'
14 x 2'x1.5'
10 x 2'x2'
10 x 2'x1'
12 x 1'x1'
Any info to : Maidstone, Kent Police station Tel: 101
Messages : Send a message
Crime Ref : YY/17519/15
No of items stolen : 70
Location : UK > KentCategory : FLAGSTONES & FLOOR TILESID : 91428User : 52513 ; Churches and Memorial Custodians ; (Registered SalvoWEB user for 6 months)Date Created : 03 Sep 2015 14:48:45Date Modified : 04 Sep 2015 09:34:13;
STOLEN : RED STANDSTONE BIRDBATH
Stolen from Watlington, Oxfordshire UK on 16 July 2015
Item : A red sandstone bird bath with applied bronze decoration and a central bronze figure of a young girl on a dolphin by Richard Garbe
Overall height 4'3" (129.5 cm), Figure height 1' 3" (38 cm)
Square at bowl 1'3" (38 cm) Square at base 1'4" (40.4 cm)
Any info to : R McIntyre, PC 0200, Wallingford
Messages : Send a message
Crime Ref : 4315016713
No of items stolen : 1
Location : UK > OxfordshireCategory : STATUARYID : 89824User : 52054 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 6 months)Date Reset : 13 Jul 2015 00:30:02Date Modified : 21 Jun 2015 20:58:33; EL from 21 Jun 2015 to 12 Jul 2015 21Jun15;EL History : 21 Jun 2015 20:50:51;
STOLEN : STONE SCULPTURE - STATUE IS OF A PROPHET
Stolen from Belton, Loughbrough on 05/05/2015
Item : Statue is of a Prophet
5ft tall, this has been sympathetically restored by stonemason using a local stone which is very similar to the original stone.
The statue is 63" tall and weighs around ½ ton
Any info to : 3046515
Messages : Send a message
Crime Ref : 3046515
No of items stolen : 1
Location : UK > LeicestershireCategory : GARDENID : 89030User : 51750 ; Professional/Architect/Designer/Media/Film/TV ; (Registered SalvoWEB user for 6 months)Date Created : 06 May 2015 15:38:24Date Modified : 06 May 2015 16:18:28;
STOLEN : WOOL CARPET/RUG IN VARIOUS COLOURS GEOMETRIC DESIGN
Stolen from Kensal Green on 21 April 2015
Item : A knotted wool rug or carpet, approx 118 by 76 inches, with a geometric design in light and dark brown, cream, pink and black.
Any info to : Metropolitan Police. Tel: 101 and quote crime ref: 6518176/15
Messages : Send a message
Crime Ref : 6518176/15
No of items stolen : 1
Location : UK > London North WestCategory : FURNITURE & MIRRORSID : 88924User : 34 ; Antique/Reclamation/Salvage Trade ; (Salvo Code Dealer)Date Created : 29 Apr 2015 21:46:11Date Modified : 29 Apr 2015 22:24:45;
STOLEN : ANOTHER HISTORIC MILESTONE STOLEN AT REDBOURN, HERTS.
Stolen from Redbourn, Hertfordshire on 15th March 2015
Item : Very distinctive square section small Milepost, well-known to local residents and others, and featured in local publications. It was on the A5183 at St.Albans Road at Redbourn, opposite the Chequers Public House. It would originally have been installed by The Dunstable-St. Albans-London Turnpike Trust, established by Act of Parliament in 1722; after the abolition of this and other similar trusts responsibility would have fallen to the Parish Council, later passing to the County Council in the 20th Century.
This milestone has the Milestone Society Identity ref. HE_LH24. Its neighbour HE_LH23 was stolen two years ago.
Any info to : Hertfordshire Police 01707 354000
Messages : Send a message
Crime Ref : WCR/41/20187/15
No of items stolen : 1
Location : UK > WarwickshireCategory : Architectural STONE & TERRACOTTAID : 88606User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)Date Created : 10 Apr 2015 14:15:35Date Modified : 28 Apr 2015 16:52:03;
STOLEN : COALBROOKDALE GARDEN BENCH
Stolen from near Lichfield, Staffordshire UK on 25 February 2015
Item : A beautiful (most likely Coalbrookdale) cast iron 'Oak and Ivy' pattern bench. White painted with wooden slatted seat - but could now be a different colour. Approximate size 155 cm wide. Taken from a garden near Lichfield but possibly now in the Kent or South East region or anywhere in the country. Unusual and striking pattern. A £100 reward for first information which leads to recovery.
Any info to : PC 3864 Lichfield Tel [phone]
Messages : Send a message
Crime Ref : 27th February 2015 No 644
No of items stolen : 1
Location : UK > North YorkshireCategory : GARDENID : 87846User : 51382 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 1 year)Date Reset : 19 Mar 2015 00:30:01Date Modified : 11 Mar 2015 18:00:05; EL from 03 Mar 2015 to 10 Mar 2015 03Mar15; EL from 11 Mar 2015 to 18 Mar 2015 11Mar15;EL History : 02 Mar 2015 10:07:07;11 Mar 2015 00:30:01;
STOLEN : MILESTONE TAKEN FROM THE SIDE OF OLD LONDON ROAD, MALDON, ESSEX
Stolen from Maldon, Essex, UK on 31/1/15 - 11/3/15
Item : This is a milestone approximately 30 cms square but only around 90 total height including below surface. It was set into a concrete socket and would probably have needed lifting equipment to extract. It had received damage to one corner and face about six years ago, probably by carelessly operated grass-cutting machinery. It was situated opposite the cemetery in Old London Road (Grid Ref TL83740712) and was registered bythe Milestone Society with the ID ref EX_MGMN37
Any info to : PS 214 Maldon District Neighbourhood Policing Sergeant Direct Dial: 101 Ext 412104
Messages : Send a message
Crime Ref : CF0205920315
No of items stolen : 1
Location : UK > EssexCategory : Architectural STONE & TERRACOTTAID : 88150User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)Date Created : 16 Mar 2015 11:31:40Date Modified : 16 Mar 2015 11:45:20;
STOLEN : MILESTONE PLATE STOLEN. A420 JUST WEST OF CHIPPENHAM, WILTSHIRE
Stolen from Chippenham, Wiltshire on Before Christmas 2014
Item : This stone was involved in a major traffic accident sometime before Christmas 2014 and is now not only leaning over at a worse angle than ever but is in three or more pieces. The Cast iron plate on its front has disappeared, presumed stolen.
Any info to : Crime reference number 5410002796 reported by Wiltshire Council. Tel: 101 to report any news, quoting the ref. number.
Messages : Send a message
Crime Ref : 5410002796 Wiltshire
No of items stolen : 1
Location : UK > WiltshireCategory : Architectural METALWORKID : 86859User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)Date Created : 10 Jan 2015 17:26:33Date Modified : 28 Apr 2015 16:54:46;
STOLEN : A LARGE TAYLORS OF LOUGHBOROUGH BELL
Stolen from Bromyard on 7 August 2014
Item : The bell has a diameter of 37 1/2" is approx 3' tall weighs just shy of half a ton and was made by Taylor's of Loughborough in 1902. It is stamped with the numbers 232 and 11.
The bell had come from Co-operative Wholesale Society's Crumpsall Biscuit Works in Manchester.
Any info to : PC 2361. Tel [phone]
Messages : Send a message
Crime Ref : 22EJ / 50213D-14
No of items stolen : 1
Location : UK > Hereford & WorcsCategory : Shop, Pub, Church, Telephone Boxes & BygonesID : 84377User : 1 ; Antique/Reclamation/Salvage Trade ; (Administrator)Date Created : 11 Aug 2014 15:27:57Date Modified : 11 Aug 2014 15:37:21;
Each section is contained in a div with the class itemspacingmodified, then all the info is inside the divs with the class itemindentmodified so you just need to pull the text from each.
The only problem is the line breaks, you can see 91278User is syuck together, we can replace the line breaks with newlines:
connection = urllib2.urlopen('http://www.theft-alerts.com')
soup = BeautifulSoup(connection.read().replace("<br>","\n"), "html.parser")
for sp in soup.select("table div.itemspacingmodified"):
for wd in sp.select("div.itemindentmodified"):
text = wd.text
if not text.startswith("Images :"):
print(text)
So now we get:
STOLEN : CHERUB IN MARBLE, PART OF A FOUNTAIN
Stolen from Canterbury, Kent, UK on 8 February 2016
Item : A copy of Verrocchio's cupid - winged cherub standing on one leg holding a dolphin - in white marble which formed the top part of a fountain. approximately 3 foot high. Item has discoloured due to weathering with some lichen growth.
Any info to : PC 12994 Canterbury. Tel [phone]
Messages : Send a message
Crime Ref : ZY - 4370 - 16
No of items stolen : 1
Location : UK > Kent
Category : STATUARY
ID : 93578
User : 53329 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 1 month)
Date Created : 10 Feb 2016 14:36:23
Date Modified : 11 Feb 2016 16:40:06;
STOLEN : OVER 70 ANTIQUE YORK STONE PAVING SLABS
Stolen from Steyning on 30th October 2015
Item : Antique York Stone paving slabs stolen from historic landscaped garden overnight. Truck driven through electric gates to gain access.
Any info to : PCSO Stewart Metcalfe. Sussex Police mob. [phone]
Messages : Send a message
Web URL : https://stmarysbramber.co.uk
Crime Ref : 47150140173
No of items stolen : 70
Recovered Details : None
Location : UK > West Sussex
Category : FLAGSTONES & FLOOR TILES
ID : 92311
User : 52866 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 3 months)
Date Created : 05 Nov 2015 12:04:50
Date Modified : 05 Nov 2015 12:15:10;
STOLEN : GARDEN STATUE OF BOY
Stolen from Bridgnorth, Shropshire on 20th / 21st Aug 2015
Item : Small lead,(I think ! ),statue of a boy standing on a stone plinth
Any info to : West Mercia Police - Crime No 22FJ59981W15
Messages : Send a message
Crime Ref : 22FJ59981W15
No of items stolen : 2
Recovered Details : NA
Location : UK > Shropshire
Category : STATUARY
ID : 91278
User : 52457 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 6 months)
Date Reset : 10 Sep 2015 00:30:01
Date Modified : 26 Aug 2015 15:07:31; EL from 26 Aug 2015 to 09 Sep 2015 26Aug15;
EL History : 26 Aug 2015 10:52:49;
STOLEN : YORKSTONE FLAGSTONES
Stolen from Nr Sevenoaks, Kent, UK on 26 Aug 2015
Item : Flagstones from St Mary's Church, Sundridge, Sevenoaks, Kent TN14 6EA. 70 in total. They are old Yorkstone flagstones approx. 2" thick, the sizes are as follows:
24 x 3'x2'
14 x 2'x1.5'
10 x 2'x2'
10 x 2'x1'
12 x 1'x1'
Any info to : Maidstone, Kent Police station Tel: 101
Messages : Send a message
Crime Ref : YY/17519/15
No of items stolen : 70
Location : UK > Kent
Category : FLAGSTONES & FLOOR TILES
ID : 91428
User : 52513 ; Churches and Memorial Custodians ; (Registered SalvoWEB user for 6 months)
Date Created : 03 Sep 2015 14:48:45
Date Modified : 04 Sep 2015 09:34:13;
STOLEN : RED STANDSTONE BIRDBATH
Stolen from Watlington, Oxfordshire UK on 16 July 2015
Item : A red sandstone bird bath with applied bronze decoration and a central bronze figure of a young girl on a dolphin by Richard Garbe
Overall height 4'3" (129.5 cm), Figure height 1' 3" (38 cm)
Square at bowl 1'3" (38 cm) Square at base 1'4" (40.4 cm)
Any info to : R McIntyre, PC 0200, Wallingford
Messages : Send a message
Crime Ref : 4315016713
No of items stolen : 1
Location : UK > Oxfordshire
Category : STATUARY
ID : 89824
User : 52054 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 6 months)
Date Reset : 13 Jul 2015 00:30:02
Date Modified : 21 Jun 2015 20:58:33; EL from 21 Jun 2015 to 12 Jul 2015 21Jun15;
EL History : 21 Jun 2015 20:50:51;
STOLEN : STONE SCULPTURE - STATUE IS OF A PROPHET
Stolen from Belton, Loughbrough on 05/05/2015
Item : Statue is of a Prophet
5ft tall, this has been sympathetically restored by stonemason using a local stone which is very similar to the original stone.
The statue is 63" tall and weighs around ½ ton
Any info to : 3046515
Messages : Send a message
Crime Ref : 3046515
No of items stolen : 1
Location : UK > Leicestershire
Category : GARDEN
ID : 89030
User : 51750 ; Professional/Architect/Designer/Media/Film/TV ; (Registered SalvoWEB user for 6 months)
Date Created : 06 May 2015 15:38:24
Date Modified : 06 May 2015 16:18:28;
STOLEN : WOOL CARPET/RUG IN VARIOUS COLOURS GEOMETRIC DESIGN
Stolen from Kensal Green on 21 April 2015
Item : A knotted wool rug or carpet, approx 118 by 76 inches, with a geometric design in light and dark brown, cream, pink and black.
Any info to : Metropolitan Police. Tel: 101 and quote crime ref: 6518176/15
Messages : Send a message
Crime Ref : 6518176/15
No of items stolen : 1
Location : UK > London North West
Category : FURNITURE & MIRRORS
ID : 88924
User : 34 ; Antique/Reclamation/Salvage Trade ; (Salvo Code Dealer)
Date Created : 29 Apr 2015 21:46:11
Date Modified : 29 Apr 2015 22:24:45;
STOLEN : ANOTHER HISTORIC MILESTONE STOLEN AT REDBOURN, HERTS.
Stolen from Redbourn, Hertfordshire on 15th March 2015
Item : Very distinctive square section small Milepost, well-known to local residents and others, and featured in local publications. It was on the A5183 at St.Albans Road at Redbourn, opposite the Chequers Public House. It would originally have been installed by The Dunstable-St. Albans-London Turnpike Trust, established by Act of Parliament in 1722; after the abolition of this and other similar trusts responsibility would have fallen to the Parish Council, later passing to the County Council in the 20th Century.
This milestone has the Milestone Society Identity ref. HE_LH24. Its neighbour HE_LH23 was stolen two years ago.
Any info to : Hertfordshire Police 01707 354000
Messages : Send a message
Crime Ref : WCR/41/20187/15
No of items stolen : 1
Location : UK > Warwickshire
Category : Architectural STONE & TERRACOTTA
ID : 88606
User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)
Date Created : 10 Apr 2015 14:15:35
Date Modified : 28 Apr 2015 16:52:03;
STOLEN : COALBROOKDALE GARDEN BENCH
Stolen from near Lichfield, Staffordshire UK on 25 February 2015
Item : A beautiful (most likely Coalbrookdale) cast iron 'Oak and Ivy' pattern bench. White painted with wooden slatted seat - but could now be a different colour. Approximate size 155 cm wide. Taken from a garden near Lichfield but possibly now in the Kent or South East region or anywhere in the country. Unusual and striking pattern. A £100 reward for first information which leads to recovery.
Any info to : PC 3864 Lichfield Tel [phone]
Messages : Send a message
Crime Ref : 27th February 2015 No 644
No of items stolen : 1
Location : UK > North Yorkshire
Category : GARDEN
ID : 87846
User : 51382 ; Diyer/Homeowner/Private ; (Registered SalvoWEB user for 1 year)
Date Reset : 19 Mar 2015 00:30:01
Date Modified : 11 Mar 2015 18:00:05; EL from 03 Mar 2015 to 10 Mar 2015 03Mar15; EL from 11 Mar 2015 to 18 Mar 2015 11Mar15;
EL History : 02 Mar 2015 10:07:07;11 Mar 2015 00:30:01;
STOLEN : MILESTONE TAKEN FROM THE SIDE OF OLD LONDON ROAD, MALDON, ESSEX
Stolen from Maldon, Essex, UK on 31/1/15 - 11/3/15
Item : This is a milestone approximately 30 cms square but only around 90 total height including below surface. It was set into a concrete socket and would probably have needed lifting equipment to extract. It had received damage to one corner and face about six years ago, probably by carelessly operated grass-cutting machinery. It was situated opposite the cemetery in Old London Road (Grid Ref TL83740712) and was registered bythe Milestone Society with the ID ref EX_MGMN37
Any info to : PS 214 Maldon District Neighbourhood Policing Sergeant Direct Dial: 101 Ext 412104
Messages : Send a message
Crime Ref : CF0205920315
No of items stolen : 1
Location : UK > Essex
Category : Architectural STONE & TERRACOTTA
ID : 88150
User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)
Date Created : 16 Mar 2015 11:31:40
Date Modified : 16 Mar 2015 11:45:20;
STOLEN : MILESTONE PLATE STOLEN. A420 JUST WEST OF CHIPPENHAM, WILTSHIRE
Stolen from Chippenham, Wiltshire on Before Christmas 2014
Item : This stone was involved in a major traffic accident sometime before Christmas 2014 and is now not only leaning over at a worse angle than ever but is in three or more pieces. The Cast iron plate on its front has disappeared, presumed stolen.
Any info to : Crime reference number 5410002796 reported by Wiltshire Council. Tel: 101 to report any news, quoting the ref. number.
Messages : Send a message
Crime Ref : 5410002796 Wiltshire
No of items stolen : 1
Location : UK > Wiltshire
Category : Architectural METALWORK
ID : 86859
User : 46089 ; Charity/Government/Institution/Plc ; (Registered SalvoWEB user for 2 years or more)
Date Created : 10 Jan 2015 17:26:33
Date Modified : 28 Apr 2015 16:54:46;
STOLEN : A LARGE TAYLORS OF LOUGHBOROUGH BELL
Stolen from Bromyard on 7 August 2014
Item : The bell has a diameter of 37 1/2" is approx 3' tall weighs just shy of half a ton and was made by Taylor's of Loughborough in 1902. It is stamped with the numbers 232 and 11.
The bell had come from Co-operative Wholesale Society's Crumpsall Biscuit Works in Manchester.
Any info to : PC 2361. Tel [phone]
Messages : Send a message
Crime Ref : 22EJ / 50213D-14
No of items stolen : 1
Location : UK > Hereford & Worcs
Category : Shop, Pub, Church, Telephone Boxes & Bygones
ID : 84377
User : 1 ; Antique/Reclamation/Salvage Trade ; (Administrator)
Date Created : 11 Aug 2014 15:27:57
Date Modified : 11 Aug 2014 15:37:21;