Basic usage¶
TextScrubber¶
The text_scrubber.text_scrubber.TextScrubber
class cleans a single or a collection of strings. It can be easily
constructed and configured with building blocks:
from text_scrubber import TextScrubber
ts = (TextScrubber().to_ascii()
.lowercase()
.tokenize()
.remove_stop_words()
.join())
which can then be used as:
ts.transform('héLlô there, WòrlD') # outputs 'hello world'
or with an iterable of input:
ts.transform(['héLlô there, WòrlD', 'slímm̀er ÀI']) # outputs ['hello world', 'slimmer AI']
For a complete list of building blocks please refer to the text_scrubber.text_scrubber.TextScrubber
API
reference.
Geo¶
The text_scrubber.geo
module contains functions to normalize geographical data which deal with spelling errors,
country name variations, etc.:
from text_scrubber.geo import normalize_country, normalize_region, normalize_city
# Countries
normalize_country('Peoples rep. of China') # [('China', 1.0)]
normalize_country('Deutschland') # [('Germany', 1.0)]
normalize_country('st Nevis and Kitties') # [('Saint Kitts and Nevis', 0.75)]
normalize_country('ira') # [('Iran', 0.857), ('Iraq', 0.857)]
# Cities
normalize_city('Leibnitz', ['Austria']) # [('Leibnitz', 'Austria', 1.0)]
normalize_city('heidelberg') # [('Heidelberg', 'Germany', 1.0),
# ('Heidelberg', 'South Africa', 1.0),
# ('Heidelberg', 'United States', 1.0)]
normalize_city('ohioo', ['US']) # [('Ohio', 'United States', 0.889)]
normalize_city('Madri', ['Spain', 'US', 'Brazil']) # [('Madrid', 'Spain', 0.909),
# ('Madrid', 'United States', 0.909),
# ('Mari', 'Brazil', 0.889)]
# Regions
normalize_region('triangle park', ['US']) # [('The Triangle Park', 'United States', 1.0)]
normalize_region('Fur', ['Denmark']) # [('Fur', 'Denmark', 1.0)]
normalize_region('texel', ['NL']) # [('Texel', 'Netherlands', 1.0)]
Each of the above normalization functions will return the match score as last entry in the tuple. These scores are
always between 0.0 and 1.0, where 1.0 is a perfect match. If a known mapping exists, like Deutschland
to
Germany
, then the match score will be 1.0.
The text_scrubber.geo
module also contains functions to find the name of places (country, region, and city) in
text dealing with spelling errors, country name variations, etc.:
from text_scrubber.geo import (find_city_in_string, find_country_in_string,
find_region_in_string)
# Countries
find_country_in_string("Institute of German study, Accra, Ghana")
# Returns: [Match(substring_range=(34, 39), substring='Ghana',
# normalized='Ghana', score=1.0),
# Match(substring_range=(13, 19), substring='German',
# normalized='Germany', score=0.923)]
find_country_in_string("Peking University, 5 Yiheyuan Rd, "
"Haidian District, Beijing, CH, 100871")
# Returns: [Match(substring_range=(61, 63), substring="CH",
# normalized="China", score=1.0)]
# Cities
find_city_in_string("Météorage Pau France", {"France"})
# Returns: [Match(substring_range=(10, 13), substring="Pau",
# normalized=("Pau", "France"), score=1.0),
# Match(substring_range=(14, 20), substring="France",
# normalized=("La Frasnée", "France"), score=0.909)]
find_city_in_string("Bavarian Environment Agency, Hans Högn Straße 12, "
"95030 Hof Saale, Bavaria, Germany", {"Germany})
# Returns: [Match(substring_range=(56, 59), substring='Hof',
# normalized=('Hof', 'Germany'), score=1.0),
# Match(substring_range=(39, 45), substring="Straße",
# normalized=("Trassem", "Germany"), score=0.857)]
# Regions
find_region_in_string("Fur Museum, 7884 Fur, Denmark.", {"Denmark"})
# Returns: [Match(substring_range=(0, 3), substring='Fur',
# normalized=('Fur', 'Denmark'), score=1.0),
# Match(substring_range=(17, 20), substring='Fur',
# normalized=('Fur', 'Denmark'), score=1.0),
# Match(substring_range=(22, 29), substring='Denmark',
# normalized=('Kingdom of Denmark', 'Denmark'), score=1.0)]
find_region_in_string("Department of Biological Oceanography, Royal Netherlands Institute "
"for Sea Research (NIOZ), Texel, The Netherlands", {"Netherlands"})
# Returns: [Match(substring_range=(45, 56), substring='Netherlands',
# normalized=('Kingdom of the Netherlands', 'Netherlands'), score=1.0),
# Match(substring_range=(92, 97), substring='Texel',
# normalized=('Texel', 'Netherlands'), score=1.0),
# Match(substring_range=(103, 114), substring='Netherlands',
# normalized=('Kingdom of the Netherlands', 'Netherlands'), score=1.0)]
Note
Whenever a country is considered part of another country normalize_country
will return the latter.
E.g., Puerto Rico
is mapped to United States
and Greenland
to Denmark
.
Cleaning¶
There are clean functions available for countries/regions/cities, which all follow the same cleaning pipeline:
from text_scrubber.geo import clean_country, clean_region, clean_city
clean_country('cent afr rep.') # 'central african republic'
clean_region('Hyōgo') # 'hyogo'
clean_city('płońsk') # 'plonsk'
clean_city('neustadt/westerwald') # 'neustadt westerwald'