How do I better clean extremely messy data with OpenRefine? - data-cleaning

Question

I'm using the latest version of OpenRefine and I am cleaning a CSV file of 33,230 rows. There are 5 rows that I'm working with (Name, Personal_email, Phone_number, Twitter_handle, and Website). I have cleaned the data by adding titles, sorting the ID, applying titlecase, etc...but I have one major problem that I can't fix, and atomization didn't work. For instance, the Name column is perfect, but for some rows, the personal email is in the Phone_number column, or the Twitter handle is in Website row. How do I fix this without having to manually check each row? Also, how would I add "http" to the beginning of each Website entry without changing the information already in that column? Thanks!

Answer 1

I am not a user of OpenRefine, but I have lots of experience to handle messy data using python and pandas.
In the data cleaning process, first, I will find the rules inside the data and filter the rows without proper format from the raw data, e.g.
Personal_email must contain '#'.
Phone_number, should only have digits and '-'.
Do Twitter_handle has length limit?
website starts with www?
Second, I will throw these rows if they are not important or the number of them is small vs the retains ones(5 vs 33,230).
Or just handle it in the raw file if the mistakes are litter.

Answer 2

I believe, OpenRefine has all features, required for this task.
This task can be solved by the following steps:
create new column "mixed_data"
using join of your columns (Name, Personal_email, Phone_number, Twitter_handle, and Website) with some unique separator character, which can not be found as a part of value text, let say "|", then cell in this column may look as
"Smith, John|s.john1#mail.com|[phone]|#sjohn|www.jsmith.com"
join menu
join parameters
consequently create copies of your original columns, based on "mixed_data" using regular expressions:
Personal_email2:
value.find(/(^|[|])([A-z0-9_.-]+[#][A-z0-9_.-]+)([|]|$)/)[0]
Phone_number2: value.find(/(^|[|])([+]?[0-9 -]+)([|]|$)/)[0]
Twitter_handle2: value.find(/(^|[|])([#][A-z0-9]+)([|]|$)/)[0]
Website2: value.find(/(^|[|])([A-z0-9_][A-z0-9_.-]+)([|]|$)/)[0]
Note. Regular expressions above can be enhanced, but in some cases would be enough.
Resulting columns will look like ones on attached picture:
OpenRefine sample sheet
Remove remaining "|" characters in columns by "Edit cells->Replace" feature
Or just copy&paste json below and apply it in "Apply Operation History" window:
[
{
"op": "core/column-addition",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"baseColumnName": "Website",
"expression": "join ([cells['Website'].value,cells['Name'].value,cells['Personal_email'].value,cells['Phone_number'].value,cells['Twitter_handle'].value],'|')",
"onError": "keep-original",
"newColumnName": "mixed_data",
"columnInsertIndex": 5,
"description": "Create column mixed_data at index 5 based on column Website using expression join ([cells['Website'].value,cells['Name'].value,cells['Personal_email'].value,cells['Phone_number'].value,cells['Twitter_handle'].value],'|')"
},
{
"op": "core/column-addition",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"baseColumnName": "mixed_data",
"expression": "grel:value.find(/(^|[|])([A-z0-9_][A-z0-9_.-]+)([|]|$)/)[0]",
"onError": "set-to-blank",
"newColumnName": "Website2",
"columnInsertIndex": 6,
"description": "Create column Website2 at index 6 based on column mixed_data using expression grel:value.find(/(^|[|])([A-z0-9_][A-z0-9_.-]+)([|]|$)/)[0]"
},
{
"op": "core/column-addition",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"baseColumnName": "mixed_data",
"expression": "grel:value.find(/(^|[|])([#][A-z0-9]+)([|]|$)/)[0]",
"onError": "set-to-blank",
"newColumnName": "Twitter_handle2",
"columnInsertIndex": 6,
"description": "Create column Twitter_handle2 at index 6 based on column mixed_data using expression grel:value.find(/(^|[|])([#][A-z0-9]+)([|]|$)/)[0]"
},
{
"op": "core/column-addition",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"baseColumnName": "mixed_data",
"expression": "grel:value.find(/(^|[|])([+]?[0-9 -]+)([|]|$)/)[0]",
"onError": "set-to-blank",
"newColumnName": "Phone_number2",
"columnInsertIndex": 6,
"description": "Create column Phone_number2 at index 6 based on column mixed_data using expression grel:value.find(/(^|[|])([+]?[0-9 -]+)([|]|$)/)[0]"
},
{
"op": "core/column-addition",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"baseColumnName": "mixed_data",
"expression": "grel:value.find(/(^|[|])([A-z0-9_.-]+[#][A-z0-9_.-]+)([|]|$)/)[0]",
"onError": "set-to-blank",
"newColumnName": "Personal_email2",
"columnInsertIndex": 6,
"description": "Create column Personal_email2 at index 6 based on column mixed_data using expression grel:value.find(/(^|[|])([A-z0-9_.-]+[#][A-z0-9_.-]+)([|]|$)/)[0]"
},
{
"op": "core/text-transform",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"columnName": "Personal_email2",
"expression": "value.replace(\"|\",\"\")",
"onError": "keep-original",
"repeat": false,
"repeatCount": 10,
"description": "Text transform on cells in column Personal_email2 using expression value.replace(\"|\",\"\")"
},
{
"op": "core/text-transform",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"columnName": "Phone_number2",
"expression": "value.replace(\"|\",\"\")",
"onError": "keep-original",
"repeat": false,
"repeatCount": 10,
"description": "Text transform on cells in column Phone_number2 using expression value.replace(\"|\",\"\")"
},
{
"op": "core/text-transform",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"columnName": "Twitter_handle2",
"expression": "value.replace(\"|\",\"\")",
"onError": "keep-original",
"repeat": false,
"repeatCount": 10,
"description": "Text transform on cells in column Twitter_handle2 using expression value.replace(\"|\",\"\")"
},
{
"op": "core/text-transform",
"engineConfig": {
"facets": [],
"mode": "row-based"
},
"columnName": "Website2",
"expression": "value.replace(\"|\",\"\")",
"onError": "keep-original",
"repeat": false,
"repeatCount": 10,
"description": "Text transform on cells in column Website2 using expression value.replace(\"|\",\"\")"
}
]