How to Clean Data at the Command Line

Learn how to clean data using command-line utility jq, xsv, and csvkit.
Share the link to this page
Copied

About the Product

Cleaning data is a widely known process that can let us explore data and see beyond its raw form. Multiple technologies can solve this task, but we have a problem.

The data-driven problem we face: Whenever you want to import a CSV file, by habit, you go to Google and see how to find the two lines that you always forget (in Python for example) so you get them to open up your text editor to make a file and paste what you found in it.

Why the command line?

The simplest data cleaning tasks might sound frustrating or time-wasting and maybe you use a higher-level library like Pandas but I bet you still write more code than just dealing with the terminal which can pack a bunch of lines of codes into just one-liner at the command line.

This ebook makes dealing with CSV files, JSON, or in general any text file much easier.

What's in it for you?

In this ebook, I'm trying to save your time and the hassle of dealing with files at the system level. You may also like the adventure of exploring command-line tools and programs that you may not have heard of. I encourage you to try these tools as I do on my workdays.

While dealing with the command line may sound a bit geeky, this ebook is simple and easy to follow, and it's a lot of fun. There are real examples from a scientific paper, Covid tracking project data, Reddit user data, and more that you can practice with and try useful programs and tools at the comfort of your command line.

Content:

  • In this ebook you'll be able to clean data using command-line tools: tr, grep, sort, uniq, sort, awk, sed, and csvlook and practice cleaning a COVID-19 CSV file using command-line programs: csvkit and xsv comparing the performance of each.
  • You'll also see how to sort and concatenate a large CSV file with csvkit and xsv, and calculate their performance with respect to Pandas.
  • In the last chapter, you'll get to know how to clean a JSON file using the command-line program jq.

Who should take this Product?

If you are a data scientist, data engineer, data analyst, software developer, or you use data a lot (like TXT, CSV, or JSON), this ebook is for you.

Requirements

You should have a basic understanding of how the terminal works.

Author

Ezzeddin Abdullah

Data Platform Engineer

School

Ezzeddinabdullah's School

One-time Fee
$8
€7.37
£6.33
CA$10.87
A$12.15
S$10.79
HK$62.63
CHF 7.21
NOK kr85.83
DKK kr55.05
NZ$13.30
د.إ29.37
৳876.79
₹666.37
RM37.98
₦10,357.08
₨2,220.32
฿293.20
₺256.57
B$40.52
R149.57
Лв14.42
₩10,810.15
₪30.10
₱452.72
¥1,212.84
MX$131.62
QR29.12
P108.69
KSh1,038.57
E£379.04
ብር452.84
Kz6,696
CLP$7,595.60
CN¥57.85
RD$473.09
DA1,077.96
FJ$17.97
Q62.23
GY$1,672.77
ISK kr1,109.44
DH80.54
L141.06
ден454.32
MOP$64.42
N$148.61
C$294.05
रु1,064.60
S/29.46
K30.62
SAR30
ZK196.72
L36.67
Kč186.88
Ft2,880.62
SEK kr85.41
ARS$6,884.46
Bs55.20
COP$30,174.29
₡4,066.54
L197.19
₲58,878.17
$U305.70
zł31.59

What's Included

File Size: 654K
Pages: 36
Language: English
Level: All levels
Skills: Data Cleaning, Command-line, Bash, Scripting, Data Manipulation
Age groups: All ages

Sign Up

Share

Share with friends, get 20% off
Invite your friends to LearnDesk learning marketplace. For each purchase they make, you get 20% off (upto $10) on your next purchase.