HDFS and YARN command cheat sheet
Topic – HDFS and YARN command cheat sheet
Let’s look into some of the top HDFS commands that you might require while working in a Big Data infrastructure.
HDFS and YARN command cheat sheetat sheet
HDFS and YARN command cheat sheet
- hadoop fs -ls
It lists the files in hadoop home directory
- hadoop fs -ls -R/
It lists the files in a recurrsive order
- hadoop fs -ls/
It lists the files in hadoop root directory
- hadoop fs -ls -t -r
It lists the files in reverse order, sorted by time
- hadoop fs -ls -s
It lists the files in the descending order of size
- hadoop fs -ls /user |grep booking
It searches the files with the name booking
- hadoop fs -tail /user/thedatamonk/restaurant.txt
It will display the last 10 rows of the file restaurant.txt
- hadoop fs -rmdir /user/thedatamonk/direct
It will remove directories
- hadoop fs -rm -R /user/thedatamonk/direct
It will remove empty and non-empty directories as well
- hadoop fs -cp <complete file location 1> <complete directory location>
It will move a file from one location to another directory.
It copies a file from one location to another
- hadoop fs -mv <complete file location 1> <complete folder location>
It cut and paste file from one location to another directory
- hadoop fs -copyFromLocal <Local folder location> <HDFS folder location>
It copies file from local to HDFS location. Remember, the keyword copyFromLocal is case sensitive.
- hadoop fs -put <Local folder location> <HDFS folder location>
Another command to copy files from local to HDFS location. All the words are keywords.
- hadoop fs -copyToLocal <HDFS data location> <Local folder location>
It copies from HDFS to Local or desktop. Remember, the key word copyToLocal is case sensitive.
- hadoop fs -get <HDFS data location> <Local folder location>
Another command to copy from HDFS to Local or desktop.
- hadoop fs -df -h <location>
The above command gives the free space in the disk.
-h converts the space into bytes
- hadoop fs -du -h <location>
The above command gets the used space in the disk
- hadoop fs -touchz
To create an empty file on the file system
- hadoop fs -cat
It copies files to stdout
- yarn node -list
list nodes in the yarn cluster
- yarn node -status <node id>
It gives status of a node (memory used, free, number of containers, etc) for (first column from command above)
- yarn application -list
It gives a list of Yarn applications and their state
- yarn logs -applicationId <app id>
It dumps the logs for a particular application
- hdfs dfsadmin -safemode get
It finds out if you’re in safe mode
- hdfs dfsadmin -report
It finds out how much disk space is used, free, under-replicated, etc.
The Data Monk Interview Books – Don’t Miss
Now we are also available on our website where you can directly download the PDF of the topic you are interested in. On Amazon, each book costs ~299, on our website we have put it at a 60-80% discount. There are ~4000 solved interview questions prepared for you.
10 e-book bundle with 1400 interview questions spread across SQL, Python, Statistics, Case Studies, and Machine Learning Algorithms – Ideal for 0-3 years experienced candidates
23 E-book with ~2000 interview questions spread across AWS, SQL, Python, 10+ ML algorithms, MS Excel, and Case Studies – Complete Package for someone between 0 to 8 years of experience (The above 10 e-book bundle has a completely different set of e-books)
12 E-books for 12 Machine Learning algorithms with 1000+ interview questions – For those candidates who want to include any Machine Learning Algorithm in their resume and to learn/revise the important concepts. These 12 e-books are a part of the 23 e-book package
Important Resources to crack interviews (Mostly Free)
There are a few things that might be very useful for your preparation
The Data Monk Youtube channel – Here you will get only those videos that are asked in interviews with Data Analysts, Data Scientists, Machine Learning Engineers, Business Intelligence Engineers, Analytics managers, etc.
Go through the watchlist which makes you uncomfortable:-
All the list of 200 videos
Complete Python Playlist for Data Science
Company-wise Data Science Interview Questions – Must Watch
All important Machine Learning Algorithm with code in Python
Complete Python Numpy Playlist
Complete Python Pandas Playlist
SQL Complete Playlist
Case Study and Guesstimates Complete Playlist
Complete Playlist of Statistics
Keep Learning !!