HDFS and YARN command cheat sheet

Topic – HDFS and YARN command cheat sheet
Let’s look into some of the top HDFS commands that you might require while working in a Big Data infrastructure.
HDFS and YARN command cheat sheetat sheet

HDFS and YARN command cheat sheet

1. hadoop fs -ls
   It lists the files in hadoop home directory
   
2. hadoop fs -ls -R/
   It lists the files in a recurrsive order
   
3. hadoop fs -ls/
   It lists the files in hadoop root directory
   
4. hadoop fs -ls -t -r
   It lists the files in reverse order, sorted by time
   
5. hadoop fs -ls -s
   It lists the files in the descending order of size
   
6. hadoop fs -ls /user |grep booking
   It searches the files with the name booking
   
7. hadoop fs -tail /user/thedatamonk/restaurant.txt
   It will display the last 10 rows of the file restaurant.txt
   
8. hadoop fs -rmdir /user/thedatamonk/direct
   It will remove directories
   
9. hadoop fs -rm -R /user/thedatamonk/direct
   It will remove empty and non-empty directories as well
   
10. hadoop fs -cp <complete file location 1> <complete directory location>
    It will move a file from one location to another directory.
    It copies a file from one location to another
    
11. hadoop fs -mv <complete file location 1> <complete folder location>
    It cut and paste file from one location to another directory
    
12. hadoop fs -copyFromLocal <Local folder location> <HDFS folder location>
    It copies file from local to HDFS location. Remember, the keyword copyFromLocal is case sensitive.
    
13. hadoop fs -put <Local folder location> <HDFS folder location>
    Another command to copy files from local to HDFS location. All the words are keywords.
    
14. hadoop fs -copyToLocal <HDFS data location> <Local folder location>
    It copies from HDFS to Local or desktop. Remember, the key word copyToLocal is case sensitive.
    
15. hadoop fs -get <HDFS data location> <Local folder location>
    Another command to copy from HDFS to Local or desktop.
    
16. hadoop fs -df -h <location>
    The above command gives the free space in the disk.
    -h converts the space into bytes
    
17. hadoop fs -du -h <location>
    The above command gets the used space in the disk
    
18. hadoop fs -touchz
    To create an empty file on the file system
    
19. hadoop fs -cat
    It copies files to stdout
    
    YARN Commands
    
20. yarn node -list
    list nodes in the yarn cluster
    
21. yarn node -status <node id>
    It gives status of a node (memory used, free, number of containers, etc) for (first column from command above)
    
22. yarn application -list
    It gives a list of Yarn applications and their state
    
23. yarn logs -applicationId <app id>
    It dumps the logs for a particular application
    
24. hdfs dfsadmin -safemode get
    It finds out if you’re in safe mode
    
25. hdfs dfsadmin -report
    It finds out how much disk space is used, free, under-replicated, etc.
    
    

The Data Monk Interview Books – Don’t Miss

Now we are also available on our website where you can directly download the PDF of the topic you are interested in. On Amazon, each book costs ~299, on our website we have put it at a 60-80% discount. There are ~4000 solved interview questions prepared for you.

10 e-book bundle with 1400 interview questions spread across SQL, Python, Statistics, Case Studies, and Machine Learning Algorithms – Ideal for 0-3 years experienced candidates

23 E-book with ~2000 interview questions spread across AWS, SQL, Python, 10+ ML algorithms, MS Excel, and Case Studies – Complete Package for someone between 0 to 8 years of experience (The above 10 e-book bundle has a completely different set of e-books)

12 E-books for 12 Machine Learning algorithms with 1000+ interview questions – For those candidates who want to include any Machine Learning Algorithm in their resume and to learn/revise the important concepts. These 12 e-books are a part of the 23 e-book package

Individual 50+ e-books on separate topics

Important Resources to crack interviews (Mostly Free)

There are a few things that might be very useful for your preparation

The Data Monk Youtube channel – Here you will get only those videos that are asked in interviews with Data Analysts, Data Scientists, Machine Learning Engineers, Business Intelligence Engineers, Analytics managers, etc.
Go through the watchlist which makes you uncomfortable:-

All the list of 200 videos
Complete Python Playlist for Data Science
Company-wise Data Science Interview Questions – Must Watch
All important Machine Learning Algorithm with code in Python
Complete Python Numpy Playlist
Complete Python Pandas Playlist
SQL Complete Playlist
Case Study and Guesstimates Complete Playlist
Complete Playlist of Statistics

Keep Learning !!

Thanks,
Nitin Kamal