Skip to main content

Filesystem is full, but can't find space being used - Solaris

I saw a strange issue once where the  /staging filesystem is almost full, but I can't find where the space is being used. 

'df -h' command shows that the filesystem is 100% used as below.

==>df -h
Filesystem             size   used  avail capacity  Mounted on
/staging                 99G   99G   0K     100%      /staging

But 'du -h' command says that only 3.7G out of 99G is used.

==> pwd
/staging
==> du -h | grep "\.$"
3.7G .



This is strange , now how will you find which files are consuming remaining 95.3G of disk space. The unix command which can be used here is 'lsof' - list of open files.

==> lsof | grep -i /staging

cat        4723  user1 txt   VREG     270,28001          50028743680      15496 /staging (/dev/vx/dsk/staging)
cat        4723  user1 3r  VREG     270,28001          50028743680      15496 /staging (/dev/vx/dsk/staging)
sed        4724  user1 1w  VREG     270,28001          50028743680      15496 /staging (/dev/vx/dsk/staging)
ksh        8796   jg8789  cwd   VDIR     270,28001              1949696         11 /app/staging/pollers
ksh       10834  collect  cwd   VDIR     270,28001              1949696         11 /app/staging/pollers
cat       15794  user1  txt   VREG     270,28001          52067398891      36877 /staging (/dev/vx/dsk/staging)
cat       15794  user1 3r  VREG     270,28001          52067398891      36877 /staging (/dev/vx/dsk/staging)
sed       15795  user1 1w  VREG     270,28001          50028743680      15496 /staging (/dev/vx/dsk/staging)

All the highlighted entries sheds light on to the processes that are using huge disk space. Here we can see many 'cat', 'sed' commands are holding huge disk space which can be released by terminating the processes.

==> kill -9 4723  4724  15794  15795  

 
 DESCRIPTION ABOUT lsof 
    Lsof revision 4.82 lists on its standard output file infor-  
    mation about files opened by processes for the following  
    UNIX dialects:  
      AIX 5.3  
      Apple Darwin 9 (Mac OS X 10.5)  
      FreeBSD 4.9 for x86-based systems  
      FreeBSD 7.[012] and 8.0 for AMD64-based systems  
      Linux 2.1.72 and above for x86-based systems  
      Solaris 9 and 10  
    (See the DISTRIBUTION section of this manual page for infor-  
    mation on how to obtain the latest lsof revision.)  
    An open file may be a regular file, a directory, a block  
    special file, a character special file, an executing text  
    reference, a library, a stream or a network file (Internet  
    socket, NFS file or UNIX domain socket.) A specific file or  
    all the files in a file system may be selected by path.

Comments

Post a Comment

Popular posts from this blog

Create SNMP Client in JAVA Using SNMP4j

Please follow the below mentioned steps to write a very simple SNMP4J application using Java and eclipse IDE. First you need to download a copy of SNMP4J.jar. You can get it from http://www.snmp4j.org/html/download.html Then you need to decide how you are going to test your application. Check whether you have access to any real network devices like Cisco routers or Juniper routers or whether you are aware of any SNMP agent running on any of your servers.There is nothing to worry if you cannot find one , you can start SNMP agent on your desktop machine as well. Here I am giving an example of how you can use your Windows machine to test your application by starting the SNMP service on the machine. I am using a Windows XP machine and below are the steps needed to start SNMP agent on the machine. 1. Go to Start->Control Panel -> Add or Remove Programs -> Add or Remove Windows Components      Select or Check "Management and Monitoring Tools". ...
16 comments
Read more

Symantec Interview questions for the role of a Senior Software Developer C++

Below I have mentioned a few interview questions for the role of a Senior Software Engineer at Symantec. They were looking for a Senior person with mainly Windows API programming experience. I had 7 rounds of interview including a 'lunch with Manager' and a Director's interview. I have mentioned the questions below. I hope this would help some Developers looking for Symantec Interview Questions. Let me know your comments... C++/ OOPS Questions Rate yourself in C++ The interviewer wrote a  class with 2 members , a char* and a HANDLE  and asked me to complete the class. He was mainly looking for proper assignment operator and copy constructors. What is polymorphism What is abstract class and when to use abstract class How is virtual function implemented What is vptr, is vptr created per class or per object What is strlen_s() call would do? When to throw or catch an exception? Explain a few Design Patterns? Where did you use singleton and Factory patterns? ...
51 comments
Read more

How to create a minikube single node cluster for learning Kubernetes

In this post I will explain how to setup a minikube single node kubernetes cluster using AWS EC2 instance which would help anyone who is trying to learn kubernetes and also help them to gain practical knowledge in kubernetes by running kubernetes commands, creating kubernetes objects etc. Minikube is a single node kubernetes cluster which means a kubernetes cluster with only one node that is a single VM. Minikube is only used for learning purposes and it is not an alternative for a real kubernetes cluster and should not be used for development and production usage. In this example I have launched an AWS EC2 instance with below configuration where I will install minikube and related tools. AWS EC2 Instance Configuration AMI: Ubuntu Free tier eligible 64 bit Instance type : t2-large ( For me t2-small or t2-micro is giving performance issues due to less memory) Once the EC2 instance is up and running, login to the instance using below command on terminal. If you are using wi...
27 comments
Read more