2

There are lot of these kinds of questions on superuser, so I'll try and keep this quick by telling you what it's not:

  • There is no application involved here. Just a user and a filesystem.
  • I can login as root and my non-root user has full sudo access.
  • As far as I know, this problem only exists in one directory (though given enough time I think I can recreate this behavior in any directory -- more on that below).
  • No strange filesystem customizations, umasks or odd user or directory permission schemes -- I own this system, and no other people interact with it.

This directory is a destination path for backup scripts written in bash. These scripts are not fancy. Rsync files from elsewhere to a working directory. Tar up all the files in the working directory and store the resulting archives in this destination path. Pretty simple stuff.

Let's say this is a fresh install, for this example. The script works fine for a while. But after some period of weeks, no files can be created in the destination path anymore, by any user, using any method. I've tried touch, tar, zip, etc. Nothing works. Tar and zip will fail with I/O errors, touch acts fine but no file gets created. In that destination path, and only that path. File creation/modification/deletion works fine everywhere else on the filesystem (if it's happening elsewhere, it's asymptomatic).

permissions on the dir look like this via an ls -al: drwxrwxr-x. -- owned by my non-root user and its group.

Lastly, if I reboot the system after this condition presents itself, I can create files in that path again for a short time. Eventually though, the problem returns.

When I said I can probably recreate this in another dir, it's because I've changed the destination path to something else trying to fix this a few different times already, and it eventually happens in every location I've tried.

System Details:

  • CentOS 2.6.32-431.3.1.el6.x86_64
  • Filesystem is ext4
  • 400GB free on / (lvm)

Any ideas or thoughts on things to try would be greatly appreciated. I'm happy to go into more detail if needed, but I wanted to keep this brief to start with.

Thanks for your time!

EDIT 10/20/14: Still no answers here. Can I get anyone any more info?

EDIT 10/8/14: Adding information as requested in comments:
$ ls -l
total 0

$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
vg_omega1 1 2 0 wz--n- 464.24g 0

EDIT 10/7/14: Adding information as requested in comments:
Mount results:
/dev/mapper/vg_omega1-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Path:
/home/[username]/savegames/sync

EDIT 10/6/14: Further testing indicates I can apparently create sub-directories and symlinks just fine. This problem appears to be limited to regular files.

  • Please add the output of mount along with the path of where you are trying to create files. – 0x7c0 Oct 7 '14 at 7:21
  • If you create a sub directory can you then create files okay in that sub directory? – 0x7c0 Oct 7 '14 at 20:06
  • Tested. No. The directory gets created fine. Cannot create files in the new sub-directory. Same behavior as the parent dir. – Bob B. Oct 7 '14 at 20:39
  • May i know the current size of the affected directory (ls -l output).. Also please add output of vgs command – VDR Oct 8 '14 at 6:26
  • The problem is related specifically to this script? You can create the directory and use it indefinitely for other purposes but the problem starts when you run the script? After the problem starts, can you create any type of file by any other means in that directory? – fixer1234 Oct 8 '14 at 18:53
0

I can't comment as I don't have the rep, but seeing your mount output it looks like you have SELinux running. From experience, it's hard to track "weird" stuff like this down unless you're familiar with SELinux. 5+ years dealing with it has taught me a trick or two.

tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")

You also mention you're using rsync, and this problem "goes away and comes back again". Is your SELinux options set to enforce by default? (applied again after a reboot), and you're (sometimes) putting SELinux into permissive mode (5 mins, or 10 days etc time) after you reboot?

sudo getenforce

If the result of that is enforcing , and If you do have SELinux set to enforce by default, chances are that you're running into SELinux policy denials.

Ref the RedHat 6 admin guide - Ch 12 - rsync

If you

ls -lZ /home/[username]/savegames/sync 

What is the context label? It should be something like public_content_t It may be user_home_t which means a lot of SELinux confined processes will be denied access. There will be denial messages in /var/log/audit/audit.log

If SELinux is indeed the culprit, ref CentOS SELinux troubleshooting guide for ways to sort this out. You could:

  • Put SELinux into permissive mode and see if your problem goes away. Put it back into enforcing mode and your problem will come back. Not a long term solution, so...
  • Either relabel the context of the /home/[username]/savegames/sync folder (to public_content_t or similar) or
  • Create policies to allow your scripts/processes to work with your folder (audit2allow). This looks daunting but is not impossible.

PLEASE. DO NOT TURN SELINUX OFF.

Well, having said that, if you're the only one using the system, and you're not serving websites or FTP or anything, okay maybe leave it in permissive mode, but good practice suggests you modify your policy to give you a key to a door, rather than just take the door off its hinges and put it aside :-)

  • I'm still researching, but this response is fantastic. Here's what I know at the moment. I'll post a followup comment when I'm done either way: $ sudo getenforce Enforcing $ ls -lZ /home/[user]/savegames drwxrwxr-x. [user] [user] unconfined_u:object_r:user_home_t:s0 sync – Bob B. Oct 9 '14 at 19:49
  • Put SELinux into permissive mode and ran the job again. No change in behavior. Fails the same way. No denials in the audit log either. – Bob B. Oct 9 '14 at 21:02
  • Ah, sorry that didn't help then. – Bert GT Oct 9 '14 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.