2.1 beta 3 import/watch - Airtime Development Discussions on Sourcefabric Forum

Wade Eilrich

I brought up a new Ubuntu 10.04 server with Airtime 2.1 beta 3. I set up a new drive (/media/music) and copied, using scp, all my music from my current server running Airtime 2.0.3. The copy created /media/music/stor and /media/music/stor/imported. I logged into the Airtime GUI and set my import folder to /media/music. I then tried to set my watch folder to /media/music/stor/imported. At first, I thought I could simply navigate, using the displayed folder hierarchy, but as soon as I navigated to /media/music the interface became busy (the spinning wheel). Next I tried to just type the directory name in the input box: /media/music/stor/imported. The interface became busy (the spinning wheel) and appears to have hung. I do have a lot of files in that directory so I though I would give it a chance to work. Three hours later, it is still busy and I see no indication of progress.

Using ssh, I logged into the new server and looked in /media/music. I see three directories: imported, organize, and stor. Both imported and organize are empty. The stor directory contains two directories (imported and organize) and two m3u playlists - this is correct and mirrors the information in my primary server (from which they were copied). The /media/music/stor/imported directory contains most of the music. The /media/music/stor/organize directory has a couple of mp3 files.

Even though the GUI indicates that the watch folder is busy on the new server, nothing is happening to the files. I would expect that as the watched folder analyses the structure of the /media/music/stor/imported folder it would start to copy the files to /media/music. Am I wrong? Do I misunderstand the watch mechanism? What is the best way to copy the existing music to the new server so Airtime will find and use it?

Wade Eilrich

I think I found out what the main problem is: I formatted the drive as FAT32 so I could share it (samba) with remote computers running Windows. This is a big problem because FAT32 does not support permissions. I reviewed my options and decided to reformat the drive with NTFS (which does support permissions). I'm copying the files back onto the newly formatted drive now using the same hierarchy. The mount point is /media/music and within that I'm creating a /stor folder that will hold /imported and /organize.

I've already added myself to the www-data group and will change the owner and group to www-data once the files have finished copying using: sudo chmod -R www-data:www-data /media/music

When all that is done, I'll go back into the 2.1 beta 3 web interface and attempt to set up the media folders. Does this sound like a reasonable approach?

Martin Konecny

Hi Wade,

When you tried to add the drive through the WebUI then you are correct that it was a permissions issue. But you could have kept using FAT32, just make sure you mount the drive so that its world readable (or at least so that www-data can read it).

Also I wasn't sure why you set the watched folder to look inside the storage directory. These are meant to be mutually exclusive. We should probably make that more clear somehow.

Wade Eilrich

I reformatted the drive as NTFS solved (some) of the permissions issues by changing /etc/fstab to use uid=www-data,gid=www-data, but I still cannot upload files by dragging and dropping them into Add Media. I get the error "The file was not uploaded, this error will occur if the computer hard drive does not have enough disk space."

My guess is that permissions issues still exist with my import media folder even though all the permissions appear to be okay because df -Th shows that I've only used 10% of the available disk space on that volume.

The output of ls -al is for the media folder is:

drwxrwxrwx 1 www-data www-data 4096 2012-04-08 20:01 .

drwxr-xr-x 4 root root 4096 2012-04-09 10:53 ..

drwxrwxrwx 1 www-data www-data 0 2012-04-09 10:55 stor

All the sub-directories have the same permissions as stor. Then I had an idea: reset the media folder through Airtime. Voila! I go to the Playlist Builder and it shows: File Import in Progress (with three little bars that strobe through a progress update). However, when the import finished, the display did not update with the new media.

This was easily rectified by clicking on Playlist Builder - some of the media showed up, but not all. The Playlist Builder shows 189 entries and I know there are over 2,000. To verify this, I sorted the entries by creator, picked one that I know has a lot of songs, and counted only 16 (out of 195). I verified the song count at the terminal by changing to that directory and got the count using: find . -type f | wc -l

Why didn't the other files get imported? Also, from 2.0.3 I know the media heirarchy within stor has two sub-directories (imported and organize). Shouldn't there be an 'organize' at the top level (there isn't).

As an aside, I was a little concerned by the warning given when I reset the media folder location because it appeared that the files would be erased, but this did not happen :-)

Next, I changed my watch folder to /home/<myusername>/Music and this appears to work correctly. I copied some files into the watch folder and they appeared in the Playlist Builder. I had thought that these files would be copied into the import directory, but after re-reading the documentation I'm guessing that is not the designed behavior. I'm a little puzzled by this.

What will happen if I move files out of the watch folder? Will they disappear from the Playlist Builder? What I really want is a location where I can copy files and have Airtime put them into the imported folder without my having to drag and drop them through the Add Media area. This would be very useful to me because each artist has his/her own folder, album, and songs. Those must be navigated and the songs selected and then placed into Add Media. It would be much simpler to use 'cp' (or 'scp') to copy a bunch of stuff all at once. I had thought to simply copy files into my media folder, but as I noted (above) this does not work correctly - the Playlist Builder doesn't pick up all the files.

Wade Eilrich

Just to see what happened, I changed my media import folder back to the default value of /srv/airtime/stor/

I then clicked on the Playlist Builder and, to my surprise, found 15 entries even though there is nothing in this directory! I double-checked using the terminal, and there is nothing in stor except a single directory: organize. Two questions:

(1) where is the imported directory?

(2) why are there 15 entries in the playlist builder?

(Note: there is nothing in my watch folder).

ls -al /srv/airtime/stor

total 12

drwxrws--- 3 www-data www-data 4096 2012-04-07 14:59 .

drwxr-xr-x 3 root root 4096 2012-04-07 14:59 ..

drwxrws--- 2 www-data www-data 4096 2012-04-07 14:59 organize

Wade Eilrich

More Info

On a whim, I restarted the media monitor (sudo service airtime-media monitor restart) and the Playlist Builder, after a few seconds, began to display progress again. I switched over to Add Media - it still won't accept any files - and then switched back to Playlist Builder. So far, it has found 795 of the media files. I don't know why the media monitor arbitrarily stopped the first time, but it appears that simply restarting it fixes the problem of picking up the files in the media folder.

I still don't know why Add Media isn't working, but I'm looking through the python files to see if I can find any clues by searching for the error string.

Wade Eilrich

I've narrowed down the problem in Add Media. First, I looked in PluploadController.php to research the sequence of events that happen when a file is queued for upload. I found that the file is not copied directly to the media folder. Instead, it goes into a temporary upload directory (defined by "upload_tmp_dir" in one of the Airtime config files - this is, unless you change it, /tmp/plupload/). The method that does this is called Application_Model_StoredFile::uploadFile(). Another method, Application_Model_StoredFile::copyFileToStor() moves the file from the temporary upload directory into the media folder.

I can see the uploader transferring files to the temporary directory and then removing them. What I'm looking for is an error message with either code 102 or 103 ("Failed to open output stream" or "Failed to move uploaded file") with a message telling me the exact reason it failed, but the problem is not here.

Next I look at checkForEnoughDiskSpaceToCopy() to see if it is failing(it should not). The calling method, copyFileToStor(), tries to place the file in the /organize media folder. A comment by Martin K. says that if the checkForEnoughDiskSpaceToCopy() succeeds, the method attempts to rename the file (because this is an atomic operation). If the rename fails, copyFileToStor() emits the error message (code 108) "The file was not uploaded, this error will occur if the computer hard drive does not have enough disk space." This is the error message I see when I try to use drag and drop in Add Media.

I think this is the exact location that is causing my problem. There is plenty of disk space (about 100GB free), but I don't see an /organize directory anywhere in the media folder hierarchy so the rename will fail with a misleading error message (misleading because there is plenty of disk space). The problem may be that the /organize directory does not exist. If this is the case, shouldn't copyFileToStor() check for the existence of the /organize directory before it attempts to call rename() (and perhaps create the directory)?

Wade Eilrich

Solved!

I created the /organize directory in the correct place in the media folder hierarchy using mkdir from the Terminal and drag and drop through Add Media now works!

Suggestion: add a check before calling rename() in copyFileToStor() to see if the /organize folder exists and, if it doesn't, create it.

Note: Airtime 2.0.1 created /imported and /organize folders when the media folder was set. Did this functionality change in 2.1?

Martin Konecny

Hi Wade, It's a bit difficult to keep track of all the points you are making I'll do my best to address most of them:

"What I really want is a location where I can copy files and have Airtime put them into the imported folder without my having to drag and drop them through the Add Media area."

This is possible by simply adding to the stor/organize directory. Any file added here should be renamed and moved to the imported directory. For the watched folders, those are simply "watched" for changes. The files there are not modified or moved in anyway.

"Why didn't the other files get imported? Also, from 2.0.3 I know the media heirarchy within stor has two sub-directories (imported and organize). Shouldn't there be an 'organize' at the top level (there isn't)."

I'm really not sure about the state of your system. Did you try tail -F /var/log/airtime/media-monitor/media-monitor.log to see if any errors are occuring during the import?

Martin Konecny

"I created the /organize directory in the correct place in the media folder hierarchy using mkdir from the Terminal and drag and drop through Add Media now works!"

Thanks for posting this. The organize directory is supposed to be created when changing the stor directory, not sure why this safeguard isn't taking place. I've created a ticket here:

http://dev.sourcefabric.org/browse/CC-3622

Wade Eilrich

Hi Martin - sorry about my rambling. I do tend to cover a lot of ground in a stream-of-consciousness flow, but that's the way my mind works :)

I didn't look at the media-monitor log because I had enough information from the errors being reported by Add Media to start looking through the source (Javascript and PHP) files to see how the upload process works. I did find the problem. It was a missing /organize directory. The default media directory is /srv/airtime/stor/ but on my system that is on a small boot disk. I added an NTFS 120GB disk (/dev/sdb1) to store my media. I made the mount point /media/music for this disk. I copied all my media from my Airtime 2.0.3 server (using scp) to /media/music. This created /media/music/stor and /media/music/stor/imported. All of my media resides in /media/music/stor/imported.

After I copied all the media over I used Airtime's web interface System->Media Folders to set the Import folder to /media/music/stor. When I opened the Playlist Builder I saw that it was importing files, but it stopped long before it should have. To fix that, I simply restarted the media-monitor using the Terminal command 'sudo service airtime-media-monitor restart'. By looking at the Playlist Builder I could see that the import continued and eventually picked up all the media in my import folder. I still don't know why the import process stopped, but this is a repeatable condition under these specific circumstances.

Next I wanted to add some new media so I went to Add Media and used drag and drop to add some new media. It failed to add the media and gave me an inaccurate error message about not having enough hard disk space. That hard disk has plenty of space. The problem was that there was no /organize folder in /media/music/stor - only the /imported folder I created when I copied the media files there. When I looked through the code in StoredFile.php I found that the /organize folder is where Add Media wanted to copy the files. All I had to do was create the /organize folder and Add Media worked correctly.

My suggestions may help to make Add Media more robust. Add Media gave me an inaccurate error message because of the missing /organize folder (it told me I didn't have enough disk space). Before rename() is called in CopyFileToStor() I think it would be wise to make sure that the /organize folder exists and optionally to create it if it doesn't exist or emit an error message about it not being there. Does that make sense?

Martin Konecny

After I copied all the media over I used Airtime's web interface System->Media Folders to set the Import folder to /media/music/stor. When I opened the Playlist Builder I saw that it was importing files, but it stopped long before it should have. To fix that, I simply restarted the media-monitor using the Terminal command 'sudo service airtime-media-monitor restart'. By looking at the Playlist Builder I could see that the import continued and eventually picked up all the media in my import folder. I still don't know why the import process stopped, but this is a repeatable condition under these specific circumstances.

I think the media-monitor log would come in handy in this situation

Next I wanted to add some new media so I went to Add Media and used drag and drop to add some new media. It failed to add the media and gave me an inaccurate error message about not having enough hard disk space. That hard disk has plenty of space. The problem was that there was no /organize folder in /media/music/stor - only the /imported folder I created when I copied the media files there. When I looked through the code in StoredFile.php I found that the /organize folder is where Add Media wanted to copy the files. All I had to do was create the /organize folder and Add Media worked correctly.

This is defn a bug on our part. I'm curious why your scp command didn't copy this directory as well though.

My suggestions may help to make Add Media more robust. Add Media gave me an inaccurate error message because of the missing /organize folder (it told me I didn't have enough disk space). Before rename() is called in CopyFileToStor() I think it would be wise to make sure that the /organize folder exists and optionally to create it if it doesn't exist or emit an error message about it not being there. Does that make sense?

This defn makes sense. We'll take care of this in the bug ticket I created.

Howdy, Stranger!

Categories

Poll

Top Posters