Upgrade from 1.8.2 to 2.0.3 stalled
  • Vote Up0Vote Down James DunnJames Dunn
    Posts: 226Member
    Hi,

    I've been trying to upgrade Airtime manually from 1.8.2 to 2.0.x for a few months and have had various problems. We recently bought a new hard drive to backup and perform the upgrade on the 350GB archive and I copied all the data across with rsync. Then I imported the database following the instructions here.

    Then I copied across the config files in /etc/airtime as the api_key was different.

    Then I followed the instructions here to upgrade to Airtime 2.0.x

    The upgrade process starts but stalls at the point of reorganising the archive:

    * Inserting stor directory location /srv/airtime/stor/ into music_dirs table
    Creating media-monitor log file
    Reorganizing files in stor directory

    If I run ls -l /srv/airtime/stor | tail, I get:

    drwxr-sr-x 2 www-data 4096 2012-03-21 18:00 fe9
    drwxr-sr-x 2 www-data 4096 2011-08-30 20:38 fea
    drwxr-sr-x 2 www-data 4096 2011-10-05 00:00 fec
    drwxr-sr-x 2 www-data 4096 2011-09-01 02:00 fef
    drwxr-sr-x 2 www-data 4096 2011-12-16 18:00 ff0
    drwxr-sr-x 2 www-data 4096 2011-09-28 21:00 ff3
    drwxr-sr-x 2 www-data 4096 2011-11-04 02:00 ff5
    drwxr-sr-x 2 www-data 4096 2012-04-05 13:00 ffd
    drwxr-sr-x 2 www-data 4096 2011-11-19 00:00 ffe
    drwxrwsrwx 2 www-data www-data 4096 2012-04-26 01:30 organize

    So the upgrade process has created the 'organize' directory, but in previous attempts there were also 'imported' and 'recorded' directories. There are no files in the 'organize' directory and the upgrade process has been running for 8 hours!!

    The first time I attempted the upgrade, the /srv/airtime directory was a symlink and it stalled. In the current upgrade situation, airtime is running in a VM with a 450GB hard drive so I can eliminate the symlink issue but the result is the same. The VM's hard drive is at 84% with 54GB free - this should be enough space to reorganise the archive right?

    How can I get airtime to complete the upgrade?

    thanks

    James
  • 9 Comments sorted by
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Hi James,

    If you use the "top" command, do you see any CPU being used? What is the size of the organize directory?
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down James DunnJames Dunn
    Posts: 226Member
    Hi Martin,


    Yes the CPU is busy, please see the attached file for details of the top command. Python seems to be the main consumer of the CPU and ps ax shows this line:

    3003 pts/1 R+ 259:39 python /home/nts-studio/airtime-2.0.3/install_minimal/upgrades/airtime-1.9.0/media-monitor-upgrade.py

    The organize directory is the same size as listed above - ie it's still empty!

    ls -l /srv/airtime/stor/organize | tail -n 1:

    drwxrwsrwx 2 www-data www-data 4096 2012-04-26 01:30 organize

    ls -l /srv/airtime/stor/organize:

    total 0

    Could it be that the audio files referenced in the postgresql database backup of the production server are pointing to the actual directory they exist in on on the production server (/media/usb/airtime/stor) rather than the symlink /srv/airtime/stor? So when I attempt an upgrade on our test machine, it's looking for /media/usb/airtime/stor rather than /srv/airtime/stor which is where the files have been copied to?

    thanks for your help,

    James


    top - airtime upgrade.log
    6K
    Post edited by James Dunn at 2012-04-26 10:26:08
  • Vote Up0Vote Down James DunnJames Dunn
    Posts: 226Member
    Hi Martin and all,

    I left the VM upgrade running and finally after 36 hours it's completed! The storage directory has been reorganized and I now have the 3 directories 'imported', 'recorded' and 'organize' containing all the files. I guess 350Gb takes a long time to process!!

    I get an error messages at the end of the install, and with sudo airtime-check-system:

    PLAYOUT_ENGINE_PROCESS_ID      = FAILED
    LIQUIDSOAP_PROCESS_ID          = FAILED
    MEDIA_MONITOR_PROCESS_ID       = FAILED
    RABBITMQ_PROCESS_ID            = FAILED

    but I think this is the same issue as bug CC-3402 mentioned here as I running airtime on a different port.

    I assume the symlink should be fine too then, thanks for you help

    James
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Great! :) We defn don't have a 350GB test setup here, so had no idea it would take that long :)
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Wade EilrichWade Eilrich
    Posts: 98Member
    I process about 15GB every time a new beta version comes out and it takes between 2-4 hours to import the media. This does seem like a long time to add items into a database. Do you have any idea what the limiting factor is?

    I know, for example, that each file must be opened to read the tags but it should not take that long to simply read the tag information because, for example, I can do that with Mp3tag on over 3,000 items in just a few minutes. Certainly, adding records to a database is very fast (assuming the transactions are crafted well). I must be missing a key component in the import process that is very time-intensive.
  • Vote Up0Vote Down James DunnJames Dunn
    Posts: 226Member
    Hi Wade,

    All I can say is that in the upgrade I performed on our VM it took 36 hours to reorganise the 350Gb of audio files. This may be slower than some machines as the VM has only 900Mb RAM, although most of the 3.2Ghz CPU should be dedicated to the upgrade. Also the VM's hard drive image was on an external USB drive. I'm currently running the upgrade on our second PC for hopefully the last time so we should be able to go live with it this week! Finger crossed. It's also using all 2Gb of the RAM so I expect it to be a bit quicker than 36 hours.

    James
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    > I process about 15GB every time a new beta version comes out and it takes between 2-4 hours to import the media. 

    This is actually pretty bad, created a ticket for this:

    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Wade EilrichWade Eilrich
    Posts: 98Member
    Thank you, Martin. Here's a summary of that server's specifications (running Ubuntu 10.04 LTS server):
    Motherboard: Intel L440GX+ V.721242-004
    CPU: Dual processor Pentium III 600MHz
    RAM: 512MB ECC SDRAM
    Network: 2 ea. Intel Pro 10/1000 Ethernet in bridge mode
    Video: Intel graphics processor
    Sound: AC'97 compatible sound processor
    Power: 500W Antec power supply,
    USB: 4 USB 2.0 ports
    Serial: DB9 serial port
    Parallel: parallel port
    BIOS: Intel BIOS version P14.

  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    For me it takes about 90 seconds to import a 3GB library, so linearly interpolating that to 15GB, would take about 450 seconds or 7.5min. 

    I have an SSD + i7 processor, so this is slightly above average :)

    I'll take a look if there's any simple improvements we can do, but we are done with major modifications for 2.1, and this will most likely have to wait for the next major release.
    Airtime Pro Hosting: http://airtime.pro
Add a Comment