pypo / silananalyzer.py / huge amout of API requests
  • Vote Up0Vote Down Stephane JourdanStephane Jourdan
    Posts: 8Member
    Hi,

    I'm running Airtime 2.3.1 (apt source) on a ubuntu 12.04 x86 fully updated. It's been running for a few weeks, but as it's still in a testing state, there's currently a very low amount of icecast listeners (< 5). I can't say since when I have this problem, but yesterday, many services stopped working, and the logs were showing strange errors about 'no space left on device' etc., which wasn't true (there was plenty of space available). To the point I checked the php sessions on /var/lib/php5 : more than 150K session files. And below that limit, sessions couldn't be written on the said directory. Right, so I removed them the way I could (you can't really rm such a huge amout of files) and went on investigating what was going wrong here.

    I still don't know exactly what my problem is, but I have three things that don't look right and generate an enormous amount of load: 
    • I get an unusually high amount of session files written: around 10 per second. And never get deleted.
    root@airtime:~# date; ls /var/lib/php5/ | wc -l
    Fri Apr  5 20:22:55 CEST 2013
    37760

    root@airtime:~# date; ls /var/lib/php5/ | wc -l
    Fri Apr  5 20:23:55 CEST 2013
    38323 

    The content looks always the same but obviously I haven't check each of them: 

    cat sess_j20sdr5bnaaop4m1ded50cho26 
    Zend_Auth|a:1:{s:7:"storage";N;}

    • the /var/log/airtime/pypo/pypo.log log file goes crazy continuously repeating the same error 4 times per second, every second, which could match the previous sessions (?): 
    2013-04-05 20:26:48,474 INFO - [silananalyzer.py : main() : line 65] - Processed: 0 songs
    2013-04-05 20:26:48,601 ERROR - [silananalyzer.py : main() : line 56] - list index out of range
    2013-04-05 20:26:48,601 ERROR - [silananalyzer.py : main() : line 57] - Traceback (most recent call last):
      File "/usr/lib/airtime/pypo/bin/media/update/silananalyzer.py", line 49, in main
        data['cuein'] = str('{0:f}'.format(info['sound'][0][0]))
    IndexError: list index out of range

    2013-04-05 20:26:48,678 INFO - [silananalyzer.py : main() : line 65] - Processed: 0 songs


    • the /var/log/apache2/other_vhosts_access.log log file goes crazy accordingly: in less than 4 days I had not less then 129892 of those two requests (names, IP, API key masked, all are correct), the servers public IP requesting to its domain name the same thing: 
    my.domain.com:80 my:external:ip - - [05/Apr/2013:20:29:02 +0200] "POST ///api/update-cue-values-by-silan/api_key/APIKEY HTTP/1.1" 200 408 "-" "Python-urllib/2.7"
    my.dnsname.com:80 my:external:ip - - [05/Apr/2013:20:29:03 +0200] "GET ///api/get-files-without-silan-value/api_key/APIKEY HTTP/1.1" 200 534 "-" "Python-urllib/2.7"

    The API and service is responding correctly as I can do: 

    {"version":"2.3.1"}

    and I can GET the same request than the logs too: 

    [{"id":1500,"fp":"\/srv\/airtime-data\/stor\/imported\/Chris Watson\/Outside the Circle of Fire\/19-Elephants-256kbps.mp3"}]

    I tried to reconfigure airtime with dpkg-reconfigure airtime and it didn't change anything. 

    I've activated the admin interface for RabbitMQ to see if there was something happening strange there, and I didn't saw anything strange but maybe I missed what to look at.

    At one point I tried the new liquidsoap version coming from this thread (http://forum.sourcefabric.org/discussion/15111/airtime-audio-fails-daily) but no chance either. It's still there. I tried to remove silan without removing airtime (dpkg -r --ignore-depends=airtime silan) and it didn't help either. 

    Do you have a clue ? Something I could try ?

    Thanks a lot for your help ! 

    for reference, my airtime-check-system output

    root@airtime:~# airtime-check-system 
    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 3.2.39-xenU-7726-i386
    MACHINE_ARCHITECTURE           = i686
    TOTAL_MEMORY_MBYTES            = 496072
    TOTAL_SWAP_MBYTES              = 370684
    AIRTIME_VERSION                = 2.3.1
    OS                             = Ubuntu 12.04.2 LTS i686
    CPU                            = Intel(R) Xeon(R) CPU           L5640  @ 2.27GHz
    WEB_SERVER                     = Apache/2.2.22 (Ubuntu)
    PLAYOUT_ENGINE_PROCESS_ID      = 1272
    PLAYOUT_ENGINE_RUNNING_SECONDS = 2046
    PLAYOUT_ENGINE_MEM_PERC        = 2.4%
    PLAYOUT_ENGINE_CPU_PERC        = 2.9%
    LIQUIDSOAP_PROCESS_ID          = 1637
    LIQUIDSOAP_RUNNING_SECONDS     = 2042
    LIQUIDSOAP_MEM_PERC            = 3.1%
    LIQUIDSOAP_CPU_PERC            = 7.9%
    MEDIA_MONITOR_PROCESS_ID       = 1576
    MEDIA_MONITOR_RUNNING_SECONDS  = 2043
    MEDIA_MONITOR_MEM_PERC         = 3.5%
    MEDIA_MONITOR_CPU_PERC         = 0.0%
    -- Your installation of Airtime looks OK!
    Post edited by Stephane Jourdan at 2013-04-05 14:57:15
  • 3 Comments sorted by
  • Vote Up0Vote Down Cliff WangCliff Wang
    Posts: 339Member, Sourcefabric Team
    Thank you for the information, we have already addressed that problem and bug ticket is created: 

    And I don't think remove silan will help, the workaround for now could be remove that file from storage temporally, I mean from /srv/airtime/stor/
    Post edited by Cliff Wang at 2013-04-05 15:35:10
    Cloud Radio -> http://airtime.pro
  • Vote Up0Vote Down Stephane JourdanStephane Jourdan
    Posts: 8Member
    Thank you so much, I didn't thought it would help to remove the named file outputed by the API, but you're right, it fixed the problem.

    And btw next time I'll check the airtime issues tracker before posting here!

    Thanks again, Cliff.
  • Vote Up0Vote Down Cliff WangCliff Wang
    Posts: 339Member, Sourcefabric Team
    You are welcome, :). Glad to hear the workaround works. You give us so many details, that really helps us to find what's the problem. Please carry on reporting problems you meet to help us build better product.
    Cloud Radio -> http://airtime.pro
Add a Comment