Airtime 2.1 Playout Problem
  • Hello, 
    I just Upgraded from Airtime 2.0.3 to 2.1
    I got no streaming although Shows are secheduled already, 
    On Air is not red,
    when going to System>Status 
    i got Playout Disabled, 
    sometimes playout turns to Enabled and On Air turns to Red, 
    Then within seconds it returns back to Disabled and Online is not red again 

    Status Sanpshot attached, 
    Also Logs Attached

    Thanks in advance,

    800 x 600 - 171K
  • 55 Comments sorted by
  • Hi Karim, Radio-Libriamoci,

    Have you guys upgraded to 2.1.1? Is everything working fine?

    Thank you
  • Have you already rebooted?

    We not yet upgraded but in the past we faced the same issue and it was fixed after reboot
  • Thanks alot for your Support, 
    I just rebooted my VPS, and nothing happened, I'm still facing the same problem, 
    Any solutions ?
  • Any Help !
  • Any Help Would be appreciated, Please !!!
  • Hi Karim,

    We see your posts. Posting 3 or 4 times here and elsewhere won't get you to a solution any quicker... please give the community time to respond, it's a weekend after all. :)

    As ever, screenshots and logs help people to help you troubleshoot.

    Best, Adam
  • I'm Really Sorry, And Thanks alot Adam :)
    Logs attached again as for the previous file is corrupted

    Thanks alot and sorry again :)
  • Reply to @Karim+Ali:

    I took a look at your logs and see the following:

    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 54] - Before ANSI_X3.4-1968
    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 58] - No locale currently set. Attempting to get default locale.
    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 62] - No default locale exists. Let's try loading from /etc/default/locale
    2012-06-09 20:15:32,076 INFO - [pypocli.py : configure_locale() : line 73] - New locale set to: C

    Seems like your VPS doesn't provide a default locale. You will need to set it up manually.

    Type:
    sudo dpkg-reconfigure locales


    Then make sure you select your locale (I selected en_CA.utf-8 for Engish/Canada). Make sure the ending has UTF-8!!

    Selecting OK will bring up another screen where you make sure your selection is the default one. After this everything should work.
    Airtime Pro Hosting: http://airtime.pro
  • Hello, 
    Thanks alot for your reply, 
    I did change my default locale to ar_EG for Arabic_Egypt
    When type locale i get:

    LANG=ar_EG
    LANGUAGE=ar_EG:ar
    LC_CTYPE="ar_EG"
    LC_NUMERIC="ar_EG"
    LC_TIME="ar_EG"
    LC_COLLATE="ar_EG"
    LC_MONETARY="ar_EG"
    LC_MESSAGES="ar_EG"
    LC_PAPER="ar_EG"
    LC_NAME="ar_EG"
    LC_ADDRESS="ar_EG"
    LC_TELEPHONE="ar_EG"
    LC_MEASUREMENT="ar_EG"
    LC_IDENTIFICATION="ar_EG"
    LC_ALL=

    The problem is still the same no any improvements

    When Type sudo airtime-check-status I get:

    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 2.6.18-274.18.1.el5.028stab098.1
    MACHINE_ARCHITECTURE           = x86_64
    TOTAL_MEMORY_MBYTES            = 1572864
    TOTAL_SWAP_MBYTES              = 0
    AIRTIME_VERSION                = 2.1.0
    OS                             = Ubuntu 11.04 x86_64
    CPU                            = Intel(R) Xeon(R) CPU           X3450  @ 2.67GHz
    WEB_SERVER                     = Apache/2.2.17 (Ubuntu)
    PLAYOUT_ENGINE_PROCESS_ID      = NOT MONITORED
    PLAYOUT_ENGINE_RUNNING_SECONDS = 1
    PLAYOUT_ENGINE_MEM_PERC        = 0%
    PLAYOUT_ENGINE_CPU_PERC        = 0%
    LIQUIDSOAP_PROCESS_ID          = 4012
    LIQUIDSOAP_RUNNING_SECONDS     = 0
    LIQUIDSOAP_MEM_PERC            = 0.0%
    LIQUIDSOAP_CPU_PERC            = 0.0%
    MEDIA_MONITOR_PROCESS_ID       = FAILED
    MEDIA_MONITOR_RUNNING_SECONDS  = -1
    MEDIA_MONITOR_MEM_PERC         = 0%
    MEDIA_MONITOR_CPU_PERC         = 0%
    RABBITMQ_PROCESS_ID            = 1559
    RABBITMQ_RUNNING_SECONDS       = 196
    RABBITMQ_MEM_PERC              = 1.9%
    RABBITMQ_CPU_PERC              = 0.0%

    Any More Help ?
    Thanks alot

  • You need to make sure the locale has "UTF-8" attached to the end.
    Airtime Pro Hosting: http://airtime.pro
  • Here is Locale Default:

    root@server1:~# sudo dpkg-reconfigure locales
    Generating locales...
      ar_EG.UTF-8... up-to-date
    Generation complete.

    How about Airtime-system-check results i put in my last message, 
    its saying MEDIA_MONITOR_PROCESS_ID       = FAILED
    PLAYOUT_ENGINE_PROCESS_ID      = NOT MONITORED

    Help really appreciated, 
    Thanks
  • Forget to say thats its still not red, and no streaming at all
  • Thank you very much Martin, 
    It Really works i added .utf-8 to my locale defaults 
    and simply works,
    Thanks alot I appreciate all your efforts to provide wonderful support for all airtimers here, 

    For the sake of helping others encountering same issue: 

    1. Add the locale to the list of 'supported locales'
    Edit /var/lib/locales/supported.d/local and add the following line:
    en_CA UTF-8

    2. Regenerate the supported locales
    Run sudo dpkg-reconfigure locales

    3. Change the default locale in /etc/default/locale to 
    LANG="en_CA.utf-8"
    LANGUAGE="en_CA.utf-8:en" 

    Reboot!

    Thanks Alot again :)
  • Not sure step one is necessary (you should be able to select the supported locale from the dpkg-reconfigure program), but glad you got it working :)
    Airtime Pro Hosting: http://airtime.pro
  • Hello, 
    Again On Air is not Red, & Airtime Offline 
    I don't really know why am getting this problem now !
    Logs Attached, 
    Please Help Martin 
    Thanks
    800 x 600 - 194K
  • Hi,

    just upgraded from 2.0.3 to 2.1.0 but at the end of the process this is the outcomes

    *** Verifying your system environment, running airtime-check-system ***
    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 2.6.32-238.19.1.el5.028stab092.2
    MACHINE_ARCHITECTURE           = x86_64
    TOTAL_MEMORY_MBYTES            = 1048800
    TOTAL_SWAP_MBYTES              = UNKNOWN
    AIRTIME_VERSION                = 2.1.0
    OS                             = Debian GNU/Linux 6.0.5 (squeeze) x86_64
    CPU                            = Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
    WEB_SERVER                     = Apache/2.2.16 (Debian)
    PLAYOUT_ENGINE_PROCESS_ID      = 28614
    PLAYOUT_ENGINE_RUNNING_SECONDS = 0
    PLAYOUT_ENGINE_MEM_PERC        = 0.1%
    PLAYOUT_ENGINE_CPU_PERC        = 0.0%
    LIQUIDSOAP_PROCESS_ID          = 28617
    LIQUIDSOAP_RUNNING_SECONDS     = 0
    LIQUIDSOAP_MEM_PERC            = 0.8%
    LIQUIDSOAP_CPU_PERC            = 0.0%
    MEDIA_MONITOR_PROCESS_ID       = FAILED
    MEDIA_MONITOR_RUNNING_SECONDS  = -1
    MEDIA_MONITOR_MEM_PERC         = 0%
    MEDIA_MONITOR_CPU_PERC         = 0%
    RABBITMQ_PROCESS_ID            = 1437
    RABBITMQ_RUNNING_SECONDS       = 11
    RABBITMQ_MEM_PERC              = 2.7%
    RABBITMQ_CPU_PERC              = 0.0%

    -- There appears to be a problem with your Airtime installation.

    dpkg: error processing airtime (--configure):
     subprocess installed post-installation script returned error exit status 1
    Processing triggers for python-support ...
    Errors were encountered while processing:
     airtime
    E: Sub-process /usr/bin/dpkg returned an error code (1)

    The schedule looks to work, icecast is not connected, the on air light is off.
    I restarted all services step by step but when I type invoke-rc.d airtime-playout the result is Restarting Airtime Playout: start-stop-daemon: warning: failed to kill 3301: No such process
     
    I tried all you said in the previous post but... nothing... please help urgently
  • Update.

    the streaming is up, the on air sign sometimes is on but no streaming
  • @Kalim: Around what time did you have problem with On Air light? Is server time and your airtime time in the same time zone? if not what are the timezone?

    @Radio-Libriamoci Web: Can you please post log files?

    Thanks
  • I just solved the problem the day before tomorrow with Martin's Help and within just next hours less than 24 hrs i got the prblem again seems around 12 - 1 PM today 

    Server time zone is ar_EG.utf-8 and Airtime time zone is EET 
    Do you have any suggestions ?

    Thanks james
  • Ok you were right James, 
    I just resetted my VPS TimeZone to be as same as Airtime Time Zone EET, 
    Now VPS & Airtime have same timezone, and runnding perfect !

    Problem Resolved

    For the sake of helping other, 
    here is how to change your ubuntu server timezone,

    sudo dpkg-reconfigure tzdata

    and Choose your preferences :)

    Thanks
  • Agaaaiin Radio Stopped !!!!! 
    On Air not red, Airtime Offline
    its the third time Problem resolved and after hours streaming stopped again, 
    Cant understand hows it happpening !

    Locale Ddefaults set, UFT-8 Added, server and airtime has same EET timezone
    Whats happening every couple of hours !

    Logs attached !
  • Hi James, here attached the log files.

    In the UI I see that liquidsoap and mediamonitor failed.

    Thanks

    dario
  • UPDATE June 13rd 8:10 GMT +1

    I reconfigured locales since in the playout log it looket to be "C" and not it_IT:UTF-8, rebooted and AIRTIME 2.1.0 is ONAIR again.

    The problem looks to me the same faced by Karim but now I'm go to monitor the enviroment to find out if it will go down again ina couple of hours.

    Further ideas or comment?


  • Hello,
    @Radio-Libriamoci 
    Please ensure that the server timezone is same as airtime timezone 
    maybe it will help you untill james or martin check our logs 

    Thanks alot,

  • Hi Karim, thank you.

    Actually if you have a look at my playout log posted in the previous post you'll see that the local default were uncorrect.
    I fixed it following the steps you suggested and checked the time zones to ensure they're aligned and they are.

    Now Airtime is working correctly since more than 2 hours and i'm monitoring it. The playout log do not highlight any issue at this time but let's monitor.

    Looking forward a feedback from James, it would be great if other users would share their experience upgrading.

    Thanks again
  • After 4h 30' from reboot it's working correctly
  • Thats Perfect, 
    Me As well, its back working and I'm just monitoring !
  • You are running it on a VPS as well Radio-Libriamoci ?
  • Yes I do Karim. It's a VPS based on Debian 6.0.4
  • Hello,

    The server time and Airtime time timezone doesn't need to be same. The problem you guys are having is something else. We found this bug in pypo and I'm suspecting that is the root causes of your problem.

    Were you guys not scheduling/modifying anything related schedule for more than a hour? and Airtime went off line?
  • James, Airtime went offline yesterday night just after the upgrade.
    Early today, after the locales adjustement end the reboot it started and never went offline anymore.
    One hour later I added couple of file and updated some show.
    Any other test to be performed?

    here attached the last playout log. it looks actually strange (I'm not an expert LOL)
  • Hi,

    That log doesn't tell me much :(.
    If it happens again, please try to modify schedule(eg, adding/deleting/editing shows or files from now playing page... anything schedule related.) and see if that will make pypo going?

  • James this is the pypo log as was few minutes ago
    root@rlwebradio:~# less +G /var/log/airtime/pypo/pypo.log
    3 to local cache /var/tmp/airtime/pypo/cache/scheduler/1234d07462ed9d0a52881e0d293154b7.mp3
    2012-06-13 19:42:20,773 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96709/schedule_id/0
    2012-06-13 19:42:20,893 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:42:20,893 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-13 19:46:16,618 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96710/schedule_id/0
    2012-06-13 19:46:18,025 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:46:18,026 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-13 19:46:18,437 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96711/schedule_id/0
    2012-06-13 19:46:19,206 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:46:19,207 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    (END)

    I just rebooted: in the nowplaying view the indication of what's on air was uncorrect, one track late. Now it looks to work correctly but for sure we'll face similar issues in the next hours
  • Hi,

    Thanks for the report. We know about this bug and we are fixing it for 2.1.1.
  • Thank you James. Your support is really great!
    Post edited by Radio-Libriamoci Web at 2012-06-13 15:28:58
  • Im Sorry, 
    but Its really not understandable, 
    I dont have reasons why i get Airtime Offline for Hours and Then Comes online again without performing any changes !
    While Im having tracks running in Airtime Already when it goes offline, 
    Now Im having it running perfect but im just monitoring as expecting going offline in the next hours 
    what is this bug you are talking about Radio-libriamoci & James is it related to our issue ?
  • Wonderful, 
    I got Airtime Offlin, 
    While having Shows scheduled & On Air is Red !!!!!!!!!!!!!!!!!!

    SnapShot for Airtime Control Panel & IceCast status at the same time !!
    there is a serious bug, I need Help !
    1366 x 768 - 231K
  • We're randomly experimenting the same.

    James, how much time for the patch?
  • Airtime 2.1.1 tarball is now available

    Debian package and official announcement will be tomorrow.
    Airtime Pro Hosting: http://airtime.pro
  • Reply to @Karim+Ali:

    Karim, next time this happens please do the following:

    tar czf airtime.log /var/log/airtime

    and post airtime.log here.
    Airtime Pro Hosting: http://airtime.pro
  • Ok Thank you very Much Martin
    What is it about this new tarball 2.1.1 a new version ?

    Thanks
  • Hi James,

    as I'm not so confortable to work with tarball I'm waiting for the UI to notifie the package availabe to upgrade via apt-get. I do not see it yet
  • Done right now!
    Airtime 2.1.1 up and running! I'll get back to you for an exhausting feedback.
    In the meanwhile a big big thanks to you and Sourcefabric Staff for the really great support!

    Dario
  • Hi Jamesm 12hours after the upgrade it looks to work fine. No stream interruption, no issues with the ON AIR light.

    Here following the Pypo log for your evaluation.

    @Karim please share your experience.

    Thanks

    Dario

    root@rlwebradio:~# less +G /var/log/airtime/pypo/pypo.log
    2012-06-16 09:33:13,721 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:37:28,334 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99290/schedule_id/0
    2012-06-16 09:37:28,439 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:37:28,439 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:40:19,257 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99291/schedule_id/0
    2012-06-16 09:40:19,375 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:40:19,375 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:44:44,099 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99292/schedule_id/0
    2012-06-16 09:44:44,196 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:44:44,196 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
     
  • Didnt yet Upgraded to the new version 2.1.1
    But i very satisfied with 2.1.0, Streaming is really fine and wonderful,
    Can some one just let me know whats 2.1.1 ChangeLog ?

    Thanks
  • Karim your should upgrade mainly to fix the issues we both faced in the last couple of days. Do it and post your feedback here please. I fixed and if you fix too it will be the confirmation of the patch working correctly.

    For help just ask
  • Hello, 
    I upgraded to 2.1.2, 
    Everything seemed to be wonderful, 
    but suddenly few minutes ago i got::
    Schedule running
    On Air Button is Red
    & Airtime Offline in Icecast Status and thus Streaming is totally OFFLINE !!

    Log Files attached as requested by Martin When this issue happens again, 
    I Need Help :)
    Thanks in advance
  • What server time did the off line happened?
  • 4 Hours & 20 Minutes ago, 
    Around 17 EET Server Time 
    Now its 21:16 EET and still Offline

    Thanks

  • Hi Karim,

    I've looked over your logs and it appears files are being push to Liquidsoap and played at regular intervals. I can see as recently as 20:06:13 EET time (the time it appears these logs were created) that the following track began to play:

    2012/06/18 20:06:13 [decoder:3] Method "MP3" accepted "/var/tmp/airtime/pypo/cache/scheduler/d215e7b8e8b43a073dabf68955def168.mp3"



    Post edited by Martin Konecny at 2012-06-18 15:27:09
    Airtime Pro Hosting: http://airtime.pro