Airtime 2.1 Playout Problem
  • Vote Up1Vote Down Karim AliKarim Ali
    Posts: 92Member
    Hello, 
    I just Upgraded from Airtime 2.0.3 to 2.1
    I got no streaming although Shows are secheduled already, 
    On Air is not red,
    when going to System>Status 
    i got Playout Disabled, 
    sometimes playout turns to Enabled and On Air turns to Red, 
    Then within seconds it returns back to Disabled and Online is not red again 

    Status Sanpshot attached, 
    Also Logs Attached

    Thanks in advance,

  • 55 Comments sorted by
  • Vote Up1Vote Down James MoonJames Moon
    Posts: 197Member
    Hi Karim, Radio-Libriamoci,

    Have you guys upgraded to 2.1.1? Is everything working fine?

    Thank you
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Have you already rebooted?

    We not yet upgraded but in the past we faced the same issue and it was fixed after reboot
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Thanks alot for your Support, 
    I just rebooted my VPS, and nothing happened, I'm still facing the same problem, 
    Any solutions ?
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Any Help !
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Any Help Would be appreciated, Please !!!
  • Vote Up0Vote Down Adam ThomasAdam Thomas
    Posts: 313Member
    Hi Karim,

    We see your posts. Posting 3 or 4 times here and elsewhere won't get you to a solution any quicker... please give the community time to respond, it's a weekend after all. :)

    As ever, screenshots and logs help people to help you troubleshoot.

    Best, Adam
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    I'm Really Sorry, And Thanks alot Adam :)
    Logs attached again as for the previous file is corrupted

    Thanks alot and sorry again :)
    airtimelog.tar.gz
    3M
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Reply to @Karim+Ali:

    I took a look at your logs and see the following:

    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 54] - Before ANSI_X3.4-1968
    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 58] - No locale currently set. Attempting to get default locale.
    2012-06-09 20:15:32,075 DEBUG - [pypocli.py : configure_locale() : line 62] - No default locale exists. Let's try loading from /etc/default/locale
    2012-06-09 20:15:32,076 INFO - [pypocli.py : configure_locale() : line 73] - New locale set to: C

    Seems like your VPS doesn't provide a default locale. You will need to set it up manually.

    Type:
    sudo dpkg-reconfigure locales


    Then make sure you select your locale (I selected en_CA.utf-8 for Engish/Canada). Make sure the ending has UTF-8!!

    Selecting OK will bring up another screen where you make sure your selection is the default one. After this everything should work.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Hello, 
    Thanks alot for your reply, 
    I did change my default locale to ar_EG for Arabic_Egypt
    When type locale i get:

    LANG=ar_EG
    LANGUAGE=ar_EG:ar
    LC_CTYPE="ar_EG"
    LC_NUMERIC="ar_EG"
    LC_TIME="ar_EG"
    LC_COLLATE="ar_EG"
    LC_MONETARY="ar_EG"
    LC_MESSAGES="ar_EG"
    LC_PAPER="ar_EG"
    LC_NAME="ar_EG"
    LC_ADDRESS="ar_EG"
    LC_TELEPHONE="ar_EG"
    LC_MEASUREMENT="ar_EG"
    LC_IDENTIFICATION="ar_EG"
    LC_ALL=

    The problem is still the same no any improvements

    When Type sudo airtime-check-status I get:

    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 2.6.18-274.18.1.el5.028stab098.1
    MACHINE_ARCHITECTURE           = x86_64
    TOTAL_MEMORY_MBYTES            = 1572864
    TOTAL_SWAP_MBYTES              = 0
    AIRTIME_VERSION                = 2.1.0
    OS                             = Ubuntu 11.04 x86_64
    CPU                            = Intel(R) Xeon(R) CPU           X3450  @ 2.67GHz
    WEB_SERVER                     = Apache/2.2.17 (Ubuntu)
    PLAYOUT_ENGINE_PROCESS_ID      = NOT MONITORED
    PLAYOUT_ENGINE_RUNNING_SECONDS = 1
    PLAYOUT_ENGINE_MEM_PERC        = 0%
    PLAYOUT_ENGINE_CPU_PERC        = 0%
    LIQUIDSOAP_PROCESS_ID          = 4012
    LIQUIDSOAP_RUNNING_SECONDS     = 0
    LIQUIDSOAP_MEM_PERC            = 0.0%
    LIQUIDSOAP_CPU_PERC            = 0.0%
    MEDIA_MONITOR_PROCESS_ID       = FAILED
    MEDIA_MONITOR_RUNNING_SECONDS  = -1
    MEDIA_MONITOR_MEM_PERC         = 0%
    MEDIA_MONITOR_CPU_PERC         = 0%
    RABBITMQ_PROCESS_ID            = 1559
    RABBITMQ_RUNNING_SECONDS       = 196
    RABBITMQ_MEM_PERC              = 1.9%
    RABBITMQ_CPU_PERC              = 0.0%

    Any More Help ?
    Thanks alot

  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    You need to make sure the locale has "UTF-8" attached to the end.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Here is Locale Default:

    root@server1:~# sudo dpkg-reconfigure locales
    Generating locales...
      ar_EG.UTF-8... up-to-date
    Generation complete.

    How about Airtime-system-check results i put in my last message, 
    its saying MEDIA_MONITOR_PROCESS_ID       = FAILED
    & PLAYOUT_ENGINE_PROCESS_ID      = NOT MONITORED

    Help really appreciated, 
    Thanks
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Forget to say thats its still not red, and no streaming at all
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Thank you very much Martin, 
    It Really works i added .utf-8 to my locale defaults 
    and simply works,
    Thanks alot I appreciate all your efforts to provide wonderful support for all airtimers here, 

    For the sake of helping others encountering same issue: 

    1. Add the locale to the list of 'supported locales'
    Edit /var/lib/locales/supported.d/local and add the following line:
    en_CA UTF-8

    2. Regenerate the supported locales
    Run sudo dpkg-reconfigure locales

    3. Change the default locale in /etc/default/locale to 
    LANG="en_CA.utf-8"
    LANGUAGE="en_CA.utf-8:en" 

    Reboot!

    Thanks Alot again :)
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Not sure step one is necessary (you should be able to select the supported locale from the dpkg-reconfigure program), but glad you got it working :)
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Hello, 
    Again On Air is not Red, & Airtime Offline 
    I don't really know why am getting this problem now !
    Logs Attached, 
    Please Help Martin 
    Thanks
    airtimelog.tar.gz
    5M
    Untitled-1.jpg
    800 x 600 - 194K
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Hi,

    just upgraded from 2.0.3 to 2.1.0 but at the end of the process this is the outcomes

    *** Verifying your system environment, running airtime-check-system ***
    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 2.6.32-238.19.1.el5.028stab092.2
    MACHINE_ARCHITECTURE           = x86_64
    TOTAL_MEMORY_MBYTES            = 1048800
    TOTAL_SWAP_MBYTES              = UNKNOWN
    AIRTIME_VERSION                = 2.1.0
    OS                             = Debian GNU/Linux 6.0.5 (squeeze) x86_64
    CPU                            = Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
    WEB_SERVER                     = Apache/2.2.16 (Debian)
    PLAYOUT_ENGINE_PROCESS_ID      = 28614
    PLAYOUT_ENGINE_RUNNING_SECONDS = 0
    PLAYOUT_ENGINE_MEM_PERC        = 0.1%
    PLAYOUT_ENGINE_CPU_PERC        = 0.0%
    LIQUIDSOAP_PROCESS_ID          = 28617
    LIQUIDSOAP_RUNNING_SECONDS     = 0
    LIQUIDSOAP_MEM_PERC            = 0.8%
    LIQUIDSOAP_CPU_PERC            = 0.0%
    MEDIA_MONITOR_PROCESS_ID       = FAILED
    MEDIA_MONITOR_RUNNING_SECONDS  = -1
    MEDIA_MONITOR_MEM_PERC         = 0%
    MEDIA_MONITOR_CPU_PERC         = 0%
    RABBITMQ_PROCESS_ID            = 1437
    RABBITMQ_RUNNING_SECONDS       = 11
    RABBITMQ_MEM_PERC              = 2.7%
    RABBITMQ_CPU_PERC              = 0.0%

    -- There appears to be a problem with your Airtime installation.

    dpkg: error processing airtime (--configure):
     subprocess installed post-installation script returned error exit status 1
    Processing triggers for python-support ...
    Errors were encountered while processing:
     airtime
    E: Sub-process /usr/bin/dpkg returned an error code (1)

    The schedule looks to work, icecast is not connected, the on air light is off.
    I restarted all services step by step but when I type invoke-rc.d airtime-playout the result is Restarting Airtime Playout: start-stop-daemon: warning: failed to kill 3301: No such process
     
    I tried all you said in the previous post but... nothing... please help urgently
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Update.

    the streaming is up, the on air sign sometimes is on but no streaming
  • Vote Up0Vote Down James MoonJames Moon
    Posts: 197Member
    @Kalim: Around what time did you have problem with On Air light? Is server time and your airtime time in the same time zone? if not what are the timezone?

    @Radio-Libriamoci Web: Can you please post log files?

    Thanks
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    I just solved the problem the day before tomorrow with Martin's Help and within just next hours less than 24 hrs i got the prblem again seems around 12 - 1 PM today 

    Server time zone is ar_EG.utf-8 and Airtime time zone is EET 
    Do you have any suggestions ?

    Thanks james
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Ok you were right James, 
    I just resetted my VPS TimeZone to be as same as Airtime Time Zone EET, 
    Now VPS & Airtime have same timezone, and runnding perfect !

    Problem Resolved

    For the sake of helping other, 
    here is how to change your ubuntu server timezone,

    sudo dpkg-reconfigure tzdata

    and Choose your preferences :)

    Thanks
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Agaaaiin Radio Stopped !!!!! 
    On Air not red, Airtime Offline
    its the third time Problem resolved and after hours streaming stopped again, 
    Cant understand hows it happpening !

    Locale Ddefaults set, UFT-8 Added, server and airtime has same EET timezone
    Whats happening every couple of hours !

    Logs attached !
    airtimelog.tar.gz
    6M
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Hi James, here attached the log files.

    In the UI I see that liquidsoap and mediamonitor failed.

    Thanks

    dario
    liquidsoap_log.txt
    2K
    media_monitor_log.txt
    3K
    playout_log.txt
    5K
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    UPDATE June 13rd 8:10 GMT +1

    I reconfigured locales since in the playout log it looket to be "C" and not it_IT:UTF-8, rebooted and AIRTIME 2.1.0 is ONAIR again.

    The problem looks to me the same faced by Karim but now I'm go to monitor the enviroment to find out if it will go down again ina couple of hours.

    Further ideas or comment?


  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Hello,
    @Radio-Libriamoci 
    Please ensure that the server timezone is same as airtime timezone 
    maybe it will help you untill james or martin check our logs 

    Thanks alot,

  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Hi Karim, thank you.

    Actually if you have a look at my playout log posted in the previous post you'll see that the local default were uncorrect.
    I fixed it following the steps you suggested and checked the time zones to ensure they're aligned and they are.

    Now Airtime is working correctly since more than 2 hours and i'm monitoring it. The playout log do not highlight any issue at this time but let's monitor.

    Looking forward a feedback from James, it would be great if other users would share their experience upgrading.

    Thanks again
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    After 4h 30' from reboot it's working correctly
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Thats Perfect, 
    Me As well, its back working and I'm just monitoring !
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    You are running it on a VPS as well Radio-Libriamoci ?
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Yes I do Karim. It's a VPS based on Debian 6.0.4
  • Vote Up0Vote Down James MoonJames Moon
    Posts: 197Member
    Hello,

    The server time and Airtime time timezone doesn't need to be same. The problem you guys are having is something else. We found this bug in pypo and I'm suspecting that is the root causes of your problem.

    Were you guys not scheduling/modifying anything related schedule for more than a hour? and Airtime went off line?
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    James, Airtime went offline yesterday night just after the upgrade.
    Early today, after the locales adjustement end the reboot it started and never went offline anymore.
    One hour later I added couple of file and updated some show.
    Any other test to be performed?

    here attached the last playout log. it looks actually strange (I'm not an expert LOL)
    playout_logs.txt
    4K
  • Vote Up0Vote Down James MoonJames Moon
    Posts: 197Member
    Hi,

    That log doesn't tell me much :(.
    If it happens again, please try to modify schedule(eg, adding/deleting/editing shows or files from now playing page... anything schedule related.) and see if that will make pypo going?

  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    James this is the pypo log as was few minutes ago
    root@rlwebradio:~# less +G /var/log/airtime/pypo/pypo.log
    3 to local cache /var/tmp/airtime/pypo/cache/scheduler/1234d07462ed9d0a52881e0d293154b7.mp3
    2012-06-13 19:42:20,773 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96709/schedule_id/0
    2012-06-13 19:42:20,893 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:42:20,893 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-13 19:46:16,618 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96710/schedule_id/0
    2012-06-13 19:46:18,025 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:46:18,026 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-13 19:46:18,437 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/96711/schedule_id/0
    2012-06-13 19:46:19,206 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-13 19:46:19,207 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    ~
    (END)

    I just rebooted: in the nowplaying view the indication of what's on air was uncorrect, one track late. Now it looks to work correctly but for sure we'll face similar issues in the next hours
  • Vote Up0Vote Down James MoonJames Moon
    Posts: 197Member
    Hi,

    Thanks for the report. We know about this bug and we are fixing it for 2.1.1.
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Thank you James. Your support is really great!
    Post edited by Radio-Libriamoci Web at 2012-06-13 15:28:58
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Im Sorry, 
    but Its really not understandable, 
    I dont have reasons why i get Airtime Offline for Hours and Then Comes online again without performing any changes !
    While Im having tracks running in Airtime Already when it goes offline, 
    Now Im having it running perfect but im just monitoring as expecting going offline in the next hours 
    what is this bug you are talking about Radio-libriamoci & James is it related to our issue ?
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Wonderful, 
    I got Airtime Offlin, 
    While having Shows scheduled & On Air is Red !!!!!!!!!!!!!!!!!!

    SnapShot for Airtime Control Panel & IceCast status at the same time !!
    there is a serious bug, I need Help !
    Untitled-1 copy.jpg
    1366 x 768 - 231K
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    We're randomly experimenting the same.

    James, how much time for the patch?
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Airtime 2.1.1 tarball is now available

    Debian package and official announcement will be tomorrow.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Reply to @Karim+Ali:

    Karim, next time this happens please do the following:

    tar czf airtime.log /var/log/airtime

    and post airtime.log here.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Ok Thank you very Much Martin
    What is it about this new tarball 2.1.1 a new version ?

    Thanks
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Hi James,

    as I'm not so confortable to work with tarball I'm waiting for the UI to notifie the package availabe to upgrade via apt-get. I do not see it yet
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Done right now!
    Airtime 2.1.1 up and running! I'll get back to you for an exhausting feedback.
    In the meanwhile a big big thanks to you and Sourcefabric Staff for the really great support!

    Dario
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Hi Jamesm 12hours after the upgrade it looks to work fine. No stream interruption, no issues with the ON AIR light.

    Here following the Pypo log for your evaluation.

    @Karim please share your experience.

    Thanks

    Dario

    root@rlwebradio:~# less +G /var/log/airtime/pypo/pypo.log
    2012-06-16 09:33:13,721 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:37:28,334 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99290/schedule_id/0
    2012-06-16 09:37:28,439 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:37:28,439 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:40:19,257 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99291/schedule_id/0
    2012-06-16 09:40:19,375 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:40:19,375 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
    2012-06-16 09:44:44,099 DEBUG - [api_client.py : notify_media_item_start_playing() : line 291] - http://localhost:80/api/notify-media-item-start-play/api_key/%%api_key%%/media_id/99292/schedule_id/0
    2012-06-16 09:44:44,196 INFO - [api_client.py : notify_media_item_start_playing() : line 296] - API-Status 1
    2012-06-16 09:44:44,196 INFO - [api_client.py : notify_media_item_start_playing() : line 297] - API-Message
     
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Didnt yet Upgraded to the new version 2.1.1
    But i very satisfied with 2.1.0, Streaming is really fine and wonderful,
    Can some one just let me know whats 2.1.1 ChangeLog ?

    Thanks
  • Vote Up0Vote Down Radio-Libriamoci WebRadio-Libriamoci Web
    Posts: 74Member
    Karim your should upgrade mainly to fix the issues we both faced in the last couple of days. Do it and post your feedback here please. I fixed and if you fix too it will be the confirmation of the patch working correctly.

    For help just ask
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    Hello, 
    I upgraded to 2.1.2, 
    Everything seemed to be wonderful, 
    but suddenly few minutes ago i got::
    Schedule running
    On Air Button is Red
    & Airtime Offline in Icecast Status and thus Streaming is totally OFFLINE !!

    Log Files attached as requested by Martin When this issue happens again, 
    I Need Help :)
    Thanks in advance
    airtimelog.tar.gz
    6M
  • Vote Up0Vote Down James MoonJames Moon
    Posts: 197Member
    What server time did the off line happened?
  • Vote Up0Vote Down Karim AliKarim Ali
    Posts: 92Member
    4 Hours & 20 Minutes ago, 
    Around 17 EET Server Time 
    Now its 21:16 EET and still Offline

    Thanks

  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Hi Karim,

    I've looked over your logs and it appears files are being push to Liquidsoap and played at regular intervals. I can see as recently as 20:06:13 EET time (the time it appears these logs were created) that the following track began to play:

    2012/06/18 20:06:13 [decoder:3] Method "MP3" accepted "/var/tmp/airtime/pypo/cache/scheduler/d215e7b8e8b43a073dabf68955def168.mp3"



    Post edited by Martin Konecny at 2012-06-18 15:27:09
    Airtime Pro Hosting: http://airtime.pro
Add a Comment