Airtime / server health monitoring
  • Vote Up0Vote Down Sergei S.Sergei S.
    Posts: 54Member
    Hi,

    what's the best practice to monitor the station server, apart from actually listening to it 24/7 ?

    Reading through logs - looking for any particular records?
    How can I find if there's enought RAM or processing power to perform all the crossfades all the time?

    This makes much sense for a shared hosting environment.
    For instance, Amazon's "micro" instance seems to experience lack of processing power from time to time.

    Kind regards,
    Sergei.
  • 7 Comments sorted by
  • Vote Up0Vote Down Andrey PodshivalovAndrey Podshivalov
    Posts: 1,526Member, Administrator, Sourcefabric Team
    hi Sergei,

    you are right, amazon's micro instance isn't suitable for airtime. I recommend using some hardware dedicated server instead of vps.
    1gb ram and 1ghz cpu should be enough for airtime.
  • Vote Up0Vote Down Andrea GiardiniAndrea Giardini
    Posts: 72Member
    Creating a point of discussion about providers / hardware / Operative System should be a good idea!

    I use a Kimsufi base server from ovh and performarces are pretty good (with more than 1200 songs loaded) Wink
    It's an Intel celeron @ 1.20Ghz with 2Gb of Ram. I know that CPU is not so good but for 15€/month that's really cool!

    However you can use a tool for monitoring, or maybe creating a daemon checking server's status each minute. You can extract loadavg, and free mem in a easy way through a bash script Smile
  • Vote Up0Vote Down Sergei S.Sergei S.
    Posts: 54Member
    Andrey Podshivalov писал(а) Ср., 15 Июнь 2011 14:28
    amazon's micro instance isn't suitable for airtime

    Privet, Andrey!
    Thanks for the reply. This was my first experience with Amazon's EC2, and the first dive into unix administration : ) And I found it quite productive and entertaining!
    Amazon's free quota allows to run that micro instance for free - a perfect platform to test.

    What's bad about the micro instance, is that is provides "a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available" - meaning it is often short of processing power, and liquidsoap produces pauses in transmission.

    However, even a standard instance should be ok for the task. But that will cost around $70/month.

    As we didn't get the financing for this project yet, we'll stick to the AWS. Later on will move on some dedicated server at a local provider, and rent some vps's as regional retranslators around the globe.

    Andrea Giardini
    you can use a tool for monitoring, or maybe creating a daemon checking server's status each minute. You can extract loadavg, and free mem in a easy way through a bash script


    Andrea, thanks for the message!

    I found the monitoring tool at Amazon's Console, and it shows the 100% CPU usage throughout the day - see attached chart screenshot.

    Maybe you know which particular log files to watch for the output stream interruptions?
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Sergei,

    I've created a ticket for monitoring of CPU usage:
    http://dev.sourcefabric.org/browse/CC-2388
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Andrey PodshivalovAndrey Podshivalov
    Posts: 1,526Member, Administrator, Sourcefabric Team
    The simplest way for linux host monitoring is nagios. 
    sudo apt-get install nagios3


    It allows to monitor hdd, cpu, mem, processes and get notifications. It works by default w/o any settings. You need just specify root email address in /etc/aliases for notifications.
    root: your_address


  • Vote Up0Vote Down Brandon MartinBrandon Martin
    Posts: 10Member
    We are using Nagios to monitor our Airtime servers. Here's some of the commands I've come up with so far.  These are just checking for core processes running and reporting on how many with critical and warnings. Given the flexibility of Nagios, custom checks could be written for Airtime.







    command[check_apache]=/usr/lib/nagios/plugins/check_procs -c 0:40 -w 5:25 -a 'apache2'
    command[check_liquidsoap]=/usr/lib/nagios/plugins/check_procs -c 0: -w 1:10 -a 'liquidsoap'
    command[check_media_monitor]=/usr/lib/nagios/plugins/check_procs -c 0:150 -w 10:120 -a 'media_monitor'
    command[check_rabbitmq]=/usr/lib/nagios/plugins/check_procs -c 0: -w 1:10 -a 'rabbitmq'

    Post edited by Brandon Martin at 2016-01-05 10:56:29
  • Vote Up0Vote Down Brandon MartinBrandon Martin
    Posts: 10Member
    I've also found this: https://github.com/mairlist/streammonitor

    It allows me to monitor the stream coming from icecast to detect silence or offline streams. Using Nagios to monitor the JSON output using this script: https://exchange.nagios.org/components/com_mtree/attachment.php?link_id=6363&cf_id=24

    The final Nagios command after it was imported: 








    define service{
            use                     generic-service
            host_name               dtphsairtimeakghp
            service_description     Airtime stream less than -20db
            check_command           check_json!-H 10.0.70.120 -P 8321 --key_equals silenceDuration,0:3 -c silenceDuration,4 -m silenceDuration
            check_interval          1
    }



    The -m adds performance data on the silenceDuration field.

    Post edited by Brandon Martin at 2016-01-07 14:32:14
Add a Comment