airtime-liquids: out of memory
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Hi all,

    I'm having problem with an Airtime installation. After a while the system runs out of memory and kills the "airtime-liquids" process.

    I'have another server running in the same webradio, it's a twin system but with less music loaded, and it does not have this problem. And on another system wich also has 2GB of RAM and about 1TB of music, no problem too.

    Attached here there are the recent Airtime logs, the syslog portion of the killing episode and the little snippet of liquidsoap code that I replaced in ls_script.liq.

    AIRTIME_STATUS_URL             = http://localhost:80/api/status/format/json/api_key/%%api_key%%
    AIRTIME_SERVER_RESPONDING      = OK
    KERNEL_VERSION                 = 3.2.0-33-generic
    MACHINE_ARCHITECTURE           = x86_64
    TOTAL_MEMORY_MBYTES            = 2018168
    TOTAL_SWAP_MBYTES              = 1951728
    AIRTIME_VERSION                = 2.2.0
    OS                             = Ubuntu 12.04.1 LTS x86_64
    CPU                            = AMD Turion(tm) II Neo N40L Dual-Core Processor
    WEB_SERVER                     = Apache
    PLAYOUT_ENGINE_PROCESS_ID      = 1098
    PLAYOUT_ENGINE_RUNNING_SECONDS = 36335
    PLAYOUT_ENGINE_MEM_PERC        = 0.2%
    PLAYOUT_ENGINE_CPU_PERC        = 0.0%
    LIQUIDSOAP_PROCESS_ID          = 13217
    LIQUIDSOAP_RUNNING_SECONDS     = 1532
    LIQUIDSOAP_MEM_PERC            = 1.4%
    LIQUIDSOAP_CPU_PERC            = 10.9%
    MEDIA_MONITOR_PROCESS_ID       = 18540
    MEDIA_MONITOR_RUNNING_SECONDS  = 21977
    MEDIA_MONITOR_MEM_PERC         = 0.3%
    MEDIA_MONITOR_CPU_PERC         = 0.4%
    RABBITMQ_PROCESS_ID            = 1304
    RABBITMQ_RUNNING_SECONDS       = 36331
    RABBITMQ_MEM_PERC              = 0.2%
    RABBITMQ_CPU_PERC              = 0.0%
    -- Your installation of Airtime looks OK!

    Any thoughts?

    crashatamente,
    Francesco P.
  • 15 Comments sorted by
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Hi Francesco,

    Have you been re-streaming a lot of third party webstreams by any chance? 
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Ok, from the logs I see that you do rebroadcast streams.

    We are looking into this...please stay tuned.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Hi Martin,

    no, that server should not rebroadcast streams! In fact, there are no "Web Streams" defined! :? It is still almost always running on "automatic mode" with the ls_script.liq patch with a direct stream to a local server... where can I check for this rebroadcast in the logs?

    Maybe I did some misconfiguration in the ls_script.liq...

    misteriosamente,
    Francesco P
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    What changes did you make to ls_script.liq? I can take a quick look but can't guarantee anything.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    The only changes I did are the ones in the snippet that I attached to the first message! Just to be sure, I did a diff with a newly downloaded tarball...

    controllatamente,
    Francesco P.

    P.S. The link "1) Download the latest version of Airtime" in the page https://www.sourcefabric.org/en/airtime/download/ still point to the 2.1.3 release!
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Strange, I don't see any attachments on your post. Please post the diff :)

    Martin
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    On the first message in this thread, there is a "ls_script.liq.insert" attach, that's the same output as the diff! :)

    ricontrollatamente,
    Francesco P.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Defn not visible from here, might be a bug in the forum software. 
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Ok! :)

    So I think you can't download the logs too? :?

    Here it is:
    #default = amplify(id="silence_src", 0.00001, noise())
    #default = rewrite_metadata([("artist","Airtime"), ("title", "offline")], default)
    default_music = playlist(mode = "random", reload = 3600, "/data/music/imported")
    default_music = amplify(1., override="replay_gain", default_music)
    default_jingle = playlist(mode = "randomize", reload = 1, reload_mode = "rounds", "/data/music/jingle")
    default_jingle = amplify(1., override="replay_gain", default_jingle)
    default = smart_crossfade(rotate(weights = [1, 5], [default_jingle, default_music]))
    default = normalize(default)
    ignore(output.dummy(default, fallible=true))

    riportatamente,
    Francesco P.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    That script looks fine. 

    Are the operating systems identical as well?
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Hi Martin,

    sorry to be late! Yes, the two machines are twins, both in hardware and in software. The only thing in which they differ is the amount of music loaded on them: about 24G on the failing one, only 9GB on the one that is working fine.

    controllatamente,
    Francesco P.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Strange. Perhaps you can ask the Liquidsoap team whether using the playlist operator on 24GB of data could cause memory issues. 
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Maybe it is so. On a third machine, with 2GB of RAM and 1TB of music, there is no "out of memory" problem, but sometime I get a segfault from airtime-liquidsoap...

    However, I'm reverting from playlist() to request() in that background-music-script, I'll see if things start going better! :)

    speratamente,
    Francesco P.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    It would be really good if you reported those problems. The liquidsoap team
    is really good at fixing bugs.


    On Tue, Dec 4, 2012 at 6:03 PM, FrancescoP <<br />airtime-support@lists.sourcefabric.org> wrote:

    > Maybe it is so. On a third machine, with 2GB of RAM and 1TB of music,
    > there is no "out of memory" problem, but sometime I get a segfault from
    > airtime-liquidsoap...
    >
    > However, I'm reverting from playlist() to request() in that
    > background-music-script, I'll see if things start going better! :)
    >
    > *speratamente,
    > Francesco P.*
    >
    >
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down FrancescoPFrancescoP
    Posts: 52Member
    Hi Martin!

    Just for the record, I think that the problem I was slowly investigating has been already found and fixed in the meanwhile (they got the same conclusion that I got in the meanwhile):

    http://dev.sourcefabric.org/browse/LS-647

    If you are ever planning to generate an updated liquidsoap package... ;)

    eventualmente,
    Francesco P.

Add a Comment