Ingesting a remote (soundcloud) audio file - Airtime Development Discussions on Sourcefabric Forum

Eric Pugh

Hello,

I'm working on a project which allows users to not only upload content from the Airtime library to SoundCloud, but also import audio from their SoundCloud account into Airtime. I've added some sample code to github which adds a "SoundCloudImport" controller and some new methods in the "SoundCloud" model

https://github.com/ericpugh/Airtime/commits/master

This successfully lists the available SoundCloud media, and ingests a single item by downloading the file (cURL) and throwing it into the watch folder. What I was wondering if the folks on the board had any ideas about the best way to handle this. Is there a better way to handle the ingest of the file? Am I duplicating any functionality here? Also, wondering what would be the best why to structure code for this functionality (in the SoundCloud Model is probably not the correct way to do it.)

Would appreciate any ideas on the project.

Martin Konecny

Hi Eric,

Really interesting project you have going on here. Regarding the upload to SoundCloud functionality - Airtime already has this feature, unless I'm misunderstanding your intent with "upload content from the Airtime library to SoundCloud"

As for adding SoundCloud files to Airtime, this would be a great feature for Airtime. I'll take a look at the code you have tomorrow and offer some suggestions, and answer your questions more thoroughly.

Albert FR

adding SoundCloud files to Airtime will be a good feature to integrate into Airtime ;-)

Eric Pugh

Hi,

yeah, sorry for the confusion. I'm talking about pulling in audio from a user's SoundCloud account, to be added to Airtime's library. :-)

Martin Konecny

Hi Eric,

Took a quick look at the functionality today and this looks like a great start! I'm pulling in tracks from my SoundCloud account to the Airtime library without any issues :)

Is there a better way to handle the ingest of the file?

I think using php curl is fine for this task,

Am I duplicating any functionality here?

No, this functionality is completely new.

Also, wondering what would be the best why to structure code for this functionality (in the SoundCloud Model is probably not the correct way to do it.)

You are correct, functionality like this should be stored in a service class, unfortunately we do not use this at the moment - services are mixed with models... Feel free to start your own services layer, a good post for working with Zend and Services can be found here:

http://stackoverflow.com/questions/2458195/how-to-implement-service-layer-in-zend-framework

As for future work, I think it would be much better if you based your work off the devel branch. There are many changes in devel (all for the better :), and it will be a bigger pain to merge down the road, if not based off the proper branch.

Albert FR

I'm really impatient to see this feature into airtime, good work Eric Pugh !

Eric Pugh

Excellent. Thanks for the input, guys!

Sounds like a "Service" class would be useful code maintenance in general. I'm going to try and setup the correct branch (as Martin suggested) and post back here with any updates.

Albert FR

hurry up then ;-)

Eric Pugh

Hi guys,

I was wondering if any of you knew what the project is doing to allow the updating of ID3 tags (in php) on imported media? I'm having some issues updating metadata of imported Soundcloud files. I'm trying to update an imported files metadata with the metadata from soundcloud, I'm able to successfully update metadata by creating an instance of "StoredFile" using recallByFilepath(), but the problem is that depending on a number of things (like filesize, i guess) the file may or may not have already been copied into the import folder by the time I try to update the md. So, sometimes it doesn't work.

I was thinking it would be much easier to edit the ID3 tags when the file is first requested. I saw some discussions on the forum about the getID3 library, which looks like it was once used in the project, but doesn't seem to be there now. There are also some PHP functions (http://php.net/manual/en/ref.id3.php) which might be useful... but those would have to been added to the initial install.

Is there another way Airtime is handling this?

Thanks

Martin Konecny

Hi Eric,

We don't use getID3 anymore as this was a poorly maintained project. We actually have a separate Python process in the background that handles all of the metadata parsing.

The way it works is as follows: When the file is download and put into a watched or stor folder, media-monitor (our background python process) is automatically notified that a file was created in one of its folders and it gets to work parsing the file's metadata, finding the file's audio length etc. When this is done, it sends a POST request to the server's ApiController (reloadMetadataGroupAction) which stores this information in the database.

What I would do is allow updating of metadata only once it appears in Airtime's library. In this case, you can be 100% sure that the above outlined process has completed - I'm assuming you're talking about the user editing metadata via the Web UI correct?

Eric Pugh

Hi Martin,

No, I was trying to programmatically update the file's metadata with some of the "track" information returned from the SoundCloud API. At the very least I was hoping to set the title, I was setting a temporary filename of the imported file which included the soundcloud track id, which isn't helpful for users that need to recognize the file they just imported. Maybe, I'll just try to set the filename to the "title" from soundcloud (cleaning problematic characters)... that would at least satisfy baseline functionallity.

Thanks

Eric Pugh

Well, adding a sleep() seems to allow the Metadata to update correctly. I've added some updates to the repo, which seems to be working pretty well. Thanks for the suggestions, guys.

G Potter

Love this idea - I would love it if AirTime could pull from a SoundCloud group http://help.soundcloud.com/customer/portal/articles/874929-where-are-groups- to an Airtime watch folder.

Martin Konecny

Eric Pugh said:
Hi Martin,
No, I was trying to programmatically update the file's metadata with some of the "track" information returned from the SoundCloud API. At the very least I was hoping to set the title, I was setting a temporary filename of the imported file which included the soundcloud track id, which isn't helpful for users that need to recognize the file they just imported. Maybe, I'll just try to set the filename to the "title" from soundcloud (cleaning problematic characters)... that would at least satisfy baseline functionallity.

Thanks

I think a better way would be to download to a temp directory first, edit the metadata, and then move the file to Airtime's stor directory. This way you avoid race conditions that the sleep() function introduces.

Eric Pugh

@G Potter: Agreed... importing content for Soundcloud Groups (which the Airtime user is a moderator of) was the next thing I want to do.

@Martin: Maybe I don't understand how this works, but how could I edit the Metadata from a temp directory? My understanding was that an imported file doesn't have metadata until it was already moved to stor. (meaning it now has an Airtime id and related metadata fields.) That's why I was looking for a way to update ID3 tags, so that I could change the metadata fields on the file itself, rather than updating the md database fields. Am I totally off base here?

Martin Konecny

My understanding was that an imported file doesn't have metadata until it was already moved to stor.

Can you verify this statement? I find it strange that Soundcloud let's you download a file, but you need to grab the metadata separately. I would assume the metadata is contained within the file???

Eric Pugh

All the tests that I've done, Soundcloud reads tags from an mp3 in order to populate the fields on Soundcloud when the file is uploaded, but it never updates them. If you fetch the actual file (via api) the original tags (if any) are still there even if you've edited/added information about the track on SC.

Martin Konecny

Ok that makes more sense to me now. Sounds like the best way to handle this would be download the file to a temporary location, use a tag editor to add the new Soundcloud tags, and then copy that file to the stor directory so media-monitor can handle the rest.

Have you used Python before? It might make sense to use taglib (http://taglib.github.com/) + python to handle this. Of course this could be for version 2 :)

Eric Pugh

Thanks Martin... I'll check it out!

Martin Konecny

Come to think of it, I think we could collaborate on this part of the feature. For example, Airtime's media-monitor already has a metadata tag reader/writer built in. I think the best way to approach this would be to write a temporary text file with the same name but an additional ".metadata" extension that contains the new metadata from soundcloud as simple text. Then when media-monitor picks up the audio, it looks for this additional file and if present, handles updating the metadata. What do you think? We could handle the part I outlined, you would be only responsible for creating this text file with the format

tag1=value1

tag2=value2

etc.

Eric Pugh

Hi Martin,

This sounds like a great idea. (and something that could be useful for other functionality besides soundcloud imports.) Do you think the .metadata file be moved to stor? Looks like the .identifier files might be doing something similar.

Martin Konecny

Exactly. This would be analogous to the .identifier file.
On Mar 11, 2013 9:58 AM, "Eric Pugh"
wrote:

> Hi Martin,
> This sounds like a great idea. (and something that could be useful for
> other functionality besides soundcloud imports.) Do you think the .metadata
> file be moved to stor? Looks like the .identifier files might be doing
> something similar.
>
>
>
>

Eric Pugh

I've created a new function that creates a unique_filename.metadata file as described above. Also added the key "created_with=soundcloud" to the metadata. I was thinking that a switch like that might make it possible for the media-monitor to also update the Soundcloud ID and Permalink of the newly created file. (something like the /utils/soundcloud-uploader script.

thx

Eric Pugh

Hello,

I've recently came back to this project, and have started looking into the media monitor python_apps that could process a special ".metadata" file for an imported audio file. Is there any documentation, or information about these scripts to get me started?

For example, looking through /media/monitor/organizer.py, metadata.py, etc. a lot of the required functionality exists, but what acts as the control structure for these classes? airtime.py? If so, can anyone give a brief description of how this is tied together? (not too familiar with python.)

Thanks.

(also, changed the fork I'm working on with a new branch, after having some issues with upstream merge)

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-25 11:15:48

May I verify: This project imports SoundCloud files into the Airtime library, yes? But Airtime has a WebStream function now. So, I would have thought the way forward would be: Instead of pulling SoundCloud files into Airtime, leave them where they are, just have some code convert them on the fly into a readable format that can be used by Airtime's WebStream URL field. Or would that perhaps be in a different project?

hoerich

I am not sure about implementation of this.
And I am not sure if converting on the fly or predownloading would be better.
But why converting? Just limit sources to already supported ones and LS should deal it.
Personally I don't think this has high priority. Visit bugtracker to vote for ticket dev.sourcefabric.org
cheers

John Chewter

Hi I am now trying to do similar - any updates from all you guys on your progress?

Thomas Hackett · Post edited by Thomas Hackett at 2013-10-15 08:49:45

John Chewter said:
Hi I am now trying to do similar - any updates from all you guys on your progress?

So far it looks like not even the Webstream function works 100%. Quirky LiquidSoap can't deal with as many stream constructs as say Shoutcast/WinAmp/VLC, and then when it does work it causes buffer problems. I see people moving away from central libraries to more distributed file solutions, so I think it would be unfortunate if providing better options in this area doesn't gain more priority, and sooner rather than later.

hoerich

Thomas Hackett said:

John Chewter said:
Hi I am now trying to do similar - any updates from all you guys on your progress?

So far it looks like not even the Webstream function works 100%. Quirky LiquidSoap can't deal with as many stream constructs as say Shoutcast/WinAmp/VLC, and then when it does work it causes buffer problems. I see people moving away from central libraries to more distributed file solutions, so I think it would be unfortunate if providing better options in this area doesn't gain more priority, and sooner rather than later.

I partly agree.
Having basics working is essential. The team did a good bugfixing-job on 2.5 yet.

But every user is free working on whatever he priorizes highest.

I did not have any problems you mentioned but if I got time, I try to reproduce with Winamp and VLC.
My suggestion regarding Shoutcast - don't use it, use Icecast.

cheers

Thomas Hackett

hoerich said:

Thomas Hackett said:

John Chewter said:
Hi I am now trying to do similar - any updates from all you guys on your progress?

So far it looks like not even the Webstream function works 100%. Quirky LiquidSoap can't deal with as many stream constructs as say Shoutcast/WinAmp/VLC, and then when it does work it causes buffer problems. I see people moving away from central libraries to more distributed file solutions, so I think it would be unfortunate if providing better options in this area doesn't gain more priority, and sooner rather than later.

I partly agree.
Having basics working is essential. The team did a good bugfixing-job on 2.5 yet.

But every user is free working on whatever he priorizes highest.

I did not have any problems you mentioned but if I got time, I try to reproduce with Winamp and VLC.
My suggestion regarding Shoutcast - don't use it, use Icecast.

cheers

I cannot say for sure if the stations that cause buffer overflows in Airtime use Shoutcast or Icecast. I doubt very much it makes a difference. As a broadcaster you cannot simply exclude shoutcast.com.

hoerich

Thomas Hackett said:

hoerich said:
I cannot say for sure if the stations that cause buffer overflows in Airtime use Shoutcast or Icecast. I doubt very much it makes a difference. As a broadcaster you cannot simply exclude shoutcast.com.

using icecast is just a suggestion.
shoutcasts current version has been released in february 2007. latest RC in 2011.
so development seems not up-to-date.
if you are dependent to shoutcast.com, then use it.

hoerich

just saw there's light in the dark...
nullsoft released a new RC on 16th october.

stella

Thanks for this information. It useful

anika

Thanks for the nice piece of information and that was quite useful.

Howdy, Stranger!

Categories

Poll

Top Posters