Hack Added! (CRON jobable) Bulk MP3 importer from multi RSS Feeds into appropriate watched folders! - Airtime Hacks Discussions on Sourcefabric Forum

John Chewter · Post edited by John Chewter at 2013-11-19 14:08:28

It is here: http://coastal-computing.com/airtime-rss-importer.zip

Ideal for collecting news pods & other regular podcasts and music feeds.

Airtime RSS Bulk MP3 Importer by John Chewter (info@coastal-computing.com).
Version 0.5b.

What is it?

A modified (for Airtime) and extended version of Ichapod by Patrick Simonds (divinitycycle@gmail.com).
This is a BASH shell script intended to be run as a Cron job, or by hand as needed.
Quite simply, Airtime-RSS-Importer downloads files from podcast feeds, formats and files them away into approriate Airtime watched folders.
It outputs files with all the ID3 tags filled in, including cover art (if supplied). The filenames are also very human readable, making it easier to archive & organize podcasts.

Setup & Use

Pre-requisites for this script are eyeD3, xsltproc, wget, and mp3gain.
If you're running Ubuntu, you should be able to just apt-get all of those.

sudo apt-get install eyeD3 xsltproc wget mp3gain

Like any other script, just chmod a+x to give it executable rights.
The script has a few settings in it, plus you need to give it a text file containing the podcasts you want to download. The example (podcasts.txt) is included.

Settings

destinationfolder: This is the root folder Airtime-RSS-Importer will move all the downloaded files to. Sub-directories are created for each feed.

podcastlist: The address in the filesystem where your podcasts.txt is located (enclose in double-quotes if the path contains spaces).

downloadlog: The address in the filesystem where you want Airtime-RSS-Importer to keep an ongoing record of what files have been downloaded. This is checked against to prevent duplicates (enclose in double-quotes if the path contains spaces).

podcasts.txt: This file should contain a list of the feeds you want Airtime-RSS-Importer to download from.
The format is very simple, with one line per feed. You can use none, one, or two labels.
With no labels specified, Airtime-RSS-Importer will attempt to parse the feed name out of the RSS.
With one label specified, Airtime-RSS-Importer will ignore the feed's Channel Name and instead use what you specify for the Artist, Album Artist, and Album tags. Depending on the feed, this may work fine.
With two labels specified, the first label is used for the Artist and AlbumArtist tags, and the second is used as the Album tag. This is my preferred method of use.

Note: DO NOT delete the last line, for some reason it gets ignored.

This script was written almost entirely within Notepad++ and tested on a headless machine running Ubuntu Server

John Chewter tailored Patrick's script to work with Airtime 2.5x

INSTALLATION

This program runs as a bash script. You will need to install the prerequisites, but if you're running Ubuntu they are all in the default repositories and a single apt-get can get you where you need to be:

1/ sudo apt-get install eyeD3 xsltproc wget mp3gain
2/ Using sFTP or FTP - create a new folder: /usr/share/airtime-rss-importer/
3/ Using sFTP or FTP - copy the following files to /usr/share/airtime-rss-importer/
   airtime-rss-importer.sh
   readpodcast.xsl
   podcasts.txt
   readme.txt
4/ Before use and after any changes - remember to use sFTP or FTP make executable (744) /usr/share/airtime-rss-importer/airtime-rss-importer.sh
5/   NOTE: Folders below will be created, if they do not exist.
6/    Edit podcasts.txt to add the RSS feeds you need.
       Example:

       Where Did The Road Go---WDTRG---http://feeds.feedburner.com/WhereDidTheRoadGoShowArchive?format=xml
       Ecoshock---Ecoshock---http://feeds.feedburner.com/EcoShockNews?format=xml
       Conspiracy Queries---Conspiracy Queries---http://feeds.feedburner.com/ConspiracyQueriesWithAlanPark?format=xml


   IMPORTANT - Leave a BLANK LINE at the end of podcasts.txt
   TIP: You may need to add ?format=xml to the end of the RSS URL

7/ Usage
   -h --help show this help
   -d --Days of RSS to fetch. eg. '-d 30' means get the last 30 days files. Default is 7"
   -o --FULL Path to Airtime STOR folder or Watched Folder Root Typically -o '/srv/airtime/stor/' or -o '/home/airtime-watched/'"
   -mp3gain '1' for auto gain adjust - '0' for no gain adjust- default is 0"
   e.g.
/usr/share/airtime-rss-importer/airtime-rss-importer.sh -d 120 -o '/home/airtime-watched/' -mp3gain 1
8/ This can easily be run as a cron job.
9/ Lastly add the folders containing the downloaded files to your watch folder list.

Daniel James

Hi John,

media-monitor is meant to notice when the length of a watched file changes, so that cheat should work (in theory). If you overbook the show with filler content at the end, it should compensate for any shorter-than-expected podcasts.

A better solution might be to have a smart block option to fill the block in date order, newest content first, instead of in random order (which is really only good for music shuffling). Please open a ticket if you think that's a good idea.

Cheers!

Daniel

Albert FR

really good job, John !

Albert FR

The next step : automatic show integration :-)

John Chewter

Er my next step - which is already working here - was to add it to the add media screen.

hoerich

amazing work!

Does Ichapod support HTTP Authentication?

John Chewter

No. Feel free to add it :) It parses public rss feeds for mp3 links, downloads, creates/recreates metatags, renames the file and optionaly normalises the audio levels.
It does not read ATOM feeds HOWEVER - passing it through feedburner which, will make ATOM into RSS 2.0, works ;)

John Chewter · Post edited by John Chewter at 2013-11-23 17:19:48

From some server feeds, it works amazingly fast - on others it is just very fast. :)

hoerich

ok, thx John

John Chewter

Why did youwant it to do that? Have I missed something?

hoerich

John Chewter said:
Why did youwant it to do that? Have I missed something?

We use a the austrian Cultural Broadcast Archive.
On http://cba.fro.at any copyrighted material is automaticall cutted and public accessible.
And it provides uncut versions (including copyrighted music) for program coordinators of community radio stations.

John Chewter · Post edited by John Chewter at 2014-01-20 21:00:30

I have spent 4 days recoding this and it is far more robust now than it was and than the original code I started with.
It handles 'mostly' standard RSS 2.0 feeds as well as standard ones now (handles & corrects common errors in feed). Now almost everything feeds works straight out of the box. Ones that met the standard completely always did, by the way.

I have added mono -> stereo conversion if a mono file is found - because thats is what I need (else liquid soap chokes and you get nice silence for the period of the show even with Voisses silence detection) - Don't get started Voisses - it doesn't ;)
Regularised the resulting mp3 filenames to ensure universal Airtime acceptance..

Would stereo -> mono be important to anybody? Lots of people?
It seems it IS possible to do secure logins for secure feeds. How important is that? I know Hoeric needs that - anybody else?

I am going to add 'set genre' in mp3 file ID3 tag to a named genre/feed option

Any other requests?

Can anybody else supply useful ideas while I am still coding this

Albert FR

A good point will be to automate podcasts integration in shows ;-)
But this feature is more complicated to integrate without danger (postgresql modifications)

John Chewter

Thats my ultimate goal.
I have an idea for a simple cheat to, maybe , get somewhere close.
If I copied the latest imported file to a file called latest.mp3 in each watched folder and we added that file to a linked show - it would work if all the series of podcasts were the same length. Some series I use are - some are not

John Chewter

Hi Daniel,

Thanks for that tip! I can see that media monitor would change the size listed in the library - but would it actually change the times in the scheduled show list? I can't find the code that would do that.

I guess it is easy enough to test today - so I will do that. If it changes the times in the actual show - then the problem is solved!

Albert FR

Daniel James said:

A better solution might be to have a smart block option to fill the block in date order, newest content first, instead of in random order (which is really only good for music shuffling). Please open a ticket if you think that's a good idea.

I'm waiting for this feature too :D

Daniel James

Hi Albert, dynamic dates are in ticket http://dev.sourcefabric.org/browse/CC-4990

John Chewter · Post edited by John Chewter at 2014-03-21 19:58:26

This is as compliant as airtime allows [dont look at me]. This has proven to be a solution for talk radio.
WANTED brave alpha testers

Hi
After a few months of coding I have a working improved super-duper alpha new version working
It makes airtime much more useable for talk radio.
Features:

bulk import from rss feeds
hugely updated widget
Title and usual stuff Imported
Feed desc -> show desc in listings widget - was not easy - digs out comment from 3 RSS tags, if present, -strips html tags - converts utf8-> iso and stuffs it in the conductor tag. If Airtime had allowed it - it would have put it in the correct tag [geeesh]
Feed Image-> show image in listings widget (was not easy and correctly thumbnail resizing ). If Airtime had allowed it - it would have put it in the correct tag [geeesh2]
Latetest mp3 feed copied to latest.mp3 in feed folder - so the latest version of feed always runs e.g.news feed in repeating show
Shedloads more
Buys the tea
Makes the Tea
Drinks the Tea

Email me at info@deprogrammedradio.com if you want an alpha test
If you did not understand my post - alas, its probably not for you. Not for the feint.

TODO
Airtime Compliance.

Robbt

John Chewter said:
I have spent 4 days recoding this and it is far more robust now than it was and than the original code I started with.
It handles 'mostly' standard RSS 2.0 feeds as well as standard ones now (handles & corrects common errors in feed). Now almost everything feeds works straight out of the box. Ones that met the standard completely always did, by the way.

I have added mono -> stereo conversion if a mono file is found - because thats is what I need (else liquid soap chokes and you get nice silence for the period of the show even with Voisses silence detection) - Don't get started Voisses - it doesn't ;)
Regularised the resulting mp3 filenames to ensure universal Airtime acceptance..

Would stereo -> mono be important to anybody? Lots of people?
It seems it IS possible to do secure logins for secure feeds. How important is that? I know Hoeric needs that - anybody else?
I am going to add 'set genre' in mp3 file ID3 tag to a named genre/feed option
Any other requests?
Can anybody else supply useful ideas while I am still coding this

Stereo to Mono would be useful for us at WCRS because our transmitter only broadcasts at Mono and I wasn't sure how to output the Audio from Airtime as Mono to the USB audio device. So if it were able to automatically convert a stereo RSS into Mono for playout that would probably get around broadcasting only one channel of audio. I think that most LPFM's will only broadcast in Mono. I'm also interested in testing the latest alpha of this script but I sent you an e-mail about that already.

What are the steps to integrate this into a future production release of Airtime ? I'm willing to help however I can because I think this feature will really make Airtime a lot more useful to a lot of LPFM's that are going to startup in the next year and would help us get away from using proprietary software that has been giving us a headache with random audio degradation the last couple of months.

John Chewter · Post edited by John Chewter at 2014-03-22 08:32:19

Airtime should (and mostly does) handle the mono-to-stereo and stereo-to-mono automatically accprding to the stream settings. I find that sometimes it gets it wrong and its very hard to reproduce - so I force stereo in mine to be on the safe side. I sometimes see an error that says '2 channels expected - found 1'.

It is very simple to force it to mono instead - so I will add a switch for that.

At the moment it does not touch airtime code and runs on a cron job - the downloaded files appearing in appropriate watch folders, are automatically imported by Airtiime and so are all ready to schedule. :)

It also will email you if there are problems with the import.

Whether Airtime adopts it is up to them - they would have to do some small extra work to get it working in Airtime Pro (no shell access).

Albert FR

Daniel James said:

A better solution might be to have a smart block option to fill the block in date order, newest content first, instead of in random order (which is really only good for music shuffling). Please open a ticket if you think that's a good idea.

I'm waiting for this feature too :D

Albert FR

your feature will be added into the next versions of Airtime/Airtime Pro ?

Daniel James

Hi Robbt,

Robbt said:

Stereo to Mono would be useful for us at WCRS because our transmitter only broadcasts at Mono and I wasn't sure how to output the Audio from Airtime as Mono to the USB audio device. So if it were able to automatically convert a stereo RSS into Mono for playout that would probably get around broadcasting only one channel of audio. I think that most LPFM's will only broadcast in Mono.

I have opened a ticket to add the Channels option available for streams to the hardware audio output:

http://dev.sourcefabric.org/browse/CC-5759

In my view this is a better option than converting files in the library to mono, because you then have the option to stream in stereo while being mono on FM. Or, if you convert to stereo FM later, you will still be able to benefit from stereo archive material.

Cheers!

Daniel

mikeibc

Big thumbs up for this hack :)

I'd like the downloads to keep their original file names is this possible ?

If it's not possible how do could i add a description to the file name? i have a number of shows with the same titles so need to include the description which has the original date that the shows we're first broadcast.

Thanks

John Chewter · Post edited by John Chewter at 2014-04-22 18:25:12

This is an open source hack and it is quite well documented. You could just copy the variable $origfilename into the variable $finishedfilename before the id3 tag rewrite. Then you can be really happy with original downloaded filenames like freddyfoxshow01022014-1931.mp3 (which is typical of many rss feed file attachment names) instead of ones that include the artist, album and song.

There are id3 tags defined for 'first played' and 'last played'. I guess you could modify my new code to add those - but I am not sure how useful that would be even if it contained anything, as most I see are empty, if present at all.

The version I am finishing, right now, puts the text from the feed description into the track description.

mikeibc

I appreciate your good work :)

I did have a poke around in the code but couldn't fathom where the renaming part was going on.
So I'll just hang tight and wait for the new release, :)

Howdy, Stranger!

Categories

Poll

Top Posters