Can airtime remove duplicate media files?
  • Vote Up2Vote Down Jesper JJesper J
    Posts: 21Member
    Subject pretty much says it all ;)

    I though the import function sorted that out, but now i have dupes throughout quite a bit of my content. Re-uploading isnt an option :/
  • 24 Comments sorted by
  • Vote Up0Vote Down Voisses TechVoisses Tech
    Posts: 1,423Member
    "I though the import function sorted that out, but now i have dupes throughout quite a bit of my content. Re-uploading isnt an option "

    "What You Talking about?"
    :-O :O

    If the  title and album are the same only one will appear in the library,no duplicates ,so I suggest
    If you are using linux the my suggestion is to use either
    • fslint
    • meld

    to find duplicates


    but  use pyrenamer for bulk renaming of files

    Anyone reading this a find it funny about my grammar , I make no apology ,Go get a translator.
    "The Problem with education today is that it takes a university degree to switch on a light bulb"
    "You learn from your mistakes but wise people learn from others mistakes avoid Making mistakes there is not sufficient rooms to make them"
    "Innuendo","If's","Assumptions" and "Fear" are for politician.Who,What,where,When and How are for those seeking knowledge and care about Humanity.
    "I might be in Mud but that does not Make me a Wild Hog(pig)"
    “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage to move in the opposite direction.”
    "The only thing that remains constant is change itself"
    May the force be with you,until our path or destiny bring us in tandem.
  • Vote Up0Vote Down John ChewterJohn Chewter
    Posts: 899Member
    Hi Jesper
    Yes, I have seen this. The following may go a long way to fixing this up quite quickly.

    I guess you have added 'watched' folders after having imported through the upload screen and hence the duplicates?

    If you drill down the 'stor' folder path you may find your oldest imported dupes are several levels down - in folders called things like '1' and 'unknown' as well as ones you will recognise. I would MOVE those to somewhere outside the stor path and any watched paths (for safety). I hope this will solve most of your problem in a few minutes - it did mine.
    Post edited by John Chewter at 2014-01-25 18:38:54
    No longer using Airtime or Libretime.
  • Vote Up0Vote Down Jesper JJesper J
    Posts: 21Member
    I uploaded 10-20 tracks for a demo in the beginning. Later on, I realized that the 20 tracks was a totally random pick from the 3000 tracks that I was going to upload, so I decided to upload all 3000 again. And now I have dupes in the library.

    I did not use the watched folder option - I used the import command in the CLI instead, which brought me to the current situation.

    Im aware of the linux options to find duplicates (but thanks though) - but im thinking that there might be some sort of correlation between the physical structure, and the airtime database, so im not too fond of the CLI utilities, unless it doesnt screw up metadata in the Airtime databases.

    I found quite a bunch of files in a "1" directory below the storage dir - like you said. But is it totally safe to move them out, maybe just merge them with the rest of the content (thus eliminating the dupes)?
  • Vote Up0Vote Down Voisses TechVoisses Tech
    Posts: 1,423Member
    Here is a simpler solution

    Change your import folder in system settings from whatever it is (/srv/airtime/stor) to something else /home/$USER$/Music/

    This will remove all your current songs from the database.(including playlist)

    Use whatever duplicate file remover in either windows or linux on the previous directory,

    Then either make it back as an import folder or watch folder

    You could use the combination of php and the database query but I do not know how safe that is
    Anyone reading this a find it funny about my grammar , I make no apology ,Go get a translator.
    "The Problem with education today is that it takes a university degree to switch on a light bulb"
    "You learn from your mistakes but wise people learn from others mistakes avoid Making mistakes there is not sufficient rooms to make them"
    "Innuendo","If's","Assumptions" and "Fear" are for politician.Who,What,where,When and How are for those seeking knowledge and care about Humanity.
    "I might be in Mud but that does not Make me a Wild Hog(pig)"
    “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage to move in the opposite direction.”
    "The only thing that remains constant is change itself"
    May the force be with you,until our path or destiny bring us in tandem.
  • Vote Up0Vote Down amsterdammackamsterdammack
    Posts: 1Member
    A station with different DJ's can run into this problem very easy. You can't just delete double entries, because it destroys others play-lists rather than replacing the missing gap with the other file... An Admin should have the option to clean the database if this can not be an automatic job.

    Post edited by amsterdammack at 2014-03-02 20:36:17
  • Vote Up0Vote Down hoerichhoerich
    Posts: 627Member, Airtime Moderator

    amsterdammack said:

    A station with different DJ's can run into this problem very easy. You can't just delete double entries, because it destroys others play-lists rather than replacing the missing gap with the other file... An Admin should have the option to clean the database if this can not be an automatic job.



    +1
    Official Airtime Forum Manager
    --------------------------
    Most of the time an issue is located between keyboard and chair.
  • Vote Up0Vote Down Jesper JJesper J
    Posts: 21Member
    +1 from me too. New feature for Airtime?
  • Vote Up0Vote Down John ChewterJohn Chewter
    Posts: 899Member
    This would require some complex mutex and would certainly make mistakes unless you can find a feature 'unique' to the tracks. Yeah Right..
    No longer using Airtime or Libretime.
  • Vote Up0Vote Down John ChewterJohn Chewter
    Posts: 899Member
    Airtime does NOT import lots of potentially useful tags. Not even UID, copyright variants, comments, lyrics (synced or otherwise) and other fields that they reserved space for in the database.- never mind the unique identfier fields :(

    So, how do you guys propose we identify duplicates - based on the metatags stored and available in Airtime?

    Be very specific.

    If you can answer this , then it should be simple, with just a few lines of PostgreSQL code, for an old dude like me, to make an external utility to sort it out. My guess is it would rip out 'similar-ish' tracks too.

    Post edited by John Chewter at 2014-03-23 20:44:28
    No longer using Airtime or Libretime.
  • Vote Up0Vote Down John ChewterJohn Chewter
    Posts: 899Member
    Hmmmm. :)

    OK. So what if we ignore the Airtime metatag system altogether for this purpose?

    I guess one could do recursive checks in the 'unique' ID3 tags fields in the stored audio file structure - and then modify the Airtime database to redirect any show content to the single remaining mp3 file instance. I guess I would pick the one with largest file size, but same length, so I didn't get some 16k/s 'bedroom mixer's' version.

    Of course, before you understand recursion, you have to understand recursion. ;)
    Post edited by John Chewter at 2014-03-23 20:45:41
    No longer using Airtime or Libretime.
  • Vote Up0Vote Down Roger WilcoRoger Wilco
    Posts: 784Banned
    Wouldn't it be better to focus on preventing duplicates from even adding to the database in the very first place though? Versus trying to play clean-up and risking a glitch or worse...
    Post edited by Roger Wilco at 2014-03-24 11:57:44
  • Vote Up0Vote Down Alexander JohnnyAlexander Johnny
    Posts: 1Member
    You can try this Duplicate File Deleter its better  than others,
    I had a similar problem many times. The duplicate files are such that they my hard drives used to end up with bad sectors. Also, there was always a shortage of free space. I regularly use "Duplicate Files Deleter" when i feel that my computer is lagging. You should try it too and i'm sure that you won't regret it.

    It can remove duplicate files,music,exe etc.
  • Vote Up0Vote Down GopalGopal
    Posts: 1Member
    please try "Duplicate Files Deleter program" to remove duplicate files. thanks 
  • Vote Up0Vote Down Andy CourcierAndy Courcier
    Posts: 22Member
    This may be an old thread but I'd like to add some information about duplicate files in Airtime.
    Firstly, you might be surprised to find that Airtime isn't to blame here, and they aren't duplicate files. What? I hear you say, how can that be?

    Ok so let me explain my findings...

    [1] Download MP3tag from http://www.mp3tag.de it is free software.
    [2] Inspect the tags of the files that appear to be creating duplicates in Airtime.
    [3] Look for anything that is different to the meta tags in Airtime, pay special attention to the year field.
    What you might find is that the date is written as 1976//1976 in the year field or similar. I'm not sure where this corruption comes from but it happens frequently. Airtime will see this file as different to a file with the year tag of 1976 and therefore will create what appears to be a duplicate. It is not a duplicate, the meta tags are different. Correct any corruptions/mistakes that you find in the meta tags and save/upload the file and it should overwrite the existing file(s) without creating what appear to be duplicates.

    This is a strange scenario and appears to be caused by the way that Airtime interprets the year field and perhaps others. When you view the meta data in Airtime you will be unaware of the corrupted year field, so it seems that Airtime 'corrects' it in some way but does not re-write the header section on the file, just the database entry.

    If Airtime used a simple regular expression like: ^([0-9][0-9][0-9][0-9])$ to validate the year field this problem would be resolved. This regular expression will allow for four decimal digits from 0-9 each and nothing else, so 1976//1976 would fail validation whilst 1976 would pass.

    I hope somebody out there finds this useful and resolves their issues. I have done a lot of research into this problem and let me tell you, there is nothing on the Internet that will help you very much.

    Airtime need to look at this post and consider their mp3 (ID3) meta data validation methods or simply have Airtime update the ID3 tag in the file, thus writing out the correct year field format and preventing apparent duplicates.

    Happy radio's :)


    Duplicates.png
    1372 x 795 - 157K
    AndyC - BlastFM Internet Radio Stations
  • Vote Up0Vote Down mikeibcmikeibc
    Posts: 41Member
    Anyone tried this ?
    Getid3 Open Source php script: http://www.getid3.org/
    Demo: http://www.getid3.org/demo/

    Duplicate removal possible and no database connection needed.
    I haven't test driven this version so create another folder with a few duplicates and see how it handles that before running on your airtime folders.

    If all's good then follow Voises post refresh airtimes database after you've removed the duplicates.

    "Change your import folder in system settings from whatever it is (/srv/airtime/stor) to something else /home/$USER$/Music/

    This will remove all your current songs from the database.(including playlist)

    Use whatever duplicate file remover in either windows or linux on the previous directory,

    Then either make it back as an import folder or watch folder"

    All future uploads should use mp3tag for tagging before uploading.
    http://www.mp3tag.de/en/
  • Vote Up0Vote Down LarsonLarson
    Posts: 1Member
    try for best result DuplicateFilesDeleter
  • Vote Up0Vote Down tina rodrigotina rodrigo
    Posts: 1Member
    Please use Dulpicate files deleter,it is very simple to use. But make it sure to keep the important files in back up. Thanks.
  • Vote Up0Vote Down fina lorenfina loren
    Posts: 3Member
    Duplicate files deleter software helps you dash off some more space by eliminating copied or 

    unnecessary files in bulk. This software gives forth truly what you would presume based on its 

    name, and it does it effectively and quickly. You would not believe how much space it raids off.


  • Vote Up0Vote Down fina lorenfina loren
    Posts: 3Member
    Duplicate files deleter software helps you dash off some more space by eliminating copied or 

    unnecessary files in bulk. This software gives forth truly what you would presume based on its 

    name, and it does it effectively and quickly. You would not believe how much space it raids off.

  • Vote Up0Vote Down fina lorenfina loren
    Posts: 3Member
    Duplicate files deleter software helps you dash off some more space by eliminating copied or 

    unnecessary files in bulk. This software gives forth truly what you would presume based on its 

    name, and it does it effectively and quickly. You would not believe how much space it raids off.

  • Vote Up0Vote Down JohnrafJohnraf
    Posts: 1Member
    I used to have similar problems too, but after using
    "long path tool" You can use to solve this problem.

  • Vote Up0Vote Down sandoval8sandoval8
    Posts: 1Member
    Duplicate Files Deleter is free, very good for other things, and also includes a duplicate files I would recommend it.
  • Vote Up0Vote Down Steven GSteven G
    Posts: 3Member
    Duplicate Files Deleter is not free https://duplicatefilesdeleter.com/Faq.php Trial version works only first 21 days.


  • Vote Up0Vote Down Steven GSteven G
    Posts: 3Member

    I would strongly recommend Duplicate Media Finder http://duplicate-media-finder.kdo-rg.com

    Duplicate Media Finder belongs to the new generation of duplicate file finders which are much more
    powerful.


Add a Comment