Duplicate files in iTunes after running script

GO TO ADMIN PANEL > ADD-ONS AND INSTALL VERTIFORO SIDEBAR TO SEE FORUMS AND SIDEBAR

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
Hi, I have created some applescript, that creates a playlist of all the files I have modified outside of iTunes, part of the reason for doing this is that iTunes then has updated metadata information for these files without having to select 'Get Info' within iTunes. The trouble is that for some reason some (but not most) of the files now appear in iTunes twice both referring to the exactly same physical file.

Can anybody see what is going wrong

thanks Paul

example applescript

tell app "iTunes"
set new_playlist to (make user playlist with properties {name:"newList12/20/08 4:01 PM"})
add(POSIX file "/Users/renaudg/Music/iTunes/iTunes Music/Astral Projection/Another World/09 Still on mars.mp3") to new_playlist
add(POSIX file "/Users/renaudg/Music/iTunes/iTunes Music/U2/October/03 I Threw A Brick Through A Window.mp3") to new_playlist
add(POSIX file "/Users/renaudg/Music/iTunes/iTunes Music/Emilie Simon/Emilie Simon (bonus disc)/01 Desert (english version).mp3") to new_playlist
repeat with nexttrack in (get every track of new_playlist)
refresh nexttrack
end repeat
end tell

Within the iTune sMusic folder it shows Still on Mars twice (wrong) but it only shows 'I Threw A Brick Through A Window.' once (correct), this is reflected in the iTunes xml file
where the value of Location is identical for the two entries. I tried to post the relevent part of Xml but this causes my forum post to fail with error 'New members are not permitted to post images or links) , but I can make it available.
 

S2_Mac

New member
Joined
Oct 24, 2006
Messages
4,876
Points
0
Location
About 3 feet in front of the monitor
The "add" command is meant to add files to the iTunes library; apparently iTunes considers some of your files to be different than the ones it already knows about, so they get re-added. Can't explain why one file gets two different entries...iTunes 8 is still a bit quirky.

This would all be easier if you could use the "duplicate" command -- the best way to add a track to a playlist -- but duplicate requires a track object, and there doesn't seem to be any way to get a track object from a file path. That is, I've had no luck trying to:
Code:
tell application "iTunes"
	set a_track to some track whose location is "some:file:path.mp3"
end tell
in any variation.
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
Hmm, thanks anybody else have any solution. I thought I could perhaps I could find the song in the iTunes library and then use duplicate , but the search command only allows to search on metadata not by filename
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
I tried your idea with aliases because location is an alias and it stlil doesnt work
tell application "iTunes"
set newalias to alias "Macintosh HD:Users:paul:Music:iTunes:iTunes Music:test.mp3"
set newtrack to (some file track of library playlist 1 whose location = newalias)
end tell

and I get error 'A descriptor type mismatch occurred'

I tried with some other propeties

set newtrack to (some file track of library playlist 1 whose album = "No Cities Left")
set newtrack to (some file track of library playlist 1 whose artist = "The Dears")
set newtrack to (some file track of library playlist 1 whose album artist = "Dearyme")
set newtrack to (some file track of library playlist 1 whose comment = "Fred")

and they all worked fine although that last two were noticeably slower, maybe location is the only one that doesnt work because it only a properties for a file track (as opposed to just track)

So what I am going to do instead is parse the iTunes xml file to generate a list of database ids that relate to the files I want to update (not using applescript so should be quick) then do something like

set my_list to (get file tracks of library playlist 1 whose database ID = 3000 or database ID = 3002)
repeat with nexttrack in my_list
duplicate (get file tracks of library playlist 1 whose database ID = 3000 or database ID = 3002) to playlist "New playlist"
end repeat

but one more question , originally I tried
set my_list to (get file tracks of library playlist 1 whose database ID is in {3000,3002})
but it complains 'iTunes got an error:Handler only handles single objects'
I thought this was standard syntax its alot neater than the Or Syntax
 

S2_Mac

New member
Joined
Oct 24, 2006
Messages
4,876
Points
0
Location
About 3 feet in front of the monitor
I tried your idea with aliases because location is an alias and it stlil doesnt work....

Yeah, tell me about ;-) Makes me crazy that iTunes Applescript falls down so badly in this one area...arrgh!

For all I know, you're a pro at calling Perl (or whatever you'll be using) as a shell script from Applescript....but just in case you're not, download this script for an example of dealing with Perl (see the get_listing() handler just beyond the halfway point for the cleanest example. (And if you are a pro, tell me how to make my code better ;-)


but one more question , originally I tried...

You p'bly meant for that repeat loop to look like this, yes?
Code:
	repeat with nexttrack in my_list
		duplicate (nexttrack) to playlist "All Playlists"
	end repeat
As you've come to see, "elegant" is not Applescript's middle name ;-)

I dunno the canonical explanation; Applescript can test objects against comparative values (file tracks of library playlist 1 whose database ID < 3000 for instance), but it can't test by iterating through a list...I think it's explained somewhere in the Language Guide.

Don't forget -- what iTunes Applescript calls "database ID" is listed in the XML file as "track ID", and the value that iTunes Applescript reports as "file track ID" doesn't exist in the XML file, but can be called as "id" ;-)
Code:
set t to some track whose database ID is 768 -- appears in XML as "Track ID"
-- the above line returns: file track id 7755 of library playlist id 7746 of source id 43
set t to some track whose id is 7755 -- this value does not appear in the XML
-- the above line returns: file track id 7755 of library playlist id 7746 of source id 43
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
S2_Mac said:
For all I know, you're a pro at calling Perl
Thanks, I dont really know Perl or scripting but I am a developer. I dont know about your script but I process the Xml file by reading it line by line and extracting (and caching) the information I require so that the file is only traversed once, there is no chance of it taking expotentially long to process it however many tracks Im looking at.

RE:getting the track objects I wish I knew how the iTunes applescript choose worked internally because if when you use choose it just looks up the value from an index its going to work very quickly BUT if the value isnt indexed then it might internally have to scan all records looking for a match, if this was the case it might be better to just loop through all the tracks once in applescript identifying matches when found. beause Database Id is a key Im assuming it is indexed.

S2_Mac said:
I dunno the canonical explanation; Applescript can test objects against comparative values (file tracks of library playlist 1 whose database ID < 3000 for instance), but it can't test by iterating through a list...I think it's explained somewhere in the Language Guide.
Yes , youre right its here in the Filter section

The filter form specifies application objects only. It cannot be used to filter the AppleScript objects list, record, or text. A term that uses the filter form is also known as a whose clause.
 

S2_Mac

New member
Joined
Oct 24, 2006
Messages
4,876
Points
0
Location
About 3 feet in front of the monitor
I process the Xml file by reading it line by line and extracting (and caching) the information I require so that the file is only traversed once, there is no chance of it taking expotentially long to process it however many tracks Im looking at.

Sounds like you've been burned by regexps at some point ;-) You're doing this in C-something? Like to see the code....

As a benchmark, how does line-by-line reading compare to this perl? (Paste into an Applescript and run; no need for iTunes to be open.)
Code:
global path_to_xml

set path_to_xml to (escape_spaces(text 3 through -3 of (do shell script "defaults read com.apple.iapps iTunesRecentDatabasePaths"))) as string

set ids_and_paths to get_ids_and_locations()

-- functions
to get_ids_and_locations()
	set the_cmd to "perl -e 'local $/ = undef;my $s=<>;my @out=();my $id;my $path;
while ($s=~m|<key>(\\d*)</key>.*?<dict>.*?<key>Location</key><string>file://(.*?)</string>.*?</dict>|sg ) {
  ($id,$path)=($1,$2);$path=~s/&/&/sg;$path=~s/%20/ /sg;
  push(@out,\"$id,$path\\n\");}
print @out;' " & path_to_xml
	return do shell script the_cmd
end get_ids_and_locations

on escape_spaces(txt)
	set {TID, text item delimiters} to {text item delimiters, " "}
	set {txt_items, text item delimiters} to {every text item of txt, "\\ "}
	set {txt, text item delimiters} to {txt_items as string, TID}
	return txt
end escape_spaces
Returns lines of ID, comma, path. Paths have the ampersand html entity replaced with actual ampersand, and hexed space-char (%20) replaced with actual space. Wicked fast; be interesting to know this regexp-based function compares to a straight read-and-parse.
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
Hi, yes Im using java, the code is built into the program so not easy to compare timings or give complete code buts heres a snippet below. Im actually the author of the Jaikoz tagger (dont know id you have heard of it) and Im rewriting the itunes integration for it.

I dont have a problem with regexp , I was making the point about the algorithm. Say that I have 10 tracks that I want to identify in the file and they are the last 10 tracks in the xml file, I dont want to make 10 seperate passes of the file to identify them whereas if I just start at the beginning of a file and process line by line Im only going to have the file I/O overhead once, however many ids I lookup. Now I know you could get all 10 ids in one pass using regex but it looked to me like your original script contained a number of seperate calls to regexp , and each one may have to parse the whole file.

BTW, If Ive understood you regexp your conversion from url to filename isnt quite right because it doesnt convert UTF8 characters that are invalid for a url and hence hex encoded i.e Don%60T should be converted back to Don'T

Code:
 public   Map<Integer, List<Integer>> getTrackDatabaseIdsforWrapper(Set<MetadataWrapper> oldWrappers)  throws URISyntaxException 
    {
        Map<Integer, String> originalPaths = new HashMap<Integer, String>();       
        Map<Integer, List<Integer>> originalIds = new HashMap<Integer, List<Integer>>();
       
        //Convert paths to url form just once
        for (MetadataWrapper wrapper : oldWrappers)
        {
            String convertedPath = convertPath(wrapper);
            originalPaths.put(wrapper.getRecNo(), convertedPath);
        }

        try
        {
            //Test xml file scanning
            File xmlFile = new File(UserPreferences.getPrefs().getSave().getItunesUpdateLibraryXmlFile().getValue());

            if (!xmlFile.exists())
            {
                //TODO shouldnt we show problem in a dialog because more important now
                MainWindow.userInfoLogger.log(Level.WARNING, InfoMessage.MSG_ITUNES_CANT_FIND_XML_FILE.getMsg(xmlFile.getPath()));
                return originalIds;
            }

            final FileInputStream fis = new FileInputStream(xmlFile);
            final FileChannel fc = fis.getChannel();
            BufferedReader reader = new BufferedReader(Channels.newReader(fc, "UTF-8"));
            String line;
            String lastTrackline = "";            
            while ((line = reader.readLine()) != null)
            {
                //We Store the track database id in case we need it, but do not process it yet
                if (line.contains("<key>Track ID</key><integer>"))
                {
                    lastTrackline = line;
                }

                //Found a location line
                if (line.contains("<key>Location</key>"))
                {
                    for (Map.Entry<Integer, String> entry : originalPaths.entrySet())
                    {
                        checkValue(line, lastTrackline, entry, originalIds,originalPaths);
                    }                   
                }
            }
        }

        catch (Exception ex)
        {
            MainWindow.logger.warning("getTrackDatabaseIdsforWrapper failed:" + ex.getMessage());
        }
        return originalIds;
    }

     protected String convertPath(MetadataWrapper wrapper) throws URISyntaxException 
    {
        URI uri = new URI("file", "localhost", wrapper.getFile().getAbsolutePath(),null);
        return uri.toString();
    }
    
    private void checkValue(String line, String lastTrackline, Map.Entry<Integer, String> entry, Map<Integer, List<Integer>> ids,Map<Integer, String> paths)   throws URISyntaxException 
    {
        //Found a Match
        if (line.contains(entry.getValue()))
        {
            //Now work out the track database id
            int startofKey = lastTrackline.indexOf("<integer>") + 9;
            int endofKey = lastTrackline.indexOf("</integer>");
            String trackIdAsString = lastTrackline.substring(startofKey, endofKey);
            int trackId = 0;
            try
            {
                 trackId = Integer.parseInt(trackIdAsString);
            }
            catch (NumberFormatException npe)
            {
                MainWindow.logger.warning("There was a problem parsing:" + trackIdAsString + " to a number:" + npe.getMessage());
            }

            //Found trackdatabase id for match ok, check if already added for this wrapper
            if (trackId > 0)
            {
                if (ids.containsKey(entry.getKey()))
                {
                    List<Integer> trackIds = ids.get(entry.getKey());
                    trackIds.add(trackId);
                    ids.put(entry.getKey(), trackIds);
                }
                else
                {
                    List<Integer> trackIds = new ArrayList<Integer>();
                    trackIds.add(trackId);
                    ids.put(entry.getKey(), trackIds);
                }
            }
            paths.remove(entry.getKey());
        }
    }
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
I cant easily compare timimgs because my code is embedded in the program, but Ive pasted a code extract for retrieving the ids for a list of files from the xml in case of any interest. I'll also paste an example of the applescript it generates for upating itunes when Ive got it working. I am the developer of the jaikoz tagger (don't know if you've heard of it) and Im redoing the iTunes integration on the Mac.

I don't have a problem with regexp but looking at your original script it looked like you were applying multiple regexps to the file, and each one could potentially have to read the whole file, so Im just saying only doing it once if possible.

BTW if Ive understood it correctly your script doesnt convert UTF8 characters back that have to be hex encoded back promperly i.e file://localhost/users/paul/Don%60T%20mess shouldb become file://localhost/users/paul/Don'T mess

Code:
 public   Map<Integer, List<Integer>> getTrackDatabaseIdsforWrapper(Set<MetadataWrapper> oldWrappers)  throws URISyntaxException
    {
        Map<Integer, String> originalPaths = new HashMap<Integer, String>();
        Map<Integer, List<Integer>> originalIds = new HashMap<Integer, List<Integer>>();

        //Convert paths to url form just once
        for (MetadataWrapper wrapper : oldWrappers)
        {
            String convertedPath = convertPath(wrapper);
            originalPaths.put(wrapper.getRecNo(), convertedPath);
        }

        try
        {
            //Test xml file scanning
            File xmlFile = new File(UserPreferences.getPrefs().getSave().getItunesUpdateLibraryXmlFile().getValue());

            if (!xmlFile.exists())
            {
                //TODO shouldnt we show problem in a dialog because more important now
                MainWindow.userInfoLogger.log(Level.WARNING, InfoMessage.MSG_ITUNES_CANT_FIND_XML_FILE.getMsg(xmlFile.getPath()));
                return originalIds;
            }

            final FileInputStream fis = new FileInputStream(xmlFile);
            final FileChannel fc = fis.getChannel();
            BufferedReader reader = new BufferedReader(Channels.newReader(fc, "UTF-8"));
            String line;
            String lastTrackline = "";
            while ((line = reader.readLine()) != null)
            {
                //We Store the track database id in case we need it, but do not process it yet
                if (line.contains("<key>Track ID</key><integer>"))
                {
                    lastTrackline = line;
                }

                //Found a location line
                if (line.contains("<key>Location</key>"))
                {
                    for (Map.Entry<Integer, String> entry : originalPaths.entrySet())
                    {
                        checkValue(line, lastTrackline, entry, originalIds,originalPaths);
                    }
                }
            }
        }

        catch (Exception ex)
        {
            MainWindow.logger.warning("getTrackDatabaseIdsforWrapper failed:" + ex.getMessage());
        }
        return originalIds;
    }

     protected String convertPath(MetadataWrapper wrapper) throws URISyntaxException
    {
        URI uri = new URI("file", "localhost", wrapper.getFile().getAbsolutePath(),null);
        return uri.toString();
    }

    private void checkValue(String line, String lastTrackline, Map.Entry<Integer, String> entry, Map<Integer, List<Integer>> ids,Map<Integer, String> paths)   throws URISyntaxException
    {
        //Found a Match
        if (line.contains(entry.getValue()))
        {
            //Now work out the track database id
            int startofKey = lastTrackline.indexOf("<integer>") + 9;
            int endofKey = lastTrackline.indexOf("</integer>");
            String trackIdAsString = lastTrackline.substring(startofKey, endofKey);
            int trackId = 0;
            try
            {
                 trackId = Integer.parseInt(trackIdAsString);
            }
            catch (NumberFormatException npe)
            {
                MainWindow.logger.warning("There was a problem parsing:" + trackIdAsString + " to a number:" + npe.getMessage());
            }

            //Found trackdatabase id for match ok, check if already added for this wrapper
            if (trackId > 0)
            {
                if (ids.containsKey(entry.getKey()))
                {
                    List<Integer> trackIds = ids.get(entry.getKey());
                    trackIds.add(trackId);
                    ids.put(entry.getKey(), trackIds);
                }
                else
                {
                    List<Integer> trackIds = new ArrayList<Integer>();
                    trackIds.add(trackId);
                    ids.put(entry.getKey(), trackIds);
                }
            }
            paths.remove(entry.getKey());
        }
    }
 

S2_Mac

New member
Joined
Oct 24, 2006
Messages
4,876
Points
0
Location
About 3 feet in front of the monitor
I dont have a problem with regexp , I was making the point about the algorithm....

First, my apologies -- thought you were looking for an Applescript-centric solution; Java puts you up in the Premier league<g> (Obviously Perl is unsuited, since you're accommodating Windows users.) On the basis of thinking you were writing for Mac-only, what strings the code was looking for, and how it was going to match up a previously-found ID with a Location, is what had me curious..thought maybe you'd whip out a Python super-routine or something. (I like perl RegExp 'cuz it saves me having to do too much thinking ;-) I'm just one notch up from illiterate when it comes to Java -- I stopped 'serious" coding way back when MacOS was moving from Pascal to C -- but it looks like you're doing lottsa excess line reading....

Don't know how the Perl runtime actually handles the read (by the literal code it's getting the entire file in one slurp; in reality it may be reading large chunks), but the purpose of the regexp is to march through the file in <dict>...</dict> blocks, looking for ID && Location in each block, and bailing at the first non-match; should be the minimum amount of work.

In the Java, it looks like you're testing every line of the XML for "<key>Track ID</key><integer>", which is also going to read all the playlist entries (each entry comprised of "<dict>\n<key>Track ID</key><integer>nnnn</integer>\n</dict>"); in my XML file, that's about 4X the number of "Track ID" keys that exist in the Tracks section (the only section that lists Locations). Looking for "<key>nnnn</key>" would limit line reads to only the Tracks section.

(Ignoring the crap required to make the perl fit in an Applescript, here's what my little block did:
Code:
"perl -e  #setup perl
'local $/ = undef;  #set read head to read entire file at once
my $s=<>;  #read the file specified by the param "path_to_xml" into scalar var $s
my @out=();  #init output array
my $id;  #init scalar var
my $path;  #init scalar var
#loop thru $s, matching on the track's ID AND Location, until there are no more matches
while ($s=~m|<key>(\\d*)</key>.*?<dict>.*?<key>Location</key><string>file://(.*?)</string>.*?</dict>|sg ) {
  ($id,$path)=($1,$2);  #load matches into vars
  $path=~s/&/&/sg;  #substitute "&" for all "&" entities
  $path=~s/%20/ /sg;  #substitute " " for all "%20"s
  push(@out,\"$id,$path\\n\");  #add ID, a comma, location, and a newline to output array
}
#return output array to Applescript (automatically flattens)
print @out;' " & path_to_xml
Just added the conversion for amps and spaces to make the output minimally human-readable; I'm still learning how to best use Perl's Unicode capabilities ;-)
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
Hi, you are right that further optimization could be done of the java code , i was just making the point that although I scan every line - I only scan each line once.

I have a working applescript and I no longer have the problem with two records pointing to the same file when the metadata has changed but it has flashed up another issues. My algorithm was as follows:

After saving files scan xml file for Track Ids of all tracks edited/deleted/edited and moved.
Create Playlist

Edited metadata only:
Duplicate edited new tracks to playlist

Moved Tracks:
Get reference to original file
Add tracks that have been moved playlist
Update new track from old track with Itunes properties such as rating
Delete the reference to the original file.

Deleted Tracks:
Delete tracks that have been deleted

New Tracks not known to iTunes:
Add tracks that did not exist in the xml files to the playlist

But suprisingly when I rename/move a track that was already in the iTunes database rather than allocating a new id it recognises the filename has changed and just adjusts the details accordingly. This is good news but means I should'nt delete the old reference because that actually deletes the new reference because they are one and the same.

Im just wondering how iTunes does this, I didnt think it monitored changes and even it if does Im suprised it can work out what the filename is changing to without being told, doe s it work in all cases ?
 

S2_Mac

New member
Joined
Oct 24, 2006
Messages
4,876
Points
0
Location
About 3 feet in front of the monitor
But suprisingly when I rename/move a track that was already in the iTunes database rather than allocating a new id it recognises the filename has changed and just adjusts the details accordingly.

On a Mac, iTunes is tracking aliases (not paths); as long as a file maintains the same file system ID iTunes is gonna know right where it is. Ways to break the alias would include moving the file to another volume, or working on a same-name copy of the file. All the XML file shows is a path, but the real index file -- "iTunes Library" -- records aliases and paths.

So you can, for instance, make a copy of an iTunes track (thereby generating a file system ID that iTunes knows nothing about) and replace its original file, and iTunes will recognize the new file by virtue of its path; OTOH you could move an iTunes track to just about anywhere on the same volume, and iTunes will recognize it by virture of the alias (file system ID).
 

ijabz

New member
Joined
Dec 31, 2008
Messages
8
Points
0
Thats good, (unfortunately I dont think it does this on Windows)
 
Top