What Tomtom does when you "submit anonymous data"

[canderson]

The discussion in this thread changed from Latest Map Guarantee, to an interesting discussion on submission of anonymous data. I moved this to its own thread as it is in interesting/important topic on its own. -MVL

When a tomtom submits anonymous data on the driving history (per this link), it must hold a hidden GPS log to do so. Seems like tracking an owner's mileage would be an easy calculation based on that log.

When you listen, the interviewer is asking primarily about Mapshare information (e.g., changes to street names) and how they apply a statistical model for validation.

The part that you're describing would be their new "road discovery" mechanism, but that is limited to the brand new "Live" units, isn't it?

 

[gatorguy]

No. The very limited miles of new roads "discovered" by data uploads in the new map pre-date the release of the first NA live device. All the newer TomTom's AFAIK do keep a tracklog, inaccessible to users, that is transmitted back from opt-in users via Home. It would be tough to log speeds driven on specific roads at specific times without a full log including coordinates, time and date-stamped. But so far, very, very little of the "undiscovered roads" data has actually resulted in a road appearing within the latest map. In fact it's really insignificant so far. It is not a replacement for physically driving a road to map it IMO.

In addition, I suspect the log couldn't store more than the last 1000 miles or so driven, maybe a bit more. So those that seldom connect to home let much of their data get overwritten, never sent to TT.

 

[mvl]

Teleatlas apparantly pulled in the link. They refer to it if you listen to the whole webcast.

They had a posted a picture last February that showed the example of the correction process.

It showed two things:
1) a road that was off from the true location by about 100 feet. The road was on the map, and a ton of dots were on a separate section. The dots were supposed to be tomtom tracklogs. Teleatlas said that the map 830 would have those locations computer-identified and corrected. (DHN - is all of Alberta still a fe hundred feet off?)

2) it showed a new development, a straight road where instead all the tracklog dots were squiggly. Again it said a new road was created.

Edit: it looks like PGPSW saved a copy here.

 

[gatorguy]

There's also a copy of it at GPSReview.
http://www.gpsreview.net/mapshare-shapes-tele-atlas-maps/

 

[dhn]

Many Albertan members state that the 'road' for the Trans-Canada Highway is off by 100's of feet.

That's like saying I-95 in the States is misplaced by 100's of feet.

 

[canderson]

In addition, I suspect the log couldn't store more than the last 1000 miles or so driven, maybe a bit more. So those that seldom connect to home let much of their data get overwritten, never sent to TT.

Even so, it should be fairly easy to spot a file increasing in size over time, and suddenly being reduced during a connect with Home. Hmmm...

 

[gatorguy]

If you discover where the file is stored, be sure to post it. I don't think it's it's in a file location we have access to, but that's only a guess.

 

[mvl]

Many Albertan members state that the 'road' for the Trans-Canada Highway is off by 100's of feet.

That's like saying I-95 in the States is misplaced by 100's of feet.

I haven't seen complaints about Alberta since map 830. I'm curious if this process actually fixed it. Do you know anyone who still has these problems?

 

[mvl]

If you discover where the file is stored, be sure to post it. I don't think it's it's in a file location we have access to, but that's only a guess.

I think it's also in the part of memory that isn't exposed to the USB-drive emulator. Probably the same mysterious place that clear-flash touches, and the place that holds the hidden birthday.txt equivalent (wasn't hidden in app 6.x) that runs the LMG timer.

You'll have to do a binary compare on an explorer backup before and after a drive around the block. I did a visual compare on timestamps and nothing popped up as suspicious.

Another thing to check is, does the internal memory's total size drop a few bytes each time you drive a long distance.

 

[mikealder]

Look in the statdata folder on the device in here should be triplog-2009-06-15 (Obviously the dates will be different), if your folder is empty then you must have answered NO to a question about collecting trip data anonymously. You cannot do much with these files though as they are in a format only TT will understand - Mike

 

[mvl]

I have anonymous sharing enabled. You know what, maybe HOME doesn't even ask for the upload and ships it everytime when you connect. It sure takes a long time to provide the update list.

I didn't find those files in statdata, just the allowtrip.dat file with the flag on whether I permit anonymous uploads or not. Perhaps I have to disable the HOME app/service as it may just auto share/delete them each time I connect.

If I remember from your PGPSW posts, Mike, I think you rarely/never use HOME, which is probably why you see them.

 

[canderson]

Look in the statdata folder on the device in here should be triplog-2009-06-15 (Obviously the dates will be different), if your folder is empty then you must have answered NO to a question about collecting trip data anonymously. You cannot do much with these files though as they are in a format only TT will understand - Mike

Mine is always just a 1 byte file containing the ASCII number "2". Not too helpful. I do have the anonymous data enabled. Then again, I just recently did an update. Need to make a point of looking BEFORE I let Home do its thing and see if the results are different.

 

[gatorguy]

My guess is still that the actual data file cannot be seen by users. The "2" in your file, Canderson, could simply be the equivalent of "yes" (opt-in) for TomTom to upload the anonymous data.

 

[mikealder]

AFAIK Home uploads the saved data as soon as you connect to it, I might have these specific files on my device due to other tests I have been running but I don't think so, the 720 hasn't been near Home in quite some time unlike some of my other units (my choice)

My usual method of using Windows Explorer and Winrar managed to wipe the 720 earlier today, so even thats not completely idiot proof, but at least my backup worked. On this occasion I am not too sure what Vista decided to do other than crash the PC during a file transfer, very interesting results all the same - Mike

 

[canderson]

My guess is still that the actual data file cannot be seen by users. The "2" in your file, Canderson, could simply be the equivalent of "yes" (opt-in) for TomTom to upload the anonymous data.

Just did a substantial run with a couple of destinations, and the file still contains only a "2", and the folder contains only that one file. So if the history is being kept, it doesn't appear to be here. Need to do the full compare on another run later.

 

[mvl]

My guess is still that the actual data file cannot be seen by users. The "2" in your file, Canderson, could simply be the equivalent of "yes" (opt-in) for TomTom to upload the anonymous data.

I have a backup before the opt-in. It doesn't have allowtrip.dat, instad it has disallowtrip.dat. So that must be how the tomtom does it.

I did scour through my backups and one of them does have the files Mike mentioned in the statdata folder. None of my other backups has them, so HOME probably uploads and deletes at the earliest opportunity, probably before you even attempt an update. The files must have been from my one backup where I used an SD reader instead of the tomtom.

I opened up the files and they're a bunch of binary gobbledygook.

 

[mvl]

Just did a substantial run with a couple of destinations, and the file still contains only a "2", and the folder contains only that one file. So if the history is being kept, it doesn't appear to be here. Need to do the full compare on another run later.

Try killing the home process and service, then plugging in the tomtom. I'm not at home now, I would have otherwise tried myself. I doubt you even need to drive anywhere, just walk around your yard.

 

[gatorguy]

Did a little investigating. The statdata file has either a 1, 2, or 3 as best I can determine. If you've opted out, it seems the number is always 3, at least on the four devices I've checked. Opt in can be either a 1 or 2, so a bit more checking to see what, if anything, it indicates. Perhaps opt-in, but no current data, or opt-in, data file exists? Just guessing.

 

[mvl]

I have a backup before the opt-in. It doesn't have allowtrip.dat, instad it has disallowtrip.dat. So that must be how the tomtom does it.

Did a little investigating. The statdata file has either a 1, 2, or 3 as best I can determine. If you've opted out, it seems the number is always 3, at least on the four devices I've checked. Opt in can be either a 1 or 2, so a bit more checking to see what, if anything, it indicates. Perhaps opt-in, but no current data, or opt-in, data file exists? Just guessing.

My old pre-optout backup was app 8.320, so perhaps tomtom used different coding on that navcore version.

 

[dhn]

FWIW, I have 3 app backups and I have a 2 in each one.

In the backup of the app I'm currently using (8.350) I have a file called triplog-5-05.dat which, as I remember, was the last time I used the unit. I just looked at the actual file on my unit today and it is now called triplog-2009-06-09.dat.

What's interesting, to me anyway, is the timestamp. It shows 8:03 pm. Since I'm typing this at 6:12 pm Eastern, I can only surmise that the time may be based on the European servers.

The only problem with that is that I connected using Home here about one hour ago, so the time in the Netherlands has to be more than 3 hours ahead. Strange.