Extracting Your Own Location Information From Google - The Hard Way
api code google programming · 23 comments · 1,350 words · read ~41,879 times.
Update! There's a new way to do this - read my latest post to find out how.
Two or three years ago, I was contacted by a recruiter from Google. They wanted to know if I'd be interested in working for the advertising giant.
We played the usual game of dancing around salary and schedules, when he dropped the bombshell - they were looking for people to work on Google+.
I asked the recruiter if he'd seen my G+ profile. At the time, it had this childish scrawl all over it.
We mutually decided not to pursue the opportunity any further.
It's a decision I've often thought about. I've been involved in my fair share of high profile failures, and I didn't feel I could survive another one on my CV. Yet, I now wonder if I would have been able to do anything to improve the awful situation Google+ finds itself in with developers.
Where Is Everybody?
This weekend, I decided I wanted to build a Weasley Family Clock. You remember the one from Harry Potter which lists where every member of the family currently is?
During our last holiday to San Francisco, my wife and I had made extensive use of Google Latitude. She was alone in an unfamiliar city and I was stuck at a conference in the suburbs. It was a simple, passive, and comforting way to keep an eye on each other's whereabouts.
Well, Google killed Latitude. Killed it stone dead. As part of their fanatical quest to drive people on to G+, they made that the only way to share the location data they were gathering.
So, my wife set up an account with the sole purpose of sharing her location with me. That's a +1 for their user numbers but -100 in good will.
Ok, so, let's get started!
Getting Location Data Through The Google+ API
You can't.
Oh, don't get me wrong, you can read the documentation which says it's supported, follow the tutorials which show you how to do it, read StackOverflow posts giving you advice, and watch YouTube videos showing how awesome it is - but you can't get any location data out of the API. Not your friends', and not your own.
I'm a Product Manager. I know that sometimes you have to make tough choices about which new features to include, or which bugs to prioritise.
What I can't understand is how anyone has the gall to actively solicit bug reports from highly engaged members of the community, and then ignore them all.
I can't understand how you can have documentation which is fundamentally contradicted by reality.
I can't understand how any manager at Google thinks that annoying the people who want to interact with you is a good idea.
It's like Google were a toddler, repeatedly punching kids in the face and then wondering why they don't want to play with it.
All the people I know who work at Google are smart, passionate, committed, and eager to do the right thing. But something in that company is stopping them. The very culture of the company seems to be "we're the best of the best, therefore everything we do is perfect. There's no need to fix our broken windows."
I practice a zero-tolerance form of Product Management. Every small customer facing defect is a signal to your customers that you don't care about them. Each bug is a slap in the face and deserves a swift remedy - or at the very least an apology and a timescale for a fix.
Google is sticking its head in the sand and hoping developers will ignore the broken functionality, and broken promises, which now damage its reputation.
How To Liberate Your Location Data From Google
You're going to be in for a fun ride - so strap in!
First, visit Google's Location History Page.
On there, you'll see a link to download your location history by exporting it to KML. Go ahead and click it.
Hey presto! A KML file with your location throughout the day.
The URL format is:
https://maps.google.com/locationhistory/b/0/kml?startTime=1398553200000&endTime=1398639600000
OK, so how do we get this programmatically?
Aha! Foolish grasshopper! Google doesn't provide any API to get this data. You must go through the Web interface.
Now, this is simply no good if we want real time access to our location. So, let's get creative.
If we have our Google cookies, we can retrieve our data fairly simply.
Using Chrome, install this Cookie Exporting add on.
In settings, allow it to access incognito windows.
Open an incognito window and log in to your Location History page.
Click the Cookie plugin, and you'll be rewarded with all your cookies for Google.
Copy and paste the data into a text file. I called mine "cookies.txt" because I'm an original and creative thinker.
To get your data using curl on the command line, run this.
curl -b cookies.txt "https://maps.google.com/locationhistory/b/0/kml?startTime=1398510000000&endTime=1398517922000"
curl will use the cookies in the text file, and retrieve your information.
The eagle-eyed among you will have noticed that the start and end times will need to be updated. They are simply the Unix timestamps for the start and end of the day.
Here's how I calculated them using PHP.
// Today's time is needed to get the bounding for the request
$currentTime = time();
// Beginning and end of the day
$startTime = strtotime("midnight", $currentTime);
$endTime = strtotime("tomorrow", $startTime);
// The Google Location history URL - gets the KML.
$locationURL = "https://maps.google.com/locationhistory/b/0/kml?startTime=" .
$startTime . "000&endTime=" .
$endTime . "000";
To request the file, using PHP, we need to tell curl where the cookies are.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $locationURL);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
$kml = curl_exec($ch);
curl_close($ch);
All I want is the last geocoded point in the file, so there's no need to interpret it as XML (your needs may vary). So I just lazily iterate over each line, looking for the data I want.
// Get the most recent location
foreach(preg_split("/((r?n)|(rn?))/", $kml) as $line)
{
if (strrpos($line, "") === 0)
{
//-1.2345678 51.7654321 0
$line = str_replace("", "", $line);
$line = str_replace("", "", $line);
$pieces = explode(" ", $line);
$long = $pieces[0];
$lat = $pieces[1];
}
}
And there you have it. Stealing cookies, scraping data, and calculating timestamps - all because Google can't get its shit together.
This should have been a simple call according to their well documented (but incorrect) API.
Seriously Google, why should I recommend or use your products and services, when you treat me so disrespectfully?
Alternatives
Some of you may be wondering why I didn't use a different product.
It's true that Glympse, OwnTracks, et al, do a pretty good job.
But that's yet another app to install, configure, update, maintain, and watch for battery life impact. I change phones regularly, and don't want the hassle of remembering another password to another service.
Others may question why I feel the need to track my wife's location all the time. It's simply a low effort way to ensure she knows when I'm stuck on the motorway, and I can see how long I have to clean the house and cook her dinner.
I'm building a dashboard to live by our front door which will give us an at-a-glance status update.
Hey, it works for us!
A Final Message To Google
For the sake of my sanity - and the mental well-being of every developer who has to interact with you - please follow these simple guidelines.
- Fix what is broken, before adding new functionality.
- Don't kill off a well used product without ensuring a robust transition plan.
- Make sure your documentation reflects reality.
- Acknowledge your bug reports - and act on them.
Thanks!
Get your Google Location History the hard way… Again!
Jörg says:
Thanks for finding out — however, it doesn't work for me. curl is returning a „Temporarily moved“ header. The returned URL points to the login screen. It looks like authentication with the „stolen“ cookies does not work (anymore).
Hey Jörg, I've just tried this an it still works for me. What happens if you try to visit the URL in your browser rather than from CURL?
Jörg says:
Result is output to screen:
The document has moved , http://www.google.com/accounts/ServiceLogin
A redirect to the login screen.
If I enter the same URL in the Chome incognito tab, I get a KML-file (looking inside: Yes, geodata). That also proofs they don't check the referer.
Since the cookie-textfile mentions an example that uses wget I try that also. wget also is redirected and then downloads the html of the website. Which is not useful, but at least its funny. 🙂
I've noticed your cookie file has 16 entries (mine has 15), however, since the „interesting parts“ are blured for good reason and my lines have a different order I cannot find out which one is missing (and if thats the reason).
Blamed the Mac. Used a VM with Debian Squeeze Linux instead. Same result.
Just to understand what's going on I use curl without the cookie file. Same result.
Hm.
Jörg says:
OK. Found the reason. This is ridiculous. Very ridiculous.
I saved the file with the name „cookie.txt“ and told curl to use „cookieS.txt“ with an „s“, and curl does not complain at all, not even in verbose mode,… because if the parameter is not a file, curl sends it as a cookie literally. So I was sending the string „cookie.txt“ to google instead of the content of the file.
Gaaah! What a silly parameter format!
However. I get geodata now. Thanks for listening.
I am creating an Open Source Tool that applies geodata to photos, mapping the google location to the image files. I think with the informations of your blogpost I now can finish it. I'd like to mention you in the credits, is that OK for you?
Hahaha! PEBCAK 😉 Yes, happy to be credited & look forward to seeing the final result.
Computers really are silly 🙂
Julian Bond says:
Note here that currentLocation has disappeared from the People API. One year on from the death of Latitude and apparently location is being dropped. AFAIK, there's no longer any way of setting a profile current location since Checkin doesn't set it and I can't find any way of changing my profile from "lives in" to "currently in". My guess is that location history is going to disappear as well.
hey ho.
And yes, I hear you about the Platform API bug tracker. It's a black hole. Try sorting on most starred, there are 100 or so with more than 2 stars most of which are dormant and have been for more than a year. See Also, https://plus.google.com/106416716945076707395/posts/VDBxM9KtQUm My plea from the beginning of the year for Google Staffers to do some pruning and cleaning of the bug tracker database.
Martin Votruba says:
Hi,
I had exactly the same idea with Weasley Family Clock, luckily I found your post pretty soon, so I didn't' have to make this research on my own, thanks a lot. Such shame that they act like that. Beside the fact, that Google Task are not able to show tasks from all your lists at once (!), it's another big disappointment that I had recently experienced.
Anyway, have you made any progress with the Clock? Are they working?
Martin.
Terence Eden says:
I can use it - but I still find that the cookies get invalidated after a little while. Which makes it a bit of a pain to have to re-authorise.
That's quite cool. I'm wondering now if it's possible to implement this in Google Apps Script. Something along these lines : http://stackoverflow.com/a/19658754/4306542 or if at all possible use https://github.com/googlesamples/apps-script-oauth2 to handle the authenticating.
I just came across this page randomly after hacking apart the same page you did. Though my script manually logs in rather than saving cookies. Seems to have the same effect.
Jörg says:
Would you be so kind to share your script (the login part)? I can imagine that a clean login is more future-proof than storing a bunch of cookies from previous logins.
Mark Hughes says:
Aieee! The uri you list above (and the one I have used for a couple years) no longer works, as of Wed Aug 26 @ 19:08:00 MDT or thereabouts.
Any idea what the new url might be? Thanks.
Terence Eden says:
Looks like they've taken it down. It would appear that you can get an export of it at https://www.google.com/settings/takeout/custom/location_history?hl=en&gl=US&expflags - but it's not a real-time feed like before.
Mark Hughes says:
Thanks for the suggestion. Unfortunately, I believe that the "takeout" downloads one's entire history, and then it puts in on google drive. I need only a segment and a direct download to my server would avoid the extra step of getting the file from google drive.
There's some discussion about alternatives in these places:
https://productforums.google.com/forum/#!topic/maps/e4t5ijeVdPs https://stackoverflow.com/questions/32332904/current-url-to-download-kml-data-from-google-location-history
Terence Eden says:
I've also asked on StackOverflow - http://stackoverflow.com/questions/32335114/extfiltrating-google-location-history-from-timeline
Mark Hughes says:
Where do you see the URL that you mention in your stackoverflow post? Is this what you see downloading on your phone? It's considerably different than what I see using chrome on a desktop.
Terence Eden says:
It was what what Firefox reported the download URL was. Having seen your product forum post, I'm inclined to go with your simpler URL!
Mark Hughes says:
Well they both lack the essential virtue of actually working.
larick says:
Hi there !
It seems that Google modified once again the data provided by the exported KML file.... When we ask to display bruteData on the map, all the data are there, but the exported KML file doesn't contains all the data : are missing all the data concerning the night. The exported data starts around 8AM and finish around 9PM...
Did someone else noticed this problem ? ... and have any idea to fix it ?!
Yom says:
Hello Terence,
I find this write up very useful. First I have to explain I am not a technical person so pardon me in advance
We are having issues with verifying users address and we thought Google might have an API for Location History but as you have mentioned they do not. We wonder if we can build a solution that will send a link to customers mobile phone and get them to click it and ask for their permission after which may be able to temporarily access their location history as a once off. I know their are some privacy issues here but we want to be transparent and let them know in the terms of use and give them the option to consent of decline
Kindly explain how you reckon we can achieve this?
Thanks in advance
I don't think Google will let you have that data. You will need to contact them directly to find out.
wissam says:
Hey, I need to implement the exact thing to our app. With the user having the option to decline it.
Did you find any solution for this? Thanks in advance
The net has been struggling with geo-social function for ages and never worked out how to balance privacy with usefulness. Plazes, Dodgeball/Foursquare, Latitude, location in G+posts/Tweets/Facebook. Broken systems, Broken APIs, deadpool. It should be useful. It never works.