Get your Google Location History the hard way… Again!
Last year, I wrote about how to extract Location History from Google. Once again, Google have changed their URLs to make it even harder to get one's current location out of their data-greedy hands.
It used to be the case that Latitude gave that information - but they killed it. Then they promised it in Google+ - but never delivered. Now they offer you a data-dump which they will email to you. Hardly convenient if you want a single day!
sigh
Recently Google released Timeline - an interactive way to view where you've been. It also offers the ability to download a single day's worth of location tracks as KML.
With a little bit of fiddling, we can get the direct URL. Strap in, it's going to get rough!
In order to get today's date (2015-09-02) the URL needs to be:
https://www.google.com/maps/timeline/kml?authuser=0&pb=!1m8!1m3!1i2015!2i8!3i2!2m3!1i2015!2i8!3i2
That URL format is… unique…
Here's how it breaks down:
pb=!1m8!1m3!1iYYYY!2iMM!3iDD!2m3!1iYYYY!2iMM!3iDD
Where MM is a Javascript style zero-based month. (Why is the first month "0" but the first day "1"? Because fuck you, that's why!)
One thing to note, when you get the KML file, the timestamps are in ISO 8601 format (Yay!) but are set to West Coast USA time (Boo!)
2015-09-02T12:15:37.836-07:00
Annoying, but not the end of the world. How do you get the data into your app?
You can't.
There's no authorised API access for this. You can't do the OAuth dance. The only way is via cookie-jacking.
Sign in to Google via your web browser. Open up your web inspector in Chrome or Firefox. Load the KML URL and choose "Copy as cURL"
You'll end up with a command like:
curl 'https://www.google.com/maps/timeline/kml?authuser=0&pb=!1m8!1m3!1i2015!2i8!3i2!2m3!1i2015!2i8!3i2' -H 'Host: www.google.com' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-GB,en;q=0.5' --compressed -H 'DNT: 1' -H 'Cookie: --------------' -H 'Connection: keep-alive'
You can use that command - or steal your own cookie - for use in your app. Your cookie will likely expire after a set time - whereupon you'll have to reauthenticate.
What a faff! Come on, Google. Rather than pissing about with your logo, why not build something that works properly? I dare ya!
Extracting Your Own Location Information From Google - The Hard Way
The Day Google Deleted Me
Ameer Ellaboudy says:
Ameer says:
J says:
Koen says:
curl "https://www.google.nl/maps/timeline/kml?authuser=1&pb=!1m8!1m3!1i2017!2i4!3i3!2m3!1i2017!2i4!3i3"
--2.0
-H "Host: http://www.google.nl"
-H "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9, /;q=0.8"
-H "Accept-Language: en-US,en;q=0.5"
--compressed
-H "Cookie: NID=102= long string of code here;
CONSENT=YES+NL.nl+V8;
SID= some code here;
HSID= code here; SSID= code here;
APISID= code here;
SAPISID=code here;
OGPC= code here;
OGP= code here;
DV= code here"
-H "Connection: keep-alive"
-H "Upgrade-Insecure-Requests: 1
How can I adapt it to make it work?
jimmy says:
- browse to http://www.google.com (and login)
- open the network monitor
- clear the results
- copy/paste the url ( https://www.google.com/maps/timeline/kml?authuser=0&pb=!1m8!1m3!1i2015!2i8!3i2!2m3!1i2015!2i8!3i2) in the browser address and hit enter
- nothing is added to the network monitor, but the kml file is downloaded.
Michael Ingraham says:
I am able to retrieve the curl command for a Timeline URL encoded to obtain the location data for a particular day. In it are all of the header (-H) fields including the cookie. The cookie itself has many sub-parameters as mentioned in other replies to this post.
I also found this post which has some Python code which can be used to download a KML. The code references a similar method of retrieving the cookie.
What I don’t understand is how to “save off” the cookie details. Should it be stored as plain text in a variable? Should if be store in a text file and referenced in that manner (I’m thinking of how my browser’s cookies get stored and used). I simply do not have the knowledge or experience regarding use of cookies to comprehend this “cookie topic” in any detail.
I actually am trying to create a slight variation of this technique. I have two Google accounts for which I want to look up the current location. Right now I have location history enabled and only turned on for the mobile device associated with each account. Interactively, I want to log on to one of those accounts, retrieve the latest location history (i.e., today’s KML), and then parse it to get the latest logged location for that account.
I’d love to be able to programmatically log in to a particular Google account (a simple snippet of Python code) becoming user ‘0’, fire off the Timeline KML HTTP request, collect the resulting output, and then use a KML parsing library (e.g., http://pythonhosted.org/pykml/). Worst case, I suppose I could log in to each account, save off that account’s specific cookie and then use the appropriate cookie depending on which account I’ve logged in. Ideally I’d like to pass the authentication information directly to the Timeline HTTP request but I haven’t gotten that to work properly even interactively. Ideally, ideally, I’d like to retrieve the cookie programmatically instead of having to store it (and have it go stale).
Another issue I’m having is making my login persistent from the login request to the KML request. Does anyone know how to do that in Python? I’d love some guidance.
Michael Ingraham says:
I was able to perform some additional testing and figured out what portion of the saved curl was the "cookie_content". Indeed, it's obvious now. The cookie is the is that element in the command including everything in that string, including the 'cookie:' label. So, I know what to save.
That git uses this method and also has some great code to capture the daily location history and then parses it and performs varying types of analysis and visualizations. There are a couple of updates to the Google Location History data (LineStrings vs. Tracks) that required a few tweaks (very minor) to the code. I also added a bit to actually save off the coordinates of each "stationary" place logged. It was part of the date. It just wasn't being parsed.
So, I have the KML for the latest day. I'm keeping the logic to just save it to a file rather than trying to figure out how to stream the data back directly. I just call get_kml_file since I'm only interested in the here and now passing the current time year, month, and day. Then I have the great parsing code to get the latest logged Place coordinates. Then I delete the file immediately afterwards.
I'll need a bit of logic to find the most current "Point" in case the latest entry is a "moving" entry.
Google sure is doing everything they can to make life difficult.
Cheers!
Mike
P.S. Now the bit about doing this with multiple Google accounts and their associated cookies. And the bit about cookies going stale and needing to be refreshed.
P.P.S. If you have any insights, I'm open to any and all suggestions.
If anyone is interested, here's the link: https://github.com/Excel-projects/Excel-Timesheet