# Using cURL to interact with Google Data services **Warning** : This page is about Google's older APIs, the Google Data APIs; it's relevant only to the APIs that are listed in the [Google Data APIs directory](https://developers.google.com/gdata/docs/directory), many of which have been replaced with newer APIs. For information about a specific new API, see the new API's documentation. For information about authorizing requests with a newer API, see [Google Accounts Authentication and Authorization](https://developers.google.com/accounts). *Ryan Boyd, Google Data APIs Team September 2007* - [Introduction](https://developers.google.com/gdata/articles/using_cURL#introduction) - [Obtaining and installing cURL](https://developers.google.com/gdata/articles/using_cURL#installation) - [Authenticating to a Google Data Service](https://developers.google.com/gdata/articles/using_cURL#authenticating) - [Using ClientLogin](https://developers.google.com/gdata/articles/using_cURL#authenticating-clientlogin) - [Retrieving feeds and entries](https://developers.google.com/gdata/articles/using_cURL#retrieving-feeds) - [Updating entries](https://developers.google.com/gdata/articles/using_cURL#updating-entries) - [Creating entries](https://developers.google.com/gdata/articles/using_cURL#creating-entries) - [Deleting entries](https://developers.google.com/gdata/articles/using_cURL#deleting-entries) - [Uploading media objects](https://developers.google.com/gdata/articles/using_cURL#media-support) - [Other command-line tools](https://developers.google.com/gdata/articles/using_cURL#other-tools) - [Conclusion](https://developers.google.com/gdata/articles/using_cURL#conclusion) ## Introduction At heart, Google Data APIs use [Atom](http://atompub.org/rfc4287.html) feeds and entries (XML) as a data format and HTTP as a protocol for data transmission - extending the [Atom Publishing Protocol](http://bitworking.org/projects/atom/draft-ietf-atompub-protocol-17.html). We publish a number of [client libraries](https://developers.google.com/gdata/clientlibs) to make interacting with Google Data APIs easier. However, you're always welcome to use lower-level tools to work with our services, and it's pretty easy to do with a little guidance. cURL is a command-line application for performing requests using a variety of protocols including HTTP. cURL is often used by developers to test Google Data services, as it supports the HTTP functionality required to interact with the APIs at a low level. cURL only provides support for performing the HTTP communication, so knowledge of the [Google Data protocol](https://developers.google.com/gdata/basics), the service-specific protocol and the XML data format used is a prerequesite for working with the application. Some other tools are mentioned in this article for making these tasks easier. This article uses examples based upon the Picasa Web Albums data API. However, all these examples can be readily applied to other [Google Data APIs](https://developers.google.com/gdata). ## Obtaining and installing cURL cURL is commonly available on a default install of many UNIX/Linux platforms. Try typing `curl` in your favorite shell to see if the tool is installed and is in your `PATH`. If you don't have the tool installed, visit the [download page](http://curl.haxx.se/download.html) on the [cURL website](http://curl.haxx.se/) to obtain the official source or a user-contributed binary package. Note that the command-line tool uses the `libcurl` library, which may be offered as a separate download package, so, if you're not compiling from the source, be sure to download a 'binary' package instead of a 'libcurl' package. The SSL-enabled packages are required if you wish to use cURL to [acquire authentication tokens](https://developers.google.com/gdata/articles/using_cURL#authenticating) or to access some Google Data services which require using SSL for requests. ## Authenticating to a Google Data Service Authenticated Google Data requests are performed by adding an HTTP header to the request which contains either a [ClientLogin](https://developers.google.com/accounts/docs/AuthForInstalledApps) (desktop/mobile apps) or [AuthSub](https://developers.google.com/accounts/docs/AuthForWebApps) (web apps) authentication token. For the purposes of testing using cURL, ClientLogin is the easier method and is documented below. AuthSub authentication headers could be used with cURL, but the more-advanced process of obtaining the tokens is out of scope for this article. ### Using ClientLogin ClientLogin is intended for installed (desktop/mobile) applications. With this method of authentication, the application using Google Data APIs directly handles the username and password of the user. An authentication request for [ClientLogin](https://developers.google.com/accounts/docs/AuthForInstalledApps) takes a username, password, and service name as form post variables. These variables are passed as the `Email`, `Passwd`, and `service` arguments, respectively. This request yields a response with several tokens, one of which can be used to make requests to the Google Data service. Note that data arguments passed with `curl` must be [URL-encoded](http://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters) if they contain non-ASCII characters, which often appear in `Email` and `Passwd` arguments. You can ask `curl` to URL-encode these arguments by using the `--data-urlencode` flag. Example Request: ``` curl https://www.google.com/accounts/ClientLogin \ --data-urlencode Email=brad.gushue@example.com --data-urlencode Passwd=new+foundland \ -d accountType=GOOGLE \ -d source=Google-cURL-Example \ -d service=lh2 ``` Example Response: ``` SID=DQAAAHYBADCv2pSv7nflacDNwz3zEDUGtrSvNVDcpkSfddi77b3U5sEaHmP8YLWhmA36F9rk85mL8J5dqo4apn0T1vKz0fPGI9Xtnuet6cuE2ZzYvrNIwbSC_HjTqF4zudNQnnlDuD2wqZT-g1qXI8KhGAQZV4NexHZoQPlabTsGuRZeIBxj1A LSID=EUBBBIaBADCl-kNxvRVmcQghpt3cqSMfEooKR9flLOUZqwgP9OrZS83gse-KSdTNeXhxsET7FYenDhceP9lIPOmesH-t9qh-AWUHjjMdZEbUNeF9mWyzln6Z-FajaiG-cVFkqW0ZJ8ZbnCP30xXj6xFK6QxaAcqy_9Pej8jhEnxS9E61ftQGPg Auth=EUBBIacAAADK-kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxsET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfagZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh ``` Please see the ClientLogin documentation for specific information on the [parameters](https://developers.google.com/accounts/docs/AuthForInstalledApps#Parameters) used in the above request. In this example, the service we're using is the Picasa Web Albums data API. The service name (`service`) is `lh2`. The service names for other Google Data services can be found in the [Google Data APIs FAQ](https://developers.google.com/gdata/faq#clientlogin) page. The value of the `Auth` token in the above response is the only value needed for authentication to Google Data services. The value of this token is formed into an HTTP header which is then used for each request to a Google Data service. ``` curl --silent --header "Authorization: GoogleLogin auth=EUBBIacAAADK\ -kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxs\ ET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfag\ ZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh" \ "http://picasaweb.google.com/data/feed/api/user/default" ``` **Note** : The method of escaping newline characters with backslash characters ('\\') above does not work in the Windows command shell, so you must enter the entire command on one line if you're running `curl` on Windows. ## Retrieving feeds and entries In Google Data APIs, retrieving feeds and entries is done by performing an HTTP `GET` on a URL, with an optional set of query parameters. Because we are performing a `GET` request, only the auth header and the URL are required to be passed to `curl`. The example below will continue using the Picasa Web Albums data API and is used to retrieve a list of albums owned by the authenticated user. Note that we have shortened the auth token to `ABCDEFG` in this example, but the full token (e.g. `EUBBIacA`...`32JKOuGh` from above) should be used in its place. ``` curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default" ``` This will return an unformatted blob of XML: ``` http://picasaweb.google.com/data/feed/base/user/brad.gushue2007-09-13T21:30:21.454Z... ``` There are some decent tools for formatting this output to make it more human-readable, including [tidy](http://tidy.sourceforge.net/). The easiest way to use tidy is to pipe the output from the curl command to tidy like the following: ``` curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default" | tidy -xml -indent -quiet ``` This results in a much more readable feed, like the following: ``` http://picasaweb.google.com/data/feed/api/user/brad.gushue 2007-09-13T21:47:07.337Z brad.gushue http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue Brad http://picasaweb.google.com/brad.gushue Picasaweb 8 1 1000 brad.gushue Brad http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593 2007-05-23T04:55:52.000Z 2007-05-23T04:55:52.000Z Trip To Italy This was the recent trip I took to Italy. public Brad http://picasaweb.google.com/brad.gushue 9810315389720904593 ... ... ``` Individual entries can be retrieved in the same way by providing the URL for the entry, as opposed to a feed URL. ## Updating entries Entries in Google Data APIs are updated by doing an `HTTP PUT` to the edit URL with a new copy of the entry's XML in the body of the request. 1. Retrieve the entry using the `atom:link/@rel='self'` URL value 2. Update the entry locally to make the needed changes 3. `PUT` the entry back to the server, using the `atom:link/@rel='edit'` URL value ### 1. Retrieving the entry The entry can be retrieved using one of the two URLs bolded in the feed block above. The URL needed is the `href` value for the `link` element with a `rel='self'`. ``` curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593" ``` ### 2. Updating the entry locally After you retrieve the entry, the entry needs to be updated using a local text editor or application to make any needed changes to the entry. In the command above to retrieve an entry, we did not pipe the results to `tidy` as we have done in the previous examples. The result is XML that represents the same data, but has different formatting than the version piped to `tidy`. For the purposes of hand editing an entry, using `tidy` can often make working with the XML easier. **Note** : Please remember to include all XML namespace definitions which are used as attributes to the `atom:entry` when you post your new entry. Omitting these will cause parsing exceptions. Also, `tidy` will replace the spaces between the namespace definitions with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using `tidy`, please be sure to add additional spaces between these attributes on the `entry` element. ### 3. Updating the entry on the server Using the `edit` URL, you need to `PUT` a copy of the entry to the service using cURL. A header to indicate the type of content being sent to the server needs to be added. The following snippet assumes that the file with the updated entry is saved in updated_entry.xml. ``` curl --silent --data-binary "@updated_entry.xml" --request PUT --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456" ``` ## Creating entries Entries in Google Data APIs are created by doing an `HTTP POST` to the post URL with a new entry. The `atom:id` is assigned by the server, and thus not necessary to include in new entries. The easiest way to create a new entry is to take an old entry and modify it. The following example will do just that. 1. Retrieve a template entry using the `atom:link/@rel='self'` 2. Modify the template entry locally to remove unnecessary information and make the needed changes 3. `POST` the entry back to the server, using the `post` URL for the feed. This is either found in the retrieved feed as the `href` value for the `link` element with a `rel='http://schemas.google.com/g/2005#post'`, or in the documentation for the service on . ### 1. Retrieve a template entry A single entry can be retrieved using the `href` value of a `link` element with a `rel='self'` in the same way as an entry was retrieved before updating it in the above example. ``` curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593" | tidy -xml -indent -quiet > template_entry.xml ``` The response, after using `tidy` will look something like: ``` http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981 2007-05-23T04:55:52.000Z 2007-05-23T04:55:52.000Z Trip To Italy This was the recent trip I took to Italy. public Brad http://picasaweb.google.com/brad.gushue 9810315389741123981 TripToItaly public 1179896152000 0 500 0 brad.gushue Brad true 0 Trip To Italy This was the recent trip I took to Italy. Brad ``` ### 2. Modify the template entry We want to create an album called "Curling in Canada" with pictures from our recent curling match. Google Data allows you to drop Atom elements that the server supplies values for, so to create this simple template entry, we'll remove the `atom:id`, `atom:published`, `atom:updated`, `atom:author`, and the various `atom:link` elements in the feed. This will give us a stripped-down template entry. The entry then needs to be modified to represent the new album we are creating: ``` Curling in Canada Pictures of all my curling matches in Canada. Canada public true ``` **Note** : Please remember to include all XML namespace definitions which are used as attributes to the `atom:entry` when you post your new entry. Omitting these will cause parsing exceptions. Also, `tidy` will replace the spaces between the namespace definitions and replace them with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using `tidy`, please be sure to add additional spaces between these attributes on the `entry` element. ### 3. Posting the new entry to the server The `curl` command for posting a new entry to the server is very similar to updating an existing entry except that the URL is different: ``` curl --silent --request POST --data-binary "@template_entry.xml" --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue" | tidy -xml -indent -quiet ``` If the post was successful, the resulting XML output is a copy of the newly-created entry. This entry will include things which the server generated at the time the entry was created, including the values for the `atom:id`, `atom:published`, `atom:updated`, and `atom:link` elements. The resulting `link` values can be used to edit or delete the entry, provided no additional changes are made [in the interim](https://developers.google.com/gdata/reference#Optimistic-concurrency). ## Deleting entries Deleting entries is very similar to updating entries, except an `HTTP DELETE` method is used instead of an `HTTP PUT` and no data is required to be sent. Also like the update request, the `edit` URL is used as the target of the HTTP request. ``` curl --silent --request DELETE --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456" ``` ## Uploading media objects An important feature of the Picasa Web Albums data API and the Documents List data API is the ability to upload binary objects. cURL can easily accomplish uploading binary data and a slug header. However, the Documents List data API currently requires posting the XML along with the binary data as a MIME multipart message. Forming the multipart message is out of scope for this article. The example below shows how to upload a picture called `sweeping_the_rock.png` to a Picasa Web Album with the title "Sweeping the rock": ``` curl --silent --request POST --data-binary "@sweeping_the_rock.png" --header "Slug: Sweeping the rock" --header "Content-Type: image/png" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/5113621341847124417" | tidy -xml -indent -quiet ``` ## Other command-line tools Some developers prefer learning or debugging using other command-line tools. Popular tools include: - **telnet, openssl** are used for making raw socket connections (plain text and ssl-based, respectively) to web servers and can be used to interact with Google Data services. Note that not all Google Data services may support SSL. Here's how you open up the connections: - `telnet picasaweb.google.com 80` (Picasa Web Albums data API) - `openssl s_client -connect www.google.com:443` (Google Calendar data API and other services on www.google.com) After the connection is made, a raw HTTP request must be sent. This request includes the HTTP verb, relative path, version, all headers and the body of the request. Here's an example: ``` POST /data/feed/api/user/brad.gushue HTTP/1.1 Host: picasaweb.google.com Authorization: GoogleLogin auth=ABCDEFG Content-Length: 493 Curling in Canada Pictures of all my curling matches in Canada. Canada public true ``` When sending raw HTTP data, be aware that the `POST` and `PUT` operations will require computing the value for a `Content-Length` header. You can use the UNIX tool `wc` to compute this value. Place all the content of the HTTP body into a text file such as `template_entry.xml` (example used above) and run `wc -c template_entry.xml`. It is often difficult to debug if you accidentally use an incorrect value for the `Content-Length` header. - **wget** is typically used to download data from a web server to a local file. However, `wget` has lots of options which make it capable of performing all the different types of requests needed to interact with Google Data services.. Here's an example of how to use `wget` to `POST` a new album entry to Picasa Web Albums: ``` wget --post-file template_entry.xml --header "Authorization: GoogleLogin auth=ABCDEFG" --header "Content-Type: application/atom+xml" "http://picasaweb.google.com/data/feed/api/user/brad.gushue" ``` - **xsltproc** is a tool to apply XSL transformations ([XSLT](http://www.w3.org/TR/xslt)) to XML documents. It can be used to easily extract desired bits of data from an XML entry or feed returned by Google Data APIs, or to generate new or updated entries. ## Conclusion As you've seen, cURL and several other command-line tools can be used to easily interact with Google Data services using raw XML and HTTP. Please join us in the [API specific forums](http://code.google.com/googleapps/support/) if you have any questions about using these tools with your favorite Google Data API.