Using the Generic feed dropbox

The Generic feed dropbox allows for multiple feed types to be uploaded to a single dropbox per environment. This guide will walk through how to use the Generic dropbox and provide links to the appropriate guide for the respective feed that you are planning to use. This guide does not apply to the Merchants, Services, or Availability feed (please refer to the Exporting Feeds section of the documentation).

*You may need to reset your ssh key to enable uploads to the Generic feed dropbox. To do so, please refer to Partner Portal feed configuration..

The Generic dropbox relies on there being two separate uploads:

  • Descriptor file: describes what feed type you will be uploading
  • Feed file(s): the content of the actual feed

Structuring the descriptor field

The descriptor file is uploaded to inform our system of what feed type you are uploading. This allows us to validate and process the feed correctly. The descriptor file should be uploaded before the feed contents and must follow these naming requirements:

  • Use the .filesetdesc.json file extension
  • Each file name must be unique and can not be re-used across uploads. We recommend including the generation timestamp and shard number in the file name.
    • Example file name: offers-1582641402-shard0.filesetdesc.json
message FilesetDescriptor {
  // The timestamp at which this feed was generated.
  //
  // In Unix time format (seconds since the epoch). (required)
  int64 generation_timestamp = 1;

  // Identifies the name of this feed. (required)
  string name = 2;

  // Paths (relative to the dropbox root) specifying data files included in this
  // feed. (required)
  repeated string data_file = 3;
}

Possible values for the name field include:

  • promote.offer

An example JSON descriptor file for an offers feed with two shards is available below:

{
  "generation_timestamp": 1524606581,
  "name": "promote.offer",
  "data_file": [
    "offers_1524606581_1.json",
    "offers_1524606581_2.json"
  ]
}

Structuring the feed content

After uploading the descriptor file, you will then upload all of the files for the feed data type corresponding to the feed configuration file named by your descriptor file. The file names and path locations (relative within the dropbox) must exactly match what was included within the data_file field. If any file is missing, improperly named, or uploaded to a different location then the entire feed will not be processed.

The contents of these feed data files must conform to the relevant spec of the feed that was specified in the descriptor file.

Feed file sizes and upload frequency

  • Keep feed file size below 200 MB (after compression). Use multiple shards if needed according to the file sharding tutorial.
  • Most integrations will only need to use a single shard. You should use as few shards as possible. There is a maximum of 1000 shards per feed.
  • Individual records sent in one shard don't need to be sent in the same shard number in future feeds.
  • For better performance, split data evenly among the shards, to make all the shard files similar in size.
  • If necessary, use gzip to compress feeds. However, do so for each individual feed shard.