Using the Generic feed dropbox

The Generic feed dropbox allows for multiple feed types to be uploaded to a single dropbox per environment. This guide will walk through how to use the Generic dropbox and provide links to the appropriate guide for the respective feed that you are planning to use.

Warning: This guide does not apply to the Merchants, Services, or Availability dropboxes. It only applies to the Generic dropbox.

(Please refer to the Exporting Feeds (end-to-end) or Feeds (starter) section of the documentation).

Note: You must be approved for each feed type that you will be uploading. If you upload a feed type that is not approved for your integration, the feed will not be processed. If you are unsure of which feed types apply to your integration, please reach out to your Actions Center contact.

Warning: You may need to reset your ssh key to enable uploads to the Generic feed dropbox. To do so, please refer to Partner Portal feed configuration (documentation).

The Generic dropbox relies on there being two separate uploads:

1. Descriptor file: describes what feed type you will be uploading
2. Feed file(s): the content of the actual feed

Note: If you make an upload to the generic dropbox and do not see any corresponding entry in the Partner Portal that may mean that your descriptor file was not uploaded or has an incorrect file name. For help debugging this issue please reach out to your Actions Center contact.

Structuring the descriptor field

The descriptor file is uploaded to inform our system of what feed type you are uploading. This allows us to validate and process the feed correctly. The descriptor file should be uploaded before the feed contents and must follow these naming requirements:

• You must use the .filesetdesc.json file extension for the descriptor file.
• Each descriptor filename must be unique and can't be re-used across uploads. We recommend including the generation timestamp and feed name in the filename.
  • Example: offers_1524606581.filesetdesc.json
• Each descriptor file must list all data files in the latest feed for the relevant feed name.

message FilesetDescriptor {
  // The timestamp at which this feed was generated, in Unix time format
  // (seconds since the epoch). (required)
  int64 generation_timestamp = 1;

  // Identifies the name of this feed. (required)
  string name = 2;

  // Paths (relative to the dropbox root) specifying data files included in this
  // feed. (required)
  repeated string data_file = 3;
}

Possible values for the name field include:

An example JSON descriptor file for an offers feed with two shards is available below:

{
  "generation_timestamp": 1524606581,
  "name": "promote.offer",
  "data_file": [
    "offers_1524606581_1.json",
    "offers_1524606581_2.json"
  ]
}

Structuring the feed content

After uploading the descriptor file, you will then upload all of the feed files for the feed data type corresponding to the feed configuration file named by your descriptor file. The file names and path locations (relative within the dropbox) must exactly match what was included within the data_file field. If any file is missing, improperly named, or uploaded to a different location then the entire feed will not be processed.

The contents of these feed data files must conform to the relevant spec of the feed that was specified in the descriptor file.

Each feed file filename must be unique and cannot be re-used across uploads. We recommend including the generation timestamp and the shard number (incremental id) in the filename.

• Example: offers_1524606581_1.json

Feed file sizes and upload frequency

• Keep feed file size below 200 MB (after compression).
• Each decompressed data file size should be less than 2 GB.
• Most integrations will only need to use a single shard. You should use as few shards as possible. There is a maximum of 1000 shards per feed.
• Individual records sent in one shard don't need to be sent in the same shard number in future feeds.
• For better performance, split data evenly among the shards, to make all the shard files similar in size.
• If necessary, use gzip to compress feeds. However, do so for each individual feed shard.

Troubleshooting and debugging

After uploading your files (descriptor and feed files) head to the Feed History dashboard (documentation) on the Partner Portal (navigate to History > Feeds) to follow the progress of your feed ingestion.

Look for the name you have input in the descriptor file in the "Feed name" column to find your feed.

Once the feed is ingested (status is Success or Fail) you can click on its row to see the details of the errors and warnings.