Deploy the Microsoft SharePoint Online connector

Warning: The Cloud Search reference connectors are provided "as is" as sample code for use in creating your own working connectors. This sample code requires substantial customization and testing before being used in proof-of-concept or production environments. For production use, we highly recommend obtaining help from one of our Cloud Search partners. For further help finding a suitable Cloud Search partner, contact your Google Account Manager.

You can set up Google Cloud Search to return results from your organization's SharePoint Online content in addition to your Google Workspace content. You use the Google Cloud Search SharePoint Online connector and configure it to access a specific SharePoint Online data source.

Important considerations

Honored SharePoint settings

The Cloud Search SharePoint Online connector always honors the Search Visibility setting on SharePoint, which can't be overridden. For draft documents, the permissions on the user account (that the connector uses to access SharePoint Online) controls what draft documents are indexed and returned. If the account has only "Full Read" permissions, the connector honors the "Draft item visibility" settings on SharePoint.

You also configure the connector to limit results based on user account access. To apply security trimming for SharePoint content, you synchronize the following external identities with the Google Directory:

• Office 365 / Azure AD Users
• Office 365 / Azure AD security Groups
• SharePoint groups (with Office 365 users and groups as members)

Note: You can make all the results from this connector public (available to anyone in your domain). To do this, specify the following parameters in the SharePoint Online connector configuration file:

defaultAcl.mode=override
defaultAcl.public=true

When you set up public results, you don't have to set up identity connectors.

Search optimization

You can improve your users' experience by configuring the connector to return more relevant search results.

To use the API, set values for HTML generation parameters in the SharePoint Online connector configuration file. These parameters let you set which fields have higher or lower impact on matches.

To set up a schema, follow the instructions in Create and register a schema. When you set up a schema:

• To map the names of SharePoint content types to corresponding object definitions, the connector normalizes the content type names by excluding unsupported characters. For object definitions, the Cloud Search API supports only A-Z, a-z, and 0-9 as valid characters. For example, the content type "Announcements" maps to the object definition "Announcements". The content type "News Article" maps to "NewsArticle" (no space).

• When the connector can't match an object definition with an object definition, the connector uses the fallback object type (itemMetadata.objectType). Learn more about metadata configuration parameters.

• The connector maps SharePoint property names to property definitions.

• When the connector encounters Microsoft Outlook .msg files as it indexes content, it overrides the content type for the files and indexes them as application/vnd.ms-outlook.

Known connector limitations

• Each connector instance can index content from only a single site collection. To index multiple site collections, set up additional connector instances.
• Memory consumption increases with the number of unique users and groups that you use in ACLs for each site collection.
• Delete notifications are not instantaneous and it can take more than 4 hours for a connector to recognize that a user deleted content from the source repository.

System requirements

System requirements
Operating system	Windows Server 2016 Ubuntu Red Hat Enterprise Linux 5.0 SUSE Enterprise Linux 10 (64 bit)
Authentication	Live Authentication
Software	SharePoint Online environment Java JRE 1.8 installed on the computer that will run the Google Cloud Search SharePoint Online connector

Deploy the connector

Prerequisites

1. Create a Google Workspace private key, which contains your service account ID. To learn how to get a private key, go to Configure access to the Google Cloud Search API.

2. Your Google Workspace administrator must add a data source to search. Record the data source ID.

3. If the connector returns results based on ACLs (results aren't public), your Google Workspace administrator must create two identity sources and give you their IDs:

   • one for Office 365 users and groups
   • one for SharePoint groups

   The admin must also get your organization's Google Workspace customer ID and give it to you.

   Learn how to get these values in Map user identities in Cloud Search.

4. Set up a user account for the connector that has SharePoint Online Site Collection Administrator privileges.

5. If the connector returns results based on ACLs (results aren't public), create credentials to read users and groups from your Office 365 account. Follow Microsoft's instructions to create an Azure Active Directory application. As you set up the app:

   1. Name the app with a descriptive name, such as "gcs-o365-identity-connector".
   2. Add permissions. The app needs "Group.Read.All" and "User.Read.All".
   3. Record the following information to use in the connector configuration:
      • Application ID
      • Tenant
      • Client Secret

Step 1. Install the SharePoint Online connector software

1. Clone the connector repository from GitHub.

   $ git clone https://github.com/google-cloudsearch/sharepoint-connector.git
   $ cd sharepoint-connector

2. Check out the desired version of the connector:

   $ git checkout tags/v1-0.0.3

3. Build the connector.

   $ mvn package

   To skip tests when you build the connector, run mvn package -DskipTests instead of mvn package.

4. Copy the connector zip file to your local installation directory:

   $ cp target/google-cloudsearch-sharepoint-connector-v1-0.0.3.zip installation-dir
   $ cd installation-dir
   $ unzip google-cloudsearch-sharepoint-connector-v1-0.0.3.zip
   $ cd google-cloudsearch-sharepoint-connector-v1-0.0.3

Step 2. Create the SharePoint Online connector configuration file

1. In the same directory as the connector installation, create a new file and name it sharepoint-online-connector.config. If you plan to run many connector instances, add more details to the name to distinguish it.

2. Add parameters as key/value pairs to the file contents, as in the following example:

   ### SharePoint Online connector configuration ###
   
   # Required parameters for data source access
   api.sourceId=08ef8becd116faa4546b8ca2c84b2879
   api.serviceAccountPrivateKeyFile=service_account.json
   
   # Required parameters for SharePoint Online access
   sharepoint.server=https://mydomain.onmicrosoft.com
   sharepoint.siteCollectionOnly=true
   sharepoint.username=[e-mail]
   sharepoint.password=pa$sw0rd
   sharepoint.formsAuthenticationMode=LIVE
   sharepoint.deploymentType=ONLINE
   
   # SharePoint identity mapping parameters
   api.identitySourceId=08ef8becd116faa475de26d9b291fed9
   api.referenceIdentitySources=defaultIdentitySource
   api.referenceIdentitySource.defaultIdentitySource.id=08ef8becd116faa5d3783f8c5a80e5aa
   
   # Optional SharePoint List Item metadata parameters
   contentTemplate.sharepointItem.title=Title
   contentTemplate.sharepointItem.unmappedColumnsMode=APPEND
   

   For detailed descriptions of each parameter, go to the configuration parameters reference.

3. (Optional) Configure additional connector parameters, as needed. For details, go to Google-supplied connector parameters.

Step 3. Set up logging for the SharePoint Online connector

1. Create a folder named logs in the same directory that contains the connector binary.

2. Create a Latin-1-encoded file named logging.properties in the same directory and add the following content:

   handlers = java.util.logging.ConsoleHandler,java.util.logging.FileHandler
   # Default log level
   .level = INFO
   # uncomment line below to increase logging level for SharePoint APIsa
   #com.google.enterprise.cloudsearch.sharepoint.level=FINE
   
   # uncomment line below to increase logging level to enable API trace
   #com.google.api.client.http.level = FINE
   java.util.logging.ConsoleHandler.level = INFO
   java.util.logging.FileHandler.pattern=logs/connector-sharepoint.%g.log
   java.util.logging.FileHandler.limit=10485760
   java.util.logging.FileHandler.count=10
   java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
   

Step 4. Configure the SharePoint Online identity connector

This step is required to apply SharePoint Online identity-based ACLs to search results. If you set up the connector with public ACLs, you can skip this step.

1. In the same directory as the SharePoint Online connector installation, create a new file and name it sharepoint-online-identity-connector.config.

2. Add parameters as key/value pairs to the file contents, as in the following example:

   ### SharePoint Online identity connector configuration ###
   # Required parameters for data source access
   api.customerId=C05d3djk8
   api.serviceAccountPrivateKeyFile=service_account.json
   
   # Required parameters for SharePoint Online access
   sharepoint.server=https://mydomain.onmicrosoft.com
   sharepoint.siteCollectionOnly=true
   sharepoint.username=[e-mail]
   sharepoint.password=pa$sw0rd
   sharepoint.formsAuthenticationMode=LIVE
   sharepoint.deploymentType=ONLINE
   
   # SharePoint identity mapping parameters
   api.identitySourceId=08ef8becd116faa475de26d9b291fed9
   api.referenceIdentitySources=defaultIdentitySource
   api.referenceIdentitySource.defaultIdentitySource.id=08ef8becd116faa5d3783f8c5a80e5aa
   

   The values are almost the same as for the SharePoint Online connector, except that instead of api.sourceId, the parameter is api.customerId. The value of api.customerId is the customer ID that you got from your Google Workspace admin.

Step 5. Install the Office 365 Identity connector

This step is required to apply SharePoint Online identity-based ACLs to search results. If you set up the connector with public ACLs, you can skip this step.

1. Clone the connector repository from GitHub.

   $ git clone https://github.com/google-cloudsearch/office-365-connector.git
   $ cd office-365-connector

2. Check out the desired version of the connector:

   $ git checkout tags/v1-0.0.3

3. Build the connector.

   $ mvn package

   To skip tests when you build the connector, run mvn package -DskipTests instead of mvn package.

4. Copy the connector zip file to your local installation directory:

   $ cp target/google-cloudsearch-office-365-connector-v1-0.0.3.zip installation-dir
   $ cd installation-dir
   $ unzip google-cloudsearch-office-365-connector-v1-0.0.3.zip
   $ cd google-cloudsearch-office-365-connector-v1-0.0.3

Step 6. Configure the Office 365 Identity connector and set up logging

1. In the same directory as the identity connector installation, create a new file and name it o365-identity-connector.config.

2. Add parameters as key/value pairs to the file contents, as in the following example. Note: All parameters are required.

   ### Office 365 Identity connector configuration ###
   
   api.identitySourceId=12345abcde
   api.customerId=abcde12345
   o365.clientId=a63c6eb3-29e7-486
   o365.tenant=185ef9ed-29e7-486
   o365.clientSecret=raHJN15vRLBKs
   

   For detailed descriptions of each parameter, go to the Office 365 Identity connector configuration reference.

3. Create a folder named logs in the same directory that contains the connector binary.

4. Create an ASCII or UTF-8 file named logging.properties in the same directory and add the following content:

   handlers = java.util.logging.ConsoleHandler,java.util.logging.FileHandler
   # Default log level
   .level = INFO
   # uncomment line below to increase logging level for Office 365 APIs
   #com.google.enterprise.cloudsearch.o365.level=FINE
   
   # uncomment line below to increase logging level to enable Google API traces
   #com.google.api.client.http.level = FINE
   java.util.logging.ConsoleHandler.level = INFO
   java.util.logging.FileHandler.pattern=logs/connector-o365.%g.log
   java.util.logging.FileHandler.limit=10485760
   java.util.logging.FileHandler.count=10
   java.util.logging.FileHandler.formatter=java.util.logging.SimpleFormatter
   

Step 7. Launch the SharePoint Online connector

In the following steps, you map the principals in both Office 365 and the SharePoint site collection to identities in the Cloud Identity service. This synchronization is done with the Office 365 Identity connector and the SharePoint Online identity connector.

After the Office 365 connector synchronizes the users and groups, run the SharePoint Online identity connector, to synchronize the SharePoint site collection groups. Then you run the SharePoint Online connector to index and serve results to your Cloud Search users.

1. Run the Office 365 identity connector:

   $ java -Djava.util.logging.config.file=logging.properties -jar google-cloudsearch-office-365-connector-v1-0.0.3.jar -Dconfig=o365-identity-connector.config

2. Run the SharePoint Online identity connector:

   $ java -Djava.util.logging.config.file=logging.properties -cp "google-cloudsearch-sharepoint-connector-v1-0.0.3.jar" com.google.enterprise.cloudsearch.sharepoint.SharePointIdentityConnector -Dconfig=sharepoint-online-identity-connector.config

3. Run the SharePoint Online connector:

   $ java -Djava.util.logging.config.file=logging.properties -jar google-cloudsearch-sharepoint-connector-v1-0.0.3.jar -Dconfig=sharepoint-online-connector.config

Configuration parameters reference

Data source access

Setting	Parameter
Data source ID	api.sourceId=1234567890abcdef Required. The Google Cloud Search data source ID set up by the Google Workspace administrator.
Path to the service account private key file	api.serviceAccountPrivateKeyFile=PrivateKey.json Required. The path to the Google Cloud Search service account key file.

SharePoint Online access

Setting	Parameter
Fully-qualified domain name for the SharePoint Site Collection	sharepoint.server=http://yoursharepoint.example.com/ Required. If the domain name is not fully-qualified, then set DNS override on the connector host.
Site Collection Only Mode	sharepoint.siteCollectionOnly=true Required. For SharePoint Online, set the value true.
SharePoint username	sharepoint.username=username Required. The username for the account used to access SharePoint Online.
SharePoint password	sharepoint.password=user_password Required. The password for the account used to access SharePoint Online.
Authentication Mode	sharepoint.formsAuthenticationMode=LIVE Required. For SharePoint Online, the value is always `LIVE`.
Deployment Type	sharepoint.deploymentType=ONLINE Required. For SharePoint Online, the value is always `ONLINE`.

SharePoint Identity Mapping

Setting	Parameter
Identity Source ID	api.identitySourceId=1234567890abcdef Required. The Cloud Search identity source ID for the SharePoint groups identity source.
Reference Identity Sources	api.referenceIdentitySources=defaultIdentitySource Required. For sharePoint Online, the value is always `defaultIdentitySource`.
Reference Identity Source IDs	api.referenceIdentitySource.defaultIdentitySource.id=112233abcd Required. The Cloud Search identity source ID for Office 365 Users and Groups identity source.

HTML content generation

Setting	Parameter
HTML template title field	contentTemplate.sharePointItem.title=Title The SharePoint field to use as the HTML template title for generated HTML.
HTML content high search quality fields	contentTemplate.sharePointItem.quality.high=highField1[,highField2,...] A comma-separated list of fields to include in the generated HTML as high-quality fields. When the search query terms match these fields, the results are ranked higher.
HTML content medium search quality fields	contentTemplate.sharePointItem.quality.medium=mediumField1[,mediumField2,...] A comma-separated list of fields to include in the generated HTML as medium-quality fields.
HTML content low search quality fields	contentTemplate.sharePointItem.quality.low=lowField1[,lowField2,...] A comma-separated list of fields to include in the generated HTML as low-quality fields.
HTML content unmapped columns	contentTemplate.sharepointItem.unmappedColumnsMode=APPEND How the connector handles unmapped columns. Value is APPEND (default) or IGNORE. APPEND—The connector generates HTML content with all fields, including those that aren't set with a quality level (high, medium, or low). IGNORE—The connector generates HTML content with only mapped fields.

Office 365 Identity connector configuration

Setting	Parameter
Identity Source ID	api.identitySourceId=1234567890abcdef Required. The Office 365 identity source ID. This value should match the value of `api.referenceIdentitySource.defaultIdentitySource.id` in the SharePoint Online connector configuration.
Google Customer Id	api.customerId=c1b1d1e1 Required. Your organization's Google Workspace customer ID.
Office 365 Application Id	o365.clientId=a63c6eb3-29e7-486 Required. The application id from the Office 365 application setup.
Office 365 Tenant	o365.tenant=185ef9ed-29e7-486 Required. The tenant for your Office 365 account (your Azure Activity Directory app)
Office 365 client secret	o365.clientSecret=raHJN15vRLBKs Required. Credential secret from Office 365 Application setup