SyncSage Data Ingestion Guide

Your Simplified Path to Seamless Data Integration

1.1 Sign-Up Options: SyncSage-Managed Account

Overview:

SyncSage provisions a pre-configured Fivetran account for users.

Action:

Just Sign In / Sign Up

2.1 Supported Methods

Method	Use Case
OAuth 2.0	SaaS applications (e.g., Salesforce, Google Analytics).
API Key/Secret	Databases, custom APIs, or legacy systems.
SSH Key Pair	Secure access to SFTP/S3 buckets.

3.1 Snowflake Requirements (If You Have One)

Provide the following in our Data Ingestion -> Destinations tab:

Parameter	Example Value
Host	xy12345.snowflakecomputing.com
Port	443
User	FIVETRAN_USER
Password	************
Database	RAW_DATA
Warehouse	FIVETRAN_WH
Role	FIVETRAN_ROLE

Note: If you don't have one, we will help you provide/create one.

SyncSage uses Fivetran under the hood to automate data ingestion from 680+ sources (like Salesforce, Google Analytics, and PostgreSQL) into your Snowflake data warehouse. Here's how to set it up in 3 easy steps:

5.1 Step 1: Connect Your Data Sources

Figure 1: Adding a data source in SyncSage

Connector selection interface in SyncSage

Figure 1.1: Selecting a connector in SyncSage

5.1.1 Add a Source

Go to Data Sources -> Add Source.
Choose from 680+ pre-built connectors (e.g., Salesforce, PostgreSQL).
Authenticate:
- For SaaS apps (e.g., Salesforce): Click Authorize with Google/Microsoft.
- For databases: Enter credentials (username, password, host).

5.1.2 Monitor Active Connectors

Real-Time Status:

Salesforce: Active | Last Sync: 2 hours ago
PostgreSQL: Active | Last Sync: 30 mins ago

Action:

Click any connector to edit settings or pause/resume syncs.

5.2 Step 2: Set Up Data Pipelines

Figure 2: Data Pipelines dashboard in SyncSage

5.2.1 Configure a Pipeline

Go to Pipelines -> Configure Pipeline.
Choose:
- Sync Type: Real-time, incremental, or batch.
- Destination: Snowflake (pre-configured).
Mapping: Map source fields to Snowflake tables.

5.2.2 Track Pipeline Progress

Active Pipelines (Example):

Salesforce -> Snowflake: 78% complete (1.2M/1.5M records)
PostgreSQL -> Snowflake: 45% complete (500K/1.1M records)

Metrics:

Successful Runs: 1,234 (Last 30 days)
Average Duration: 45 minutes

5.3 Step 3: Monitor & Optimize

Figure 3: SyncSage's monitoring dashboard

5.3.1 Real-Time Monitoring Dashboard

Health Overview:

Successful Syncs: 245/250 (98% success rate)
Failed Syncs: 5 (2% failure rate)

Resource Usage:

CPU: 65%
Memory: 48%
Network: 72%

5.3.2 Fix Issues Instantly

Alerts:

MongoDB Connection Failed: Check credentials
High CPU Usage: Optimize queries or scale resources

Action:

Click any alert for guided troubleshooting.

6.1 Simplified Schema Management

Auto-Mapping:
SyncSage detects schemas and maps fields automatically.
Custom Overrides:
Rename tables or exclude sensitive fields (e.g., credit_card_number).

6.2 Supported Sources: Connector Categories

Category	Example Sources	Sync Type
SaaS Applications	Salesforce, HubSpot, Zendesk	Full + Incremental
Databases	PostgreSQL, MySQL, MongoDB	CDC (Change Data Capture)
Cloud Storage	S3, Google Cloud Storage	File-based
APIs	REST, GraphQL	Polling
Legacy Systems	FTP, SFTP	Batch

7.1 Security

🔒
Rotate Credentials:
Change passwords/API keys every 90 days.
🔒
Whitelist IPs:
Allow SyncSage's IP ranges in your firewall.

7.2 Cost Control

Incremental Syncs:
Set for large datasets (e.g., sync only new Salesforce leads).
Example Configuration:
```
{
  "sync_mode": "incremental",
  "cursor_field": "last_modified_date",
  "cursor_granularity": "seconds"
}
```

Auto-Suspend:

Use in Snowflake to save costs during idle periods.

Snowflake Command:

ALTER WAREHOUSE FIVETRAN_WH 
SET AUTO_SUSPEND = 300; -- Suspend after 5 minutes of inactivity

7.3 Optimize Performance

Off-Peak Syncs:
Schedule syncs during off-peak hours (e.g., 2 AM).

Query History:

Use the Query History tab in Snowflake to identify slow queries.

Snowflake Query:

SELECT query_id, query_text, execution_time, error_code
FROM table(information_schema.query_history())
WHERE execution_time > 60000 -- Queries taking more than 60 seconds
ORDER BY start_time DESC
LIMIT 20;

8.1 Common Issues & Fixes

Issue	How to Fix in SyncSage
Sync Failed	Check the Alerts tab for error details.
Missing Data	Verify field mappings in the pipeline.
Slow Syncs	Reduce sync frequency or contact support.

SyncSage Data Ingestion Guide

1.1 Sign-Up Options: SyncSage-Managed Account

2.1 Supported Methods

3.1 Snowflake Requirements (If You Have One)

4.1 Steps to Validate and Finalize

1. Test Connection

2. Save & Finalize

3. Snowflake Setup

5.1 Step 1: Connect Your Data Sources

5.1.1 Add a Source

5.1.2 Monitor Active Connectors

5.2 Step 2: Set Up Data Pipelines

5.2.1 Configure a Pipeline

5.2.2 Track Pipeline Progress

5.3 Step 3: Monitor & Optimize

5.3.1 Real-Time Monitoring Dashboard

5.3.2 Fix Issues Instantly

6.1 Simplified Schema Management

6.2 Supported Sources: Connector Categories

7.1 Security

7.2 Cost Control

7.3 Optimize Performance

8.1 Common Issues & Fixes