Overview
Airbyte is an open-source data integration platform used to move data from SaaS tools, databases, APIs, and files into warehouses, lakes, and analytics platforms.
In a Shakudo environment, Airbyte usually sits at the ingestion layer. It brings data into storage or analytics systems such as object storage, Postgres, Snowflake, BigQuery, Superset, or downstream transformation pipelines.
This page is written for onboarding and deployment calls. It focuses on what customers need to understand, provide, validate, and troubleshoot in a real environment.
Where it fits in the stack
- Primary role: Airbyte provides a reusable platform capability rather than a one-off application.
- Typical deployment model: Kubernetes + Helm, with customer-specific values and secrets.
- Typical access model: private internal endpoint or customer-approved external route.
- Typical support model: validate deployment health first, then validate user workflow and integrations.
Getting Started
Start with one safe workflow in Airbyte before enabling production usage. The goal is to prove connectivity, permissions, and operational ownership.
What the customer needs to provide
- source system credentials such as database user, API token, or SaaS OAuth access
- destination credentials such as warehouse, object storage, or database access
- sync schedule and expected data volume
- network allow-listing between Airbyte workers and source/destination systems
First workflow
- Open the Airbyte UI
- Create a source connector such as Postgres, Salesforce, S3, or an API source
- Create a destination such as Postgres, S3/MinIO, Snowflake, or BigQuery
- Create a connection, choose tables/streams, and run the first sync
- Check sync history and row counts before enabling a schedule
Administration and Best Practices
Airbyte is most effective when connectors are organized, monitored, and configured with clear ownership. Since Shakudo manages the underlying infrastructure, your focus should be on building reliable data pipelines and ensuring the data arriving in your destinations is accurate and up to date.
Organize Connectors Clearly
Use descriptive names for your connectors, such as salesforce-prod or postgres-analytics, so team members can quickly understand their purpose. Add notes or descriptions to document what data is being synced and where it is being used.
Use Secure Credentials
Store API keys, database credentials, and tokens using Shakudo’s secret management capabilities. Use dedicated service accounts instead of personal credentials and update them periodically according to your organization’s security policies.
Choose the Right Sync Frequency
Set schedules based on how often your source data changes.
- Real-time operational data: every 15–30 minutes
- Databases and application data: hourly
- Reporting data: daily
Avoid running syncs more frequently than necessary, as this can increase API usage and processing time.
Prefer Incremental Syncs
Whenever supported, use incremental syncs rather than full refreshes. This reduces runtime, minimizes load on source systems, and improves reliability for large datasets.
Monitor Pipeline Health
Review sync history regularly to check for:
- Failed jobs
- Authentication errors
- Unexpected drops in record counts
- Longer-than-usual runtimes
Address recurring failures quickly to ensure downstream dashboards and AI applications continue to receive fresh data.
Review Schema Changes
When new fields or tables appear in your source systems, review them before enabling synchronization. This helps prevent unexpected changes in downstream reports and analytics pipelines.
Document Ownership
Each production connector should have a clearly defined owner who is responsible for monitoring its health and validating the data being delivered.
Troubleshooting & FAQ
Use this section during customer debugging calls. Format: Problem → What to check → Fix.
Sync fails immediately
- What to check: Check source credentials, network access, and whether the connector can reach the source host
- Fix: Update the credential, allow-list the worker egress IP, then rerun the connection check
Sync is slow or times out
- What to check: Check table size, connector logs, worker CPU/memory, and source-side rate limits
- Fix: Reduce selected streams, increase worker resources, or move to incremental sync
Destination table has missing rows
- What to check: Check sync mode, primary key, cursor field, and the latest job logs
- Fix: Switch to incremental/deduped mode only when primary keys and cursors are correct
Connector upgrade breaks a connection
- What to check: Check connector version, release notes, and saved connection config
- Fix: Rollback connector version if available or re-save the source/destination config
