Availability

Airbyte Cloud Airbyte OSS

Support Level

Certified

Latest Version

2.0.0

GitLab

This page contains the setup guide and reference information for the Gitlab Source connector.

Prerequisites

• Gitlab instance or an account at Gitlab

For Airbyte Cloud:

• Personal Access Token (see personal access token)
• OAuth

For Airbyte Open Source:

• Personal Access Token (see personal access token)

Setup guide

Step 1: Set up GitLab

Create a GitLab Account or set up a local instance of GitLab.

Airbyte Open Source additional setup steps

Log into GitLab and then generate a personal access token. Your token should have the read_api scope, that Grants read access to the API, including all groups and projects, the container registry, and the package registry.

Step 2: Set up the GitLab connector in Airbyte

For Airbyte Cloud:

1. Log into your Airbyte Cloud account.
2. In the left navigation bar, click Sources. In the top-right corner, click + new source.
3. On the source setup page, select GitLab from the Source type dropdown and enter a name for this connector.
4. Click Authenticate your GitLab account by selecting Oauth or Personal Access Token for Authentication.
5. Log in and Authorize to the GitLab account.
6. API URL (Optional) - The URL to access your self-hosted GitLab instance or gitlab.com (default).
7. Start date (Optional) - The date from which you'd like to replicate data for streams.
8. Groups (Optional) - List of GitLab group IDs, e.g. airbytehq for single group, airbytehq another-repo for multiple groups.
9. Projects (Optional) - List of GitLab projects to pull data for, e.g. airbytehq/airbyte.
10. Click Set up source.

Note: You can specify either Group IDs or Project IDs in the source configuration. If both fields are blank, the connector will retrieve a list of all the groups that are accessible to the configured token and ingest as normal.

For Airbyte Open Source:

1. Authenticate with Personal Access Token.

Supported sync modes

The Gitlab Source connector supports the following sync modes:

• Full Refresh - Overwrite
• Full Refresh - Append
• Incremental - Append
• Incremental - Append + Deduped

Supported Streams

This connector outputs the following streams:

• Branches
• Commits (Incremental)
• Issues (Incremental)
• Group Issue Boards
• Pipelines (Incremental)
• Jobs
• Projects
• Project Milestones
• Project Merge Requests (Incremental)
• Users
• Groups
• Group Milestones
• Group and Project members
• Tags
• Releases
• Deployments
• Group Labels
• Project Labels
• Epics (only available for GitLab Ultimate and GitLab.com Gold accounts. Stream Epics uses iid field as primary key for more convenient search and matching with UI. Iid is the internal ID of the epic, number of Epic on UI.)
• Epic Issues (only available for GitLab Ultimate and GitLab.com Gold accounts)

Additional information

GitLab source works with GitLab API v4. It can also work with self-hosted GitLab API v4.

Performance considerations

Gitlab has the rate limits, but the Gitlab connector should not run into Gitlab API limitations under normal usage. Please create an issue if you see any rate limit issues that are not automatically retried successfully.

Changelog

Version	Date	Pull Request	Subject
2.0.0	2023-10-23	31700	Add correct date-time format for Deployments, Projects and Groups Members streams
1.8.4	2023-10-19	31599	Base image migration: remove Dockerfile and use the python-connector-base image
1.8.3	2023-10-18	31547	Add validation for invalid groups_list and/or projects_list
1.8.2	2023-10-17	31492	Expand list of possible error status codes when handling expired access_token
1.8.1	2023-10-12	31375	Mark start_date as optional, migrate groups and projects to array
1.8.0	2023-10-12	31339	Add undeclared fields to streams schemas, validate date/date-time format in stream schemas
1.7.1	2023-10-10	31210	Added expired access_token handling, while checking the connection
1.7.0	2023-08-08	27869	Add Deployments stream
1.6.0	2023-06-30	27869	Add shared_runners_setting field to groups
1.5.1	2023-06-24	27679	Fix formatting
1.5.0	2023-06-15	27392	Make API URL an optional parameter in spec.
1.4.2	2023-06-15	27346	Partially revert changes made in version 1.0.4, disallow http calls in cloud.
1.4.1	2023-06-13	27351	Fix OAuth token expiry date.
1.4.0	2023-06-12	27234	Skip stream slices on 403/404 errors, do not fail syncs.
1.3.1	2023-06-08	27147	Improve connectivity check for connections with no projects/groups
1.3.0	2023-06-08	27150	Update stream schemas
1.2.1	2023-06-02	26947	New field name added to Pipelines and PipelinesExtended stream schema
1.2.0	2023-05-17	22293	Preserve data in records with flattened keys
1.1.1	2023-05-23	26422	Fix error 404 Repository Not Found when syncing project with Repository feature disabled
1.1.0	2023-05-10	25948	Introduce two new fields in the Projects stream schema
1.0.4	2023-04-20	21373	Accept api_url with or without scheme
1.0.3	2023-02-14	22992	Specified date formatting in specification
1.0.2	2023-01-27	22001	Set AvailabilityStrategy for streams explicitly to None
1.0.1	2023-01-23	21713	Fix missing data issue
1.0.0	2022-12-05	7506	Add OAuth2.0 authentication option
0.1.12	2022-12-15	20542	Revert HttpAvailability changes, run on cdk 0.15.0
0.1.11	2022-12-14	20479	Use HttpAvailabilityStrategy + add unit tests
0.1.10	2022-12-12	20384	Fetch groups along with their subgroups
0.1.9	2022-12-11	20348	Fix 403 error when syncing EpicIssues stream
0.1.8	2022-12-02	20023	Fix duplicated records issue for Projects stream
0.1.7	2022-12-01	19986	Fix GroupMilestones stream schema
0.1.6	2022-06-23	13252	Add GroupIssueBoards stream
0.1.5	2022-05-02	11907	Fix null projects param and container_expiration_policy
0.1.4	2022-03-23	11140	Ingest All Accessible Groups if not Specified in Config
0.1.3	2021-12-21	8991	Update connector fields title/description
0.1.2	2021-10-18	7108	Allow all domains to be used as api_url
0.1.1	2021-10-12	6932	Fix pattern field in spec file, remove unused fields from config files, use cache from CDK
0.1.0	2021-07-06	4174	Initial Release