# The Ultimate GitLab Import/Export Toolkit for Engineers
---
> *“I’m a DevOps engineer - if I’m doing it manually twice, I’m writing a script.”*
---
A few weeks ago, I was handed a deceptively simple‑sounding task: **spin up an entire playground environment, pipelines, history, everything for a complex micro‑services platform, from scratch.**
The catch? All source code lived in **GitLab**, and there were **dozens of projects** spread across nested groups.
I scoured the web for a clean *“export → import”* recipe that would:
* move **commits, MRs, releases, tags, wikis** - the whole lineage
* preserve **branch protections & default branches**
* run reliably for **tens of repos** without babysitting
Even GitLab’s docs stopped short of a full answer. Most blog posts boiled down to *“click this button N times”* or *“write a bash loop and pray.”*
**Manual migration? Over my dead keyboard.**
So I cracked open the GitLab API, dusted off my Python muscle memory, and the **GitLab Migration & Branch‑Housekeeping Toolkit** was born.
---
## Why Another Solution?
| **Existing Options** | **What This Toolkit Delivers** |
| --- | --- |
| UI‑only exports (one project at a time) | **Bulk, unattended export/import** of entire groups |
| 3rd‑party “all‑in‑one” SaaS (pricey, opaque) | **100 % open‑source** Python - reads every line |
| Incomplete migrations (no MRs, issues) | Captures **full project state,** Inc. metadata |
| Risk of projects landing in `root/Administrator` | **Namespace‑safe import** with post‑audit cleanup |
| Branch sprawl post‑import | **Automated branch hygiene**—keeps `demo` → spawns `demo2` |
---
## A 50,000-ft View
```mermaid
flowchart TD
subgraph Source GitLab
A>All projects
in `group-1`] --> B[export.py]
end
B -->|*.tar.gz| C((Secure
storage))
subgraph Destination GitLab
D[import.py / selected...py] --> E((Group:
group-1))
E -->|Fix stray imports| F[cleanup.py]
E -->|Delete stale branches| G[remove_obsolete_branches.py]
end
```
* [**export.py**](https://github.com/SubhanshuMG/gitlab-import-export/blob/main/scripts/export.py) triggers asynchronous exports and downloads resulting archives
* [**import.py**](https://github.com/SubhanshuMG/gitlab-import-export/blob/main/scripts/import.py) **/** [**selected\_import.py**](https://github.com/SubhanshuMG/gitlab-import-export/blob/main/scripts/selected_import.py) stream archives into the correct destination group
* [**cleanup.py**](https://github.com/SubhanshuMG/gitlab-import-export/blob/main/scripts/cleanup.py) deletes projects accidentally imported under default user namespaces
* [**Branch scripts**](https://github.com/SubhanshuMG/gitlab-import-export/blob/main/scripts/remove_obsolete_branches.py) enforce a *single source of truth* `demo` branch and spawn `demo2`
---
## Under the Hood: Key Design Decisions
1. **Official** `python‑gitlab` SDK: *No scraping, no unofficial endpoints.*
2. **Idempotency First:** Every script can be re-run safely, as existing projects or branches are detected and skipped unless you opt in to overwrite.
3. **Streaming Downloads & Uploads:** Exports are streamed in 1 MiB chunks; imports upload file-like objects, keeping memory steady.
4. **Namespace Double‑Lock:** Imports supply **both** `namespace` (string) *and* `namespace_id` (int) to the API, all but eliminating the dreaded *“imported to root”* scenario.
5. **Branch Hygiene as First-Class Citizen:** Post-import, the toolkit unprotects stale branches, deletes them, re-protects the guardians, and sets defaults *because CI breaks are not an option.*
---
## Running the Toolkit in 6 Commands
```bash
git clone https://github.com/SubhanshuMG/gitlab-import-export.git
cd gitlab-import-export && python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# 1 · bulk export
SRC_GITLAB=... SRC_TOKEN=... python scripts/export.py
# 2 · bulk import
DST_GITLAB=... DST_TOKEN=... python scripts/import.py # or selected_import.py
# 3 · fix stray namespaces & branches
python scripts/cleanup.py
python scripts/remove_obsolete_branches.py
```
```bash
python scripts/selected_import.py # edit SELECTED_PROJECTS = ["payment-service"]
python scripts/specific_project_remove_branches.py # TARGET_PROJECT_NAME="payment-service"
```
---
## Real‑World Payoff
* **4× Speed‑up** vs manual UI export/import (12 repos → 5 min)
* **Zero branch‑related pipeline failures** post‑migration
* **Repeatable** same scripts now run nightly to refresh our playground
---
## Troubleshooting Cheat‑Sheet
| Symptom | Diagnosis & Fix |
| --- | --- |
| `GitlabGetError: 404` on group | Check PAT scope (`read_api`) and group path correctness |
| Import stuck at `scheduled` | Sidekiq busy; verify destination runners aren’t paused |
| `Popen tmp/no space left` | Exports bigger than `/tmp`; set `TMPDIR` to a larger mount |
| Protected branch won’t delete | You’re not a **Maintainer**; branch script auto‑unprotects but needs rights |
---
## Extending the Toolkit
* **Parallelize exports** → Wrap the call loop in `concurrent.futures.ThreadPoolExecutor`
* **CI Variables & Releases** → After import, iterate `/projects/:id/variables` & `/releases`
* **SaaS → Self‑Managed** → Add mapping for group paths that changed between instances
* **GitLab => GitHub?** → Swap endpoints; the logic stays 90 % identical
PRs are welcome, just fork and raise an issue!
---
## The Repository
☑️ MIT‑licensed ☑️ Zero external dependencies (beyond `python‑gitlab`)
Check out the full source code and Python scripts w/ instructions:
> **GitHub →** [https://github.com/SubhanshuMG/gitlab-import-export.git](https://github.com/SubhanshuMG/gitlab-import-export.git)
---
### Final Words
If you’ve ever copy‑pasted repos one by one or worse, **lost commit history** during a migration; this toolkit is for you.
Fork it, bend it to your needs, and ship playground environments (or entire prod clones) with a single command.
Happy automating,
Signing off
[***Subhanshu Mohan Gupta***](https://www.linkedin.com/in/subhanshu-mohan-gupta-039559123/) ***(DevOps & Automation addict)***