From af8ab19f5def6f00081b0a6d1e5b20b76683f720 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Tue, 24 Jun 2025 10:00:28 +0300 Subject: refactor: use value semantics for GitHub and Codeberg clients MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Changed github.NewClient() to return Client instead of *Client - Changed codeberg.NewClient() to return Client instead of *Client - Updated sync_handlers.go to handle value semantics properly - Both clients only contain immutable string fields, making value semantics more appropriate docs: add comprehensive documentation - Added doc/ directory with full documentation - Created architecture overview explaining system design - Added complete API reference for all packages, types, and functions - Created configuration guide with examples - Added usage examples and common workflows - Created development guide for contributors - Updated README with links to documentation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- doc/README.md | 34 ++++ doc/api-reference.md | 514 +++++++++++++++++++++++++++++++++++++++++++++++++++ doc/architecture.md | 207 +++++++++++++++++++++ doc/configuration.md | 278 ++++++++++++++++++++++++++++ doc/development.md | 447 ++++++++++++++++++++++++++++++++++++++++++++ doc/examples.md | 373 +++++++++++++++++++++++++++++++++++++ 6 files changed, 1853 insertions(+) create mode 100644 doc/README.md create mode 100644 doc/api-reference.md create mode 100644 doc/architecture.md create mode 100644 doc/configuration.md create mode 100644 doc/development.md create mode 100644 doc/examples.md (limited to 'doc') diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 0000000..02ea0f0 --- /dev/null +++ b/doc/README.md @@ -0,0 +1,34 @@ +# GitSyncer Documentation + +Welcome to the GitSyncer documentation. This directory contains comprehensive documentation about the GitSyncer project structure, architecture, and API reference. + +## Table of Contents + +1. [Architecture Overview](architecture.md) - High-level system design and architecture +2. [API Reference](api-reference.md) - Complete reference of all packages, types, and functions +3. [Configuration Guide](configuration.md) - How to configure GitSyncer +4. [Usage Examples](examples.md) - Common usage patterns and examples +5. [Development Guide](development.md) - Guide for contributors + +## Quick Links + +- [Project README](../README.md) - Main project documentation +- [Source Code](https://codeberg.org/snonux/gitsyncer) - Repository on Codeberg + +## Overview + +GitSyncer is a tool for synchronizing Git repositories across multiple platforms (GitHub, Codeberg, self-hosted Git servers). It supports: + +- Bidirectional synchronization between multiple Git hosts +- Automatic branch management and filtering +- Repository creation on supported platforms +- Conflict detection and reporting +- Abandoned branch analysis + +## Getting Started + +1. Install GitSyncer +2. Create a configuration file +3. Run `gitsyncer --sync-all` to sync all configured repositories + +See the [Configuration Guide](configuration.md) for detailed setup instructions. \ No newline at end of file diff --git a/doc/api-reference.md b/doc/api-reference.md new file mode 100644 index 0000000..cc4f69c --- /dev/null +++ b/doc/api-reference.md @@ -0,0 +1,514 @@ +# GitSyncer API Reference + +This document provides a complete reference for all packages, types, and functions in GitSyncer. + +## Table of Contents + +- [Package main](#package-main) +- [Package cli](#package-cli) +- [Package codeberg](#package-codeberg) +- [Package config](#package-config) +- [Package github](#package-github) +- [Package sync](#package-sync) +- [Package version](#package-version) + +--- + +## Package main + +**Location**: `cmd/gitsyncer/main.go` + +The main package provides the application entry point. + +### Functions + +#### func main() +Application entry point that: +- Parses command-line flags +- Routes to appropriate handlers +- Manages exit codes + +--- + +## Package cli + +**Location**: `internal/cli/` + +The cli package handles all command-line interface operations. + +### Types + +#### type Flags +```go +type Flags struct { + VersionFlag bool // Show version information + ConfigPath string // Path to configuration file + ListOrgs bool // List configured organizations + ListRepos bool // List configured repositories + SyncRepo string // Single repository to sync + SyncAll bool // Sync all configured repositories + SyncCodebergPublic bool // Sync all public Codeberg repos + SyncGitHubPublic bool // Sync all public GitHub repos + FullSync bool // Full bidirectional sync + CreateGitHubRepos bool // Auto-create GitHub repositories + CreateCodebergRepos bool // Auto-create Codeberg repositories + DryRun bool // Preview mode without changes + WorkDir string // Working directory for operations + TestGitHubToken bool // Test GitHub authentication +} +``` + +### Functions + +#### func ParseFlags() *Flags +Parses command-line arguments and returns a Flags struct with all options. + +#### func HandleVersion() int +Displays version information and returns exit code 0. + +#### func HandleTestGitHubToken() int +Tests GitHub token authentication: +- Loads token from config/env/file +- Validates token with API call +- Returns 0 on success, 1 on failure + +#### func LoadConfig(configPath string) (*config.Config, error) +Loads configuration from specified path or default locations: +- `./gitsyncer.json` +- `~/.config/gitsyncer/config.json` +- `~/.gitsyncer.json` + +#### func ShowConfigHelp() +Displays help for creating configuration files with example. + +#### func HandleListOrgs(cfg *config.Config) int +Lists all configured organizations from config. + +#### func HandleListRepos(cfg *config.Config) int +Lists all configured repositories from config. + +#### func ShowUsage(cfg *config.Config) +Displays comprehensive usage information. + +#### func HandleSync(cfg *config.Config, flags *Flags) int +Synchronizes a single repository specified by `--sync` flag. + +#### func HandleSyncAll(cfg *config.Config, flags *Flags) int +Synchronizes all repositories listed in configuration. + +#### func HandleSyncCodebergPublic(cfg *config.Config, flags *Flags) int +Discovers and syncs all public Codeberg repositories to other platforms. + +#### func HandleSyncGitHubPublic(cfg *config.Config, flags *Flags) int +Discovers and syncs all public GitHub repositories to other platforms. + +#### func ShowFullSyncMessage() +Displays information about full sync mode. + +### Helper Functions (sync_handlers.go) + +#### func createGitHubRepoIfNeeded(cfg *config.Config, repoName string) error +Creates GitHub repository if it doesn't exist and token is available. + +#### func initGitHubClient(cfg *config.Config) *github.Client +Initializes GitHub client with token from configuration. + +#### func createRepoWithClient(client *github.Client, repoName, description string) error +Creates repository using provided GitHub client. + +#### func showReposToSync(repoNames []string) +Displays list of repositories that will be synced. + +#### func syncCodebergRepos(cfg *config.Config, flags *Flags, repos []codeberg.Repository, repoNames []string) int +Synchronizes discovered Codeberg repositories. + +#### func syncGitHubRepos(cfg *config.Config, flags *Flags, repos []github.Repository, repoNames []string) int +Synchronizes discovered GitHub repositories. + +--- + +## Package codeberg + +**Location**: `internal/codeberg/codeberg.go` + +The codeberg package provides a client for interacting with Codeberg's Gitea API. + +### Types + +#### type Repository +```go +type Repository struct { + ID int64 `json:"id"` // Repository ID + Name string `json:"name"` // Repository name + FullName string `json:"full_name"` // Full name (org/repo) + Description string `json:"description"` // Repository description + Private bool `json:"private"` // Is private repository + Fork bool `json:"fork"` // Is fork + CreatedAt time.Time `json:"created_at"` // Creation timestamp + UpdatedAt time.Time `json:"updated_at"` // Last update timestamp + CloneURL string `json:"clone_url"` // HTTPS clone URL + SSHURL string `json:"ssh_url"` // SSH clone URL + Size int `json:"size"` // Repository size + Archived bool `json:"archived"` // Is archived + Empty bool `json:"empty"` // Is empty repository +} +``` + +#### type Client +```go +type Client struct { + baseURL string // API base URL (https://codeberg.org/api/v1) + org string // Organization or username +} +``` + +### Functions + +#### func NewClient(org string) Client +Creates a new Codeberg API client for the specified organization/user. + +### Methods + +#### func (c *Client) ListPublicRepos() ([]Repository, error) +Lists all public repositories for an organization: +- Handles pagination automatically +- Filters out private, fork, archived, and empty repos +- Returns error on API failure + +#### func (c *Client) ListUserPublicRepos() ([]Repository, error) +Lists all public repositories for a user: +- Same filtering as ListPublicRepos +- Use when org endpoint fails (for user accounts) + +#### func GetRepoNames(repos []Repository) []string +Extracts repository names from a slice of Repository structs. + +--- + +## Package config + +**Location**: `internal/config/config.go` + +The config package handles configuration loading and validation. + +### Types + +#### type Organization +```go +type Organization struct { + Host string `json:"host"` // Git host (e.g., "git@github.com") + Name string `json:"name"` // Organization/username + GitHubToken string `json:"github_token"` // Optional GitHub API token +} +``` + +#### type Config +```go +type Config struct { + Organizations []Organization `json:"organizations"` // List of git organizations + Repositories []string `json:"repositories"` // Specific repos to sync + ExcludeBranches []string `json:"exclude_branches"` // Regex patterns for branch exclusion +} +``` + +### Functions + +#### func Load(path string) (*Config, error) +Loads configuration from JSON file: +- Validates JSON structure +- Calls Validate() on loaded config +- Returns error on failure + +### Methods + +#### func (c *Config) Validate() error +Validates configuration: +- Ensures at least one organization exists +- Returns error if validation fails + +#### func (o *Organization) GetGitURL() string +Returns the Git URL for the organization in format `host:name`. + +#### func (c *Config) FindOrganization(host string) *Organization +Finds organization by host string. + +#### func (o *Organization) IsCodeberg() bool +Returns true if organization host contains "codeberg.org". + +#### func (c *Config) FindCodebergOrg() *Organization +Finds first Codeberg organization in config. + +#### func (o *Organization) IsGitHub() bool +Returns true if organization host contains "github.com". + +#### func (c *Config) FindGitHubOrg() *Organization +Finds first GitHub organization in config. + +--- + +## Package github + +**Location**: `internal/github/github.go` + +The github package provides a client for GitHub API operations. + +### Types + +#### type Client +```go +type Client struct { + token string // GitHub personal access token + org string // Organization or username +} +``` + +#### type Repository +```go +type Repository struct { + Name string `json:"name"` // Repository name + Description string `json:"description"` // Repository description + Private bool `json:"private"` // Is private repository + Fork bool `json:"fork"` // Is fork + Archived bool `json:"archived"` // Is archived + Disabled bool `json:"disabled"` // Is disabled + Size int `json:"size"` // Repository size in KB +} +``` + +#### type CreateRepoRequest +```go +type CreateRepoRequest struct { + Name string `json:"name"` // Repository name + Description string `json:"description"` // Repository description + Private bool `json:"private"` // Create as private + AutoInit bool `json:"auto_init"` // Initialize with README +} +``` + +#### type CreateRepoResponse +```go +type CreateRepoResponse struct { + ID int64 `json:"id"` // Repository ID + Name string `json:"name"` // Repository name + FullName string `json:"full_name"` // Full name (owner/repo) + Private bool `json:"private"` // Is private + SSHURL string `json:"ssh_url"` // SSH clone URL + CloneURL string `json:"clone_url"` // HTTPS clone URL +} +``` + +#### type ErrorResponse +```go +type ErrorResponse struct { + Message string `json:"message"` // Error message + Errors []struct { + Resource string `json:"resource"` // Resource type + Field string `json:"field"` // Field with error + Code string `json:"code"` // Error code + } `json:"errors,omitempty"` +} +``` + +### Functions + +#### func NewClient(token, org string) Client +Creates new GitHub API client: +- If token is empty, tries GITHUB_TOKEN env var +- If still empty, tries ~/.gitsyncer_github_token file +- Returns client with loaded token + +### Methods + +#### func (c *Client) HasToken() bool +Returns true if client has a token configured. + +#### func (c *Client) RepoExists(repoName string) (bool, error) +Checks if repository exists: +- Returns (true, nil) if exists +- Returns (false, nil) if not found (404) +- Returns (false, error) for other errors + +#### func (c *Client) CreateRepo(repoName, description string, private bool) error +Creates a new repository: +- Checks if repo already exists first +- Creates with provided settings +- Returns nil if already exists or created successfully + +#### func (c *Client) ListPublicRepos() ([]Repository, error) +Lists all public repositories: +- Handles pagination automatically +- Filters out private, fork, archived, disabled, and empty repos +- Requires authentication token + +#### func GetRepoNames(repos []Repository) []string +Extracts repository names from Repository slice. + +--- + +## Package sync + +**Location**: `internal/sync/` + +The sync package contains the core synchronization logic. + +### Types + +#### type Syncer +```go +type Syncer struct { + config *config.Config // Configuration + workDir string // Working directory + repoName string // Current repository name + abandonedReports map[string]*AbandonedBranchReport // Abandoned branch reports + branchFilter *BranchFilter // Branch exclusion filter +} +``` + +#### type BranchInfo +```go +type BranchInfo struct { + Name string // Branch name + LastCommit time.Time // Last commit timestamp + Remote string // Remote name + IsAbandoned bool // Whether branch is abandoned + AbandonReason string // Reason for abandonment +} +``` + +#### type AbandonedBranchReport +```go +type AbandonedBranchReport struct { + MainBranchUpdated bool // Is main branch active + MainBranchLastCommit time.Time // Main branch last commit + AbandonedBranches []BranchInfo // List of abandoned branches + TotalBranches int // Total number of branches +} +``` + +#### type BranchFilter +```go +type BranchFilter struct { + excludePatterns []*regexp.Regexp // Compiled regex patterns +} +``` + +### Functions + +#### func New(cfg *config.Config, workDir string) *Syncer +Creates new Syncer instance with configuration and working directory. + +### Syncer Methods + +#### func (s *Syncer) SyncRepository(repoName string) error +Main synchronization method: +1. Creates work directory +2. Sets up repository (clone or configure remotes) +3. Fetches from all remotes +4. Gets and filters branches +5. Syncs each branch +6. Analyzes abandoned branches +7. Returns error on failure + +#### func (s *Syncer) GenerateAbandonedBranchSummary() string +Generates summary report of abandoned branches across all synced repositories. + +### Branch Filter Functions + +#### func NewBranchFilter(excludePatterns []string) (*BranchFilter, error) +Creates new branch filter with compiled regex patterns. + +### BranchFilter Methods + +#### func (f *BranchFilter) ShouldExclude(branchName string) bool +Returns true if branch matches any exclusion pattern. + +#### func (f *BranchFilter) FilterBranches(branches []string) []string +Returns branches that don't match exclusion patterns. + +#### func (f *BranchFilter) GetExcludedBranches(branches []string) []string +Returns branches that match exclusion patterns. + +#### func FormatExclusionReport(excludedBranches []string, patterns []string) string +Formats a report of excluded branches with patterns used. + +### Git Operation Functions (git_operations.go) + +#### func checkForMergeConflicts() (bool, string, error) +Checks if repository has merge conflicts. + +#### func stashChanges() error +Stashes uncommitted changes. + +#### func popStash() +Pops the last stash (called via defer). + +#### func getRemotesList() (map[string]bool, error) +Returns map of configured remotes. + +#### func getAllUniqueBranches(gitOutput []byte) []string +Parses git branch output and returns unique branch names. + +#### func changeToRepoDirectory(repoPath string) (func(), error) +Changes to repository directory and returns restore function. + +#### func fetchRemote(remote string) error +Fetches from a specific remote with prune. + +#### func checkoutExistingBranch(branch string) error +Checks out an existing local branch. + +#### func createTrackingBranch(branch, remoteName string) error +Creates new branch tracking a remote branch. + +#### func mergeFromRemotes(branch string, remotesWithBranch map[string]bool) error +Merges changes from all remotes that have the branch. + +#### func pushToAllRemotes(branch string, remotes map[string]*config.Organization, remotesWithBranch map[string]bool) error +Pushes branch to all configured remotes. + +### Internal Helper Functions + +#### func (s *Syncer) setupRepository(repoPath string) error +Sets up repository by cloning or adding remotes. + +#### func (s *Syncer) analyzeAbandonedBranches() (*AbandonedBranchReport, error) +Analyzes branches for abandonment (6+ months inactive). + +#### func (s *Syncer) findMainBranch(branches []string) string +Finds the main branch (main, master, or develop). + +#### func (s *Syncer) trackRemotesWithBranch(branch string, remotes map[string]*config.Organization) map[string]bool +Returns map of remotes that have the specified branch. + +--- + +## Package version + +**Location**: `internal/version/version.go` + +The version package provides version information. + +### Variables + +```go +var ( + Version = "0.1.0" // Application version + GitCommit = "unknown" // Git commit hash (set at build time) + BuildDate = "unknown" // Build date (set at build time) + GoVersion = runtime.Version() // Go version used for build +) +``` + +### Functions + +#### func GetVersion() string +Returns full version string with all metadata: +``` +gitsyncer version 0.1.0 + Git commit: abc123 + Built: 2024-01-15 + Go version: go1.21.5 +``` + +#### func GetShortVersion() string +Returns just the version number: `0.1.0` \ No newline at end of file diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 0000000..ccb5cbd --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,207 @@ +# GitSyncer Architecture + +## Overview + +GitSyncer is designed as a command-line tool that synchronizes Git repositories across multiple platforms. It follows a modular architecture with clear separation of concerns between CLI handling, API clients, configuration management, and core synchronization logic. + +## High-Level Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ CLI Layer │ +│ (cmd/gitsyncer/main.go) │ +└─────────────────────────┬───────────────────────────────────┘ + │ +┌─────────────────────────┴───────────────────────────────────┐ +│ CLI Handlers │ +│ (internal/cli/handlers.go) │ +│ (internal/cli/sync_handlers.go) │ +└──────┬──────────────────┬──────────────────┬────────────────┘ + │ │ │ +┌──────┴──────┐ ┌──────┴──────┐ ┌─────┴──────┐ +│ Config │ │ API Clients │ │ Sync │ +│ Manager │ │ │ │ Engine │ +│(config.go) │ │ - GitHub │ │ (sync.go) │ +│ │ │ - Codeberg │ │ │ +└─────────────┘ └──────────────┘ └────────────┘ +``` + +## Component Architecture + +### 1. Entry Point (cmd/gitsyncer/main.go) + +The main function serves as the application entry point and: +- Parses command-line flags +- Routes to appropriate handlers based on flags +- Manages application lifecycle and exit codes + +### 2. CLI Layer (internal/cli/) + +The CLI layer is responsible for user interaction and consists of: + +#### flags.go +- Defines all command-line flags +- Provides flag parsing logic +- Returns a structured `Flags` object + +#### handlers.go +- General command handlers (version, config, list operations) +- Configuration loading and validation +- Error presentation to users + +#### sync_handlers.go +- Sync-specific operations +- Orchestrates API clients and sync engine +- Handles batch operations + +### 3. Configuration Management (internal/config/) + +- Loads JSON configuration files +- Validates configuration structure +- Provides helper methods for finding organizations +- Supports multiple configuration file locations + +### 4. API Clients + +#### GitHub Client (internal/github/) +- Authenticates using personal access tokens +- Creates repositories via GitHub API +- Lists public repositories with pagination +- Handles multiple token sources (config, env, file) + +#### Codeberg Client (internal/codeberg/) +- Interacts with Codeberg's Gitea API +- Lists public repositories for users/organizations +- Supports pagination for large repository lists +- No authentication required for public operations + +### 5. Sync Engine (internal/sync/) + +The core synchronization logic is divided into several components: + +#### sync.go - Main Orchestrator +- Coordinates the entire sync process +- Manages working directory +- Handles repository-level operations + +#### repository_setup.go +- Clones repositories or ensures they exist +- Configures Git remotes +- Handles initial repository setup + +#### branch_sync.go +- Manages branch-level synchronization +- Tracks which remotes have which branches +- Orchestrates merge and push operations + +#### git_operations.go +- Low-level Git command wrappers +- Handles merge conflicts +- Manages stashing and checkout operations + +#### branch_filter.go +- Implements regex-based branch filtering +- Excludes branches based on patterns +- Provides filtering reports + +#### branch_analyzer.go +- Detects abandoned branches (6+ months inactive) +- Generates abandonment reports +- Analyzes branch activity + +### 6. Version Management (internal/version/) + +- Provides version information +- Supports build-time metadata injection +- Formats version strings for display + +## Data Flow + +### Sync Operation Flow + +1. **Configuration Loading** + ``` + User → CLI → Config Loader → Config Validation + ``` + +2. **Repository Discovery** + ``` + Config → API Clients → Repository Lists → Filtering + ``` + +3. **Synchronization Process** + ``` + For each repository: + └→ Setup/Clone Repository + └→ Configure Remotes + └→ Fetch All Remotes + └→ Get All Branches + └→ Filter Branches + └→ For each branch: + └→ Checkout/Create Branch + └→ Merge from Remotes + └→ Push to All Remotes + └→ Analyze Abandoned Branches + └→ Generate Reports + ``` + +## Design Principles + +### 1. Modularity +Each package has a single, well-defined responsibility: +- CLI handling is separate from business logic +- API clients are independent and interchangeable +- Core sync logic is platform-agnostic + +### 2. Configuration-Driven +- All behavior is controlled via configuration +- No hard-coded organization or repository names +- Flexible remote naming based on hosts + +### 3. Error Handling +- Graceful degradation (missing repos don't stop sync) +- Clear error messages with actionable guidance +- Proper exit codes for scripting + +### 4. Extensibility +- New platforms can be added by implementing API clients +- Branch filtering is regex-based for flexibility +- Sync strategies can be extended + +## Security Considerations + +### Token Management +- GitHub tokens are never logged or displayed +- Multiple token sources for flexibility +- Tokens are loaded on-demand + +### Git Operations +- All operations use standard Git commands +- No custom Git protocol implementation +- Respects Git's security model + +## Performance Characteristics + +### Scalability +- Handles multiple repositories in sequence +- Pagination support for large repository lists +- Efficient branch filtering + +### Resource Usage +- Minimal memory footprint +- Disk usage proportional to repository sizes +- Network usage optimized with selective fetching + +## Future Architecture Considerations + +### Planned Enhancements +1. **Parallel Synchronization** - Sync multiple repos concurrently +2. **Webhook Support** - Trigger syncs on push events +3. **More Platforms** - GitLab, Bitbucket, Gitea support +4. **Conflict Resolution** - Automated conflict resolution strategies + +### Extension Points +- Platform interface for new Git hosts +- Pluggable authentication mechanisms +- Custom sync strategies +- Hook system for pre/post sync actions \ No newline at end of file diff --git a/doc/configuration.md b/doc/configuration.md new file mode 100644 index 0000000..c4dcc50 --- /dev/null +++ b/doc/configuration.md @@ -0,0 +1,278 @@ +# GitSyncer Configuration Guide + +## Overview + +GitSyncer uses a JSON configuration file to define organizations, repositories, and sync behavior. The configuration file can be placed in several locations or specified via command line. + +## Configuration File Locations + +GitSyncer looks for configuration files in the following order: + +1. Path specified by `--config` flag +2. `./gitsyncer.json` (current directory) +3. `~/.config/gitsyncer/config.json` +4. `~/.gitsyncer.json` + +## Configuration Structure + +### Basic Structure + +```json +{ + "organizations": [ + { + "host": "git@github.com", + "name": "myorg", + "github_token": "ghp_xxxxxxxxxxxx" + }, + { + "host": "git@codeberg.org", + "name": "myorg" + } + ], + "repositories": [ + "repo1", + "repo2" + ], + "exclude_branches": [ + "^temp-", + "-wip$" + ] +} +``` + +### Configuration Fields + +#### organizations (required) +Array of organization objects. At least one organization must be configured. + +##### Organization Object +- **host** (string, required): Git host URL + - Format: `git@hostname` for SSH + - Format: `file:///path/to/repos` for local repositories + - Examples: `git@github.com`, `git@codeberg.org`, `git@gitlab.com` +- **name** (string, required): Organization or username +- **github_token** (string, optional): GitHub personal access token + - Only needed for GitHub organizations + - Can also be set via environment variable or file + +#### repositories (optional) +Array of repository names to sync. If empty, use `--sync-codeberg-public` or `--sync-github-public` to discover repositories. + +#### exclude_branches (optional) +Array of regex patterns for branches to exclude from synchronization. + +## Examples + +### Minimal Configuration + +Sync between GitHub and Codeberg: + +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "myusername"}, + {"host": "git@codeberg.org", "name": "myusername"} + ] +} +``` + +### With Specific Repositories + +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "myorg"}, + {"host": "git@codeberg.org", "name": "myorg"} + ], + "repositories": [ + "project1", + "project2", + "project3" + ] +} +``` + +### With Branch Filtering + +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "myorg"}, + {"host": "git@codeberg.org", "name": "myorg"} + ], + "repositories": ["myproject"], + "exclude_branches": [ + "^feature/experimental-", + "^temp-", + "-wip$", + "^old-" + ] +} +``` + +### Multiple Organizations + +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "personal"}, + {"host": "git@github.com", "name": "work"}, + {"host": "git@codeberg.org", "name": "personal"}, + {"host": "git@gitlab.com", "name": "personal"} + ], + "repositories": ["shared-project"] +} +``` + +### Local Mirror Configuration + +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "myorg"}, + {"host": "file:///home/user/git-mirror", "name": "myorg"} + ], + "repositories": ["important-project"] +} +``` + +## GitHub Token Configuration + +GitHub tokens are required for: +- Creating repositories (`--create-github-repos`) +- Listing private repositories +- Higher API rate limits + +### Token Sources (in order of precedence) + +1. **Configuration file**: `github_token` field in organization object +2. **Environment variable**: `GITHUB_TOKEN` +3. **Token file**: `~/.gitsyncer_github_token` + +### Creating a GitHub Token + +1. Go to GitHub Settings → Developer settings → Personal access tokens +2. Click "Generate new token (classic)" +3. Select scopes: + - `repo` (full control of private repositories) + - `read:org` (read organization membership) +4. Save the token securely + +### Setting the Token + +#### Method 1: Configuration File +```json +{ + "organizations": [ + { + "host": "git@github.com", + "name": "myorg", + "github_token": "ghp_xxxxxxxxxxxx" + } + ] +} +``` + +#### Method 2: Environment Variable +```bash +export GITHUB_TOKEN="ghp_xxxxxxxxxxxx" +gitsyncer --sync-all +``` + +#### Method 3: Token File +```bash +echo "ghp_xxxxxxxxxxxx" > ~/.gitsyncer_github_token +chmod 600 ~/.gitsyncer_github_token +``` + +### Testing Token + +```bash +gitsyncer --test-github-token +``` + +## Branch Exclusion Patterns + +The `exclude_branches` field accepts regular expressions to filter out branches from synchronization. + +### Common Patterns + +- `^temp-` - Exclude branches starting with "temp-" +- `-wip$` - Exclude branches ending with "-wip" +- `^feature/experimental-` - Exclude experimental feature branches +- `^(dev|development)$` - Exclude specific branch names +- `^release/\d+\.` - Exclude release branches (e.g., release/1.x) + +### Pattern Testing + +To see which branches are excluded: +```bash +gitsyncer --sync repo-name +# Output will show excluded branches and patterns +``` + +## Best Practices + +### 1. Start Simple +Begin with a minimal configuration and add complexity as needed. + +### 2. Use Dry Run +Test your configuration with `--dry-run` before actual synchronization: +```bash +gitsyncer --sync-all --dry-run +``` + +### 3. Secure Your Tokens +- Never commit tokens to version control +- Use environment variables or token files for sensitive data +- Restrict token permissions to minimum required + +### 4. Regular Expressions +- Test regex patterns before adding to configuration +- Use online regex testers to validate patterns +- Document complex patterns with comments + +### 5. Organization Naming +- Keep organization names consistent across platforms +- Use the same name on GitHub and Codeberg when possible + +## Troubleshooting + +### Configuration Not Found +```bash +$ gitsyncer --sync myrepo +No configuration file found. Please create one of: + - ./gitsyncer.json + - /home/user/.config/gitsyncer/config.json + - /home/user/.gitsyncer.json +``` + +**Solution**: Create a configuration file in one of the suggested locations. + +### Invalid JSON +```bash +$ gitsyncer --list-orgs +Failed to load configuration: invalid character '}' looking for beginning of object key string +``` + +**Solution**: Validate your JSON syntax using a JSON validator. + +### No Organizations Configured +```bash +$ gitsyncer --sync myrepo +Configuration must have at least one organization +``` + +**Solution**: Add at least one organization to the `organizations` array. + +### Token Issues +```bash +$ gitsyncer --test-github-token +ERROR: Token test failed: authentication failed (401) +``` + +**Solution**: +- Verify token is correct and not expired +- Check token has required permissions +- Ensure no extra whitespace in token \ No newline at end of file diff --git a/doc/development.md b/doc/development.md new file mode 100644 index 0000000..b22fbd4 --- /dev/null +++ b/doc/development.md @@ -0,0 +1,447 @@ +# GitSyncer Development Guide + +This guide is for contributors who want to help develop GitSyncer. + +## Table of Contents + +- [Development Setup](#development-setup) +- [Project Structure](#project-structure) +- [Building](#building) +- [Testing](#testing) +- [Code Style](#code-style) +- [Adding Features](#adding-features) +- [Contributing](#contributing) + +## Development Setup + +### Prerequisites + +- Go 1.21 or later +- Git +- Make (optional, for Makefile) +- Task (optional, for Taskfile) + +### Clone the Repository + +```bash +# Clone from Codeberg +git clone https://codeberg.org/snonux/gitsyncer.git +cd gitsyncer + +# Or from GitHub mirror +git clone https://github.com/snonux/gitsyncer.git +cd gitsyncer +``` + +### Install Dependencies + +```bash +# Download Go modules +go mod download + +# Verify modules +go mod verify +``` + +### Install Development Tools + +```bash +# Install Task runner (optional) +# macOS +brew install go-task + +# Linux +sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin + +# Install golangci-lint for linting +go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest +``` + +## Project Structure + +``` +gitsyncer/ +├── cmd/gitsyncer/ # Application entry point +│ └── main.go # Main function +├── internal/ # Private packages +│ ├── cli/ # CLI handling +│ │ ├── flags.go # Command-line flags +│ │ ├── handlers.go # General handlers +│ │ └── sync_handlers.go # Sync handlers +│ ├── codeberg/ # Codeberg API client +│ │ └── codeberg.go # Codeberg implementation +│ ├── config/ # Configuration +│ │ └── config.go # Config structures +│ ├── github/ # GitHub API client +│ │ └── github.go # GitHub implementation +│ ├── sync/ # Core sync logic +│ │ ├── sync.go # Main syncer +│ │ ├── branch_analyzer.go # Branch analysis +│ │ ├── branch_filter.go # Branch filtering +│ │ ├── branch_sync.go # Branch operations +│ │ ├── git_operations.go # Git commands +│ │ └── repository_setup.go # Repo setup +│ └── version/ # Version info +│ └── version.go # Version constants +├── test/ # Integration tests +│ ├── run_integration_tests.sh # Main test runner +│ └── test_*.sh # Individual tests +├── doc/ # Documentation +├── go.mod # Go module definition +├── go.sum # Go module checksums +├── LICENSE # BSD 2-Clause License +├── README.md # Project README +├── CLAUDE.md # AI assistant hints +└── Taskfile.yaml # Task automation +``` + +## Building + +### Using Task (Recommended) + +```bash +# Build for current platform +task build + +# Build for all platforms +task build-all + +# Run directly +task run + +# Run with arguments +task run -- --version +``` + +### Using Go Directly + +```bash +# Build binary +go build -o gitsyncer ./cmd/gitsyncer + +# Build with version info +go build -ldflags "\ + -X codeberg.org/snonux/gitsyncer/internal/version.Version=0.1.0 \ + -X codeberg.org/snonux/gitsyncer/internal/version.GitCommit=$(git rev-parse --short HEAD) \ + -X codeberg.org/snonux/gitsyncer/internal/version.BuildDate=$(date -u +%Y-%m-%d)" \ + -o gitsyncer ./cmd/gitsyncer + +# Cross-compile for Linux +GOOS=linux GOARCH=amd64 go build -o gitsyncer-linux-amd64 ./cmd/gitsyncer + +# Cross-compile for macOS +GOOS=darwin GOARCH=amd64 go build -o gitsyncer-darwin-amd64 ./cmd/gitsyncer +GOOS=darwin GOARCH=arm64 go build -o gitsyncer-darwin-arm64 ./cmd/gitsyncer +``` + +## Testing + +### Unit Tests + +Currently, the project has no unit tests. When adding new features, please include tests. + +```bash +# Run all tests (when available) +go test ./... + +# Run with coverage +go test -cover ./... + +# Run specific package tests +go test ./internal/sync/... +``` + +### Integration Tests + +```bash +# Run all integration tests +cd test +./run_integration_tests.sh + +# Run specific test +./test_branch_creation.sh +./test_conflict.sh +``` + +### Writing Tests + +Example test structure: +```go +// internal/sync/sync_test.go +package sync + +import ( + "testing" + "codeberg.org/snonux/gitsyncer/internal/config" +) + +func TestNew(t *testing.T) { + cfg := &config.Config{ + Organizations: []config.Organization{ + {Host: "git@github.com", Name: "test"}, + }, + } + + syncer := New(cfg, "/tmp/test") + if syncer == nil { + t.Fatal("New() returned nil") + } + + if syncer.workDir != "/tmp/test" { + t.Errorf("workDir = %q, want %q", syncer.workDir, "/tmp/test") + } +} +``` + +## Code Style + +### Go Code + +Follow standard Go conventions: + +1. **Formatting**: Use `gofmt` or `goimports` + ```bash + # Format all files + task fmt + # or + gofmt -w . + ``` + +2. **Naming**: + - Exported names start with capital letter + - Use CamelCase, not snake_case + - Acronyms should be all caps (URL, API, ID) + +3. **Comments**: + - Export functions/types need comments starting with the name + - Keep comments up-to-date with code changes + +4. **Error Handling**: + ```go + // Good + if err := doSomething(); err != nil { + return fmt.Errorf("failed to do something: %w", err) + } + + // Avoid bare returns + if err != nil { + return err // Missing context + } + ``` + +5. **Interfaces**: Accept interfaces, return structs + ```go + // Good + func ProcessRepo(r Repository) error { ... } + + // Avoid + func ProcessRepo(r *GitHubRepository) error { ... } + ``` + +### Commit Messages + +Follow conventional commits: +``` +type(scope): description + +[optional body] + +[optional footer] +``` + +Examples: +``` +feat(sync): add support for GitLab repositories +fix(config): handle missing organizations gracefully +docs: update API reference for new methods +refactor(cli): extract common handler logic +test: add integration tests for branch filtering +``` + +## Adding Features + +### Adding a New Git Platform + +1. Create new package in `internal/`: + ```bash + mkdir internal/gitlab + touch internal/gitlab/gitlab.go + ``` + +2. Implement platform client: + ```go + package gitlab + + type Client struct { + baseURL string + token string + org string + } + + func NewClient(token, org string) Client { + return Client{ + baseURL: "https://gitlab.com/api/v4", + token: token, + org: org, + } + } + + func (c *Client) ListPublicRepos() ([]Repository, error) { + // Implementation + } + ``` + +3. Update config to recognize platform: + ```go + func (o *Organization) IsGitLab() bool { + return strings.Contains(o.Host, "gitlab.com") + } + ``` + +4. Add sync support in handlers + +### Adding a New Command Flag + +1. Add flag in `internal/cli/flags.go`: + ```go + type Flags struct { + // ... existing flags + NewFeature bool // Add new flag + } + + func ParseFlags() *Flags { + flags := &Flags{} + // ... existing flags + flag.BoolVar(&flags.NewFeature, "new-feature", false, "Enable new feature") + } + ``` + +2. Handle flag in `cmd/gitsyncer/main.go`: + ```go + if flags.NewFeature { + os.Exit(cli.HandleNewFeature(cfg, flags)) + } + ``` + +3. Implement handler in `internal/cli/handlers.go` + +### Adding a New Configuration Option + +1. Update config struct in `internal/config/config.go`: + ```go + type Config struct { + Organizations []Organization `json:"organizations"` + Repositories []string `json:"repositories"` + ExcludeBranches []string `json:"exclude_branches"` + NewOption string `json:"new_option"` // Add new field + } + ``` + +2. Add validation if needed: + ```go + func (c *Config) Validate() error { + // ... existing validation + if c.NewOption != "" && !isValidOption(c.NewOption) { + return fmt.Errorf("invalid new_option: %s", c.NewOption) + } + } + ``` + +3. Use in sync logic as needed + +## Contributing + +### Before Submitting + +1. **Test your changes**: + ```bash + # Run integration tests + cd test && ./run_integration_tests.sh + + # Test manually + ./gitsyncer --sync test-repo + ``` + +2. **Format code**: + ```bash + task fmt + ``` + +3. **Update documentation**: + - Update relevant docs in `doc/` + - Update README if adding user-facing features + - Add examples to `doc/examples.md` + +4. **Update CLAUDE.md** if adding development commands + +### Pull Request Process + +1. Fork the repository +2. Create feature branch: + ```bash + git checkout -b feat/my-feature + ``` + +3. Make changes and commit: + ```bash + git add . + git commit -m "feat: add my feature" + ``` + +4. Push to your fork: + ```bash + git push origin feat/my-feature + ``` + +5. Create pull request with: + - Clear description of changes + - Link to related issues + - Test results + +### Code Review Guidelines + +- Respond to feedback constructively +- Make requested changes promptly +- Keep PR scope focused +- Update PR description as changes evolve + +## Debugging + +### Debug Output + +Add debug logging: +```go +import "log" + +func (s *Syncer) debugOperation() { + if os.Getenv("GITSYNCER_DEBUG") != "" { + log.Printf("Debug: operation details: %+v", s) + } +} +``` + +Use: +```bash +GITSYNCER_DEBUG=1 gitsyncer --sync test-repo +``` + +### Common Issues + +1. **Import cycle**: Keep dependencies acyclic +2. **Nil pointer**: Always check returns from New functions +3. **Git operations**: Ensure working directory is clean + +## Release Process + +1. Update version in `internal/version/version.go` +2. Update CHANGELOG.md +3. Tag release: + ```bash + git tag -a v0.1.0 -m "Release v0.1.0" + git push origin v0.1.0 + ``` +4. Build releases: + ```bash + task build-all + ``` +5. Create GitHub/Codeberg release with binaries \ No newline at end of file diff --git a/doc/examples.md b/doc/examples.md new file mode 100644 index 0000000..4f8dba4 --- /dev/null +++ b/doc/examples.md @@ -0,0 +1,373 @@ +# GitSyncer Usage Examples + +This guide provides practical examples of using GitSyncer for various scenarios. + +## Table of Contents + +- [Basic Operations](#basic-operations) +- [Repository Discovery](#repository-discovery) +- [Advanced Synchronization](#advanced-synchronization) +- [Automation Examples](#automation-examples) +- [Troubleshooting Scenarios](#troubleshooting-scenarios) + +## Basic Operations + +### Sync a Single Repository + +```bash +# Sync a specific repository +gitsyncer --sync my-project + +# Sync with custom working directory +gitsyncer --sync my-project --work-dir /tmp/gitsyncer-work + +# Dry run to preview changes +gitsyncer --sync my-project --dry-run +``` + +### Sync All Configured Repositories + +```bash +# Sync all repositories in config +gitsyncer --sync-all + +# Create missing GitHub repos automatically +gitsyncer --sync-all --create-github-repos +``` + +### List Operations + +```bash +# List configured organizations +gitsyncer --list-orgs + +# List configured repositories +gitsyncer --list-repos + +# Show version +gitsyncer --version +``` + +## Repository Discovery + +### Sync All Public Codeberg Repositories to GitHub + +```bash +# Discover and sync all public repos from Codeberg +gitsyncer --sync-codeberg-public + +# Also create repos on GitHub if they don't exist +gitsyncer --sync-codeberg-public --create-github-repos + +# Dry run to see what would be synced +gitsyncer --sync-codeberg-public --dry-run +``` + +### Sync All Public GitHub Repositories to Codeberg + +```bash +# Discover and sync all public repos from GitHub +gitsyncer --sync-github-public + +# Note: Codeberg repos must already exist +gitsyncer --sync-github-public +``` + +### Full Bidirectional Sync + +```bash +# Sync all public repos in both directions +gitsyncer --full + +# Equivalent to: +# gitsyncer --sync-codeberg-public --sync-github-public --create-github-repos +``` + +## Advanced Synchronization + +### Working with Branch Filters + +Configuration with branch exclusions: +```json +{ + "organizations": [ + {"host": "git@github.com", "name": "myorg"}, + {"host": "git@codeberg.org", "name": "myorg"} + ], + "repositories": ["my-project"], + "exclude_branches": [ + "^temp-", + "^feature/experimental-", + "-wip$" + ] +} +``` + +Output shows excluded branches: +```bash +$ gitsyncer --sync my-project + +🚫 Excluded 3 branches based on patterns: + Patterns: '^temp-', '^feature/experimental-', '-wip$' + Excluded branches: + - temp-fix + - feature/experimental-ai + - feature-wip +``` + +### Handling Merge Conflicts + +When conflicts occur: +```bash +$ gitsyncer --sync my-project + +ERROR: repository has unresolved merge conflicts +Please resolve conflicts in: /home/user/.gitsyncer-work/my-project +Or delete the directory to start fresh: rm -rf /home/user/.gitsyncer-work/my-project +``` + +Resolution options: +```bash +# Option 1: Manually resolve conflicts +cd /home/user/.gitsyncer-work/my-project +git status +# Fix conflicts +git add . +git commit -m "Resolved conflicts" +cd - +gitsyncer --sync my-project + +# Option 2: Start fresh +rm -rf /home/user/.gitsyncer-work/my-project +gitsyncer --sync my-project +``` + +### Abandoned Branch Detection + +GitSyncer detects branches inactive for 6+ months: +```bash +$ gitsyncer --sync-all + +[1/3] Syncing project1... +Repository project1 synchronized successfully! + +🔍 Abandoned branches in project1: + Main branch (main) is abandoned - last commit: 2023-01-15 + Other abandoned branches: + - feature/old-feature (2023-02-01) - abandoned: main branch is abandoned + - bugfix/old-fix (2023-03-15) - abandoned: main branch is abandoned + +=== Summary of Abandoned Branches === +Total repositories with abandoned branches: 1 + +Repository: project1 + - feature/old-feature + - bugfix/old-fix +``` + +## Automation Examples + +### Cron Job for Regular Sync + +```bash +# Add to crontab (crontab -e) +# Sync all repos every 6 hours +0 */6 * * * /usr/local/bin/gitsyncer --sync-all --config /home/user/.gitsyncer.json >> /var/log/gitsyncer.log 2>&1 + +# Sync public repos daily at 2 AM +0 2 * * * /usr/local/bin/gitsyncer --full >> /var/log/gitsyncer-public.log 2>&1 +``` + +### Shell Script Wrapper + +```bash +#!/bin/bash +# sync-repos.sh + +set -e + +CONFIG_FILE="$HOME/.config/gitsyncer/config.json" +LOG_FILE="$HOME/.gitsyncer/sync.log" + +echo "Starting sync at $(date)" >> "$LOG_FILE" + +# Test GitHub token first +if ! gitsyncer --test-github-token; then + echo "GitHub token test failed" >> "$LOG_FILE" + exit 1 +fi + +# Sync all repos +if gitsyncer --sync-all --config "$CONFIG_FILE" >> "$LOG_FILE" 2>&1; then + echo "Sync completed successfully at $(date)" >> "$LOG_FILE" +else + echo "Sync failed at $(date)" >> "$LOG_FILE" + exit 1 +fi +``` + +### CI/CD Integration + +GitHub Actions example: +```yaml +name: Sync Repositories + +on: + schedule: + - cron: '0 */6 * * *' # Every 6 hours + workflow_dispatch: # Manual trigger + +jobs: + sync: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Install GitSyncer + run: | + wget https://github.com/yourusername/gitsyncer/releases/latest/download/gitsyncer-linux-amd64 + chmod +x gitsyncer-linux-amd64 + sudo mv gitsyncer-linux-amd64 /usr/local/bin/gitsyncer + + - name: Create config + run: | + cat > gitsyncer.json << EOF + { + "organizations": [ + {"host": "git@github.com", "name": "${{ github.repository_owner }}"}, + {"host": "git@codeberg.org", "name": "${{ secrets.CODEBERG_ORG }}"} + ] + } + EOF + + - name: Sync repositories + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: gitsyncer --sync-all --create-github-repos +``` + +## Troubleshooting Scenarios + +### Testing GitHub Authentication + +```bash +# Test token is valid +$ gitsyncer --test-github-token +Testing GitHub token authentication... + Loaded token from env var (length: 40) + Checking URL: https://api.github.com/repos/myorg/gitsyncer + Token present: true (length: 40) +SUCCESS: Token is valid! Repository check returned: true +``` + +### Debugging Sync Issues + +```bash +# Check git status in work directory +cd ~/.gitsyncer-work/my-project +git status +git remote -v +git branch -a + +# Check for stashed changes +git stash list + +# View recent operations +git reflog +``` + +### Repository Not Found + +```bash +$ gitsyncer --sync nonexistent-repo +ERROR: Failed to clone from any organization +``` + +Solutions: +1. Verify repository exists on at least one platform +2. Check repository name spelling +3. For private repos, ensure proper authentication + +### Working Directory Issues + +```bash +# Permission denied +$ gitsyncer --sync my-project --work-dir /root/work +ERROR: failed to create work directory: permission denied + +# Solution: Use accessible directory +gitsyncer --sync my-project --work-dir ~/gitsyncer-work + +# Disk space issues +$ gitsyncer --sync large-repo +ERROR: write error: no space left on device + +# Solution: Clean up or use different disk +df -h +rm -rf ~/.gitsyncer-work/old-repo +gitsyncer --sync large-repo --work-dir /mnt/storage/gitsyncer +``` + +### Network and Connectivity + +```bash +# SSH key issues +$ gitsyncer --sync my-project +ERROR: git@github.com: Permission denied (publickey) + +# Solution: Add SSH key to agent +ssh-add ~/.ssh/id_rsa + +# Firewall/proxy issues +$ gitsyncer --sync my-project +ERROR: Failed to connect to github.com port 22: Connection timed out + +# Solution: Use HTTPS URLs in config +{ + "organizations": [ + {"host": "https://github.com", "name": "myorg"}, + {"host": "https://codeberg.org", "name": "myorg"} + ] +} +``` + +## Best Practices + +### 1. Start with Dry Run +Always test with `--dry-run` first: +```bash +gitsyncer --sync-all --dry-run +``` + +### 2. Use Specific Working Directories +Organize syncs by project or purpose: +```bash +gitsyncer --sync personal-projects --work-dir ~/sync/personal +gitsyncer --sync work-projects --work-dir ~/sync/work +``` + +### 3. Monitor Sync Operations +Keep logs for troubleshooting: +```bash +gitsyncer --sync-all 2>&1 | tee -a ~/gitsyncer.log +``` + +### 4. Regular Maintenance +Clean up old working directories: +```bash +# Remove repos no longer in config +cd ~/.gitsyncer-work +ls -la +rm -rf old-project-name +``` + +### 5. Handle Secrets Securely +Never put tokens in scripts directly: +```bash +# Bad +gitsyncer --config config-with-token.json + +# Good +export GITHUB_TOKEN="$(pass show github/token)" +gitsyncer --sync-all +``` \ No newline at end of file -- cgit v1.2.3