Index of every page on this site: llms.txt
All skills
github avatar

/sql-server-table-reconciliation

@f10fdb8 official
by githubgithub/awesome-copilot41k stars
5,399
Run it yourself

Use when: comparing SQL Server tables across instances, data migration validation, ETL verification, row mismatch detection, schema drift, reconciliation report, production vs staging comparison. Uses mssql-python driver with Apache Arrow for fast columnar data transfer and comparison.

Use this Skill: https://skilld.dev/gh/github/awesome-copilot/sql-server-table-reconciliation

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

SKILL.md

≈80 tokens for metadata: the name and description. ≈1.4k when used: this file.

Description uses 4.0% of example budget

Before choosing a Skill, your Agent reads its name and description. All available Skills share that space.

  • A shorter description leaves more room for other Skills. This entry exceeds our 1% size suggestion.

In our Claude Code example, all Skill names and descriptions share 8,000 characters. This Skill uses ≈320 characters, or 4.0%.

The 1% threshold is a size suggestion. Longer descriptions can still fit.

Your model, settings, and other Skills decide how much text your Agent can read.

Example settings and source

The example uses a 200k-token context and default Claude Code settings. The count includes the name, description, separators, and when_to_use when present. Codex also counts local file paths.

Skit's source and limits: Codex 0.160.1, Claude Code 2.1.292.

SQL Server Table Reconciliation

Compare identical tables across two SQL Server instances using Python with mssql-python driver and Apache Arrow. Detect missing rows, column mismatches, schema drift, and produce a reconciliation report.

Workflow

  1. Collect connection details for source and target
  2. Identify primary key / composite key
  3. Detect schema differences
  4. Extract data via Arrow for efficient columnar transfer
  5. Compare rows and columns
  6. Generate reconciliation report

Collect Inputs

Parameter Required Description
Source server Yes Source SQL Server (e.g. prod-server.database.windows.net)
Source database Yes Source database name
Target server Yes Target SQL Server (e.g. staging-server.database.windows.net)
Target database Yes Target database name
Tables Yes Comma-separated schema.table names, or schema.* wildcard (e.g. dbo.Orders,dbo.Items or dbo.*)
Auth mode Yes sql (user/password) or entra (Azure AD/token)
Primary key Auto-detect Column(s) forming the row identity. Auto-detect from metadata if not provided.
Columns to compare All Subset of columns, or all non-PK columns
Chunk size 100000 Rows per batch for large tables
Output format console console, csv, parquet, or json

Bundled Script

The reconciliation logic is provided as a standalone script at scripts/reconcile.py. Invoke it with the appropriate arguments based on user inputs:

python scripts/reconcile.py \
    --source-server <source_server> \
    --source-database <source_database> \
    --target-server <target_server> \
    --target-database <target_database> \
    --tables "<table_spec>" \
    --auth <sql|entra> \
    --chunk-size <chunk_size> \
    --output <console|csv|json>

Optional arguments

Argument Description
--primary-key Comma-separated PK column(s). Omit to auto-detect.
--columns Comma-separated columns to compare. Omit to compare all non-PK columns.

Example invocations

Single table with SQL auth:

python scripts/reconcile.py \
    --source-server prod-server.database.windows.net \
    --source-database ProdDB \
    --target-server staging-server.database.windows.net \
    --target-database StagingDB \
    --tables "dbo.Orders" \
    --auth sql \
    --output console

Wildcard with Entra auth and CSV output:

python scripts/reconcile.py \
    --source-server prod-server.database.windows.net \
    --source-database ProdDB \
    --target-server staging-server.database.windows.net \
    --target-database StagingDB \
    --tables "dbo.*" \
    --auth entra \
    --output csv

Prerequisites

Install required packages before running:

pip install mssql-python pyarrow pandas

Comparison Rules

  • Normalize types before comparing: cast decimals to same precision, trim strings, normalize datetime to UTC
  • NULL handling: NULL == NULL is considered a match (both sides missing = no diff)
  • Ignore row order: always compare by PK join, never positional
  • Large tables: chunk extraction with OFFSET/FETCH or ROW_NUMBER() partitioning

Hash-Based Optimization (for large tables)

When table has >1M rows, generate a hash pre-check:

SELECT {pk_cols},
       HASHBYTES('SHA2_256', CONCAT_WS('|', col1, col2, ...)) AS row_hash
FROM {table}

Compare hashes first; only fetch full rows for mismatched hashes. This reduces data transfer significantly.

Report Format

Reconciling dbo.EMPLOYEES...
Reconciling dbo.DEPARTMENTS...
Reconciling dbo.JOBS...

--- dbo.EMPLOYEES ---
  Source: 107  Target: 107
  Missing: 0  Extra: 0  Mismatches: 0
  Result: ✓ IDENTICAL

--- dbo.DEPARTMENTS ---
  Source: 27  Target: 27
  Missing: 0  Extra: 0  Mismatches: 3
  Result: ✗ DIFFERENCES FOUND

--- dbo.JOBS ---
  Source: 19  Target: 19
  Missing: 0  Extra: 0  Mismatches: 0
  Result: ✓ IDENTICAL

=== Summary: 2 passed, 1 failed, 0 skipped / 3 tables ===

When a single table is provided, include full detail (schema drift, sample rows, mismatches). When multiple tables, use the compact per-table format above with full detail only for tables with FAIL status.

Performance Considerations

Scenario Strategy
< 100K rows Single Arrow fetch, in-memory pandas compare
100K–1M rows Chunked extraction (100K batches), streaming comparison
> 1M rows Hash pre-check → only fetch mismatched rows
Wide tables (100+ cols) Compare PK + hash first, drill into specific columns on mismatch
Network-constrained Use Arrow columnar format (10-50x smaller than row-by-row)

Constraints

  • Always use mssql-python driver (not pyodbc, pymssql)
  • Always use Apache Arrow via cursor (cursor.arrow()) for data extraction
  • Connection MUST use connection string format, not keyword arguments (kwargs like encrypt=True throw errors)
  • Never compare without identifying PK first — ask user if auto-detect fails
  • Handle connection failures gracefully with retry logic
  • Never hardcode credentials in generated scripts — use os.environ / getpass (env vars: MSSQL_USER, MSSQL_PASSWORD)
  • Do not print credentials in output or logs
  • Use parameterized queries (? placeholders) for metadata lookups — never f-string interpolate user input into SQL

Source: SKILL.md on GitHub

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

3 warnings2mo3 checks · Risk MEDIUM
  • Gen Agent Trust Hub2mo

    The skill facilitates SQL Server table reconciliation but contains a significant security flaw in its Python script. The script is vulnerable to SQL injection because it uses f-string interpolation to insert table and column names directly into SQL queries. This could allow an attacker who controls the database schema or the skill's input parameters to execute unauthorized SQL commands on the connected database instances.

  • Socket2mo

    1 alert: gptAnomaly

  • Snyk2mo

    Risk: MEDIUM · 1 issue

Signed by skilld at f10fdb8. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 6 months ago
  • Python
  • sql-server
  • data-migration
  • reconciliation
  • etl
  • apache-arrow
  • schema-drift
  • mssql-python

README badge

README badge for github/awesome-copilot/sql-server-table-reconciliation

Compares identical tables across two SQL Server instances using the mssql-python driver and Apache Arrow, detecting missing rows, column mismatches, schema drift, and generating a reconciliation report. Handles single or wildcard table specs, supports both SQL and Azure AD authentication, and optimizes for large tables via hash pre-checks and chunked extraction.

Generated from the current SKILL.md.

Does this skill support MySQL or PostgreSQL?
No. The skill is designed specifically for SQL Server using the mssql-python driver and does not support other database systems.
How does it handle large tables with millions of rows?
For tables over 1M rows, the skill uses hash-based pre-checks (SHA2_256) to compare row hashes first, then fetches only mismatched rows to reduce data transfer. Smaller tables are chunked in 100K-row batches.
What authentication methods are supported?
The skill supports both SQL authentication (username/password) and Entra/Azure AD token-based authentication via the --auth flag.
Can I compare just a subset of columns?
Yes. Use the --columns argument to specify a comma-separated list of columns to compare, or omit it to compare all non-primary-key columns.
What output formats are available?
The skill can output to console, CSV, JSON, or Parquet format via the --output flag.

Generated from the current SKILL.md. These answers refresh after source changes.