# OpenMetadata Connector Standards

## Architecture: Schema-First

OpenMetadata connectors follow a **schema-first** architecture. One JSON Schema definition cascades through 6 layers:

```
JSON Schema (single source of truth)
    ├── Python Pydantic models     (make generate)
    ├── Java models                (mvn install -pl openmetadata-spec)
    ├── TypeScript types           (yarn parse-schema)
    ├── UI config forms            (RJSF auto-renders from schema)
    ├── API request validation     (server uses Java models)
    └── Test fixtures              (tests import Pydantic models)
```

**Never hand-write config classes.** Define the JSON Schema; everything else is generated.

## Connector Anatomy

Every connector lives at `ingestion/src/metadata/ingestion/source/{service_type}/{name}/` and has:

| File | Purpose | Required |
|------|---------|----------|
| `__init__.py` | Module marker | Always |
| `connection.py` | Create and test connections | Always |
| `metadata.py` | Extract metadata from the source | Always |
| `service_spec.py` | Register connector with the framework | Always |
| `client.py` | REST/SDK client wrapper | Non-database |
| `queries.py` | SQL query templates | Database |
| `lineage.py` | Lineage extraction | If lineage capability |
| `usage.py` | Usage extraction | If usage capability |
| `query_parser.py` | Query log parsing | If lineage or usage |
| `CONNECTOR_CONTEXT.md` | AI implementation brief | Generated by scaffold |

## Service Types

| Service Type | Base Class | Reference |
|---|---|---|
| `database` | `CommonDbSourceService` | `mysql/` |
| `dashboard` | `DashboardServiceSource` | `metabase/` |
| `pipeline` | `PipelineServiceSource` | `airflow/` |
| `messaging` | `MessagingServiceSource` | `kafka/` |
| `mlmodel` | `MlModelServiceSource` | `mlflow/` |
| `storage` | `StorageServiceSource` | `s3/` |
| `search` | `SearchServiceSource` | `elasticsearch/` |
| `api` | `ApiServiceSource` | `rest/` |

## Connection Types (Database Only)

| Type | Base Class | Pattern |
|------|-----------|---------|
| `sqlalchemy` | `BaseConnection[Config, Engine]` | SQLAlchemy dialect + engine |
| `rest_api` | `get_connection()` / `test_connection()` | Custom REST client |
| `sdk_client` | `get_connection()` / `test_connection()` | Vendor SDK wrapper |

Non-database connectors always use `get_connection()` / `test_connection()` functions.

## ServiceSpec System

Every connector declares a `ServiceSpec` in `service_spec.py`:

- **Database**: `DefaultDatabaseSpec(metadata_source_class=..., connection_class=..., lineage_source_class=..., usage_source_class=...)`
- **All others**: `BaseSpec(metadata_source_class=...)`

The framework resolves specs dynamically via: `metadata.ingestion.source.{service_type}.{name}.service_spec.ServiceSpec`

## Registration Checklist

To register a new connector, modify these files:

1. **Service enum**: `openmetadata-spec/.../entity/services/{serviceType}Service.json` — add type to enum + connection `oneOf`
2. **Test connection**: `openmetadata-service/.../testConnections/{serviceType}/{name}.json` — create file
3. **UI utils**: `openmetadata-ui/.../utils/{ServiceType}ServiceUtils.tsx` — import schema + add switch case
4. **Localization**: `openmetadata-ui/.../locale/languages/` — add i18n display name keys

## Code Generation Commands

```bash
source env/bin/activate
make generate                                # Python Pydantic models
mvn clean install -pl openmetadata-spec      # Java models
cd openmetadata-ui/src/main/resources/ui && yarn parse-schema  # UI schemas
make py_format                               # Format Python
mvn spotless:apply                           # Format Java
```