Docs

This document defines the Source Commons Framework CSV files, required fields, allowed values, formats, and examples.

The framework uses CSV because it is easy to review in pull requests, easy to validate, and compatible with many data tools. Multiple values in one cell must be separated with a semicolon and a space: value one; value two.

Field Prefixes

CSV Files

General Formats

Inter-record Links

Every CSV can link to existing Source Commons items without a fixed limit. Use these optional fields when a relationship is useful but is not already represented by a more specific field such as source_ids, tools_used, datasets, use_cases, related_skills, or crosswalk_source_ids.

Relationship fields are identifiers, not public contributor names. The website may also mirror these fields into a relational database so the catalogue can show connected records without reading every CSV cell.

Controlled Values

These values are Source Commons controlled values unless a field notes an external standard.

Source Levels

DCAT Connections

For structured open data, DCAT fields can be prefilled from open data portal APIs. Examples include CKAN package APIs, data.gov harvest records, data.gouv.fr dataset APIs, and open.canada.ca catalogue APIs.

When a portal exposes DCAT or DCAT-like JSON, contributors should use the official API or catalogue export to populate fields such as title, description, publisher, identifier, landing page, access URL, download URL, media type, license, spatial coverage, temporal coverage, themes, and keywords.

Source Commons does not replace DCAT. It keeps DCAT-compatible fields and adds operational fields for access method, pricing, rate limits, legal risk, extraction difficulty, join keys, crosswalks, and workflow context.

Wikidata Connections

Wikidata fields make Source Commons records easier to reconcile with a shared knowledge graph. They should be treated as identifiers and assisted relationships, not as free-text tags.

Level 1 is the optional wikidata_id field on entity-bearing tables. It stores one primary Wikidata item for the source, tool, or use case when a clear item exists. Examples include Q142 for France, Q95 for Google, Q8908 for Wikimedia Foundation, and Q720467 for Hugging Face.

Evaluation rows do not get a separate wikidata_id in this version because they describe dated observations about another record. They inherit entity context through source_id.

Level 2 adds assisted source relationships with wikidata_main_topics and wikidata_related_entities. Autocomplete should search Wikidata labels and aliases, while the stored value should preserve the QID. For example, a France Travail open data source could use Employment (Q12737077) as a main topic and France Travail (Q124556307); Ministry of Labour (Q3406276) as related entities.

Level 3 uses wikidata_advanced_relations for curated property-value relationships. Store each relation as a compact object such as {"property":"P137","value":"Q95"}, where P137 is a Wikidata property and Q95 is a Wikidata item. Because relation objects contain commas, quote the CSV cell when a relation is present. The UI should offer a small curated property list for common relationships such as owner, publisher, funder, operator, and covered area, with property search available only inside the advanced relation control.

Contributors should not choose freely from all Wikidata properties in normal editing. Properties such as P31, P279, P361, P527, P749, P1269, and P2578 may be useful in expert curation, but they are too broad or specialized for default contributor workflows.

GitHub Connections

GitHub fields connect a source, tool, or use case to a primary repository, person account, or organization account when that link helps explain provenance, implementation, maintenance, issue tracking, or reproducibility.

Use github_resource_type to choose repository, user, organization, or N/A. Use github_resource_id for the repository full name or account login. Examples include openrefine/openrefine for a repository, github for an organization, and octocat for a user account.

Use github_url for the public GitHub URL and github_api_url for the REST API metadata endpoint when checked. Examples include https://github.com/openrefine/openrefine, https://api.github.com/repos/openrefine/openrefine, https://api.github.com/orgs/github, and https://api.github.com/users/octocat.

GitHub links should be factual and relevant. A record should not link to a promotional account when an official repository, maintainer organization, or reproducibility repository is available.

Hugging Face Connections

Hugging Face metadata appears only in fields beginning with hf_.

These fields can point to models, datasets, Spaces, discussions, or workflow dependencies. They should be filled from public Hugging Face Hub API responses or repository metadata. Contributors should still verify licenses, gated access, model cards, dataset cards, and redistribution limits before relying on a repository.

MCP Connections

MCP fields document Model Context Protocol servers that help users or agents access, inspect, transform, or reuse sources, tools, and skills.

MCP servers should usually be recorded in data/tools.csv with tool_category set to MCP server. A server should be added when it provides a concrete operational interface, not merely because a project mentions MCP.

Use mcp_server_id for a stable identifier such as MCP-DATAGOUV. Use mcp_transport to describe how the server is accessed, for example stdio, http, or sse. Use mcp_tools_exposed, mcp_resources_exposed, and mcp_prompts_exposed to summarize what the server makes available.

Skills may reference supporting MCP servers through the existing mcp_servers field. Use cases may reference MCP servers through tools_used when the server is part of the workflow.

MCP records should include security notes when the server requires credentials, accesses private files, calls external APIs, or can trigger write actions.

Required Fields

Required fields are the minimum fields needed for a pull request to be reviewable. Operational details such as account requirements, authentication, rate limits, pricing, join keys, and legal notes are encouraged when known, but they are optional so public datasets can be submitted without unnecessary friction.

data/sources.csv

data/tools.csv

data/evaluations.csv

data/use-cases.csv

data/skills.csv

Skills are reusable capability descriptions: a method, recipe, promptable workflow, context package, or operational procedure that helps someone do a recurring task. A skill can mention tools and datasets, but it is not itself a software tool. A skill can support many use cases.

Every skill is represented twice:

• one row in data/skills.csv for indexing, search, and review;
• one Markdown file in skills/{skill_id}.md with generated YAML frontmatter and editable Markdown body.

The generated YAML frontmatter should preserve the same field names as the CSV where possible. Multi-value fields are YAML arrays in Markdown and semicolon-separated lists in CSV.

Example Source Row

The CSV files in this repository remain empty except for headers. This example shows the expected style for a future source pull request.

source_id,source_level,dct_title,dct_description,dct_publisher,dct_identifier,wikidata_id,dcat_landing_page,dcat_access_url,dcat_download_url,dcat_distribution_format,dcat_media_type,dcat_endpoint_url,dcat_prefill_url,dcat_prefill_status,dct_access_rights,dct_license,dct_spatial,dct_temporal,dcat_theme,dcat_keywords,wikidata_main_topics,wikidata_related_entities,wikidata_advanced_relations,access_method,access_account_required,auth_method,rate_limit_declared,rate_limit_scope,rate_limit_value,rate_limit_period,rate_limit_notes,access_cost_type,pricing_model,pricing_currency,pricing_amount,pricing_unit,pricing_notes,update_frequency,structuredness,fragmentation_level,extraction_difficulty,stable_identifiers,join_keys,legal_risk_level,legal_risk_notes,scraping_position,documented_exception_cases,dct_conforms_to,join_key_types,join_key_examples,join_key_granularity,join_key_confidence,crosswalk_source_ids,crosswalk_notes,related_source_ids,related_tool_ids,related_use_case_ids,related_skill_ids,related_evaluation_ids,github_resource_type,github_resource_id,github_url,github_api_url,github_metadata_status,hf_repo_type,hf_repo_id,hf_api_url,hf_metadata_status
SRC-CA-TORONTO-BUILDING-PERMITS,L2_dataset,Building Permits - Active Permits,Information on currently active building applications and permits in Toronto.,City of Toronto,toronto-building-permits-active,,https://open.toronto.ca/dataset/building-permits-active-permits/,https://ckan0.cf.opendata.inter.prod-toronto.ca/api/3/action/package_show?id=building-permits-active-permits,,CSV; JSON; API,application/json,https://ckan0.cf.opendata.inter.prod-toronto.ca/api/3/action/package_show?id=building-permits-active-permits,https://open.toronto.ca/dataset/building-permits-active-permits/,Partial,Public,City of Toronto Open Data Licence,CA-ON,Current,Urban planning,building permits; development; construction; addresses,,,,Open data API; file download,false,None,Unknown,IP,,,No public limit found; check API behavior before monitoring.,Free,free_open_access,N/A,N/A,N/A,No fee declared for official API or downloads.,daily,Structured,Low,Low,Permit numbers and addresses,permit_number; address; ward; application_date,Low,Open data license; geocoding and personal data minimization should be checked.,No scraping needed,Use official open data API or downloads.,DCAT-3; DQV; SourceCommons-ODSP-0.1,address; geography; date,permit_number=24 123456; ward=10; application_date=2026-01-12,Permit and parcel level,High,N/A,Address and ward keys allow joins with planning and zoning sources.,,,,,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A

Identifier Rules

Identifiers should be stable and human-readable. Suggested prefixes:

• SRC- for sources.
• TOOL- for tools.
• EVAL- for evaluations.
• UC- for use cases.
• scf-skill- plus a UUID for skills.

Records should not reuse an identifier for a different entity after publication.