Documentation

What the portal is today

Source Commons Framework (SCF) is collaborative infrastructure for finding, documenting, connecting, evaluating, and reusing data sources, tools, use cases, evaluations, and docs. Spaces, lists, and data models organize collective work. The catalog is a view of the graph, not a replacement for metadata already maintained by an authoritative provider.

When a legitimate provider already maintains a resource, SCF primarily stores its canonical reference, provenance, SCF relationships, annotations, and evaluations. Full provider metadata remains with that provider and may be cached for display. When no stable reference exists, SCF can maintain a native object and its original metadata.

Available capabilities

Collaboration

• Spaces: public or closed workspaces with members, permissions, discussions, pinned lists, and data models.
• Lists: shareable multi-resource collections. A list can contain sources and datasets, tools, use cases, docs, evaluations, and data models, displayed in separate sections. Lists also support visibility, sub-lists, themes, viewer/editor/admin roles, and owner/invite/followers/everyone contribution policies.
• Data Models: collaborative field definitions, types, required and identifier flags, possible values, examples, votes, comments, responsibilities, validation, and contribution merging.
• Community: recent activity, statistics, newsletter, and governance.

Resources and contribution

• Explore displays sources, tools, use cases, evaluations, and docs with search, categories, and filters.
• Each type also has a directory and detail page: sources, tools, use cases, evaluations, and docs.
• Contribute creates or edits all five types, connects resources, and imports references through the available connectors.
• Assisted import currently covers data.gov, open.canada.ca, data.gouv.fr, data.europa.eu, World Bank Data360, Huwise/OpenDataSoft, GitHub, Hugging Face, Kaggle, and Google Dataset Search. Imported values must still be reviewed.
• Evaluations target a source, tool, use case, or doc and use DQV dimensions and metrics, a 1–5 rating, confidence, and an annotation.
• A signed-in person can also add a direct 1–5 community rating from a resource page. One active rating is retained per person and resource. Detail pages and cards show the public average, rating count, and approved-comment count; individual ratings remain private. The displayed average combines direct ratings with ratings from historical DQV evaluation records without replacing detailed evaluations.
• Search finds public Spaces, lists, and resources, plus items available to the signed-in account.

Studio and open data

• Studio contains Source2API, Archive Data Point, API Merge, Source2Parquet, Data Update Detector, AI Agent Data Extractor, and Collaborative Data Models according to their release level and the account's permissions.
• The Web extension creates an updated JSON API or archives one data point from the page being viewed.
• The public graph can be queried as JSON at GET /api/open-data and exported as Parquet. It contains public records, relations, public lists, public Spaces, models, and observed fields. Full external-provider metadata is not re-exported as canonical SCF truth.

Architecture and source of truth

The site uses Next.js App Router and React. Supabase provides authentication, PostgreSQL, Row Level Security policies, and the operational source for contributions, relations, Spaces, lists, comments, data models, and Studio jobs.

Two record modes are explicit:

• external_reference: SCF references a URL and provider that maintain the resource;
• native_object: SCF maintains original metadata because no stable external reference exists.

The storage policy is link_only, hybrid_minimal, or native. external_metadata_cache is a rebuildable technical cache, not canonical truth. During the progressive migration, some readers retain compatibility with historical GitHub/CSV files and merge them with framework_records. Every new contribution is synchronized to Supabase and its relationships are materialized in framework_record_relations. List items use resource_type, resource_id, and resource_title; legacy source_id and source_title remain compatibility aliases. Direct ratings are stored in resource_ratings with one row per resource and user; only anonymous aggregates are public.

Operational record columns

Resource types and fields

The following tables are generated from lib/sourceSchema.js, the same definition used by the contribution workflow. csv_row is the historical name of the JSON document containing these fields; it does not mean CSV remains the operational database. Lists use semicolon-separated values or JSON depending on the displayed control.

Sources

Tools

Evaluations

Use cases

Docs

Controlled values

Empty values are accepted for optional fields. The following non-empty values are currently controlled by the contribution form.

Standards, relations, and provenance

• dct_* follows Dublin Core Terms; dcat_* follows DCAT; dqv_* follows the Data Quality Vocabulary.
• wikidata_* stores QIDs and, for advanced relations, property/value pairs.
• github_*, hf_*, and mcp_* document ecosystem-specific provenance and capabilities.
• related_* fields and manual relations connect records. Derived relations can be rebuilt without deleting manual links.
• Identifiers, authors, dates, canonical URLs, confidence levels, and notes are retained to make contributions traceable.

Access and permissions

Row Level Security limits public reads to public content. Closed content requires membership or an explicit permission. Space permissions can separately allow list creation, pinning, model creation or commenting, and discussion opening. In a private list, owners/admins manage settings, editors/admins change items, and viewers can read. Pages and actions enforce these rights on the server; the interface alone is never the access control.

Licenses and governance

The code, catalog content, and data model do not all use the same license. The exact allocation, decision process, maintenance model, and economic model are documented in community governance.