Primary key versus grain: why de-duplicating China filing data deletes real rows

Short answer: the column that uniquely identifies a row and the columns that describe what the row is about are different things, and in filing data the second set legitimately repeats. De-duplicate on it and you delete real disclosures. The usual symptom is a dataset that quietly gets smaller every time someone "cleans" it.

The distinction, stated once

In ChinaScope Financial Notes, every row carries id, the vendor's own declared primary key, measured unique inside every delivery of every table. That is what identifies a row, and what to match on when the vendor restates a figure.

Company by reporting period by standardized line item says what a row is about, and it repeats, because a schedule discloses several lines per company, period and item. De-duplicating on it silently discards real rows. Aggregate on those columns. Never treat them as a key. (Data cut 2026-04-09.)

The same trap, four more times

SAM's supply-chain graph. 70,900 rows carry 66,053 distinct (primary_code, related_code, relation_category) edges. 3,122 edges arrive on two or three rows, so 4,847 rows, 6.8%, are repeats. Every row has its own id, genuinely unique, which is the declared grain. If you join or count on the product pair, dedupe on the triple first or you inflate by that margin. (Data cut 2026-02-26.)

SmarTag tags. Each tag table's grain is newsid plus the entity code or codes. A company can appear once per article via newsid plus stockcode plus companyid. Aggregate on the grain, not on newsid alone, when counting entities. (Data cut 2026-09-16.)

RFP-Bids graph. It is a monthly panel, so a company and entity pair recurs across months. Dedupe on the pair for a relationship view, or sum across months for a period total. Those are two different questions and the same rows answer both. (Data cut 2026-05-29.)

Multi-listed companies, everywhere. In SAM, Affiliate and C2C Disclosed, a company's facts attach to each of its listed securities, so a company with two listings appears twice per period. Aggregate or dedupe on the company key, not on secu. In SAM that key is company_id2; in Affiliate and C2C Disclosed it is company_id. This has its own page, because it is the most common way A-share company counts come out too high.

The evaluation copy is not shaped like the purchased one

Worth knowing before you write code against a sample. In both C2C (PIT) and Relationships (PIT), the evaluation copy in the Matrix is pre-resolved: collapsed to a single current view, the latest _known_date per grain with delete-groups dropped. It carries _known_date but not _panel_op. The purchased export carries the full panel with both, which is what you need to replay history.

Evaluate the shape in the Matrix. Buy the export for the bitemporal work. Code written against the evaluation copy's grain will not survive contact with the full panel. (Data cuts 2026-08-22 and 2026-08-23.)