Sweddknowlege plateforme 20 août, 2026

Confidentiality and Census microdata 30.11.2022

Data Confidentiality Principles and Census Microdata Mose Job NyandwakiCensus Technical Advisor-ECA

Outline
Data Confidentiality

Census Microdata

Why share census microdata

Census Metadata

Challenges facing dissemination of census microdata

Going forward

Discussion

Confidentiality
Confidentiality assurance to respondents is the first and most important issue that should be considered by national statistical offices in terms of both perception and reality (P&RPHC Rev. 3, para. 2.145) for NSOs to maintain the trust of respondents;
Data collection depends on goodwill and trust of the members of the public;
The sixth UN Fundamental Principle of Official Statistics about confidentiality.
“Individual data collected by statistical agencies for statistical compilation, whether or not they refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical purposes.”
Statistical Agencies have an obligation to protect the, privacy, security, and confidentiality of collected data so that people, households, and organizations can’t be identified without their permission;
Statistical Agencies MUST protect the privacy and confidentiality of data throughout its life cycle — whenever we collect, use, store, and distribute it;
Data confidentiality is often a legal requirement as per the statistics act;

Confidentiality
The Statistics Act (in most countries) mandates that government data and information should be open, readily available, well managed, reusable unless there are necessary reasons for its protection;
Statistical Agencies must be accountable and are expected to balance the principles of transparency – maximizing the availability of disaggregated data – confidentiality, and privacy-ensuring personal data is not abused, misused, or putting anyone at risk of identification or discrimination, in accordance with national laws and the Fundamental Principles of Official Statistics;
Government businesses, institutions, and organizations rely on high-quality, timely, and accurate data for planning, research, and information. Good data helps country to grow and prosper through informed programming;
Statistical Agencies are expected to make official statistics available on an impartial basis to honour citizens’ entitlement to public information—Principle 1;

Confidentiality
The terms privacy, security, and confidentiality are often used interchangeably, but each term has a different meaning:
Privacy-refers to a person’s ability to control the availability of data about themselves;
Security-refers to how an organization stores and controls access to the data it holds;
Confidentiality-refers to the protection of data from, and about, individuals and organizations; and how we ensure that data is not made available or disclosed without authorization;

Confidentiality
Statistical Agencies are expected to have policies in place on confidentiality before sharing of census microdata;
The policy should have the following components: description; why is it good practice(provide assurance to the DGs/CEO); target audience; detailed description (data confidentiality committee, the system of rules and procedures regarding the release of microdata to eligible users, rules for the release of anonymized microdata, managing breaches by researchers); supporting legislation; strengths, weakness;
The developed data sharing policies should provide for NSOs to make their findings available for re-examination, and advocate for willingness to improve their data products;
There is a need for NSOs to develop particular protocol for disseminating census microdata that would spell out the necessary requirements and components of such dissemination;
Data and information held by government should be open for public access unless grounds for refusal or limitations exist under the Official Information Act or other government policy. In such cases they should be protected—public good;
Ref. OECD Principles and Guidelines for Access to Research Data from Public Funding, Organization for Economic Co-operation and Development (2007).

Confidentiality
Before sharing of microdata, NSOs should take into considerations the following principles for managing the confidentiality of microdata:
Principle 1: Appropriate use of microdata: It is appropriate for microdata collected for official statistical purposes to be used for statistical analysis to support research as long as confidentiality is protected– There may be other concerns (for example, quality) that make it inappropriate to provide access to microdata;
Principle 2: Microdata should only be made available for statistical purposes– If the use of the microdata is incompatible with statistical or analytical purposes, then microdata access should not be provided. Like in the case of administrative use, the aim is to derive information about a particular person or legal entity to make a decision that may bring benefit or harm to the individual;
Principle 3: Provision of microdata should be consistent with legal and other necessary arrangements that ensure that confidentiality of the released microdata is protected— legal arrangements to protect confidentiality should be in place before any microdata are released. In some countries, authorizing legislation does not exist. At a minimum, release of microdata should be supported by some form of authority;
Principle 4: The procedures for researcher access to microdata, as well as the uses and users of microdata, should be transparent and publicly available.– It is important to increase public confidence that microdata are being used appropriately and to show that decisions about microdata release are taken on an objective basis. It is up to the NSO to decide whether, how and to whom microdata can be released. But their decisions should be transparent;
Reference: https://unece.org/fileadmin/DAM/stats/publications/Managing.statistical.confidentiality.and.microdata.access.pdf

Confidentiality
Compliance with data protection legislation should not prevent census microdata sharing;
It is important to have legislation supporting sharing of census microdata as highlighted in Principle 3. This helps:
To provide public confidence in the arrangements – that there are legal constraints that determine what can and cannot be done;
To provide mutual understanding between NSOs and researchers on the arrangements;
To provide for greater consistency in the way research proposals are treated; and
To provide a basis for dealing with breaches.
the legislation should guarantee:
the privacy of data providers (households, businesses, or other respondents) and the confidentiality of the information they provide
the security of information received from data providers
the use of the data for statistical purposes
penalties and sanctions in case of breach.
Ref:https://unece.org/fileadmin/DAM/stats/publications/Managing.statistical.confidentiality.and.microdata.access.pdf

How to ensure confidentiality
1. Technical arrangements
Remote Access Facilities (RAFs)-The key characteristic is that users do not have access to the microdata itself but tasks using that microdata can be submitted remotely over the Internet.
Data laboratories /Safe sites:-Effective in controlling identification risk whilst enabling users access, particularly for data sets where release of a confidential microdata file is not possible. Lack of convenience to the user, including sometimes being forced to use unfamiliar data analysis software. They are also expensive for the NSO to manage.
2. Anonymization techniques-Removing or modifying the identifying variables contained in the micro dataset
Anonymizing the data consists determining which variables are potential identifiers (this relies on one's personal judgment), and in modifying the level of precision of these variables to reduce the risk of re-identification to an acceptable level.
Anonymization can mainly be achieved through:
data reduction: Removing records, Global recoding, Top and bottom coding, etc.
data perturbation: Micro-aggregation, Data swapping, Post-randomization (PRAM), Adding noise, and Resampling: .
Synthetic microdata: Synthetic microdata are an alternative approach to data protection, and are produced by using data simulation algorithms. Users are not keen to work with synthetic data as they cannot be confident of the results of their statistical analysis;
3. Review your confidentiality methods regularly

Ref: Anonymized Microdata Files, Eurostat; IPUMS-International Statistical Disclosure Controls: 159 Census Microdata Samples in Dissémination

Anonymization tools
International Household Survey Network (IHSN) has developed tools and guidelines on microdata anonymization. These tools are developed as Stata, SPSS and/or SAS programs;
A free software for data swapping, the Data Swapping Toolkit (DSTK), is also available. The DSTK is a comprehensive software package for performing and analyzing data swapping on categorical data.
Several other software are also used in specialized areas like health, retailers, ecommerce, etc.

Census Microdata
The term microdata can refer to data about an individual, person, household, business or other entity;
Microdata may be data collected through surveys, censuses or obtained from administrative records;
Microdata are usually of significance yet access to countries’ microdata is often limited by the fear that confidentiality protection cannot be guaranteed;
Census data has identification variables linked to location and identification of respondents and needs to be anonymized before dissemination;
When a census database is constructed maintenance of confidentiality and the protection of individual privacy must be a primary consideration (P&R PHC Rv 3, para. 3.286)-identifiers should be removed from the database;

Census Microdata
Official statistics are collected not just for governments but for use by the research community (researchers, academia, NGOs, international agencies) that will need microdata to enable them undertake further analysis that can help in making informed decisions;
Statistical Agencies have an obligation to support research and generally they could do more to satisfy these needs by providing access to microdata;
Publicly-funded data producing agencies have a mandate to make the data they collect as broadly available as practicable—public good;
Before dissemination of census microdata the data should be anonymized by removing all direct identifiers by applying statistical disclosure control techniques. Ref. http://ihsn.org/sites/default/files/resources/IHSN-WP005.pdf;
There is need for NSOs to respond to the pressure to move towards open data, that is, to make more data available free of charge, free of licensing restrictions, and in machine-readable formats that increase data use— https://opendatacharter.net/wp-content/uploads/2015/10/opendatacharter-charter_F.pdf;

Why share census microdata
Dissemination is one of the key responsibilities of a statistical agency;
Supporting research community work: access to census microdata assists and encourages informed decision making through enabling wider use of census data for social and economic research analysis;
Enhancing the credibility of official statistics-broader access to microdata demonstrates producers’ confidence in the data by making possible their replication or correction by independent parties;
Improves the reliability and relevance of census data-user feedback result in improvement of methodology over time;
Reduces duplication in data collection. This reduces the burden to respondents and minimizes the risk of inconsistent studies of the same topic;
Increasing return on investment though wider use of the data;
Leveraging funding for statistics –the more data files are disseminated and used the more valuable they will appear to funding bodies;
Reduces the cost of census data dissemination;
Complying with contractual and legal obligations;
Promoting development of new tools for using data like open-data (data collected using public funds or under the auspices of public agency are a “public good” )
Allows the aggregated products to be checked and confirmed-which builds trust in government and government data;
Disseminating microdata to a wide audience can enhance the quality of official data through the re-use cycle. “Data accessibility is the ultimate benchmark of data quality (IMF, 2008)”

Census Metadata
“A crucial part of creating a good dataset with long-lasting usability is ensuring that the data are easy to understand and analyze. This requires accompanying data description and documentation that is user-friendly, clear and detailed, yet comprehensive.” (http://www.data archive.ac.uk);

If users are to make effective use of microdata, they must have access to the appropriate metadata;

“In order to assist data users to better understand and interpret the data, it is important that there is adequate documentation providing a complete and clear description of the production process, including data sources, concepts, definitions and methods used” ( P&R PHC. prg 3.290);

“Producing good data documentation is easiest when planned from the start of a project and considered throughout the course of research (during the data lifecycle). Advanced planning can significantly reduce the time and money needed to prepare documentation.” (http://www.data archive.ac.uk);
A good census metadata should have:
A description of the survey including any information on quality: this ensures that the microdata are not used if the data are not fit for the intended purpose and provides an opportunity to users to assess the quality of data;
Definitions of the data items, information about dataset structure,
Technical information (computer system used to generate the files, software packages used, etc;
Description of all variables and values, coding and classification schemes, information about derived variables, confidentiality and anonymization
Data errors, problems encountered etc.
Methodological note on the rules and methods of suppression
A list of the data items and the classifications used (sometimes referred to as a data dictionary);
Good documentation reduces the amount of user support statistical staff must offer external users of their microdata and helps build the institutional memory of data collection and can assist in training new staff and improving data consistency overtime;

Challenges facing dissemination of census micro-data in Africa
legal obstacles (in many countries the statistical legislation still in use is out dated and does not recognise the dissemination of electronic data particularly microdata)-No sufficient authority to support researchers access to census microdata;
technical and financial obstacles including in-house capacity to handle the complex aspects of census microdata dissemination such as data anonymization and curation of microdata to ensure its long-term availability;
psychological obstacles – the tendency to control access perhaps because of concerns over its mis-interpretation or because ‘data is power’;
Some NSOs are concerned that the quality of their microdata may not be good enough for further dissemination–Fear of inconsistencies between census results based on microdata and published aggregate data;
Inadequate resources to support: undertaking of documentation of microdata files, meet the costs of creating access tools and safeguards, and of supporting and authorizing enquiries made by the research community; helping data users of microdata files to navigate complex file structures and variable definitions;
Technological challenges e.g. no adequate computers, no website/out dated website/websites are not optimally functional, no/outdated microdata dissemination platforms, lack of training in creating metadata records for microdata, no capacity to administer existing microdata dissemination platforms;
Bureaucratic nature of government institutions involved in data production;

Challenges facing dissemination of census micro-data in Africa
Government and donor expenditure is allocated mainly for data collection, and very little funding is provided for the long-term preservation and sharing of micro-data;
low levels of literacy, low trust in quality of public micro-data, and lack of infrastructure to access and use the data;
Many trained IT staff are lured away from the government sector to better paid private sector employment;
Skills constraints: technical skills, microdata archiving skills, knowledge of anonymization techniques;
Some NSOs commercialize micro-data making it difficult for most stakeholders to access;
Inadequate bandwidth allocation to Statistical Agencies by government hampering data dissemination

Ref:https://www.researchgate.net/publication/281969484_Leveraging_data_in_African_countries_Curating_government_microdata_for_research

Going forward
NSOs should mobilize for long-term financing, investments in human capital, and laws conducive to the safe production, exchange, and use of micro-data;
Policy decisions should be made at Ministerial level to support the placing of national data in the public domain to make it easier for National Statistical Agencies to be able to share the microdata through public domains;
NSOs should ensure easy and free optimal data access, or at least not prohibit microdata access. Data access policies which place unreasonable demands on researchers need to be adjusted to make data sharing viable;
NSOs should avoid charging for microdata as this is not beneficial to the growth of empirical research in Africa. Commercialization of microdata reduces considerably the number of potential users and hence the real value of the data;
Micro-data should be easily discoverable and accessible, and made available without bureaucratic or administrative barriers, which can prevent people from accessing the data– https://opendatacharter.net/wp-content/uploads/2015/10/opendatacharter-charter_F.pdf ;
NSOs should release data in open formats to ensure that the data is available to the widest range of users to find, access, and use. In many cases, this will include providing data in multiple, standardized formats, so that it can be processed by computers and used by people; Try to release data free of charge, under an open and unrestrictive license;
NSOS should put in place proper technological infrastructure for the various components of microdata archiving i.e. data documentation, cataloguing and dissemination (users need to be properly informed about the existence and characteristics of the datasets made available and should be researchable online), anonymization and preservation etc.
Currently, donor agencies have begun to provide funding support for micro-data reporting for further research and this has assisted African governments to improve their data resources for better national planning;

18
18

Points of discussion
Does you NSO have microdata dissemination policy? If yes, does it cover dissemination of census microdata?;
Does your NSO have data dissemination platforms?, If Yes, How many data dissemination platforms does your NSO have for sharing microdata? Which ones are dedicated for dissemination of census microdata?
If No, what plans has your NSOs put in place to disseminate census microdata?
Do you have mechanisms of clarifying to the users which platforms are for aggregated data and which ones are for microdata?
What are the challenges that your NSO face while sharing census microdata?
What solutions do you suggest for your NSO?