Background
The Administration for Digital Industries, Ministry of Digital Affairs, with the development of Taiwan's domestic AI industry as its blueprint, has cooperated with the National Institute of Cyber Security (NICS) and the Industrial Technology Research Institute (ITRI) to establish the Artificial Intelligence Evaluation Center (AIEC). It has formulated AI evaluation-related policies, standards, and regulations to promote the development of Taiwan's domestic AI industry and expand business opportunities at home and abroad.
Promoting AI Evaluation
With the rapid development of the field of artificial intelligence (AI), AI technologies are increasingly being applied to a wide range of products. In order to promote AI development and ensure the correct use of AI technologies, Taiwan needs to establish a domestic AI products and systems evaluation system. To this end, the AIEC will formulate " Draft Systems for Evaluating AI Products and Systems " and "Draft Guidelines for Evaluating AI Products and Systems" to provide AI product evaluation services and evaluation criteria, and promote the effectiveness and safety of domestic AI products.
Our Goals
AIEC aims to establish a domestic AI products and systems evaluation system and provide AI evaluation services. We have analyzed international AI standards, including NIST, ISO, and the European Parliament, and plan to establish the following 10 evaluation items:
- Safety: AI systems should not under defined conditions, lead to a state in which human life, health, property, or the environment is endangered.
- Explainability: A representation of the mechanisms underlying AI systems’ operation and the meaning of AI systems’ output in the context of their designed functional purposes.
- Resiliency: AI systems can maintain their functions and structure in the face of internal and external change and degrade safely and gracefully when this is necessary.
- Fairness: Concerns for equality and equity by addressing issues such as harmful bias and discrimination.
- Accuracy: Closeness of results of observations, computations, or estimates to the true values or the values accepted as being true.
- Transparency: The extent to which information about an AI system and its outputs is available to individuals interacting with such a system.
- Accountability: Ensure that the person responsible for the AI system can trace and explain its decisions and actions, and bear corresponding responsibilities and consequences.
- Reliability: Ability of AI systems to perform as required, without failure, for a given time interval, under given conditions.
- Privacy: The norms and practices that help to safeguard human autonomy, identity, and dignity in AI systems.
- Security: AI systems that can maintain confidentiality, integrity, and availability through protection mechanisms that prevent unauthorized access and use.
AI Evaluation System
To provide AI product evaluation services, the AIEC will establish a national AI evaluation system. The AI evaluation system consists of the following three entities:
- AIEC: Developing AI Evaluation Syetem and Methods.
- AI Evaluation Institution: Executor of AI evaluation system, providing evaluation reports and managing the evaluation reports.
- AI Testing Laboratory: Testing product and providing test report.
| Entity | AIEC | AI Evaluation Institution | AI Testing Laboratory |
|---|---|---|---|
| Tasks and Responsibilities |
|
|
|
AIEC Structure
The AIEC is composed of Institutional Promotion Committee and Technical Advisory Group.
緣起
數位發展部數位產業署以發展國內AI產業為規劃藍圖，與國家資通安全研究院、工業技術研究院合作設立AI產品與系統評測中心，制定AI評測相關制度、標準及評測體系，旨在推動台灣國內AI產業的發展，拓展國內外商機。
AI評測推動
隨著人工智慧(Artificial Intelligence, AI)領域快速發展，AI領域的各項技術逐漸應用於各式產品之中，而AI技術所帶來的利益與威脅也漸漸被各界重視。我國為推動AI發展與確保AI技術能被正確使用，亦須建立國內AI產品與系統之評測制度。為此，AI產品與系統評測中心將制定「AI產品與系統評測制度」與「AI產品與系統評測指引」，提供AI產品評測服務與評測基準，推動國內AI產品效能與安全。
AI評測中心目標
AI評測中心之目標為建立國內AI產品與系統評測體系，為國內的AI產品與系統提供評測服務。AI評測中心研析國際AI規範內容，包含NIST、ISO及歐盟等，擬建立下列10項評測項目。
- 安全性(Safety) : 安全性是指AI系統本身如果發生某些功能失效的狀況下，所需要評估的風險評估與回應措施，一般安全性的評估，往往透過法規規範相關檢測條件與測試的場域，如工廠或是道路上的場域驗證，確保AI系統的運作不會對人類、環境或資產造成傷害或損害。
- 可解釋性(Explainability) : 指對於AI模型的輸入與輸出的關係，是否能到找到因果關係或是關係的描述性呈現，解釋其決策和行為的原因和邏輯，如果AI系統具備這樣的特性，便能更容易實現除錯或是監控的功能，並且能夠向使用者和利益相關者提供透明且可理解的解釋。
- 韌性(Resiliency) : 指AI系統能夠適應不同的環境、需求和條件。彈性強調系統能夠靈活調整和擴展，以滿足不斷變化的需求和挑戰。
- 公平性(Fairness) : 指AI系統在對待不同群體和個體時能夠公正和平等，要求系統避免偏見、歧視或不公正對待，並確保公平的機會和結果，確保個人或特定族群不受到歧視與偏見之侵害。如：種族、性別、政治傾向、身體/精神殘疾等。
- 準確性(Accuracy):衡量AI系統輸出結果與真實結果之間的接近程度，可透過計算AI模型本身是否能反應出根據資料所呈現的關係，其包含了評估指標的選取與模型訓練當中如何減少發生低度擬合(模型準確率低、測試結果準確率低，意即完全不準)或是過度擬合(模型準確率高、測試結果準確率低，意即只對訓練資料集有效)的情況。
- 透明性(Transparency) : 透明度旨在糾正AI系統運營商和消費者之間普遍存在的資訊不平衡，避免使用者因為對於設計目的和訓練資料、模型架構等資訊的不足而做出不可靠的假設或運用，並且可以據此做出對應的補救措施等，但透明度不代表AI系統是公平或安全的。
- 當責性(Accountability): AI系統開發者和使用者需對系統的行為或操作負責，可問責性強調確保系統的負責方能夠追溯和解釋系統的決策和行為，並承擔相應的責任和後果，並建立組織實踐與治理的架構來持續減少可能的傷害，如利用風險管理等來協助達成更負責任的系統。
- 可靠性(Reliability) : 可靠性是指評量模型敏感度的指標，面對不同類型的干擾、噪音或異常情況時，模型仍可以保有最小化的敏感變異，意即系統在面對未預期的狀況時能夠維持良好的表現和預測能力。
- 隱私(Privacy) : 隱私是指個人免於被入侵或是透過有限的觀察而獲得個體的事實(如：身體、資料與信用)。然而在AI模型建立過程中，因為需要讀取資料進行訓練與分析的狀況，這樣的狀況往往存在可能的隱私議題，因此須將可能造成隱私的衝擊嚴重程度分級，以便做到風險評估與掌控。
- 資安(Security) : 指AI系統在面對外部攻擊、未授權訪問或不當使用時能夠保護其資源、功能和資料的完整性和機密性，並要求系統能夠有效地防止和應對安全威脅和對抗攻擊，以確保系統的正常運行而不影響其整體表現。
AI評測體系
為提供國內AI產品與系統評測服務，AI評測中心將規劃並建立國內AI評測體系，AI評測體系擬由以下三個角色組成
- AI評測中心：AI評測制度與方法制定
- AI驗證機構：AI評測制度之執行單位，管理評價報告之核發與使用
- AI測試實驗室：AI評測中心認可之測試報告產出者
| 單位 | AI評測中心 | AI驗證機構 | AI測試實驗室 |
|---|---|---|---|
| 任務與職責 |
|
|
|
AI評測中心組織
AI評測中心由制度推動委員會與技術審議小組組成
