Metrum AI Bench Privacy Policy

This Privacy Policy explains how Metrum AI, Inc. (“Metrum AI”, “we”, “us”, or “our”) collects, uses, shares, and protects personal data when you use the Metrum AI Bench platform, the Metrum agent, and Metrum AI command-line tools that connect to the platform, together the “Service”.

The Service is intended for professional and business use. Self-service signup and purchase are not currently offered to customers established in the European Economic Area or the United Kingdom.

1. Who we are

Metrum AI, Inc., 1717 N I-35, Suite 300, Round Rock, Texas 78664, United States, is the controller of the personal data described in this Privacy Policy except where Section 2 states otherwise.

You can contact our privacy team at privacy@metrum.ai.

We have not appointed a data protection officer under the GDPR because our current processing does not require one. Where applicable law requires a designated privacy contact, our privacy team performs that role.

2. Our role for Customer Content

When an organization uses the Service, that organization decides which workloads, datasets, prompts, endpoints, machines, run configurations, and other content to place in its workspaces and which benchmarks to run.

We call the following information Customer Content:

• workloads, datasets, and prompts;
• model and endpoint responses;
• judge outputs, scores, and reasoning;
• model, dataset, and endpoint identifiers;
• machine registrations and run configurations;
• benchmark results and performance results;
• run logs; and
• other information that a customer or its users submit to or generate through their workspace.

For Customer Content, the customer organization is normally the controller and Metrum AI acts as its processor under our Data Processing Addendum.

Metrum AI is an independent controller for account, billing, security, fraud-prevention, and Service-operations data described in this Privacy Policy.

If you use the Service through an organization and contact us about personal data contained in that organization’s Customer Content, we may refer the request to the organization and assist it in responding.

If you use an individual account, you are responsible for the Customer Content you submit.

3. Who this policy covers

This Privacy Policy applies to:

• people who create an account or are invited to a workspace;
• people who use the Service, agent, or connected command-line tools;
• people who read run logs;
• people who contact support or receive operational email from the Service; and
• people whose personal data appears in Customer Content.

Where personal data about another person is included in Customer Content, the customer that supplied the data is responsible for having an appropriate basis to process it and for providing any notice required by applicable law.

The Service is for professional use and is not directed to children under 16. We do not knowingly collect personal data from children through the Service.

4. Personal data we collect

4.1 Account and identity information

We may collect:

• your email address, display name, and user ID from Auth0 when you sign in;
• the workspaces to which you belong and your role in each workspace; and
• where an organization enables another identity provider, such as Google or GitHub, the account information that provider supplies to us.

If a colleague invites you to a workspace, we receive the email address they provide and use it to deliver the invitation.

An email address is required to create an account. Other information depends on how you use the Service.

4.2 Billing information

For paid plans, we may collect:

• billing email address;
• organization and plan information;
• billing address;
• tax information; and
• invoices and payment status.

Payment card information is entered into Stripe’s payment forms. Metrum AI does not receive or store full payment-card numbers.

4.3 Benchmark content

When you run a benchmark, the Service may process:

• prompts and datasets used by the workload;
• model or endpoint responses;
• judge-model inputs, outputs, scores, and reasoning;
• model, dataset, and endpoint identifiers;
• latency, throughput, power, GPU utilization, and other performance measurements;
• hardware and software configuration associated with the run; and
• run logs produced by model servers and benchmark tools.

Customer Content can contain information selected or supplied by a customer. Customers should not place personal data, confidential information, or sensitive data into prompts or datasets unless it is required for the intended benchmark and they are authorized to process it.

4.4 Usage, activity, and Service telemetry

We collect information needed to operate the Service, including:

• projects, workloads, and runs created in a workspace;
• cloud-machine requests, start and stop times, and associated costs;
• workspace activity records, such as membership and configuration changes;
• usage counts required for billing;
• error rates;
• availability and uptime information; and
• similar operational measurements needed to operate, secure, and troubleshoot the Service.

For purposes of this Privacy Policy, Service telemetry does not include Customer Content such as prompts, model responses, benchmark performance results, run configurations, or run logs.

4.5 Service logs

Our systems and identity provider may record:

• IP address;
• browser and device information;
• request timestamps;
• requested pages and API endpoints;
• authentication events; and
• application or API errors.

When someone reads a run log, we also record information about that access, including the IP address and browser type.

4.6 Machine and agent data

When a machine is registered with the Service, the Metrum agent may send:

• hostname;
• machine serial number;
• network-interface MAC addresses;
• CPU, GPU, memory, storage, and network hardware information;
• operating-system, driver, and runtime versions;
• network addresses;
• GPU, CPU, memory, power, temperature, and network telemetry collected during a run; and
• run logs required by the Service.

The agent communicates with the Service over HTTPS and authenticates using a token issued to the registered machine.

Before run logs are uploaded, we apply redaction for common secret patterns such as API keys and bearer tokens. Customers should nevertheless avoid intentionally placing credentials or other secrets in logs.

When a Metrum command-line tool connects to the Service, information needed to perform the requested operation is processed under the categories described in this Privacy Policy.

4.7 Credentials

Depending on how you configure the Service, we may store:

• cloud-provider credentials;
• Hugging Face tokens;
• model-endpoint API keys;
• judge-model API keys; and
• personal API tokens created for the Service.

Cloud-provider credentials, Hugging Face tokens, model-endpoint keys, and judge-model keys are encrypted before storage using AES-256-GCM.

Personal API tokens are stored only as hashes.

4.8 Support requests

When you contact support through the Service, we receive the message and support category.

The support dialog may attach a screenshot of the page you are viewing. A screenshot can contain Customer Content or other information displayed on that page. The dialog identifies that a screenshot is attached and allows you to remove it before submission.

Support requests and attached screenshots are delivered to our support inbox. Screenshots are not stored separately in the Service’s application database.

5. How we use personal data

Providing the Service

This includes creating and administering accounts and workspaces, running benchmarks, provisioning machines, displaying results, sending operational messages, and providing requested functionality. Where GDPR or UK GDPR applies, the legal basis is contract performance for individual accounts and our legitimate interest in providing the Service for users who access it through an organization.

Support

We process support requests to investigate and respond to problems. Where applicable, the legal basis is contract performance or our legitimate interest in supporting users of organizational accounts.

Billing and legal records

We process billing information to administer paid plans and issue invoices. Where applicable, contract performance is the legal basis for providing paid services, and legal obligation is the basis for records that tax, accounting, or other laws require us to retain.

Security and fraud prevention

We use authentication information, activity records, Service logs, and similar operational data to detect unauthorized access, investigate suspicious activity, protect accounts, and secure the Service. Where applicable, the legal basis is our legitimate interest in protecting the Service, our customers, and our systems.

Reliability and Service operation

We use Service telemetry, error reports, and operational logs to diagnose failures, maintain availability, and improve the reliability and security of the Service. Where applicable, the legal basis is our legitimate interest in operating a reliable and secure Service.

Customer Content is not used for Metrum public benchmarks

Metrum AI does not use Customer Content, including de-identified or aggregated customer performance results or run configurations, to create public Metrum leaderboards, published benchmarks, marketing materials, or public performance comparisons.

Customers may create leaderboards from their own runs inside their workspace. Those leaderboards remain Customer Content and are private by default. A customer may choose to share or publish its own results.

Any public Metrum AI leaderboard or benchmark uses runs executed by Metrum AI itself, with partner approval where partner hardware or services are involved.

No AI training on Customer Content

Unless a customer expressly agrees otherwise in writing, we do not use Customer Content, including prompts, datasets, model responses, judge outputs, run logs, performance results, or run configurations, to train or fine-tune AI models.

We also contractually prohibit subprocessors that process Customer Content on our behalf from using that Customer Content to train or fine-tune their models for their own purposes.

Using logs or other Customer Content to provide the Service, respond to a support request, investigate an incident, or debug a customer’s run is not AI model training.

Other restrictions

We do not:

• sell personal data;
• share personal data for cross-context behavioral advertising;
• use Customer Content for advertising;
• send marketing email through the Service today; or
• make decisions about individuals based solely on automated processing that produce legal or similarly significant effects.

If we introduce marketing email, we will provide any notice required by law and an unsubscribe mechanism.

We do not currently rely on consent as the primary legal basis for the processing described in this Privacy Policy. If we request consent for a new activity, you may withdraw it as provided by applicable law.

6. How we share personal data

We use service providers to operate the Service. The providers currently include:

Stripe may process payment information as an independent controller for its own fraud-prevention, legal, and compliance purposes. Stripe’s own privacy policy applies to that processing.

The processing region for a Metrum-provided cloud machine depends on the region selected for the run.

We may also disclose personal data:

• to members of your organization according to workspace permissions;
• where required by applicable law, legal process, or a valid governmental request;
• where reasonably necessary to protect the rights, security, property, or safety of Metrum AI, our users, or others;
• to lawyers, auditors, accountants, insurers, and other professional advisers subject to appropriate confidentiality obligations; or
• as part of a merger, financing, acquisition, reorganization, bankruptcy, or sale of assets, subject to applicable legal requirements.

We maintain contractual protections for service providers that process personal data on our behalf as required by applicable law and our Data Processing Addendum.

7. Data sent to providers selected by customers

Some providers are selected and controlled by the customer rather than Metrum AI.

If you connect your own AWS, Google Cloud, Microsoft Azure, Lambda, DataCrunch, RunPod, Shadeform, or another cloud account, that provider operates under your agreement with it.

If you add your own model endpoint or judge model, the Service sends that provider the prompts, model responses, credentials, or other information required to perform the requested evaluation.

If a run downloads models or datasets from Hugging Face using a token you provide, Hugging Face receives the token and the identifiers of the requested resources under your relationship with Hugging Face.

BYOC and customer-controlled machines

For bring-your-own-cloud or customer-controlled machines, model execution takes place in the environment selected by the customer.

The Service may send the prompts, datasets, workload settings, and credentials required to perform the run to that environment. The Service may receive the run results, performance measurements, telemetry, status information, and run logs required to administer the run and display its results.

Information that remains solely inside the customer’s environment and is never transmitted to Metrum AI is not collected by Metrum AI.

8. International processing and transfers

Metrum AI is based in the United States.

The Service is hosted on Amazon Web Services in the United States, with primary production infrastructure in us-east-1, Northern Virginia.

Metrum AI personnel in the United States and India may access data when required to operate, secure, and support the Service. Production access is role-limited, protected by multi-factor authentication, and logged.

When a benchmark runs on a cloud machine, the machine and the data processed on it are located in the region selected for that run.

Metrum AI does not currently offer self-service signup or checkout to customers established in the European Economic Area or the United Kingdom.

An organization outside those regions may nevertheless have users or Customer Content originating there. Where European, UK, Swiss, or other data-transfer law applies to Customer Personal Data, the Metrum AI Data Processing Addendum provides the applicable contractual transfer mechanism.

For transfers governed by European data-protection law, this may include the European Commission’s Standard Contractual Clauses. For UK transfers, it may include the UK International Data Transfer Addendum.

Where a service provider participates in the EU-US Data Privacy Framework or its applicable UK or Swiss extension, we may rely on that framework for transfers to that provider and maintain contractual safeguards where appropriate.

You may contact privacy@metrum.ai for information about the safeguards applicable to a particular transfer.

9. How long we keep data

When a member leaves a workspace, runs remain with the organization. We remove the departing member’s name and email address from those runs within 30 days where they are not required for security or legal records.

We do not restore deleted data into normal production use except where necessary to recover from an incident.

10. Your privacy rights

Where applicable law gives you privacy rights, you may ask us to:

• provide access to personal data we hold about you;
• provide a portable copy where the law requires it;
• correct inaccurate personal data;
• delete personal data;
• restrict certain processing; or
• stop or object to certain processing.

We do not discriminate against you for exercising a privacy right.

If the request concerns Customer Content controlled by an organization, we may refer the request to that organization and assist it with responding.

How to make a request

Email privacy@metrum.ai.

If you have a Metrum account, contact us from the email address associated with the account where possible.

If you do not have an account, explain how you believe your personal data reached us. We may ask for information reasonably necessary to verify your identity and authority.

We respond within the period required by applicable law. Where GDPR or UK GDPR applies, this is ordinarily one month, subject to any extension permitted by law.

If we decline a request, we will explain why and provide information about any applicable appeal or complaint mechanism.

European Economic Area and United Kingdom

Where GDPR or UK GDPR applies to the processing, applicable rights may include access, rectification, erasure, restriction, portability, and objection. You may complain to the data-protection authority responsible for you. In the United Kingdom, this is the Information Commissioner’s Office.

Switzerland

Where the Swiss Federal Act on Data Protection applies, you have the rights provided by that law and may contact the Federal Data Protection and Information Commissioner.

India

Where the Digital Personal Data Protection Act, 2023 and its implementing rules apply, you may exercise the rights provided by that law through privacy@metrum.ai and use the applicable grievance process.

United States

Where a US state privacy law applies, you may have rights to access, know, correct, delete, or obtain a copy of personal data and to appeal certain decisions.

Metrum AI does not sell personal data or share it for cross-context behavioral advertising.

Credentials used to access accounts or infrastructure are used only as necessary to provide, secure, and support the Service.

Other jurisdictions

If another privacy law applies to you, you may exercise the rights provided by that law by contacting privacy@metrum.ai.

11. How we protect data

Our technical and organizational security measures include:

• HTTPS and TLS encryption between browsers and the Service and between the Metrum agent and the Service;
• AES-256-GCM encryption of stored cloud-provider credentials, Hugging Face tokens, model-endpoint keys, and judge-model API keys;
• hashed storage of personal API tokens;
• encryption of databases, storage volumes, and backups;
• redaction of common secret patterns from run logs before agent upload;
• role-based access control for organizations, workspaces, and projects;
• role-limited production access for authorized personnel;
• multi-factor authentication for production access;
• logging of production access and relevant workspace activity;
• machine-specific authentication tokens for registered agents;
• regular backups; and
• documented security-incident response procedures.

No method of transmission or storage can provide absolute security. We maintain safeguards appropriate to the nature of the Service and the data processed.

If a security incident affects personal data, we provide legally required notifications. For Customer Personal Data that we process on behalf of an organization, we notify that organization in accordance with our Data Processing Addendum.

12. Cookies and local storage

The Service uses cookies and browser storage necessary to provide its functionality, including:

• session cookies used to keep you signed in;
• cookies used by Auth0 for authentication;
• cookies used by Stripe on checkout or payment pages for payment security and fraud prevention; and
• local storage used for interface preferences such as theme or command-palette history.

We do not currently use analytics, advertising, or behavioral-tracking cookies in the Service.

13. Changes to this Privacy Policy

We may update this Privacy Policy as the Service or applicable law changes.

We will publish the updated version and change the “Last updated” date.

For a material change, we will notify affected users by email or through the Service at least 30 days before the change takes effect where reasonably practicable or where required by law.

We will not apply a materially different use of personal data already collected where applicable law requires additional notice or consent without first providing that notice or obtaining that consent.

We maintain dated copies of previous versions and will provide them on request.

14. Contact us