Add Files API v1 and Data Sources API support

README.md

@@ -394,85 +394,166 @@ Deletes an app token.
 
 - `app_token`: The app token to delete.
 
-##### `MathpixClient.scs_file_new`
+#### Files API (async document processing)
 
-Upload a file via files-api v1 for async processing. Returns an ScsFile instance.
+The Files API processes documents asynchronously — local files, or remote URIs (`s3://`, `gs://`, public `https://`, or Azure Blob HTTPS URLs) one at a time or in bulk batch calls. Non-public sources require a registered [data source](https://docs.mathpix.com/reference/files-v1-data-sources) connecting your Mathpix account to the bucket; register it once via the API following the linked guide.
 
-Supports three upload modes (exactly one must be provided):
+Submit a single document and download the result:
 
-- `file_path`: Multipart upload from local file
-- `url`: Upload from HTTP URL or S3 presigned URL
-- `source_s3_uri`: Copy from S3 bucket (requires IAM role access)
+```python
+from mpxpy.mathpix_client import MathpixClient
+
+client = MathpixClient()
+file = client.file_new(
+    source_uri="https://cdn.mathpix.com/examples/cs229-notes1.pdf",
+    conversion_formats={"docx": True, "md": True},
+)
+file.wait_until_complete(timeout=120)
+markdown = file.to_md_text()
+file.to_docx_file("output.docx")
+```
+
+Submit a batch as a job, then collect results and failures:
+
+```python
+job = client.file_job_new(
+    files=[
+        {"source_uri": "s3://your-bucket/docs/contract-1.pdf", "custom_id": "contract-1"},
+        {"source_uri": "https://example.com/manual.pdf", "custom_id": "manual"},
+    ],
+    job_id="contracts-2026-07",
+    conversion_formats={"docx": True, "md": True},
+)
+job.wait_until_complete(timeout=3600, interval=15.0)
+for errored in job.files_iter(status="error"):
+    print("failed:", errored["custom_id"])
+file = job.file_by_custom_id("contract-1")
+```
+
+Batch submission is accept-and-defer: the call returns immediately and per-item failures (bad URIs, missing data sources) surface as per-file `error` statuses when you poll the job, not as request errors.
 
-###### `MathpixClient.scs_file_new` Arguments
+##### `MathpixClient.file_new`
 
+Submit a single document for async processing, from a remote URI (`POST /files/v1/uri`) or a local file (multipart upload). Returns a `File` instance.
+
+###### `MathpixClient.file_new` Arguments
+
+- `source_uri`: Remote location of the source document (`s3://`, `gs://`, public `https://`, or Azure Blob HTTPS URL). Exactly one of `source_uri` or `file_path` is required.
 - `file_path`: Path to a local file to upload.
-- `url`: URL of a remote file (HTTP/HTTPS or S3 presigned URL).
-- `source_s3_uri`: S3 URI (s3://bucket/key) to copy from.
-- `filename`: Optional filename to use (defaults to file basename).
-- `scs_job_id`: Optional job ID to group files together.
-- `conversion_formats`: Dict of format names to enable (e.g., `{'mmd': True, 'docx': True}`).
-- `conversion_options`: Additional conversion options dict.
-- `destination_s3_uri`: Optional S3 URI to write output files.
-- `destination_basename`: Optional basename for output files (defaults to file_id).
-- `s3_region`: Optional AWS region for S3 operations (default us-east-1).
-- `image_output_mode`: Image output mode (e.g., 'local' to upload to destination_s3_uri).
-- `include_page_info`: Include page info in output (default None).
+- `job_id`: Optional job to associate this file with. Required whenever `custom_id` is supplied.
+- `custom_id`: Optional case-sensitive customer-supplied identifier. `(job_id, custom_id)` is the idempotency key: re-submitting the same pair returns the original file.
+- `idempotency_key`: Optional client-generated key sent as the `Idempotency-Key` header; makes a standalone submission safe to retry.
+- `filename`: Optional display name for the file.
+- `conversion_formats`: Dict of format names to enable (e.g., `{'docx': True, 'md': True}`). Mathpix Markdown (`mmd`) is always produced.
+- `extra_options`: Additional request options dict merged into the request body — an escape hatch for API options this SDK version does not model yet (validated server-side). May not override the validated request fields.
+- `destination_uri`: Optional destination for results; must be backed by a registered data source. When omitted, results stay in Mathpix storage and are fetched via the download helpers.
+- `destination_basename`: Optional basename for output objects (defaults to the file_id).
+- `s3_region`: Optional region of the `destination_uri` S3 bucket.
+- `image_output_mode`: Set to `'local'` to write cropped images into `destination_uri` storage instead of the Mathpix CDN.
+- `include_page_info`: Include per-page information in the output.
+- `metadata`: Optional dict to attach metadata to the request.
+- Plus the same OCR options as `pdf_new` (`alphabets_allowed`, `rm_spaces`, `include_smiles`, `math_inline_delimiters`, `page_ranges`, etc.).
+
+##### `MathpixClient.file_job_new`
+
+Submit a batch of documents in one call (the server enforces an items-per-call ceiling). Returns a `FileJob` instance.
+
+###### `MathpixClient.file_job_new` Arguments
+
+- `files`: List of `FileSubmission` instances or dicts. Each item takes `source_uri` (required) plus optional `custom_id`, `filename`, `destination_uri`, `s3_region`, `destination_basename`, and `page_ranges`.
+- `job_id`: Optional caller-supplied job id (server-generated when omitted). Required whenever any item carries a `custom_id`.
+- `idempotency_key`: Optional key making the whole batch safe to retry; honored only when no `job_id` is supplied.
+- `conversion_formats`: Job-wide conversion formats, applied to every file.
+- `image_output_mode`: Job-wide; `'local'` writes cropped images to each file's `destination_uri`.
 - `metadata`: Optional dict to attach metadata to the request.
-- `alphabets_allowed`: Optional dict to list alphabets allowed in the output.
-- `rm_spaces`: Remove extra white space from equations (default True).
-- `rm_fonts`: Remove font commands from equations (default False).
-- `idiomatic_eqn_arrays`: Use aligned/gathered/cases instead of array (default False).
-- `include_equation_tags`: Include equation number tags in LaTeX (default False).
-- `include_smiles`: Enable chemistry diagram OCR via SMILES (default True).
-- `include_chemistry_as_image`: Return image crop for chemical diagrams (default False).
-- `include_diagram_text`: Enable text extraction from diagrams (default False).
-- `numbers_default_to_math`: Numbers are always math (default False).
-- `math_inline_delimiters`: Tuple of (begin, end) delimiters for inline math.
-- `math_display_delimiters`: Tuple of (begin, end) delimiters for display math.
-- `page_ranges`: Page range string (e.g., "2,4-6" or "2--2").
-- `enable_spell_check`: Enable predictive mode for English handwriting (default False).
-- `auto_number_sections`: Auto-number sections (default False).
-- `remove_section_numbering`: Remove existing section numbering (default False).
-- `preserve_section_numbering`: Keep existing section numbering (default True).
-- `enable_tables_fallback`: Enable advanced table processing (default False).
-- `fullwidth_punctuation`: Use fullwidth Unicode punctuation (default None).
+- `extra_options`: Additional request options dict merged into the request body — an escape hatch for API options this SDK version does not model yet (validated server-side). May not override the validated request fields.
+- Plus the same OCR options as `pdf_new`, applied to every file in the request.
+
+##### `MathpixClient.file_job_list`
+
+List submitted jobs, newest first. Returns a dict with `jobs` and `next_page_token`.
+
+###### `MathpixClient.file_job_list` Arguments
+
+- `start`: Earliest submission date to include, `yyyy-MM-dd` (UTC).
+- `end`: Latest submission date to include, `yyyy-MM-dd` (UTC).
+- `limit`: Maximum jobs per page (default 100).
+- `paging_state`: Pagination cursor from the previous response's `next_page_token`.
+
+##### `MathpixClient.file_get` / `MathpixClient.file_delete` / `MathpixClient.file_job_get`
 
-##### `MathpixClient.list_scs_files`
+- `file_get(file_id)`: Fetches an existing file and returns a `File` instance seeded with its status; raises `FilesApiError` for unknown ids.
+- `file_delete(file_id)`: Permanently removes a file and its results from Mathpix-owned storage. Only files in a terminal state can be deleted; repeat deletes are idempotent.
+- `file_job_get(job_id)`: Fetches an existing job and returns a `FileJob` instance seeded with its `file_count`; raises `FilesApiError` for unknown ids.
 
-List files from files-api v1. Requires exactly one filter: scs_job_id or filename.
+##### `File`
 
-###### `MathpixClient.list_scs_files` Arguments
+Returned by `file_new`, `file_get`, and `FileJob.file_by_custom_id`. Methods:
 
-- `scs_job_id`: Filter by job ID.
-- `filename`: Filter by filename.
-- `limit`: Maximum number of results (default 100).
-- `paging_state`: Optional paging state for pagination.
+- `status()`: Current status (`pending` | `split` | `completed` | `error`), progress, and per-format conversion statuses. When the status is `error`, the `error` and `error_info` attributes carry the failure details.
+- `wait_until_complete(timeout)`: Poll until processing finishes.
+- `wait_for_format(format, timeout)`: Poll until a specific conversion finishes. Conversions complete independently of the file status and can lag behind it; download a format only after it is ready.
+- `text_result` / `bytes_result` / `json_result` / `save_file`, plus the `to_*` convenience methods (`to_md_text`, `to_docx_bytes`, `to_docx_file`, ...).
+- `delete()`: Permanently remove the file and its results from Mathpix-owned storage.
 
-Returns a dict containing 'file_ids' list and 'next_page_token' for pagination.
+##### `FileJob`
+
+Returned by `file_job_new` and `file_job_get`. Methods:
+
+- `status()`: Job status and counters (`file_count`, `files_completed`, `files_errored`).
+- `wait_until_complete(timeout, interval=5.0)`: Poll until every file reaches a terminal state. Per-file failures don't fail the job.
+- `files(status=None, limit=None, paging_state=None)`: One page of the job's file listing, optionally filtered to `pending`, `completed`, or `error`.
+- `files_iter(status=None, limit=None)`: Iterate over all files, following pagination.
+- `file_by_custom_id(custom_id)`: Fetch one file by the `(job_id, custom_id)` you supplied at submission.
+
+##### Data sources (cloud storage setup)
+
+Non-public `source_uri`/`destination_uri` buckets (`s3://`, `gs://`, Azure Blob) require a registered **data source**: a pointer from your Mathpix account to a bucket you own, with an access grant. The grant model is keyless — no secrets are uploaded to Mathpix (AWS supports a legacy `access_key` fallback). Set up the cloud-side grant following the [per-provider guides](https://docs.mathpix.com/reference/files-v1-data-sources), then register and verify:
+
+```python
+identities = client.onboarding_identities()  # fetch BEFORE creating grants
+external_id = identities["aws"]["external_id"]  # goes in your IAM trust policy
+
+data_source = client.data_source_new(
+    provider="aws",
+    bucket="your-bucket",
+    auth_method="iam_role",
+    provider_specific_details={
+        "iam_role_arn": "arn:aws:iam::123456789012:role/MathpixReader",
+        "aws_external_id": external_id,
+    },
+    region="us-east-1",
+)
+probe = data_source.test()  # {'result': 'ok', 'checks': {'read': True, 'write': True}, ...}
+```
 
-##### `MathpixClient.list_scs_jobs`
+##### `MathpixClient.onboarding_identities`
 
-List SCS jobs from files-api v1.
+Returns the Mathpix identities you grant access to (AWS trust account, Azure app/tenant, and — when available — the GCS impersonator service account) plus your per-group `external_id`, a stable value used in the AWS IAM trust policy and GCS bucket-control verification. Call it before setting up cloud-side grants.
 
-###### `MathpixClient.list_scs_jobs` Arguments
+##### `MathpixClient.data_source_new`
 
-- `start`: Optional start date filter (ISO format).
-- `end`: Optional end date filter (ISO format).
-- `limit`: Maximum number of results (default 100).
-- `paging_state`: Optional paging state for pagination.
+Register a bucket as a data source. Returns a `DataSource` instance.
 
-Returns a dict containing 'jobs' list and optionally 'paging_state' for next page.
+###### `MathpixClient.data_source_new` Arguments
 
-##### `MathpixClient.scs_job_status`
+- `provider`: Storage provider, e.g. `'aws'`, `'azure'`, `'gcp'` — see the [per-provider guides](https://docs.mathpix.com/reference/files-v1-data-sources) for the supported set.
+- `bucket`: Bucket / container name.
+- `auth_method`: Grant type for the provider, e.g. `'iam_role'` or `'access_key'` (aws), `'azure_ad'` (azure), `'service_account'` (gcp). Invalid combinations are rejected server-side.
+- `provider_specific_details`: Provider-shaped metadata (e.g. `iam_role_arn` + `aws_external_id` for aws/`iam_role`, `aws_access_key_id` for aws/`access_key`).
+- `name`: Optional human-readable label.
+- `region`: Bucket region (required for aws `access_key` only).
+- `secret`: Only for aws `access_key` (legacy); keyless grant types reject it server-side.
 
-Get the current status of an SCS job.
+A registration that conflicts with an existing data source raises `FilesApiError` (`'conflict'`); the server message identifies the conflict.
 
-###### `MathpixClient.scs_job_status` Arguments
+For AWS and Azure, call `DataSource.test()` afterward to verify the grant end-to-end. GCS registration verifies bucket control up front, so a successful return already confirms the grant.
 
-- `scs_job_id`: The job ID to get status for.
+##### `MathpixClient.data_source_list` / `data_source_test` / `data_source_delete`
 
-Returns JSON response containing job status information.
+- `data_source_list()`: List the group's registered data sources (secrets are never returned).
+- `data_source_test(data_source_id)`: Runs the read/write probe; returns `{'result', 'checks', 'message'}` and does **not** raise on a failed probe — use it to diagnose grant issues after customer-side IAM changes.
+- `data_source_delete(data_source_id)`: Removes the registration; already-started work is not interrupted, and the bucket can be registered again later. Deleting the registration does not revoke access on the cloud side — remove the grant separately to revoke access.
 
 ##### `MathpixClient.query_usage`
 
@@ -677,16 +758,17 @@ Returns a dict with 'documents' list containing conversion results. Each documen
 - `results`: Get the results dict mapping url_key to OCR result for each processed item.
 - `keys`: Get the list of URL keys in this batch.
 
-### `ScsFile`
+### `File`
 
-#### `ScsFile` Properties
+#### `File` Properties
 
 - `auth`: An Auth instance with Mathpix credentials.
 - `file_id`: The unique identifier for this file.
 
-#### `ScsFile` Methods
+#### `File` Methods
 
 - `status`: Get the current status of the file processing (file_id, status, num_pages, num_pages_completed, percent_done, formats).
+- `delete`: Permanently remove the file and its results from Mathpix-owned storage.
 - `wait_until_complete`: Wait for the file processing to complete.
 - `wait_for_format`: Wait for a specific format conversion to complete.
 - `to_mmd_text`: Get the processed file result as Mathpix Markdown string.
@@ -714,7 +796,6 @@ Returns a dict with 'documents' list containing conversion results. Each documen
 - `to_pdf_file`: Save the processed file result to a PDF file at a local path.
 - `to_html_file`: Save the processed file result to an HTML file at a local path.
 - `to_tex_zip_file`: Save the processed file result to a tex.zip file at a local path.
-- `cropped_image`: Get a cropped region from a specific page as JPEG bytes.
 
 ## Error Handling
 

changelog.md

@@ -1,5 +1,21 @@
 # mpxpy changelog
 
+## July 24, 2026
+
+- Support the public Files API v1
+  - `file_new` submits a single document, taking exactly one of `source_uri` (a remote `s3://`, `gs://`, public `https://`, or Azure Blob URI, via `POST /files/v1/uri`) or `file_path` (a local file, via multipart `POST /files/v1`); both support `Idempotency-Key` retry safety, `(job_id, custom_id)` idempotency, and the `filename`, `s3_region`, and `include_page_info` options
+  - `file_job_new` submits documents in bulk (with a server-enforced per-call ceiling) via `POST /files/v1/jobs`; the new `FileJob` class polls job status, lists files with a status filter and pagination iterator, and fetches files by `custom_id`
+  - `file_job_list`, `file_get`, `file_delete`, and `file_job_get` cover the jobs listing and file lifecycle endpoints; the `*_get` methods fetch the resource and return a seeded handle, raising `FilesApiError` for unknown ids
+  - New `File` class adds `delete()` and lazy status attributes, raises on failed status requests, and disambiguates download errors per the public API (`format_not_ready` vs `not_found` vs `unsupported_format`)
+  - New `FilesApiError` exception carries the Files API error code (`error_id`) and HTTP status; it is raised only for responses with a recognizable Files API error body, other failures raise `MathpixClientError`
+  - `file_new` and `file_job_new` accept an `extra_options` dict merged into the request body — an escape hatch for API options the SDK does not model yet, validated server-side; it may not override the validated request fields (`source_uri`, `files`, `job_id`, `custom_id`, `metadata`)
+  - The client no longer duplicates server-enforced constraints (identifier charset/length, items-per-call ceiling, status filter values); the server's `FilesApiError` is authoritative for those
+  - `ScsFile`, `scs_file_new`, `list_scs_files`, `list_scs_jobs`, and `scs_job_status` are deprecated. `scs_file_new` is now a thin wrapper that translates its legacy argument names and forwards to `file_new`; the listing/status methods keep their legacy endpoints and response shapes during the deprecation window. Migrate to `File`, `file_new`, `file_job_get(job_id).files()`, `file_job_list`, and `file_job_get(job_id).status()`
+  - Data Sources API: `onboarding_identities`, `data_source_new`, `data_source_list`, `data_source_test`, and `data_source_delete` register and manage cloud storage (S3, GCS, Azure Blob) for use as `source_uri`/`destination_uri`; new `DataSource` class with `test()` and `delete()`
+  - Debug logging redacts remote URIs to scheme and host, since signed URLs carry credentials in their query strings
+  - `files_api_url` now defaults to the resolved `api_url` as documented, so a custom `api_url` applies to Files API requests too
+- All requests now send a `User-Agent: mpxpy/<version>` header
+
 ## June 5, 2025
 
 - Add improve_mathpix argument to the Client, Image, and Pdf classes

mpxpy/__init__.py

@@ -1,4 +1,7 @@
 from mpxpy.mathpix_client import MathpixClient
+from mpxpy.file import File
+from mpxpy.file_job import FileJob, FileSubmission
+from mpxpy.data_source import DataSource
 from mpxpy.scs_file import ScsFile
 from mpxpy.pdf import Pdf
 from mpxpy.image import Image
@@ -10,10 +13,15 @@
     ValidationError,
     ConversionIncompleteError,
     FilesystemError,
+    FilesApiError,
 )
 
 __all__ = [
     "MathpixClient",
+    "File",
+    "FileJob",
+    "FileSubmission",
+    "DataSource",
     "ScsFile",
     "Pdf",
     "Image",
@@ -24,4 +32,5 @@
     "ValidationError",
     "ConversionIncompleteError",
     "FilesystemError",
+    "FilesApiError",
 ]