mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-10-01 22:14:51 +02:00
* [eric] ci: gitleaks-ignore the known historical secrets so our branch stops failing on leaks it didnt add * [eric] workflows: restore scheduled-tasks on the workflow line (revert removal, keep windows fixes + 1.1.69) * [eric] workflows: re-apply uncommitted scheduling wip (schedule pill, calendar view, slice) * [eric] ops: gitignore dev-team local state files * [eric] ops: backlog item for download-tracking visibility * [eric] ci: allowlist the cdp-routes redaction-test token in gitleaks * [aidan] feat/scheduled-tasks: keep step labels in sync on edit and show chevron on every step * [aidan] fix: schedule time in chat * [aidan] ux/workflows: add workflow step removal (#91) * [aidan] feat/scheduled-tasks: remember workflow tool permissions across runs * [aidan] feat/task-scheduling: add hourly and minute (15-min minimum) schedule intervals (#93) * [aidan] feat/scheduled-tasks: calendar, rename, and edit workflows (#94) * [aidan] bug: fix schedule button * [aidan] fix/agent-errors: surface provider rate limits * [aidan] ux/cards: click-to-rename for chat and workflow titles Single-click a card's title to enter edit mode inline. Commit on Enter/blur, cancel on Escape. Rename persists via PATCH for workflows and sessions. * [aidan] feat/workflows: seed build prompt for zero-step workflows When a new workflow has no steps, seed the agent with a prompt asking the user to describe what the workflow should do, rather than starting blank. * [aidan] feat/workflows: add-to-schedule popover for unscheduled workflows Clicking the "+" on an unscheduled workflow row opens a popover with two options: - Keep this schedule: enables the workflow's existing cadence and moves it to Scheduled - Change schedule: opens the scheduling editor to pick a different time * [aidan] ux/workflows: wire add-to-schedule popover and simplify New button - Made the "+" icon on unscheduled workflow rows clickable, opening a popover to keep or change the schedule - Removed AddIcon from toolbar "New" button (now reads "New" instead of "+ New") * [aidan] fix/scheduled-tasks: open schedule calendar when Schedule pill clicked Fixed the Schedule pill click being swallowed by the toolbar's dismiss handler. Exempted the toolbar pills via data-toolbar-pills so their click handlers fire. * [aidan] ux/workflows: open New workflow in agent build chat instead of empty card When creating a new workflow from the hub, open it in edit_agent view (with the agent builder chat) instead of a preview card. The workflow is created on the backend first so the embedded session has a real ID. * [aidan] feat/workflow-edit: add draft testing save flow * [aidan] ux/chat: remove continue chat button * [aidan] ux/workflows: polish workflow card interactions * [aidan] fix/workflow-scheduling: save unscheduled workflows as drafts * aidan ui: schedule naming changes * [aidan] ui: tool calling desc/naming * [aidan] ui: calendar sidebar naming * [aidan] ui: fix stop viewing closing chat * [aidan] feat/workflows: auto-name workflows and polish the build flow (#95) * [aidan] feat/workflow-auto-naming: auto-generate workflow titles from steps Generate a title + description from a workflow's steps (one aux call, reused for step labels) whenever it is still auto_named, so a workflow built in the Edit Agent names itself on commit instead of staying "New workflow". A manual rename sets auto_named=False and is never overwritten. Stream the aux call (non-streaming drops content on some 9router lanes) and fall back to a step-derived title when the model is unavailable. * [aidan] feat/workflows: hide unsaved new workflows until first save A brand-new "+ New" workflow is created with unsaved=true and kept out of the hub's scheduled/unscheduled lists while the user is still building it in the Edit Agent. The first commit (Save) clears the flag and the workflow appears. Every other create path stays visible immediately. * [aidan] ux/workflows: remove redundant save workflow button The Edit Agent already has Discard/Save controls in its strip, so the header "Save Workflow" button was a duplicate save path. Remove it and its pulse/edit-session-id wiring; the model/time subtitle stays. * [aidan] ux/workflows: animate title on auto-rename Wrap the workflow card title in the same Typewriter the chat card uses, so when the auto-generated name replaces the placeholder after Save it retypes letter-by-letter. Gated on a real (non-placeholder) title so it never animates on mount or for already-named workflows. * [aidan] ux/workflows: animate sidebar title on auto-rename Wrap the calendar hub's sidebar row title in the same Typewriter the workflow card uses, so a title that auto-renames retypes letter-by-letter in the sidebar too. Extract the placeholder/isRealTitle guard into the shared workflowVisuals so the card and sidebar stay in sync. * [aidan] fix/workflows: connect watch tether, keep watched chat open, wire draft run/history * [aidan] ui: grey out chat pill when not selected * [aidan] ui: fix running agent display * [aidan] feat/history-popover: add chat history and scheduled tasks run log tabs (#96) * [aidan] ux/schedule: toast when calendar view already open on expand * [aidan] ui: fix popover descs * [aidan] feat/workflow-runs: add pause, resume, and stop controls for live runs * [aidan] feat/schedule-calendar: add calendar occurrences endpoint and concrete timezones * [aidan] feat/workflows: require at least one step to save a workflow * [aidan] ux/edit-agent: hide Discard for an unsaved new workflow * [aidan] ux/edit-agent: move fix-prefix card below the step list * [aidan] feat/workflows: toast when an unattended scheduled run starts * [aidan] fix/dashboard-tethers: anchor workflow-sidecar tethers to measured card rects * [aidan] feat/workflows: validate steps before scheduling and keep chat tool memory * [aidan] feat/mcp-suggestions: dismissable integration banner with per-session cooldown * [aidan] feat/workflows: add scheduled-run "running now" toast with click-to-view (#97) * [aidan] ux/workflows: surface paused state on card, sidebar, and calendar; tidy run history * [aidan] feat/mcp-suggestions: suggest both Google and Microsoft when provider is ambiguous * [aidan] fix/agent-tokens: friendly out-of-tokens card across all agent surfaces * [aidan] feat/workflow-model: persist edit-agent model on save with switch notice and fresh drafts * [aidan] fix/workflow-chat: force stop on watched run mirrors workflow card stop * [aidan] fix/workflow-cards: keep watched run tethered on finish to avoid duplicate chat * [aidan] feat/schedule-calendar: mark current time with a now line in week view * [aidan] refactor/private-names: rename error and schedule classifiers from _ to p_ * [aidan] feat/scheduled-tasks: agent workflow scheduling and in-chat convert (#98) * [aidan] feat/workflow-suggest: nudge user to convert repeatable chat to workflow Add SuggestConvertToWorkflow MCP tool that agents call at the end of a task when they've completed something worth repeating (daily report, weekly check, recurring data pull). Frontend detects the tool call and glows the "Convert to workflow" button 3 times to draw the eye. When user clicks it, the suggested cadence (e.g. "every weekday at 9am") is stored in the draft and seeded into the scheduling agent's first prompt, so the agent can act on the suggestion rather than asking the user again. Tool is never auto-called — agents decide when a task is genuinely repeatable (not debugging, creative work, one-off lookup). Tool description emphasizes sparse, high-confidence use only (once per session max). Files changed: - backend/apps/agents/schedule_mcp_server.py: add SuggestConvertToWorkflow tool - frontend/src/shared/mcpToolMeta.ts: add label for new tool - frontend/src/app/pages/Dashboard/cards/AgentCard.tsx: detect suggestion in session messages, show+glow "Convert to workflow" button, pass cadence to draft - frontend/src/shared/state/workflowsSlice.ts: add suggested_cadence field to Workflow interface - frontend/src/app/pages/Workflows/SchedulingView.tsx: seed scheduling agent prompt with suggested cadence hint * [aidan] feat/agent-scheduling: route recurring asks through native workflows, deny claude cron skill * [aidan] feat/workflow-convert: in-chat convert popup and auto-open scheduled workflow card * [aidan] ux/calendar-page: schedule calendar restyle + popover fixes (#99) * [aidan] fix/dashboard-delete: remove workflows calendar panel on delete key * [aidan] ux/workflows-calendar: restyle hub, fix today highlight, add toolbar toggle * [aidan] ux/schedule-popover: compact density, fix sticky header bleed, add header spacing * [aidan] ux/schedule-calendar: hollow ring dot for past fires in month view * [aidan] ux/schedule-calendar: clickable +N more opens day's full run list * [aidan] feat/run-log-filters: add success and skipped pills to scheduled task history * [aidan] fix/convert-button: stop drag capture so convert-to-workflow click fires * [aidan] ux/calendar-card: match border color and radius to chat and workflow cards * [aidan] fix/minimap: render missed-runs card on the minimap * [aidan] ux/run-sparkline: simplify tooltip to plain run tally * [aidan] ux/run-history: collapse expanded run view to one clickable line * [aidan] ux/calendar-card: match corner radius to browser cards * [aidan] feat/workflows: launch-time scheduling UX and workflow-card polish (#101) * [aidan] feat/schedule-list: lazy-load list view via scroll sentinel * [aidan] feat/missed-runs: launch toast with per-workflow counts and pan-to-card * [aidan] fix/dashboard-tethers: keep watching line anchored on canvas zoom * [aidan] feat/scheduled-tasks: review missed runs at launch instead of auto-firing on_missed * [aidan] refactor/workflow-cards: use radius and status design tokens, polish card chrome * [aidan] ux/agent-card: keep convert-to-workflow visible during runs with mid-turn toast * [aidan] ux/mcp-bubble: drop redundant verb label when a workflow label is shown * [aidan] chore/backend: remove stale explanatory comments * [aidan] fix/workflows-hub: load workflows on hub mount so calendar fills at launch * [aidan] feat/workflows: generate title, description, step labels at convert time * [aidan] fix/tidy-layout: include workflows hub in tidy and fit-to-view * [aidan] feat/schedule-list: window long list via measured-height virtualizer * [aidan] ux/workflows-hub: remove time-saved badge from calendar header * [aidan] fix/types: add missing semantic-type labels and drop stray fade arg * [aidan] feat/schedule: pin monthly day-of-month and honor repeat-every intervals * [aidan] feat/schedule: inherit source-session tool surface for scheduled runs * [aidan] ux/calendar: restack hour-cell events as bars with overflow affordance * [aidan] feat/calendar: open the run card when clicking a scheduled occurrence * [aidan] ux/missed-runs: add per-group select-all toggle and rename skip action * [aidan] feat: new scheduled task design ported * [aidan] ui: sidebar reorder, repeat controls on schedule card * [aidan] ui: sidebar, scheduling time * [aidan] feat/schedule: pin monthly last-day-of-month * [aidan] feat/steps: per-step enable toggle * [aidan] feat/workflows: per-workflow color swatch * [aidan] feat/trash: soft-delete workflows with restore and purge * [aidan] feat/run-monitor: live run monitor card on the canvas * [aidan] feat/run-context: attach a run as removable chat context * [aidan] feat/compose: new-workflow landing page and auto-commit build flow * [aidan] ui/workflows: dark mode and design-system cohesion * [aidan] ui/calendar: overflow popover, condensed week view, scroll fix * [aidan] feat/home: ongoing runs, missed review, and accurate Coming-up counts * [aidan] fix/run-status: sync ongoing runs and heal stuck/interrupted runs * [aidan] ux/schedule: last-day-of-month UI, Run-at time typing, interval input * [aidan] ux/workflows: default window size and toolbar icon * [aidan] fix/schedule: measure ran_late from start and anchor recurrences to created_at * [aidan] feat/calendar: render fire times from backend, drop JS recurrence reimpl * [aidan] chore/dashboard: drop dead configure/missed-run cards, refetch on reconnect * [aidan] chore/agent-card: remove unreachable convert-to-workflow action * [aidan] fix/workflows: don't bump updated_at on a no-op draft commit so viewing a workflow doesn't reorder the sidebar * [aidan] feat/schedule: warn when scheduling a workflow that has no steps * [aidan] fix/selection-tool: never select the workflows app, and exit the tool on Escape without dropping selections * [aidan] ux/compose: diversify new-workflow starter prompts across personas * [aidan] ux/run-monitor: spawn the run card a bit farther right of the workflows app * [aidan] fix/schedule: harden run recovery and storage writes against crashes * [aidan] ux/compose: restyle new-workflow starters as a clean pill cluster with rich prompts * [aidan] fix/workflows: optimistically apply edits so the schedule banner updates instantly * [aidan] ui/workflows: three-tone surface depth so the window lifts off the canvas in both themes * [aidan] ui/workflows: close buttons turn red on hover, matching the chat card * [aidan] test/schedule: cover executor pipeline, storage durability, and recurrence gaps * [aidan] fix: remove package-lock json * [aidan] fix/workflows-compose: keep compose view until edit agent replies * [aidan] feat/workflows: auto-generate workflow + step titles with typewriter animation * [eric] deps: restore frontend/package-lock.json (PR #105 deletion broke npm ci) --------- Co-authored-by: Eric <ciregenz@berkeley.edu> Co-authored-by: cire <134991075+ciregenz@users.noreply.github.com>
936 lines
29 KiB
Python
936 lines
29 KiB
Python
#
|
|
# The Python Imaging Library.
|
|
# $Id$
|
|
#
|
|
# base class for image file handlers
|
|
#
|
|
# history:
|
|
# 1995-09-09 fl Created
|
|
# 1996-03-11 fl Fixed load mechanism.
|
|
# 1996-04-15 fl Added pcx/xbm decoders.
|
|
# 1996-04-30 fl Added encoders.
|
|
# 1996-12-14 fl Added load helpers
|
|
# 1997-01-11 fl Use encode_to_file where possible
|
|
# 1997-08-27 fl Flush output in _save
|
|
# 1998-03-05 fl Use memory mapping for some modes
|
|
# 1999-02-04 fl Use memory mapping also for "I;16" and "I;16B"
|
|
# 1999-05-31 fl Added image parser
|
|
# 2000-10-12 fl Set readonly flag on memory-mapped images
|
|
# 2002-03-20 fl Use better messages for common decoder errors
|
|
# 2003-04-21 fl Fall back on mmap/map_buffer if map is not available
|
|
# 2003-10-30 fl Added StubImageFile class
|
|
# 2004-02-25 fl Made incremental parser more robust
|
|
#
|
|
# Copyright (c) 1997-2004 by Secret Labs AB
|
|
# Copyright (c) 1995-2004 by Fredrik Lundh
|
|
#
|
|
# See the README file for information on usage and redistribution.
|
|
#
|
|
from __future__ import annotations
|
|
|
|
import abc
|
|
import io
|
|
import itertools
|
|
import logging
|
|
import os
|
|
import struct
|
|
from typing import IO, Any, NamedTuple, cast
|
|
|
|
from . import ExifTags, Image
|
|
from ._util import DeferredError, is_path
|
|
|
|
TYPE_CHECKING = False
|
|
if TYPE_CHECKING:
|
|
from ._typing import StrOrBytesPath
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
MAXBLOCK = 65536
|
|
"""
|
|
By default, Pillow processes image data in blocks. This helps to prevent excessive use
|
|
of resources. Codecs may disable this behaviour with ``_pulls_fd`` or ``_pushes_fd``.
|
|
|
|
When reading an image, this is the number of bytes to read at once.
|
|
|
|
When writing an image, this is the number of bytes to write at once.
|
|
If the image width times 4 is greater, then that will be used instead.
|
|
Plugins may also set a greater number.
|
|
|
|
User code may set this to another number.
|
|
"""
|
|
|
|
SAFEBLOCK = 1024 * 1024
|
|
|
|
LOAD_TRUNCATED_IMAGES = False
|
|
"""Whether or not to load truncated image files. User code may change this."""
|
|
|
|
ERRORS = {
|
|
-1: "image buffer overrun error",
|
|
-2: "decoding error",
|
|
-3: "unknown error",
|
|
-8: "bad configuration",
|
|
-9: "out of memory error",
|
|
}
|
|
"""
|
|
Dict of known error codes returned from :meth:`.PyDecoder.decode`,
|
|
:meth:`.PyEncoder.encode` :meth:`.PyEncoder.encode_to_pyfd` and
|
|
:meth:`.PyEncoder.encode_to_file`.
|
|
"""
|
|
|
|
|
|
#
|
|
# --------------------------------------------------------------------
|
|
# Helpers
|
|
|
|
|
|
def _get_oserror(error: int, *, encoder: bool) -> OSError:
|
|
try:
|
|
msg = Image.core.getcodecstatus(error)
|
|
except AttributeError:
|
|
msg = ERRORS.get(error)
|
|
if not msg:
|
|
msg = f"{'encoder' if encoder else 'decoder'} error {error}"
|
|
msg += f" when {'writing' if encoder else 'reading'} image file"
|
|
return OSError(msg)
|
|
|
|
|
|
def _tilesort(t: _Tile) -> int:
|
|
# sort on offset
|
|
return t[2]
|
|
|
|
|
|
class _Tile(NamedTuple):
|
|
codec_name: str
|
|
extents: tuple[int, int, int, int] | None
|
|
offset: int = 0
|
|
args: tuple[Any, ...] | str | None = None
|
|
|
|
|
|
#
|
|
# --------------------------------------------------------------------
|
|
# ImageFile base class
|
|
|
|
|
|
class ImageFile(Image.Image):
|
|
"""Base class for image file format handlers."""
|
|
|
|
def __init__(
|
|
self, fp: StrOrBytesPath | IO[bytes], filename: str | bytes | None = None
|
|
) -> None:
|
|
super().__init__()
|
|
|
|
self._min_frame = 0
|
|
|
|
self.custom_mimetype: str | None = None
|
|
|
|
self.tile: list[_Tile] = []
|
|
""" A list of tile descriptors """
|
|
|
|
self.readonly = 1 # until we know better
|
|
|
|
self.decoderconfig: tuple[Any, ...] = ()
|
|
self.decodermaxblock = MAXBLOCK
|
|
|
|
self.fp: IO[bytes] | None
|
|
self._fp: IO[bytes] | DeferredError
|
|
if is_path(fp):
|
|
# filename
|
|
self.fp = open(fp, "rb")
|
|
self.filename = os.fspath(fp)
|
|
self._exclusive_fp = True
|
|
else:
|
|
# stream
|
|
self.fp = cast(IO[bytes], fp)
|
|
self.filename = filename if filename is not None else ""
|
|
# can be overridden
|
|
self._exclusive_fp = False
|
|
|
|
try:
|
|
try:
|
|
self._open()
|
|
|
|
if isinstance(self, StubImageFile):
|
|
if loader := self._load():
|
|
loader.open(self)
|
|
except (
|
|
IndexError, # end of data
|
|
TypeError, # end of data (ord)
|
|
KeyError, # unsupported mode
|
|
EOFError, # got header but not the first frame
|
|
struct.error,
|
|
) as v:
|
|
raise SyntaxError(v) from v
|
|
|
|
if not self.mode or self.size[0] <= 0 or self.size[1] <= 0:
|
|
msg = "not identified by this driver"
|
|
raise SyntaxError(msg)
|
|
except BaseException:
|
|
# close the file only if we have opened it this constructor
|
|
if self._exclusive_fp:
|
|
self.fp.close()
|
|
raise
|
|
|
|
def _open(self) -> None:
|
|
pass
|
|
|
|
# Context manager support
|
|
def __enter__(self) -> ImageFile:
|
|
return self
|
|
|
|
def _close_fp(self) -> None:
|
|
if getattr(self, "_fp", False) and not isinstance(self._fp, DeferredError):
|
|
if self._fp != self.fp:
|
|
self._fp.close()
|
|
self._fp = DeferredError(ValueError("Operation on closed image"))
|
|
if self.fp:
|
|
self.fp.close()
|
|
|
|
def __exit__(self, *args: object) -> None:
|
|
if getattr(self, "_exclusive_fp", False):
|
|
self._close_fp()
|
|
self.fp = None
|
|
|
|
def close(self) -> None:
|
|
"""
|
|
Closes the file pointer, if possible.
|
|
|
|
This operation will destroy the image core and release its memory.
|
|
The image data will be unusable afterward.
|
|
|
|
This function is required to close images that have multiple frames or
|
|
have not had their file read and closed by the
|
|
:py:meth:`~PIL.Image.Image.load` method. See :ref:`file-handling` for
|
|
more information.
|
|
"""
|
|
try:
|
|
self._close_fp()
|
|
self.fp = None
|
|
except Exception as msg:
|
|
logger.debug("Error closing: %s", msg)
|
|
|
|
super().close()
|
|
|
|
def get_child_images(self) -> list[ImageFile]:
|
|
child_images = []
|
|
exif = self.getexif()
|
|
ifds = []
|
|
if ExifTags.Base.SubIFDs in exif:
|
|
subifd_offsets = exif[ExifTags.Base.SubIFDs]
|
|
if subifd_offsets:
|
|
if not isinstance(subifd_offsets, tuple):
|
|
subifd_offsets = (subifd_offsets,)
|
|
ifds = [
|
|
(exif._get_ifd_dict(subifd_offset), subifd_offset)
|
|
for subifd_offset in subifd_offsets
|
|
]
|
|
ifd1 = exif.get_ifd(ExifTags.IFD.IFD1)
|
|
if ifd1 and ifd1.get(ExifTags.Base.JpegIFOffset):
|
|
assert exif._info is not None
|
|
ifds.append((ifd1, exif._info.next))
|
|
|
|
offset = None
|
|
for ifd, ifd_offset in ifds:
|
|
assert self.fp is not None
|
|
current_offset = self.fp.tell()
|
|
if offset is None:
|
|
offset = current_offset
|
|
|
|
fp = self.fp
|
|
if ifd is not None:
|
|
thumbnail_offset = ifd.get(ExifTags.Base.JpegIFOffset)
|
|
if thumbnail_offset is not None:
|
|
thumbnail_offset += getattr(self, "_exif_offset", 0)
|
|
self.fp.seek(thumbnail_offset)
|
|
|
|
length = ifd.get(ExifTags.Base.JpegIFByteCount)
|
|
assert isinstance(length, int)
|
|
data = self.fp.read(length)
|
|
fp = io.BytesIO(data)
|
|
|
|
with Image.open(fp) as im:
|
|
from . import TiffImagePlugin
|
|
|
|
if thumbnail_offset is None and isinstance(
|
|
im, TiffImagePlugin.TiffImageFile
|
|
):
|
|
im._frame_pos = [ifd_offset]
|
|
im._seek(0)
|
|
im.load()
|
|
child_images.append(im)
|
|
|
|
if offset is not None:
|
|
assert self.fp is not None
|
|
self.fp.seek(offset)
|
|
return child_images
|
|
|
|
def get_format_mimetype(self) -> str | None:
|
|
if self.custom_mimetype:
|
|
return self.custom_mimetype
|
|
if self.format is not None:
|
|
return Image.MIME.get(self.format.upper())
|
|
return None
|
|
|
|
def __getstate__(self) -> list[Any]:
|
|
return super().__getstate__() + [self.filename]
|
|
|
|
def __setstate__(self, state: list[Any]) -> None:
|
|
self.tile = []
|
|
if len(state) > 5:
|
|
self.filename = state[5]
|
|
super().__setstate__(state)
|
|
|
|
def verify(self) -> None:
|
|
"""Check file integrity"""
|
|
|
|
# raise exception if something's wrong. must be called
|
|
# directly after open, and closes file when finished.
|
|
if self._exclusive_fp and self.fp:
|
|
self.fp.close()
|
|
self.fp = None
|
|
|
|
def load(self) -> Image.core.PixelAccess | None:
|
|
"""Load image data based on tile list"""
|
|
|
|
if not self.tile and self._im is None:
|
|
msg = "cannot load this image"
|
|
raise OSError(msg)
|
|
|
|
pixel = Image.Image.load(self)
|
|
if not self.tile:
|
|
return pixel
|
|
|
|
self.map: mmap.mmap | None = None
|
|
use_mmap = self.filename and len(self.tile) == 1
|
|
|
|
assert self.fp is not None
|
|
readonly = 0
|
|
|
|
# look for read/seek overrides
|
|
if hasattr(self, "load_read"):
|
|
read = self.load_read
|
|
# don't use mmap if there are custom read/seek functions
|
|
use_mmap = False
|
|
else:
|
|
read = self.fp.read
|
|
|
|
if hasattr(self, "load_seek"):
|
|
seek = self.load_seek
|
|
use_mmap = False
|
|
else:
|
|
seek = self.fp.seek
|
|
|
|
if use_mmap:
|
|
# try memory mapping
|
|
decoder_name, extents, offset, args = self.tile[0]
|
|
if isinstance(args, str):
|
|
args = (args, 0, 1)
|
|
if (
|
|
decoder_name == "raw"
|
|
and isinstance(args, tuple)
|
|
and len(args) >= 3
|
|
and args[0] == self.mode
|
|
and args[0] in Image._MAPMODES
|
|
):
|
|
if offset < 0:
|
|
msg = "Tile offset cannot be negative"
|
|
raise ValueError(msg)
|
|
try:
|
|
# use mmap, if possible
|
|
import mmap
|
|
|
|
with open(self.filename) as fp:
|
|
self.map = mmap.mmap(fp.fileno(), 0, access=mmap.ACCESS_READ)
|
|
if offset + self.size[1] * args[1] > self.map.size():
|
|
msg = "buffer is not large enough"
|
|
raise OSError(msg)
|
|
self.im = Image.core.map_buffer(
|
|
self.map, self.size, decoder_name, offset, args
|
|
)
|
|
readonly = 1
|
|
# After trashing self.im,
|
|
# we might need to reload the palette data.
|
|
if self.palette:
|
|
self.palette.dirty = 1
|
|
except (AttributeError, OSError, ImportError):
|
|
self.map = None
|
|
|
|
self.load_prepare()
|
|
err_code = -3 # initialize to unknown error
|
|
if not self.map:
|
|
# sort tiles in file order
|
|
self.tile.sort(key=_tilesort)
|
|
|
|
# FIXME: This is a hack to handle TIFF's JpegTables tag.
|
|
prefix = getattr(self, "tile_prefix", b"")
|
|
|
|
# Remove consecutive duplicates that only differ by their offset
|
|
self.tile = [
|
|
list(tiles)[-1]
|
|
for _, tiles in itertools.groupby(
|
|
self.tile, lambda tile: (tile[0], tile[1], tile[3])
|
|
)
|
|
]
|
|
for i, (decoder_name, extents, offset, args) in enumerate(self.tile):
|
|
seek(offset)
|
|
decoder = Image._getdecoder(
|
|
self.mode, decoder_name, args, self.decoderconfig
|
|
)
|
|
try:
|
|
decoder.setimage(self.im, extents)
|
|
if decoder.pulls_fd:
|
|
decoder.setfd(self.fp)
|
|
err_code = decoder.decode(b"")[1]
|
|
else:
|
|
b = prefix
|
|
while True:
|
|
read_bytes = self.decodermaxblock
|
|
if i + 1 < len(self.tile):
|
|
next_offset = self.tile[i + 1].offset
|
|
if next_offset > offset:
|
|
read_bytes = next_offset - offset
|
|
try:
|
|
s = read(read_bytes)
|
|
except (IndexError, struct.error) as e:
|
|
# truncated png/gif
|
|
if LOAD_TRUNCATED_IMAGES:
|
|
break
|
|
else:
|
|
msg = "image file is truncated"
|
|
raise OSError(msg) from e
|
|
|
|
if not s: # truncated jpeg
|
|
if LOAD_TRUNCATED_IMAGES:
|
|
break
|
|
else:
|
|
msg = (
|
|
"image file is truncated "
|
|
f"({len(b)} bytes not processed)"
|
|
)
|
|
raise OSError(msg)
|
|
|
|
b = b + s
|
|
n, err_code = decoder.decode(b)
|
|
if n < 0:
|
|
break
|
|
b = b[n:]
|
|
finally:
|
|
# Need to cleanup here to prevent leaks
|
|
decoder.cleanup()
|
|
|
|
self.tile = []
|
|
self.readonly = readonly
|
|
|
|
self.load_end()
|
|
|
|
if self._exclusive_fp and self._close_exclusive_fp_after_loading:
|
|
self.fp.close()
|
|
self.fp = None
|
|
|
|
if not self.map and not LOAD_TRUNCATED_IMAGES and err_code < 0:
|
|
# still raised if decoder fails to return anything
|
|
raise _get_oserror(err_code, encoder=False)
|
|
|
|
return Image.Image.load(self)
|
|
|
|
def load_prepare(self) -> None:
|
|
# create image memory if necessary
|
|
if self._im is None:
|
|
self.im = Image.core.new(self.mode, self.size)
|
|
# create palette (optional)
|
|
if self.mode == "P":
|
|
Image.Image.load(self)
|
|
|
|
def load_end(self) -> None:
|
|
# may be overridden
|
|
pass
|
|
|
|
# may be defined for contained formats
|
|
# def load_seek(self, pos: int) -> None:
|
|
# pass
|
|
|
|
# may be defined for blocked formats (e.g. PNG)
|
|
# def load_read(self, read_bytes: int) -> bytes:
|
|
# pass
|
|
|
|
def _seek_check(self, frame: int) -> bool:
|
|
if (
|
|
frame < self._min_frame
|
|
# Only check upper limit on frames if additional seek operations
|
|
# are not required to do so
|
|
or (
|
|
not (hasattr(self, "_n_frames") and self._n_frames is None)
|
|
and frame >= getattr(self, "n_frames") + self._min_frame
|
|
)
|
|
):
|
|
msg = "attempt to seek outside sequence"
|
|
raise EOFError(msg)
|
|
|
|
return self.tell() != frame
|
|
|
|
|
|
class StubHandler(abc.ABC):
|
|
def open(self, im: StubImageFile) -> None:
|
|
pass
|
|
|
|
@abc.abstractmethod
|
|
def load(self, im: StubImageFile) -> Image.Image:
|
|
pass
|
|
|
|
|
|
class StubImageFile(ImageFile, metaclass=abc.ABCMeta):
|
|
"""
|
|
Base class for stub image loaders.
|
|
|
|
A stub loader is an image loader that can identify files of a
|
|
certain format, but relies on external code to load the file.
|
|
"""
|
|
|
|
@abc.abstractmethod
|
|
def _open(self) -> None:
|
|
pass
|
|
|
|
def load(self) -> Image.core.PixelAccess | None:
|
|
loader = self._load()
|
|
if loader is None:
|
|
msg = f"cannot find loader for this {self.format} file"
|
|
raise OSError(msg)
|
|
image = loader.load(self)
|
|
assert image is not None
|
|
# become the other object (!)
|
|
self.__class__ = image.__class__ # type: ignore[assignment]
|
|
self.__dict__ = image.__dict__
|
|
return image.load()
|
|
|
|
@abc.abstractmethod
|
|
def _load(self) -> StubHandler | None:
|
|
"""(Hook) Find actual image loader."""
|
|
pass
|
|
|
|
|
|
class Parser:
|
|
"""
|
|
Incremental image parser. This class implements the standard
|
|
feed/close consumer interface.
|
|
"""
|
|
|
|
incremental = None
|
|
image: Image.Image | None = None
|
|
data: bytes | None = None
|
|
decoder: Image.core.ImagingDecoder | PyDecoder | None = None
|
|
offset = 0
|
|
finished = 0
|
|
|
|
def reset(self) -> None:
|
|
"""
|
|
(Consumer) Reset the parser. Note that you can only call this
|
|
method immediately after you've created a parser; parser
|
|
instances cannot be reused.
|
|
"""
|
|
assert self.data is None, "cannot reuse parsers"
|
|
|
|
def feed(self, data: bytes) -> None:
|
|
"""
|
|
(Consumer) Feed data to the parser.
|
|
|
|
:param data: A string buffer.
|
|
:exception OSError: If the parser failed to parse the image file.
|
|
"""
|
|
# collect data
|
|
|
|
if self.finished:
|
|
return
|
|
|
|
if self.data is None:
|
|
self.data = data
|
|
else:
|
|
self.data = self.data + data
|
|
|
|
# parse what we have
|
|
if self.decoder:
|
|
if self.offset > 0:
|
|
# skip header
|
|
skip = min(len(self.data), self.offset)
|
|
self.data = self.data[skip:]
|
|
self.offset = self.offset - skip
|
|
if self.offset > 0 or not self.data:
|
|
return
|
|
|
|
n, e = self.decoder.decode(self.data)
|
|
|
|
if n < 0:
|
|
# end of stream
|
|
self.data = None
|
|
self.finished = 1
|
|
if e < 0:
|
|
# decoding error
|
|
self.image = None
|
|
raise _get_oserror(e, encoder=False)
|
|
else:
|
|
# end of image
|
|
return
|
|
self.data = self.data[n:]
|
|
|
|
elif self.image:
|
|
# if we end up here with no decoder, this file cannot
|
|
# be incrementally parsed. wait until we've gotten all
|
|
# available data
|
|
pass
|
|
|
|
else:
|
|
# attempt to open this file
|
|
try:
|
|
with io.BytesIO(self.data) as fp:
|
|
im = Image.open(fp)
|
|
except OSError:
|
|
pass # not enough data
|
|
else:
|
|
flag = hasattr(im, "load_seek") or hasattr(im, "load_read")
|
|
if not flag and len(im.tile) == 1:
|
|
# initialize decoder
|
|
im.load_prepare()
|
|
d, e, o, a = im.tile[0]
|
|
im.tile = []
|
|
self.decoder = Image._getdecoder(im.mode, d, a, im.decoderconfig)
|
|
self.decoder.setimage(im.im, e)
|
|
|
|
# calculate decoder offset
|
|
self.offset = o
|
|
if self.offset <= len(self.data):
|
|
self.data = self.data[self.offset :]
|
|
self.offset = 0
|
|
|
|
self.image = im
|
|
|
|
def __enter__(self) -> Parser:
|
|
return self
|
|
|
|
def __exit__(self, *args: object) -> None:
|
|
self.close()
|
|
|
|
def close(self) -> Image.Image:
|
|
"""
|
|
(Consumer) Close the stream.
|
|
|
|
:returns: An image object.
|
|
:exception OSError: If the parser failed to parse the image file either
|
|
because it cannot be identified or cannot be
|
|
decoded.
|
|
"""
|
|
# finish decoding
|
|
if self.decoder:
|
|
# get rid of what's left in the buffers
|
|
self.feed(b"")
|
|
self.data = self.decoder = None
|
|
if not self.finished:
|
|
msg = "image was incomplete"
|
|
raise OSError(msg)
|
|
if not self.image:
|
|
msg = "cannot parse this image"
|
|
raise OSError(msg)
|
|
if self.data:
|
|
# incremental parsing not possible; reopen the file
|
|
# not that we have all data
|
|
with io.BytesIO(self.data) as fp:
|
|
try:
|
|
self.image = Image.open(fp)
|
|
finally:
|
|
self.image.load()
|
|
return self.image
|
|
|
|
|
|
# --------------------------------------------------------------------
|
|
|
|
|
|
def _save(im: Image.Image, fp: IO[bytes], tile: list[_Tile], bufsize: int = 0) -> None:
|
|
"""Helper to save image based on tile list
|
|
|
|
:param im: Image object.
|
|
:param fp: File object.
|
|
:param tile: Tile list.
|
|
:param bufsize: Optional buffer size
|
|
"""
|
|
|
|
im.load()
|
|
if not hasattr(im, "encoderconfig"):
|
|
im.encoderconfig = ()
|
|
tile.sort(key=_tilesort)
|
|
# FIXME: make MAXBLOCK a configuration parameter
|
|
# It would be great if we could have the encoder specify what it needs
|
|
# But, it would need at least the image size in most cases. RawEncode is
|
|
# a tricky case.
|
|
bufsize = max(MAXBLOCK, bufsize, im.size[0] * 4) # see RawEncode.c
|
|
try:
|
|
fh = fp.fileno()
|
|
fp.flush()
|
|
_encode_tile(im, fp, tile, bufsize, fh)
|
|
except (AttributeError, io.UnsupportedOperation) as exc:
|
|
_encode_tile(im, fp, tile, bufsize, None, exc)
|
|
if hasattr(fp, "flush"):
|
|
fp.flush()
|
|
|
|
|
|
def _encode_tile(
|
|
im: Image.Image,
|
|
fp: IO[bytes],
|
|
tile: list[_Tile],
|
|
bufsize: int,
|
|
fh: int | None,
|
|
exc: BaseException | None = None,
|
|
) -> None:
|
|
for encoder_name, extents, offset, args in tile:
|
|
if offset > 0:
|
|
fp.seek(offset)
|
|
encoder = Image._getencoder(im.mode, encoder_name, args, im.encoderconfig)
|
|
try:
|
|
encoder.setimage(im.im, extents)
|
|
if encoder.pushes_fd:
|
|
encoder.setfd(fp)
|
|
errcode = encoder.encode_to_pyfd()[1]
|
|
else:
|
|
if exc:
|
|
# compress to Python file-compatible object
|
|
while True:
|
|
errcode, data = encoder.encode(bufsize)[1:]
|
|
fp.write(data)
|
|
if errcode:
|
|
break
|
|
else:
|
|
# slight speedup: compress to real file object
|
|
assert fh is not None
|
|
errcode = encoder.encode_to_file(fh, bufsize)
|
|
if errcode < 0:
|
|
raise _get_oserror(errcode, encoder=True) from exc
|
|
finally:
|
|
encoder.cleanup()
|
|
|
|
|
|
def _safe_read(fp: IO[bytes], size: int) -> bytes:
|
|
"""
|
|
Reads large blocks in a safe way. Unlike fp.read(n), this function
|
|
doesn't trust the user. If the requested size is larger than
|
|
SAFEBLOCK, the file is read block by block.
|
|
|
|
:param fp: File handle. Must implement a <b>read</b> method.
|
|
:param size: Number of bytes to read.
|
|
:returns: A string containing <i>size</i> bytes of data.
|
|
|
|
Raises an OSError if the file is truncated and the read cannot be completed
|
|
|
|
"""
|
|
if size <= 0:
|
|
return b""
|
|
if size <= SAFEBLOCK:
|
|
data = fp.read(size)
|
|
if len(data) < size:
|
|
msg = "Truncated File Read"
|
|
raise OSError(msg)
|
|
return data
|
|
blocks: list[bytes] = []
|
|
remaining_size = size
|
|
while remaining_size > 0:
|
|
block = fp.read(min(remaining_size, SAFEBLOCK))
|
|
if not block:
|
|
break
|
|
blocks.append(block)
|
|
remaining_size -= len(block)
|
|
if sum(len(block) for block in blocks) < size:
|
|
msg = "Truncated File Read"
|
|
raise OSError(msg)
|
|
return b"".join(blocks)
|
|
|
|
|
|
class PyCodecState:
|
|
def __init__(self) -> None:
|
|
self.xsize = 0
|
|
self.ysize = 0
|
|
self.xoff = 0
|
|
self.yoff = 0
|
|
|
|
def extents(self) -> tuple[int, int, int, int]:
|
|
return self.xoff, self.yoff, self.xoff + self.xsize, self.yoff + self.ysize
|
|
|
|
|
|
class PyCodec:
|
|
fd: IO[bytes] | None
|
|
|
|
def __init__(self, mode: str, *args: Any) -> None:
|
|
self.im: Image.core.ImagingCore | None = None
|
|
self.state = PyCodecState()
|
|
self.fd = None
|
|
self.mode = mode
|
|
self.init(args)
|
|
|
|
def init(self, args: tuple[Any, ...]) -> None:
|
|
"""
|
|
Override to perform codec specific initialization
|
|
|
|
:param args: Tuple of arg items from the tile entry
|
|
:returns: None
|
|
"""
|
|
self.args = args
|
|
|
|
def cleanup(self) -> None:
|
|
"""
|
|
Override to perform codec specific cleanup
|
|
|
|
:returns: None
|
|
"""
|
|
pass
|
|
|
|
def setfd(self, fd: IO[bytes]) -> None:
|
|
"""
|
|
Called from ImageFile to set the Python file-like object
|
|
|
|
:param fd: A Python file-like object
|
|
:returns: None
|
|
"""
|
|
self.fd = fd
|
|
|
|
def setimage(
|
|
self,
|
|
im: Image.core.ImagingCore,
|
|
extents: tuple[int, int, int, int] | None = None,
|
|
) -> None:
|
|
"""
|
|
Called from ImageFile to set the core output image for the codec
|
|
|
|
:param im: A core image object
|
|
:param extents: a 4 tuple of (x0, y0, x1, y1) defining the rectangle
|
|
for this tile
|
|
:returns: None
|
|
"""
|
|
|
|
# following c code
|
|
self.im = im
|
|
|
|
if extents:
|
|
x0, y0, x1, y1 = extents
|
|
|
|
if x0 < 0 or y0 < 0 or x1 > self.im.size[0] or y1 > self.im.size[1]:
|
|
msg = "Tile cannot extend outside image"
|
|
raise ValueError(msg)
|
|
|
|
self.state.xoff = x0
|
|
self.state.yoff = y0
|
|
self.state.xsize = x1 - x0
|
|
self.state.ysize = y1 - y0
|
|
else:
|
|
self.state.xsize, self.state.ysize = self.im.size
|
|
|
|
if self.state.xsize <= 0 or self.state.ysize <= 0:
|
|
msg = "Size must be positive"
|
|
raise ValueError(msg)
|
|
|
|
|
|
class PyDecoder(PyCodec):
|
|
"""
|
|
Python implementation of a format decoder. Override this class and
|
|
add the decoding logic in the :meth:`decode` method.
|
|
|
|
See :ref:`Writing Your Own File Codec in Python<file-codecs-py>`
|
|
"""
|
|
|
|
_pulls_fd = False
|
|
|
|
@property
|
|
def pulls_fd(self) -> bool:
|
|
return self._pulls_fd
|
|
|
|
def decode(self, buffer: bytes | Image.SupportsArrayInterface) -> tuple[int, int]:
|
|
"""
|
|
Override to perform the decoding process.
|
|
|
|
:param buffer: A bytes object with the data to be decoded.
|
|
:returns: A tuple of ``(bytes consumed, errcode)``.
|
|
If finished with decoding return -1 for the bytes consumed.
|
|
Err codes are from :data:`.ImageFile.ERRORS`.
|
|
"""
|
|
msg = "unavailable in base decoder"
|
|
raise NotImplementedError(msg)
|
|
|
|
def set_as_raw(
|
|
self, data: bytes, rawmode: str | None = None, extra: tuple[Any, ...] = ()
|
|
) -> None:
|
|
"""
|
|
Convenience method to set the internal image from a stream of raw data
|
|
|
|
:param data: Bytes to be set
|
|
:param rawmode: The rawmode to be used for the decoder.
|
|
If not specified, it will default to the mode of the image
|
|
:param extra: Extra arguments for the decoder.
|
|
:returns: None
|
|
"""
|
|
|
|
if not rawmode:
|
|
rawmode = self.mode
|
|
d = Image._getdecoder(self.mode, "raw", rawmode, extra)
|
|
assert self.im is not None
|
|
d.setimage(self.im, self.state.extents())
|
|
s = d.decode(data)
|
|
|
|
if s[0] >= 0:
|
|
msg = "not enough image data"
|
|
raise ValueError(msg)
|
|
if s[1] != 0:
|
|
msg = "cannot decode image data"
|
|
raise ValueError(msg)
|
|
|
|
|
|
class PyEncoder(PyCodec):
|
|
"""
|
|
Python implementation of a format encoder. Override this class and
|
|
add the decoding logic in the :meth:`encode` method.
|
|
|
|
See :ref:`Writing Your Own File Codec in Python<file-codecs-py>`
|
|
"""
|
|
|
|
_pushes_fd = False
|
|
|
|
@property
|
|
def pushes_fd(self) -> bool:
|
|
return self._pushes_fd
|
|
|
|
def encode(self, bufsize: int) -> tuple[int, int, bytes]:
|
|
"""
|
|
Override to perform the encoding process.
|
|
|
|
:param bufsize: Buffer size.
|
|
:returns: A tuple of ``(bytes encoded, errcode, bytes)``.
|
|
If finished with encoding return 1 for the error code.
|
|
Err codes are from :data:`.ImageFile.ERRORS`.
|
|
"""
|
|
msg = "unavailable in base encoder"
|
|
raise NotImplementedError(msg)
|
|
|
|
def encode_to_pyfd(self) -> tuple[int, int]:
|
|
"""
|
|
If ``pushes_fd`` is ``True``, then this method will be used,
|
|
and ``encode()`` will only be called once.
|
|
|
|
:returns: A tuple of ``(bytes consumed, errcode)``.
|
|
Err codes are from :data:`.ImageFile.ERRORS`.
|
|
"""
|
|
if not self.pushes_fd:
|
|
return 0, -8 # bad configuration
|
|
bytes_consumed, errcode, data = self.encode(0)
|
|
if data:
|
|
assert self.fd is not None
|
|
self.fd.write(data)
|
|
return bytes_consumed, errcode
|
|
|
|
def encode_to_file(self, fh: int, bufsize: int) -> int:
|
|
"""
|
|
:param fh: File handle.
|
|
:param bufsize: Buffer size.
|
|
|
|
:returns: If finished successfully, return 0.
|
|
Otherwise, return an error code. Err codes are from
|
|
:data:`.ImageFile.ERRORS`.
|
|
"""
|
|
errcode = 0
|
|
while errcode == 0:
|
|
status, errcode, buf = self.encode(bufsize)
|
|
if status > 0:
|
|
os.write(fh, buf[status:])
|
|
return errcode
|