mirror of
https://github.com/openswarm-ai/openswarm.git
synced 2026-09-23 10:04:53 +02:00
[aidan] feat: scheduled tasks (#105)
* [eric] ci: gitleaks-ignore the known historical secrets so our branch stops failing on leaks it didnt add * [eric] workflows: restore scheduled-tasks on the workflow line (revert removal, keep windows fixes + 1.1.69) * [eric] workflows: re-apply uncommitted scheduling wip (schedule pill, calendar view, slice) * [eric] ops: gitignore dev-team local state files * [eric] ops: backlog item for download-tracking visibility * [eric] ci: allowlist the cdp-routes redaction-test token in gitleaks * [aidan] feat/scheduled-tasks: keep step labels in sync on edit and show chevron on every step * [aidan] fix: schedule time in chat * [aidan] ux/workflows: add workflow step removal (#91) * [aidan] feat/scheduled-tasks: remember workflow tool permissions across runs * [aidan] feat/task-scheduling: add hourly and minute (15-min minimum) schedule intervals (#93) * [aidan] feat/scheduled-tasks: calendar, rename, and edit workflows (#94) * [aidan] bug: fix schedule button * [aidan] fix/agent-errors: surface provider rate limits * [aidan] ux/cards: click-to-rename for chat and workflow titles Single-click a card's title to enter edit mode inline. Commit on Enter/blur, cancel on Escape. Rename persists via PATCH for workflows and sessions. * [aidan] feat/workflows: seed build prompt for zero-step workflows When a new workflow has no steps, seed the agent with a prompt asking the user to describe what the workflow should do, rather than starting blank. * [aidan] feat/workflows: add-to-schedule popover for unscheduled workflows Clicking the "+" on an unscheduled workflow row opens a popover with two options: - Keep this schedule: enables the workflow's existing cadence and moves it to Scheduled - Change schedule: opens the scheduling editor to pick a different time * [aidan] ux/workflows: wire add-to-schedule popover and simplify New button - Made the "+" icon on unscheduled workflow rows clickable, opening a popover to keep or change the schedule - Removed AddIcon from toolbar "New" button (now reads "New" instead of "+ New") * [aidan] fix/scheduled-tasks: open schedule calendar when Schedule pill clicked Fixed the Schedule pill click being swallowed by the toolbar's dismiss handler. Exempted the toolbar pills via data-toolbar-pills so their click handlers fire. * [aidan] ux/workflows: open New workflow in agent build chat instead of empty card When creating a new workflow from the hub, open it in edit_agent view (with the agent builder chat) instead of a preview card. The workflow is created on the backend first so the embedded session has a real ID. * [aidan] feat/workflow-edit: add draft testing save flow * [aidan] ux/chat: remove continue chat button * [aidan] ux/workflows: polish workflow card interactions * [aidan] fix/workflow-scheduling: save unscheduled workflows as drafts * aidan ui: schedule naming changes * [aidan] ui: tool calling desc/naming * [aidan] ui: calendar sidebar naming * [aidan] ui: fix stop viewing closing chat * [aidan] feat/workflows: auto-name workflows and polish the build flow (#95) * [aidan] feat/workflow-auto-naming: auto-generate workflow titles from steps Generate a title + description from a workflow's steps (one aux call, reused for step labels) whenever it is still auto_named, so a workflow built in the Edit Agent names itself on commit instead of staying "New workflow". A manual rename sets auto_named=False and is never overwritten. Stream the aux call (non-streaming drops content on some 9router lanes) and fall back to a step-derived title when the model is unavailable. * [aidan] feat/workflows: hide unsaved new workflows until first save A brand-new "+ New" workflow is created with unsaved=true and kept out of the hub's scheduled/unscheduled lists while the user is still building it in the Edit Agent. The first commit (Save) clears the flag and the workflow appears. Every other create path stays visible immediately. * [aidan] ux/workflows: remove redundant save workflow button The Edit Agent already has Discard/Save controls in its strip, so the header "Save Workflow" button was a duplicate save path. Remove it and its pulse/edit-session-id wiring; the model/time subtitle stays. * [aidan] ux/workflows: animate title on auto-rename Wrap the workflow card title in the same Typewriter the chat card uses, so when the auto-generated name replaces the placeholder after Save it retypes letter-by-letter. Gated on a real (non-placeholder) title so it never animates on mount or for already-named workflows. * [aidan] ux/workflows: animate sidebar title on auto-rename Wrap the calendar hub's sidebar row title in the same Typewriter the workflow card uses, so a title that auto-renames retypes letter-by-letter in the sidebar too. Extract the placeholder/isRealTitle guard into the shared workflowVisuals so the card and sidebar stay in sync. * [aidan] fix/workflows: connect watch tether, keep watched chat open, wire draft run/history * [aidan] ui: grey out chat pill when not selected * [aidan] ui: fix running agent display * [aidan] feat/history-popover: add chat history and scheduled tasks run log tabs (#96) * [aidan] ux/schedule: toast when calendar view already open on expand * [aidan] ui: fix popover descs * [aidan] feat/workflow-runs: add pause, resume, and stop controls for live runs * [aidan] feat/schedule-calendar: add calendar occurrences endpoint and concrete timezones * [aidan] feat/workflows: require at least one step to save a workflow * [aidan] ux/edit-agent: hide Discard for an unsaved new workflow * [aidan] ux/edit-agent: move fix-prefix card below the step list * [aidan] feat/workflows: toast when an unattended scheduled run starts * [aidan] fix/dashboard-tethers: anchor workflow-sidecar tethers to measured card rects * [aidan] feat/workflows: validate steps before scheduling and keep chat tool memory * [aidan] feat/mcp-suggestions: dismissable integration banner with per-session cooldown * [aidan] feat/workflows: add scheduled-run "running now" toast with click-to-view (#97) * [aidan] ux/workflows: surface paused state on card, sidebar, and calendar; tidy run history * [aidan] feat/mcp-suggestions: suggest both Google and Microsoft when provider is ambiguous * [aidan] fix/agent-tokens: friendly out-of-tokens card across all agent surfaces * [aidan] feat/workflow-model: persist edit-agent model on save with switch notice and fresh drafts * [aidan] fix/workflow-chat: force stop on watched run mirrors workflow card stop * [aidan] fix/workflow-cards: keep watched run tethered on finish to avoid duplicate chat * [aidan] feat/schedule-calendar: mark current time with a now line in week view * [aidan] refactor/private-names: rename error and schedule classifiers from _ to p_ * [aidan] feat/scheduled-tasks: agent workflow scheduling and in-chat convert (#98) * [aidan] feat/workflow-suggest: nudge user to convert repeatable chat to workflow Add SuggestConvertToWorkflow MCP tool that agents call at the end of a task when they've completed something worth repeating (daily report, weekly check, recurring data pull). Frontend detects the tool call and glows the "Convert to workflow" button 3 times to draw the eye. When user clicks it, the suggested cadence (e.g. "every weekday at 9am") is stored in the draft and seeded into the scheduling agent's first prompt, so the agent can act on the suggestion rather than asking the user again. Tool is never auto-called — agents decide when a task is genuinely repeatable (not debugging, creative work, one-off lookup). Tool description emphasizes sparse, high-confidence use only (once per session max). Files changed: - backend/apps/agents/schedule_mcp_server.py: add SuggestConvertToWorkflow tool - frontend/src/shared/mcpToolMeta.ts: add label for new tool - frontend/src/app/pages/Dashboard/cards/AgentCard.tsx: detect suggestion in session messages, show+glow "Convert to workflow" button, pass cadence to draft - frontend/src/shared/state/workflowsSlice.ts: add suggested_cadence field to Workflow interface - frontend/src/app/pages/Workflows/SchedulingView.tsx: seed scheduling agent prompt with suggested cadence hint * [aidan] feat/agent-scheduling: route recurring asks through native workflows, deny claude cron skill * [aidan] feat/workflow-convert: in-chat convert popup and auto-open scheduled workflow card * [aidan] ux/calendar-page: schedule calendar restyle + popover fixes (#99) * [aidan] fix/dashboard-delete: remove workflows calendar panel on delete key * [aidan] ux/workflows-calendar: restyle hub, fix today highlight, add toolbar toggle * [aidan] ux/schedule-popover: compact density, fix sticky header bleed, add header spacing * [aidan] ux/schedule-calendar: hollow ring dot for past fires in month view * [aidan] ux/schedule-calendar: clickable +N more opens day's full run list * [aidan] feat/run-log-filters: add success and skipped pills to scheduled task history * [aidan] fix/convert-button: stop drag capture so convert-to-workflow click fires * [aidan] ux/calendar-card: match border color and radius to chat and workflow cards * [aidan] fix/minimap: render missed-runs card on the minimap * [aidan] ux/run-sparkline: simplify tooltip to plain run tally * [aidan] ux/run-history: collapse expanded run view to one clickable line * [aidan] ux/calendar-card: match corner radius to browser cards * [aidan] feat/workflows: launch-time scheduling UX and workflow-card polish (#101) * [aidan] feat/schedule-list: lazy-load list view via scroll sentinel * [aidan] feat/missed-runs: launch toast with per-workflow counts and pan-to-card * [aidan] fix/dashboard-tethers: keep watching line anchored on canvas zoom * [aidan] feat/scheduled-tasks: review missed runs at launch instead of auto-firing on_missed * [aidan] refactor/workflow-cards: use radius and status design tokens, polish card chrome * [aidan] ux/agent-card: keep convert-to-workflow visible during runs with mid-turn toast * [aidan] ux/mcp-bubble: drop redundant verb label when a workflow label is shown * [aidan] chore/backend: remove stale explanatory comments * [aidan] fix/workflows-hub: load workflows on hub mount so calendar fills at launch * [aidan] feat/workflows: generate title, description, step labels at convert time * [aidan] fix/tidy-layout: include workflows hub in tidy and fit-to-view * [aidan] feat/schedule-list: window long list via measured-height virtualizer * [aidan] ux/workflows-hub: remove time-saved badge from calendar header * [aidan] fix/types: add missing semantic-type labels and drop stray fade arg * [aidan] feat/schedule: pin monthly day-of-month and honor repeat-every intervals * [aidan] feat/schedule: inherit source-session tool surface for scheduled runs * [aidan] ux/calendar: restack hour-cell events as bars with overflow affordance * [aidan] feat/calendar: open the run card when clicking a scheduled occurrence * [aidan] ux/missed-runs: add per-group select-all toggle and rename skip action * [aidan] feat: new scheduled task design ported * [aidan] ui: sidebar reorder, repeat controls on schedule card * [aidan] ui: sidebar, scheduling time * [aidan] feat/schedule: pin monthly last-day-of-month * [aidan] feat/steps: per-step enable toggle * [aidan] feat/workflows: per-workflow color swatch * [aidan] feat/trash: soft-delete workflows with restore and purge * [aidan] feat/run-monitor: live run monitor card on the canvas * [aidan] feat/run-context: attach a run as removable chat context * [aidan] feat/compose: new-workflow landing page and auto-commit build flow * [aidan] ui/workflows: dark mode and design-system cohesion * [aidan] ui/calendar: overflow popover, condensed week view, scroll fix * [aidan] feat/home: ongoing runs, missed review, and accurate Coming-up counts * [aidan] fix/run-status: sync ongoing runs and heal stuck/interrupted runs * [aidan] ux/schedule: last-day-of-month UI, Run-at time typing, interval input * [aidan] ux/workflows: default window size and toolbar icon * [aidan] fix/schedule: measure ran_late from start and anchor recurrences to created_at * [aidan] feat/calendar: render fire times from backend, drop JS recurrence reimpl * [aidan] chore/dashboard: drop dead configure/missed-run cards, refetch on reconnect * [aidan] chore/agent-card: remove unreachable convert-to-workflow action * [aidan] fix/workflows: don't bump updated_at on a no-op draft commit so viewing a workflow doesn't reorder the sidebar * [aidan] feat/schedule: warn when scheduling a workflow that has no steps * [aidan] fix/selection-tool: never select the workflows app, and exit the tool on Escape without dropping selections * [aidan] ux/compose: diversify new-workflow starter prompts across personas * [aidan] ux/run-monitor: spawn the run card a bit farther right of the workflows app * [aidan] fix/schedule: harden run recovery and storage writes against crashes * [aidan] ux/compose: restyle new-workflow starters as a clean pill cluster with rich prompts * [aidan] fix/workflows: optimistically apply edits so the schedule banner updates instantly * [aidan] ui/workflows: three-tone surface depth so the window lifts off the canvas in both themes * [aidan] ui/workflows: close buttons turn red on hover, matching the chat card * [aidan] test/schedule: cover executor pipeline, storage durability, and recurrence gaps * [aidan] fix: remove package-lock json * [aidan] fix/workflows-compose: keep compose view until edit agent replies * [aidan] feat/workflows: auto-generate workflow + step titles with typewriter animation * [eric] deps: restore frontend/package-lock.json (PR #105 deletion broke npm ci) --------- Co-authored-by: Eric <ciregenz@berkeley.edu> Co-authored-by: cire <134991075+ciregenz@users.noreply.github.com>
This commit is contained in:
@@ -0,0 +1,34 @@
|
||||
"""Parse docstrings as per Sphinx notation."""
|
||||
|
||||
from .common import (
|
||||
Docstring,
|
||||
DocstringDeprecated,
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringRaises,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
ParseError,
|
||||
RenderingStyle,
|
||||
)
|
||||
from .parser import compose, parse, parse_from_object
|
||||
from .util import combine_docstrings
|
||||
|
||||
Style = DocstringStyle # backwards compatibility
|
||||
|
||||
__all__ = [
|
||||
"parse",
|
||||
"parse_from_object",
|
||||
"combine_docstrings",
|
||||
"compose",
|
||||
"ParseError",
|
||||
"Docstring",
|
||||
"DocstringMeta",
|
||||
"DocstringParam",
|
||||
"DocstringRaises",
|
||||
"DocstringReturns",
|
||||
"DocstringDeprecated",
|
||||
"DocstringStyle",
|
||||
"RenderingStyle",
|
||||
"Style",
|
||||
]
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Attribute docstrings parsing.
|
||||
|
||||
.. seealso:: https://peps.python.org/pep-0257/#what-is-a-docstring
|
||||
"""
|
||||
|
||||
import ast
|
||||
import inspect
|
||||
import textwrap
|
||||
import typing as T
|
||||
from types import ModuleType
|
||||
|
||||
from .common import Docstring, DocstringParam
|
||||
|
||||
|
||||
def ast_get_constant_value(node: ast.AST) -> T.Any:
|
||||
"""Return the constant's value if the given node is a constant."""
|
||||
return getattr(node, "value")
|
||||
|
||||
|
||||
def ast_unparse(node: ast.AST) -> T.Optional[str]:
|
||||
"""Convert the AST node to source code as a string."""
|
||||
if hasattr(ast, "unparse"):
|
||||
return ast.unparse(node)
|
||||
# Support simple cases in Python < 3.9
|
||||
if isinstance(node, ast.Constant):
|
||||
return str(ast_get_constant_value(node))
|
||||
if isinstance(node, ast.Name):
|
||||
return node.id
|
||||
return None
|
||||
|
||||
|
||||
def ast_is_literal_str(node: ast.AST) -> bool:
|
||||
"""Return True if the given node is a literal string."""
|
||||
return (
|
||||
isinstance(node, ast.Expr)
|
||||
and isinstance(node.value, ast.Constant)
|
||||
and isinstance(ast_get_constant_value(node.value), str)
|
||||
)
|
||||
|
||||
|
||||
def ast_get_attribute(
|
||||
node: ast.AST,
|
||||
) -> T.Optional[T.Tuple[str, T.Optional[str], T.Optional[str]]]:
|
||||
"""Return name, type and default if the given node is an attribute."""
|
||||
if isinstance(node, (ast.Assign, ast.AnnAssign)):
|
||||
target = (
|
||||
node.targets[0] if isinstance(node, ast.Assign) else node.target
|
||||
)
|
||||
if isinstance(target, ast.Name):
|
||||
type_str = None
|
||||
if isinstance(node, ast.AnnAssign):
|
||||
type_str = ast_unparse(node.annotation)
|
||||
default = None
|
||||
if node.value:
|
||||
default = ast_unparse(node.value)
|
||||
return target.id, type_str, default
|
||||
return None
|
||||
|
||||
|
||||
class AttributeDocstrings(ast.NodeVisitor):
|
||||
"""An ast.NodeVisitor that collects attribute docstrings."""
|
||||
|
||||
attr_docs = None
|
||||
prev_attr = None
|
||||
|
||||
def visit(self, node):
|
||||
if self.prev_attr and ast_is_literal_str(node):
|
||||
attr_name, attr_type, attr_default = self.prev_attr
|
||||
self.attr_docs[attr_name] = (
|
||||
ast_get_constant_value(node.value),
|
||||
attr_type,
|
||||
attr_default,
|
||||
)
|
||||
self.prev_attr = ast_get_attribute(node)
|
||||
if isinstance(node, (ast.ClassDef, ast.Module)):
|
||||
self.generic_visit(node)
|
||||
|
||||
def get_attr_docs(
|
||||
self, component: T.Any
|
||||
) -> T.Dict[str, T.Tuple[str, T.Optional[str], T.Optional[str]]]:
|
||||
"""Get attribute docstrings from the given component.
|
||||
|
||||
:param component: component to process (class or module)
|
||||
:returns: for each attribute docstring, a tuple with (description,
|
||||
type, default)
|
||||
"""
|
||||
self.attr_docs = {}
|
||||
self.prev_attr = None
|
||||
try:
|
||||
source = textwrap.dedent(inspect.getsource(component))
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
tree = ast.parse(source)
|
||||
if inspect.ismodule(component):
|
||||
self.visit(tree)
|
||||
elif isinstance(tree, ast.Module) and isinstance(
|
||||
tree.body[0], ast.ClassDef
|
||||
):
|
||||
self.visit(tree.body[0])
|
||||
return self.attr_docs
|
||||
|
||||
|
||||
def add_attribute_docstrings(
|
||||
obj: T.Union[type, ModuleType], docstring: Docstring
|
||||
) -> None:
|
||||
"""Add attribute docstrings found in the object's source code.
|
||||
|
||||
:param obj: object from which to parse attribute docstrings
|
||||
:param docstring: Docstring object where found attributes are added
|
||||
:returns: list with names of added attributes
|
||||
"""
|
||||
params = set(p.arg_name for p in docstring.params)
|
||||
for arg_name, (description, type_name, default) in (
|
||||
AttributeDocstrings().get_attr_docs(obj).items()
|
||||
):
|
||||
if arg_name not in params:
|
||||
param = DocstringParam(
|
||||
args=["attribute", arg_name],
|
||||
description=description,
|
||||
arg_name=arg_name,
|
||||
type_name=type_name,
|
||||
is_optional=default is not None,
|
||||
default=default,
|
||||
)
|
||||
docstring.meta.append(param)
|
||||
@@ -0,0 +1,229 @@
|
||||
"""Common methods for parsing."""
|
||||
|
||||
import enum
|
||||
import typing as T
|
||||
|
||||
PARAM_KEYWORDS = {
|
||||
"param",
|
||||
"parameter",
|
||||
"arg",
|
||||
"argument",
|
||||
"attribute",
|
||||
"key",
|
||||
"keyword",
|
||||
}
|
||||
RAISES_KEYWORDS = {"raises", "raise", "except", "exception"}
|
||||
DEPRECATION_KEYWORDS = {"deprecation", "deprecated"}
|
||||
RETURNS_KEYWORDS = {"return", "returns"}
|
||||
YIELDS_KEYWORDS = {"yield", "yields"}
|
||||
EXAMPLES_KEYWORDS = {"example", "examples"}
|
||||
|
||||
|
||||
class ParseError(RuntimeError):
|
||||
"""Base class for all parsing related errors."""
|
||||
|
||||
|
||||
class DocstringStyle(enum.Enum):
|
||||
"""Docstring style."""
|
||||
|
||||
REST = 1
|
||||
GOOGLE = 2
|
||||
NUMPYDOC = 3
|
||||
EPYDOC = 4
|
||||
AUTO = 255
|
||||
|
||||
|
||||
class RenderingStyle(enum.Enum):
|
||||
"""Rendering style when unparsing parsed docstrings."""
|
||||
|
||||
COMPACT = 1
|
||||
CLEAN = 2
|
||||
EXPANDED = 3
|
||||
|
||||
|
||||
class DocstringMeta:
|
||||
"""Docstring meta information.
|
||||
|
||||
Symbolizes lines in form of
|
||||
|
||||
:param arg: description
|
||||
:raises ValueError: if something happens
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self, args: T.List[str], description: T.Optional[str]
|
||||
) -> None:
|
||||
"""Initialize self.
|
||||
|
||||
:param args: list of arguments. The exact content of this variable is
|
||||
dependent on the kind of docstring; it's used to distinguish
|
||||
between custom docstring meta information items.
|
||||
:param description: associated docstring description.
|
||||
"""
|
||||
self.args = args
|
||||
self.description = description
|
||||
|
||||
|
||||
class DocstringParam(DocstringMeta):
|
||||
"""DocstringMeta symbolizing :param metadata."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
args: T.List[str],
|
||||
description: T.Optional[str],
|
||||
arg_name: str,
|
||||
type_name: T.Optional[str],
|
||||
is_optional: T.Optional[bool],
|
||||
default: T.Optional[str],
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
super().__init__(args, description)
|
||||
self.arg_name = arg_name
|
||||
self.type_name = type_name
|
||||
self.is_optional = is_optional
|
||||
self.default = default
|
||||
|
||||
|
||||
class DocstringReturns(DocstringMeta):
|
||||
"""DocstringMeta symbolizing :returns or :yields metadata."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
args: T.List[str],
|
||||
description: T.Optional[str],
|
||||
type_name: T.Optional[str],
|
||||
is_generator: bool,
|
||||
return_name: T.Optional[str] = None,
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
super().__init__(args, description)
|
||||
self.type_name = type_name
|
||||
self.is_generator = is_generator
|
||||
self.return_name = return_name
|
||||
|
||||
|
||||
class DocstringRaises(DocstringMeta):
|
||||
"""DocstringMeta symbolizing :raises metadata."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
args: T.List[str],
|
||||
description: T.Optional[str],
|
||||
type_name: T.Optional[str],
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
super().__init__(args, description)
|
||||
self.type_name = type_name
|
||||
self.description = description
|
||||
|
||||
|
||||
class DocstringDeprecated(DocstringMeta):
|
||||
"""DocstringMeta symbolizing deprecation metadata."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
args: T.List[str],
|
||||
description: T.Optional[str],
|
||||
version: T.Optional[str],
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
super().__init__(args, description)
|
||||
self.version = version
|
||||
self.description = description
|
||||
|
||||
|
||||
class DocstringExample(DocstringMeta):
|
||||
"""DocstringMeta symbolizing example metadata."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
args: T.List[str],
|
||||
snippet: T.Optional[str],
|
||||
description: T.Optional[str],
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
super().__init__(args, description)
|
||||
self.snippet = snippet
|
||||
self.description = description
|
||||
|
||||
|
||||
class Docstring:
|
||||
"""Docstring object representation."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
style=None, # type: T.Optional[DocstringStyle]
|
||||
) -> None:
|
||||
"""Initialize self."""
|
||||
self.short_description = None # type: T.Optional[str]
|
||||
self.long_description = None # type: T.Optional[str]
|
||||
self.blank_after_short_description = False
|
||||
self.blank_after_long_description = False
|
||||
self.meta = [] # type: T.List[DocstringMeta]
|
||||
self.style = style # type: T.Optional[DocstringStyle]
|
||||
|
||||
@property
|
||||
def description(self) -> T.Optional[str]:
|
||||
"""Return the full description of the function
|
||||
|
||||
Returns None if the docstring did not include any description
|
||||
"""
|
||||
ret = []
|
||||
if self.short_description:
|
||||
ret.append(self.short_description)
|
||||
if self.blank_after_short_description:
|
||||
ret.append("")
|
||||
if self.long_description:
|
||||
ret.append(self.long_description)
|
||||
|
||||
if not ret:
|
||||
return None
|
||||
|
||||
return "\n".join(ret)
|
||||
|
||||
@property
|
||||
def params(self) -> T.List[DocstringParam]:
|
||||
"""Return a list of information on function params."""
|
||||
return [item for item in self.meta if isinstance(item, DocstringParam)]
|
||||
|
||||
@property
|
||||
def raises(self) -> T.List[DocstringRaises]:
|
||||
"""Return a list of information on the exceptions that the function
|
||||
may raise.
|
||||
"""
|
||||
return [
|
||||
item for item in self.meta if isinstance(item, DocstringRaises)
|
||||
]
|
||||
|
||||
@property
|
||||
def returns(self) -> T.Optional[DocstringReturns]:
|
||||
"""Return a single information on function return.
|
||||
|
||||
Takes the first return information.
|
||||
"""
|
||||
for item in self.meta:
|
||||
if isinstance(item, DocstringReturns):
|
||||
return item
|
||||
return None
|
||||
|
||||
@property
|
||||
def many_returns(self) -> T.List[DocstringReturns]:
|
||||
"""Return a list of information on function return."""
|
||||
return [
|
||||
item for item in self.meta if isinstance(item, DocstringReturns)
|
||||
]
|
||||
|
||||
@property
|
||||
def deprecation(self) -> T.Optional[DocstringDeprecated]:
|
||||
"""Return a single information on function deprecation notes."""
|
||||
for item in self.meta:
|
||||
if isinstance(item, DocstringDeprecated):
|
||||
return item
|
||||
return None
|
||||
|
||||
@property
|
||||
def examples(self) -> T.List[DocstringExample]:
|
||||
"""Return a list of information on function examples."""
|
||||
return [
|
||||
item for item in self.meta if isinstance(item, DocstringExample)
|
||||
]
|
||||
@@ -0,0 +1,284 @@
|
||||
"""Epyoc-style docstring parsing.
|
||||
|
||||
.. seealso:: http://epydoc.sourceforge.net/manual-fields.html
|
||||
"""
|
||||
|
||||
import inspect
|
||||
import re
|
||||
import typing as T
|
||||
|
||||
from .common import (
|
||||
Docstring,
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringRaises,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
ParseError,
|
||||
RenderingStyle,
|
||||
)
|
||||
|
||||
|
||||
def _clean_str(string: str) -> T.Optional[str]:
|
||||
string = string.strip()
|
||||
if len(string) > 0:
|
||||
return string
|
||||
return None
|
||||
|
||||
|
||||
def parse(text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the epydoc-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
ret = Docstring(style=DocstringStyle.EPYDOC)
|
||||
if not text:
|
||||
return ret
|
||||
|
||||
text = inspect.cleandoc(text)
|
||||
match = re.search("^@", text, flags=re.M)
|
||||
if match:
|
||||
desc_chunk = text[: match.start()]
|
||||
meta_chunk = text[match.start() :]
|
||||
else:
|
||||
desc_chunk = text
|
||||
meta_chunk = ""
|
||||
|
||||
parts = desc_chunk.split("\n", 1)
|
||||
ret.short_description = parts[0] or None
|
||||
if len(parts) > 1:
|
||||
long_desc_chunk = parts[1] or ""
|
||||
ret.blank_after_short_description = long_desc_chunk.startswith("\n")
|
||||
ret.blank_after_long_description = long_desc_chunk.endswith("\n\n")
|
||||
ret.long_description = long_desc_chunk.strip() or None
|
||||
|
||||
param_pattern = re.compile(
|
||||
r"(param|keyword|type)(\s+[_A-z][_A-z0-9]*\??):"
|
||||
)
|
||||
attribute_pattern = re.compile(r"(ivar|cvar|var)(\s+[_A-z][_A-z0-9]*\??):")
|
||||
raise_pattern = re.compile(r"(raise)(\s+[_A-z][_A-z0-9]*\??)?:")
|
||||
return_pattern = re.compile(r"(return|rtype|yield|ytype):")
|
||||
meta_pattern = re.compile(
|
||||
r"([_A-z][_A-z0-9]+)((\s+[_A-z][_A-z0-9]*\??)*):"
|
||||
)
|
||||
|
||||
# tokenize
|
||||
stream: T.List[T.Tuple[str, str, T.List[str], str]] = []
|
||||
for match in re.finditer(
|
||||
r"(^@.*?)(?=^@|\Z)", meta_chunk, flags=re.S | re.M
|
||||
):
|
||||
chunk = match.group(0)
|
||||
if not chunk:
|
||||
continue
|
||||
|
||||
param_match = re.search(param_pattern, chunk)
|
||||
attribute_match = re.search(attribute_pattern, chunk)
|
||||
raise_match = re.search(raise_pattern, chunk)
|
||||
return_match = re.search(return_pattern, chunk)
|
||||
meta_match = re.search(meta_pattern, chunk)
|
||||
|
||||
match = (
|
||||
param_match
|
||||
or attribute_match
|
||||
or raise_match
|
||||
or return_match
|
||||
or meta_match
|
||||
)
|
||||
if not match:
|
||||
raise ParseError(f'Error parsing meta information near "{chunk}".')
|
||||
|
||||
desc_chunk = chunk[match.end() :]
|
||||
if param_match:
|
||||
base = "param"
|
||||
key: str = match.group(1)
|
||||
args = [match.group(2).strip()]
|
||||
elif attribute_match:
|
||||
base = "attribute"
|
||||
key: str = match.group(1)
|
||||
args = [match.group(2).strip()]
|
||||
elif raise_match:
|
||||
base = "raise"
|
||||
key: str = match.group(1)
|
||||
args = [] if match.group(2) is None else [match.group(2).strip()]
|
||||
elif return_match:
|
||||
base = "return"
|
||||
key: str = match.group(1)
|
||||
args = []
|
||||
else:
|
||||
base = "meta"
|
||||
key: str = match.group(1)
|
||||
token = _clean_str(match.group(2).strip())
|
||||
args = [] if token is None else re.split(r"\s+", token)
|
||||
|
||||
# Make sure we didn't match some existing keyword in an incorrect
|
||||
# way here:
|
||||
if key in [
|
||||
"param",
|
||||
"ivar",
|
||||
"cvar",
|
||||
"var",
|
||||
"keyword",
|
||||
"type",
|
||||
"return",
|
||||
"rtype",
|
||||
"yield",
|
||||
"ytype",
|
||||
]:
|
||||
raise ParseError(
|
||||
f'Error parsing meta information near "{chunk}".'
|
||||
)
|
||||
|
||||
desc = desc_chunk.strip()
|
||||
if "\n" in desc:
|
||||
first_line, rest = desc.split("\n", 1)
|
||||
desc = first_line + "\n" + inspect.cleandoc(rest)
|
||||
stream.append((base, key, args, desc))
|
||||
|
||||
# Combine type_name, arg_name, and description information
|
||||
params: T.Dict[str, T.Dict[str, T.Any]] = {}
|
||||
for base, key, args, desc in stream:
|
||||
if base not in ["param", "attribute", "return"]:
|
||||
continue # nothing to do
|
||||
|
||||
(arg_name,) = args or ("return",)
|
||||
info = params.setdefault(arg_name, {})
|
||||
info_key = "type_name" if "type" in key else "description"
|
||||
info[info_key] = desc
|
||||
|
||||
if base == "return":
|
||||
is_generator = key in {"ytype", "yield"}
|
||||
if info.setdefault("is_generator", is_generator) != is_generator:
|
||||
raise ParseError(
|
||||
f'Error parsing meta information for "{arg_name}".'
|
||||
)
|
||||
|
||||
is_done: T.Dict[str, bool] = {}
|
||||
for base, key, args, desc in stream:
|
||||
if base in ["param", "attribute"] and not is_done.get(args[0], False):
|
||||
(arg_name,) = args
|
||||
info = params[arg_name]
|
||||
type_name = info.get("type_name")
|
||||
|
||||
if type_name and type_name.endswith("?"):
|
||||
is_optional = True
|
||||
type_name = type_name[:-1]
|
||||
else:
|
||||
is_optional = False
|
||||
|
||||
match = re.match(r".*defaults to (.+)", desc, flags=re.DOTALL)
|
||||
default = match.group(1).rstrip(".") if match else None
|
||||
|
||||
meta_item = DocstringParam(
|
||||
args=[key, arg_name],
|
||||
description=info.get("description"),
|
||||
arg_name=arg_name,
|
||||
type_name=type_name,
|
||||
is_optional=is_optional,
|
||||
default=default,
|
||||
)
|
||||
is_done[arg_name] = True
|
||||
elif base == "return" and not is_done.get("return", False):
|
||||
info = params["return"]
|
||||
meta_item = DocstringReturns(
|
||||
args=[key],
|
||||
description=info.get("description"),
|
||||
type_name=info.get("type_name"),
|
||||
is_generator=info.get("is_generator", False),
|
||||
)
|
||||
is_done["return"] = True
|
||||
elif base == "raise":
|
||||
(type_name,) = args or (None,)
|
||||
meta_item = DocstringRaises(
|
||||
args=[key] + args,
|
||||
description=desc,
|
||||
type_name=type_name,
|
||||
)
|
||||
elif base == "meta":
|
||||
meta_item = DocstringMeta(
|
||||
args=[key] + args,
|
||||
description=desc,
|
||||
)
|
||||
else:
|
||||
(key, *_) = args or ("return",)
|
||||
assert is_done.get(key, False)
|
||||
continue # don't append
|
||||
|
||||
ret.meta.append(meta_item)
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
def compose(
|
||||
docstring: Docstring,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
indent: str = " ",
|
||||
) -> str:
|
||||
"""Render a parsed docstring into docstring text.
|
||||
|
||||
:param docstring: parsed docstring representation
|
||||
:param rendering_style: the style to render docstrings
|
||||
:param indent: the characters used as indentation in the docstring string
|
||||
:returns: docstring text
|
||||
"""
|
||||
|
||||
def process_desc(desc: T.Optional[str], is_type: bool) -> str:
|
||||
if not desc:
|
||||
return ""
|
||||
|
||||
if rendering_style == RenderingStyle.EXPANDED or (
|
||||
rendering_style == RenderingStyle.CLEAN and not is_type
|
||||
):
|
||||
(first, *rest) = desc.splitlines()
|
||||
return "\n".join(
|
||||
["\n" + indent + first] + [indent + line for line in rest]
|
||||
)
|
||||
|
||||
(first, *rest) = desc.splitlines()
|
||||
return "\n".join([" " + first] + [indent + line for line in rest])
|
||||
|
||||
parts: T.List[str] = []
|
||||
if docstring.short_description:
|
||||
parts.append(docstring.short_description)
|
||||
if docstring.blank_after_short_description:
|
||||
parts.append("")
|
||||
if docstring.long_description:
|
||||
parts.append(docstring.long_description)
|
||||
if docstring.blank_after_long_description:
|
||||
parts.append("")
|
||||
|
||||
for meta in docstring.meta:
|
||||
if isinstance(meta, DocstringParam):
|
||||
if meta.type_name:
|
||||
type_name = (
|
||||
f"{meta.type_name}?"
|
||||
if meta.is_optional
|
||||
else meta.type_name
|
||||
)
|
||||
text = f"@type {meta.arg_name}:"
|
||||
text += process_desc(type_name, True)
|
||||
parts.append(text)
|
||||
text = f"@param {meta.arg_name}:" + process_desc(
|
||||
meta.description, False
|
||||
)
|
||||
parts.append(text)
|
||||
elif isinstance(meta, DocstringReturns):
|
||||
(arg_key, type_key) = (
|
||||
("yield", "ytype")
|
||||
if meta.is_generator
|
||||
else ("return", "rtype")
|
||||
)
|
||||
if meta.type_name:
|
||||
text = f"@{type_key}:" + process_desc(meta.type_name, True)
|
||||
parts.append(text)
|
||||
if meta.description:
|
||||
text = f"@{arg_key}:" + process_desc(meta.description, False)
|
||||
parts.append(text)
|
||||
elif isinstance(meta, DocstringRaises):
|
||||
text = f"@raise {meta.type_name}:" if meta.type_name else "@raise:"
|
||||
text += process_desc(meta.description, False)
|
||||
parts.append(text)
|
||||
else:
|
||||
text = f'@{" ".join(meta.args)}:'
|
||||
text += process_desc(meta.description, False)
|
||||
parts.append(text)
|
||||
return "\n".join(parts)
|
||||
@@ -0,0 +1,414 @@
|
||||
"""Google-style docstring parsing."""
|
||||
|
||||
import inspect
|
||||
import re
|
||||
import typing as T
|
||||
from collections import OrderedDict, namedtuple
|
||||
from enum import IntEnum
|
||||
|
||||
from .common import (
|
||||
EXAMPLES_KEYWORDS,
|
||||
PARAM_KEYWORDS,
|
||||
RAISES_KEYWORDS,
|
||||
RETURNS_KEYWORDS,
|
||||
YIELDS_KEYWORDS,
|
||||
Docstring,
|
||||
DocstringExample,
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringRaises,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
ParseError,
|
||||
RenderingStyle,
|
||||
)
|
||||
|
||||
|
||||
class SectionType(IntEnum):
|
||||
"""Types of sections."""
|
||||
|
||||
SINGULAR = 0
|
||||
"""For sections like examples."""
|
||||
|
||||
MULTIPLE = 1
|
||||
"""For sections like params."""
|
||||
|
||||
SINGULAR_OR_MULTIPLE = 2
|
||||
"""For sections like returns or yields."""
|
||||
|
||||
|
||||
class Section(namedtuple("SectionBase", "title key type")):
|
||||
"""A docstring section."""
|
||||
|
||||
|
||||
GOOGLE_TYPED_ARG_REGEX = re.compile(r"\s*(.+?)\s*\(\s*(.*[^\s]+)\s*\)")
|
||||
GOOGLE_ARG_DESC_REGEX = re.compile(r".*\. Defaults to (.+)\.")
|
||||
MULTIPLE_PATTERN = re.compile(r"(\s*[^:\s]+:)|([^:]*\]:.*)")
|
||||
|
||||
DEFAULT_SECTIONS = [
|
||||
Section("Arguments", "param", SectionType.MULTIPLE),
|
||||
Section("Args", "param", SectionType.MULTIPLE),
|
||||
Section("Parameters", "param", SectionType.MULTIPLE),
|
||||
Section("Params", "param", SectionType.MULTIPLE),
|
||||
Section("Raises", "raises", SectionType.MULTIPLE),
|
||||
Section("Exceptions", "raises", SectionType.MULTIPLE),
|
||||
Section("Except", "raises", SectionType.MULTIPLE),
|
||||
Section("Attributes", "attribute", SectionType.MULTIPLE),
|
||||
Section("Example", "examples", SectionType.SINGULAR),
|
||||
Section("Examples", "examples", SectionType.SINGULAR),
|
||||
Section("Returns", "returns", SectionType.SINGULAR_OR_MULTIPLE),
|
||||
Section("Yields", "yields", SectionType.SINGULAR_OR_MULTIPLE),
|
||||
]
|
||||
|
||||
|
||||
class GoogleParser:
|
||||
"""Parser for Google-style docstrings."""
|
||||
|
||||
def __init__(
|
||||
self, sections: T.Optional[T.List[Section]] = None, title_colon=True
|
||||
):
|
||||
"""Setup sections.
|
||||
|
||||
:param sections: Recognized sections or None to defaults.
|
||||
:param title_colon: require colon after section title.
|
||||
"""
|
||||
if not sections:
|
||||
sections = DEFAULT_SECTIONS
|
||||
self.sections = {s.title: s for s in sections}
|
||||
self.title_colon = title_colon
|
||||
self._setup()
|
||||
|
||||
def _setup(self):
|
||||
if self.title_colon:
|
||||
colon = ":"
|
||||
else:
|
||||
colon = ""
|
||||
self.titles_re = re.compile(
|
||||
"^("
|
||||
+ "|".join(f"({t})" for t in self.sections)
|
||||
+ ")"
|
||||
+ colon
|
||||
+ "[ \t\r\f\v]*$",
|
||||
flags=re.M,
|
||||
)
|
||||
|
||||
def _build_meta(self, text: str, title: str) -> DocstringMeta:
|
||||
"""Build docstring element.
|
||||
|
||||
:param text: docstring element text
|
||||
:param title: title of section containing element
|
||||
:return:
|
||||
"""
|
||||
|
||||
section = self.sections[title]
|
||||
|
||||
if (
|
||||
section.type == SectionType.SINGULAR_OR_MULTIPLE
|
||||
and not MULTIPLE_PATTERN.match(text)
|
||||
) or section.type == SectionType.SINGULAR:
|
||||
return self._build_single_meta(section, text)
|
||||
|
||||
if ":" not in text:
|
||||
raise ParseError(f"Expected a colon in {text!r}.")
|
||||
|
||||
# Split spec and description
|
||||
before, desc = text.split(":", 1)
|
||||
|
||||
if before and "\n" in before:
|
||||
# If there is a newline in the first line, clean it up
|
||||
first_line, rest = before.split("\n", 1)
|
||||
before = first_line + inspect.cleandoc(rest)
|
||||
|
||||
if desc:
|
||||
desc = desc[1:] if desc[0] == " " else desc
|
||||
if "\n" in desc:
|
||||
first_line, rest = desc.split("\n", 1)
|
||||
desc = first_line + "\n" + inspect.cleandoc(rest)
|
||||
desc = desc.strip("\n")
|
||||
|
||||
return self._build_multi_meta(section, before, desc)
|
||||
|
||||
@staticmethod
|
||||
def _build_single_meta(section: Section, desc: str) -> DocstringMeta:
|
||||
if section.key in RETURNS_KEYWORDS | YIELDS_KEYWORDS:
|
||||
return DocstringReturns(
|
||||
args=[section.key],
|
||||
description=desc,
|
||||
type_name=None,
|
||||
is_generator=section.key in YIELDS_KEYWORDS,
|
||||
)
|
||||
if section.key in RAISES_KEYWORDS:
|
||||
return DocstringRaises(
|
||||
args=[section.key], description=desc, type_name=None
|
||||
)
|
||||
if section.key in EXAMPLES_KEYWORDS:
|
||||
return DocstringExample(
|
||||
args=[section.key], snippet=None, description=desc
|
||||
)
|
||||
if section.key in PARAM_KEYWORDS:
|
||||
raise ParseError("Expected paramenter name.")
|
||||
return DocstringMeta(args=[section.key], description=desc)
|
||||
|
||||
@staticmethod
|
||||
def _build_multi_meta(
|
||||
section: Section, before: str, desc: str
|
||||
) -> DocstringMeta:
|
||||
if section.key in PARAM_KEYWORDS:
|
||||
match = GOOGLE_TYPED_ARG_REGEX.match(before)
|
||||
if match:
|
||||
arg_name, type_name = match.group(1, 2)
|
||||
if type_name.endswith(", optional"):
|
||||
is_optional = True
|
||||
type_name = type_name[:-10]
|
||||
elif type_name.endswith("?"):
|
||||
is_optional = True
|
||||
type_name = type_name[:-1]
|
||||
else:
|
||||
is_optional = False
|
||||
else:
|
||||
arg_name, type_name = before, None
|
||||
is_optional = None
|
||||
|
||||
match = GOOGLE_ARG_DESC_REGEX.match(desc)
|
||||
default = match.group(1) if match else None
|
||||
|
||||
return DocstringParam(
|
||||
args=[section.key, before],
|
||||
description=desc,
|
||||
arg_name=arg_name,
|
||||
type_name=type_name,
|
||||
is_optional=is_optional,
|
||||
default=default,
|
||||
)
|
||||
if section.key in RETURNS_KEYWORDS | YIELDS_KEYWORDS:
|
||||
return DocstringReturns(
|
||||
args=[section.key, before],
|
||||
description=desc,
|
||||
type_name=before,
|
||||
is_generator=section.key in YIELDS_KEYWORDS,
|
||||
)
|
||||
if section.key in RAISES_KEYWORDS:
|
||||
return DocstringRaises(
|
||||
args=[section.key, before], description=desc, type_name=before
|
||||
)
|
||||
return DocstringMeta(args=[section.key, before], description=desc)
|
||||
|
||||
def add_section(self, section: Section):
|
||||
"""Add or replace a section.
|
||||
|
||||
:param section: The new section.
|
||||
"""
|
||||
|
||||
self.sections[section.title] = section
|
||||
self._setup()
|
||||
|
||||
def parse(self, text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the Google-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
ret = Docstring(style=DocstringStyle.GOOGLE)
|
||||
if not text:
|
||||
return ret
|
||||
|
||||
# Clean according to PEP-0257
|
||||
text = inspect.cleandoc(text)
|
||||
|
||||
# Find first title and split on its position
|
||||
match = self.titles_re.search(text)
|
||||
if match:
|
||||
desc_chunk = text[: match.start()]
|
||||
meta_chunk = text[match.start() :]
|
||||
else:
|
||||
desc_chunk = text
|
||||
meta_chunk = ""
|
||||
|
||||
# Break description into short and long parts
|
||||
parts = desc_chunk.split("\n", 1)
|
||||
ret.short_description = parts[0] or None
|
||||
if len(parts) > 1:
|
||||
long_desc_chunk = parts[1] or ""
|
||||
ret.blank_after_short_description = long_desc_chunk.startswith(
|
||||
"\n"
|
||||
)
|
||||
ret.blank_after_long_description = long_desc_chunk.endswith("\n\n")
|
||||
ret.long_description = long_desc_chunk.strip() or None
|
||||
|
||||
# Split by sections determined by titles
|
||||
matches = list(self.titles_re.finditer(meta_chunk))
|
||||
if not matches:
|
||||
return ret
|
||||
splits = []
|
||||
for j in range(len(matches) - 1):
|
||||
splits.append((matches[j].end(), matches[j + 1].start()))
|
||||
splits.append((matches[-1].end(), len(meta_chunk)))
|
||||
|
||||
chunks = OrderedDict() # type: T.Mapping[str,str]
|
||||
for j, (start, end) in enumerate(splits):
|
||||
title = matches[j].group(1)
|
||||
if title not in self.sections:
|
||||
continue
|
||||
|
||||
# Clear Any Unknown Meta
|
||||
# Ref: https://github.com/rr-/docstring_parser/issues/29
|
||||
meta_details = meta_chunk[start:end]
|
||||
unknown_meta = re.search(r"\n\S", meta_details)
|
||||
if unknown_meta is not None:
|
||||
meta_details = meta_details[: unknown_meta.start()]
|
||||
|
||||
chunks[title] = meta_details.strip("\n")
|
||||
if not chunks:
|
||||
return ret
|
||||
|
||||
# Add elements from each chunk
|
||||
for title, chunk in chunks.items():
|
||||
# Determine indent
|
||||
indent_match = re.search(r"^\s*", chunk)
|
||||
if not indent_match:
|
||||
raise ParseError(f'Can\'t infer indent from "{chunk}"')
|
||||
indent = indent_match.group()
|
||||
|
||||
# Check for singular elements
|
||||
if self.sections[title].type in [
|
||||
SectionType.SINGULAR,
|
||||
SectionType.SINGULAR_OR_MULTIPLE,
|
||||
]:
|
||||
part = inspect.cleandoc(chunk)
|
||||
ret.meta.append(self._build_meta(part, title))
|
||||
continue
|
||||
|
||||
# Split based on lines which have exactly that indent
|
||||
_re = "^" + indent + r"(?=\S)"
|
||||
c_matches = list(re.finditer(_re, chunk, flags=re.M))
|
||||
if not c_matches:
|
||||
raise ParseError(f'No specification for "{title}": "{chunk}"')
|
||||
c_splits = []
|
||||
for j in range(len(c_matches) - 1):
|
||||
c_splits.append((c_matches[j].end(), c_matches[j + 1].start()))
|
||||
c_splits.append((c_matches[-1].end(), len(chunk)))
|
||||
for j, (start, end) in enumerate(c_splits):
|
||||
part = chunk[start:end].strip("\n")
|
||||
ret.meta.append(self._build_meta(part, title))
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
def parse(text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the Google-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
return GoogleParser().parse(text)
|
||||
|
||||
|
||||
def compose(
|
||||
docstring: Docstring,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
indent: str = " ",
|
||||
) -> str:
|
||||
"""Render a parsed docstring into docstring text.
|
||||
|
||||
:param docstring: parsed docstring representation
|
||||
:param rendering_style: the style to render docstrings
|
||||
:param indent: the characters used as indentation in the docstring string
|
||||
:returns: docstring text
|
||||
"""
|
||||
|
||||
def process_one(
|
||||
one: T.Union[DocstringParam, DocstringReturns, DocstringRaises],
|
||||
):
|
||||
head = ""
|
||||
|
||||
if isinstance(one, DocstringParam):
|
||||
head += one.arg_name or ""
|
||||
elif isinstance(one, DocstringReturns):
|
||||
head += one.return_name or ""
|
||||
|
||||
if isinstance(one, DocstringParam) and one.is_optional:
|
||||
optional = (
|
||||
"?"
|
||||
if rendering_style == RenderingStyle.COMPACT
|
||||
else ", optional"
|
||||
)
|
||||
else:
|
||||
optional = ""
|
||||
|
||||
if one.type_name and head:
|
||||
head += f" ({one.type_name}{optional}):"
|
||||
elif one.type_name:
|
||||
head += f"{one.type_name}{optional}:"
|
||||
else:
|
||||
head += ":"
|
||||
head = indent + head
|
||||
|
||||
if one.description and rendering_style == RenderingStyle.EXPANDED:
|
||||
body = f"\n{indent}{indent}".join(
|
||||
[head] + one.description.splitlines()
|
||||
)
|
||||
parts.append(body)
|
||||
elif one.description:
|
||||
(first, *rest) = one.description.splitlines()
|
||||
body = f"\n{indent}{indent}".join([head + " " + first] + rest)
|
||||
parts.append(body)
|
||||
else:
|
||||
parts.append(head)
|
||||
|
||||
def process_sect(name: str, args: T.List[T.Any]):
|
||||
if args:
|
||||
parts.append(name)
|
||||
for arg in args:
|
||||
process_one(arg)
|
||||
parts.append("")
|
||||
|
||||
parts: T.List[str] = []
|
||||
if docstring.short_description:
|
||||
parts.append(docstring.short_description)
|
||||
if docstring.blank_after_short_description:
|
||||
parts.append("")
|
||||
|
||||
if docstring.long_description:
|
||||
parts.append(docstring.long_description)
|
||||
if docstring.blank_after_long_description:
|
||||
parts.append("")
|
||||
|
||||
process_sect(
|
||||
"Args:", [p for p in docstring.params or [] if p.args[0] == "param"]
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Attributes:",
|
||||
[p for p in docstring.params or [] if p.args[0] == "attribute"],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Returns:",
|
||||
[p for p in docstring.many_returns or [] if not p.is_generator],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Yields:", [p for p in docstring.many_returns or [] if p.is_generator]
|
||||
)
|
||||
|
||||
process_sect("Raises:", docstring.raises or [])
|
||||
|
||||
if docstring.returns and not docstring.many_returns:
|
||||
ret = docstring.returns
|
||||
parts.append("Yields:" if ret else "Returns:")
|
||||
parts.append("-" * len(parts[-1]))
|
||||
process_one(ret)
|
||||
|
||||
for meta in docstring.meta:
|
||||
if isinstance(
|
||||
meta, (DocstringParam, DocstringReturns, DocstringRaises)
|
||||
):
|
||||
continue # Already handled
|
||||
parts.append(meta.args[0].replace("_", "").title() + ":")
|
||||
if meta.description:
|
||||
lines = [indent + l for l in meta.description.splitlines()]
|
||||
parts.append("\n".join(lines))
|
||||
parts.append("")
|
||||
|
||||
while parts and not parts[-1]:
|
||||
parts.pop()
|
||||
|
||||
return "\n".join(parts)
|
||||
@@ -0,0 +1,568 @@
|
||||
"""Numpydoc-style docstring parsing.
|
||||
|
||||
:see: https://numpydoc.readthedocs.io/en/latest/format.html
|
||||
"""
|
||||
|
||||
import inspect
|
||||
import itertools
|
||||
import re
|
||||
import typing as T
|
||||
from textwrap import dedent
|
||||
|
||||
from .common import (
|
||||
Docstring,
|
||||
DocstringDeprecated,
|
||||
DocstringExample,
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringRaises,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
RenderingStyle,
|
||||
)
|
||||
|
||||
|
||||
def _pairwise(iterable: T.Iterable, end=None) -> T.Iterable:
|
||||
left, right = itertools.tee(iterable)
|
||||
next(right, None)
|
||||
return itertools.zip_longest(left, right, fillvalue=end)
|
||||
|
||||
|
||||
def _clean_str(string: str) -> T.Optional[str]:
|
||||
string = string.strip()
|
||||
if len(string) > 0:
|
||||
return string
|
||||
return None
|
||||
|
||||
|
||||
KV_REGEX = re.compile(r"^[^\s].*$", flags=re.M)
|
||||
PARAM_KEY_REGEX = re.compile(r"^(?P<name>.*?)(?:\s*:\s*(?P<type>.*?))?$")
|
||||
PARAM_OPTIONAL_REGEX = re.compile(r"(?P<type>.*?)(?:, optional|\(optional\))$")
|
||||
|
||||
# Ideally, default value will be specified in the type declaration,
|
||||
# for which the following are supported:
|
||||
#
|
||||
# copy : bool, default True
|
||||
# copy : bool, default=True
|
||||
# copy : bool, default: True
|
||||
#
|
||||
PARAM_DEFAULT_REGEX = re.compile(
|
||||
r"(?P<type>.*?)(?:, default|\(default\))(?: | |=| = |= |: |)*(?P<value>.*)$" # pylint: disable=C0301
|
||||
)
|
||||
|
||||
# If the default value isn't specified in the type declaration,
|
||||
# it might be in the description. There isn't any formal grammar for this
|
||||
# in numpydoc, but we can make some educated guesses.
|
||||
PARAM_DEFAULT_REGEX_IN_DESC = re.compile(
|
||||
r"(?<!\S)[Dd]efault(?:s to |(?:\s*(?:is|[=:])\s*|\s+))(?P<value>(?:['\"]).*?(?:['\"])|[\w\-\.]*\w)" # pylint: disable=C0301
|
||||
)
|
||||
|
||||
RETURN_KEY_REGEX = re.compile(r"^(?:(?P<name>.*?)\s*:\s*)?(?P<type>.*?)$")
|
||||
|
||||
|
||||
class Section:
|
||||
"""Numpydoc section parser.
|
||||
|
||||
:param title: section title. For most sections, this is a heading like
|
||||
"Parameters" which appears on its own line, underlined by
|
||||
en-dashes ('-') on the following line.
|
||||
:param key: meta key string. In the parsed ``DocstringMeta`` instance this
|
||||
will be the first element of the ``args`` attribute list.
|
||||
"""
|
||||
|
||||
def __init__(self, title: str, key: str) -> None:
|
||||
self.title = title
|
||||
self.key = key
|
||||
|
||||
@property
|
||||
def title_pattern(self) -> str:
|
||||
"""Regular expression pattern matching this section's header.
|
||||
|
||||
This pattern will match this instance's ``title`` attribute in
|
||||
an anonymous group.
|
||||
"""
|
||||
dashes = "-" * len(self.title)
|
||||
return rf"^({self.title})\s*?\n{dashes}\s*$"
|
||||
|
||||
def parse(self, text: str) -> T.Iterable[DocstringMeta]:
|
||||
"""Parse ``DocstringMeta`` objects from the body of this section.
|
||||
|
||||
:param text: section body text. Should be cleaned with
|
||||
``inspect.cleandoc`` before parsing.
|
||||
"""
|
||||
yield DocstringMeta([self.key], description=_clean_str(text))
|
||||
|
||||
|
||||
class _KVSection(Section):
|
||||
"""Base parser for numpydoc sections with key-value syntax.
|
||||
|
||||
E.g. sections that look like this:
|
||||
key
|
||||
value
|
||||
key2 : type
|
||||
values can also span...
|
||||
... multiple lines
|
||||
"""
|
||||
|
||||
def _parse_item(self, key: str, value: str) -> DocstringMeta:
|
||||
pass
|
||||
|
||||
def parse(self, text: str) -> T.Iterable[DocstringMeta]:
|
||||
for match, next_match in _pairwise(KV_REGEX.finditer(text)):
|
||||
start = match.end()
|
||||
end = next_match.start() if next_match is not None else None
|
||||
value = text[start:end]
|
||||
yield self._parse_item(
|
||||
key=match.group(), value=inspect.cleandoc(value)
|
||||
)
|
||||
|
||||
|
||||
class _SphinxSection(Section):
|
||||
"""Base parser for numpydoc sections with sphinx-style syntax.
|
||||
|
||||
E.g. sections that look like this:
|
||||
.. title:: something
|
||||
possibly over multiple lines
|
||||
"""
|
||||
|
||||
@property
|
||||
def title_pattern(self) -> str:
|
||||
return rf"^\.\.\s*({self.title})\s*::"
|
||||
|
||||
|
||||
class ParamSection(_KVSection):
|
||||
"""Parser for numpydoc parameter sections.
|
||||
|
||||
E.g. any section that looks like this:
|
||||
arg_name
|
||||
arg_description
|
||||
arg_2 : type, optional
|
||||
descriptions can also span...
|
||||
... multiple lines
|
||||
"""
|
||||
|
||||
def _parse_item(self, key: str, value: str) -> DocstringParam:
|
||||
match = PARAM_KEY_REGEX.match(key)
|
||||
arg_name = type_name = is_optional = default = None
|
||||
if match is not None:
|
||||
arg_name = match.group("name")
|
||||
type_name = match.group("type")
|
||||
if type_name is not None:
|
||||
optional_match = PARAM_OPTIONAL_REGEX.match(type_name)
|
||||
if optional_match is not None:
|
||||
type_name = optional_match.group("type")
|
||||
is_optional = True
|
||||
else:
|
||||
is_optional = False
|
||||
|
||||
default_match = PARAM_DEFAULT_REGEX.match(type_name)
|
||||
if default_match is not None:
|
||||
is_optional = True
|
||||
type_name = default_match.group("type")
|
||||
default = default_match.group("value")
|
||||
|
||||
# If the default wasn't specifified in the type declaration,
|
||||
# try and see if we can find it in the description.
|
||||
if len(value) > 0 and default is None:
|
||||
default_match = PARAM_DEFAULT_REGEX_IN_DESC.search(value)
|
||||
if default_match is not None:
|
||||
default = default_match.group("value")
|
||||
|
||||
return DocstringParam(
|
||||
args=[self.key, arg_name],
|
||||
description=_clean_str(value),
|
||||
arg_name=arg_name,
|
||||
type_name=type_name,
|
||||
is_optional=is_optional,
|
||||
default=default,
|
||||
)
|
||||
|
||||
|
||||
class RaisesSection(_KVSection):
|
||||
"""Parser for numpydoc raises sections.
|
||||
|
||||
E.g. any section that looks like this:
|
||||
ValueError
|
||||
A description of what might raise ValueError
|
||||
"""
|
||||
|
||||
def _parse_item(self, key: str, value: str) -> DocstringRaises:
|
||||
return DocstringRaises(
|
||||
args=[self.key, key],
|
||||
description=_clean_str(value),
|
||||
type_name=key if len(key) > 0 else None,
|
||||
)
|
||||
|
||||
|
||||
class ReturnsSection(_KVSection):
|
||||
"""Parser for numpydoc returns sections.
|
||||
|
||||
E.g. any section that looks like this:
|
||||
return_name : type
|
||||
A description of this returned value
|
||||
another_type
|
||||
Return names are optional, types are required
|
||||
"""
|
||||
|
||||
is_generator = False
|
||||
|
||||
def _parse_item(self, key: str, value: str) -> DocstringReturns:
|
||||
match = RETURN_KEY_REGEX.match(key)
|
||||
if match is not None:
|
||||
return_name = match.group("name")
|
||||
type_name = match.group("type")
|
||||
else:
|
||||
return_name = None
|
||||
type_name = None
|
||||
|
||||
return DocstringReturns(
|
||||
args=[self.key],
|
||||
description=_clean_str(value),
|
||||
type_name=type_name,
|
||||
is_generator=self.is_generator,
|
||||
return_name=return_name,
|
||||
)
|
||||
|
||||
|
||||
class YieldsSection(ReturnsSection):
|
||||
"""Parser for numpydoc generator "yields" sections."""
|
||||
|
||||
is_generator = True
|
||||
|
||||
|
||||
class DeprecationSection(_SphinxSection):
|
||||
"""Parser for numpydoc "deprecation warning" sections."""
|
||||
|
||||
def parse(self, text: str) -> T.Iterable[DocstringDeprecated]:
|
||||
version, desc, *_ = text.split(sep="\n", maxsplit=1) + [None, None]
|
||||
|
||||
if desc is not None:
|
||||
desc = _clean_str(inspect.cleandoc(desc))
|
||||
|
||||
yield DocstringDeprecated(
|
||||
args=[self.key], description=desc, version=_clean_str(version)
|
||||
)
|
||||
|
||||
|
||||
class ExamplesSection(Section):
|
||||
"""Parser for numpydoc examples sections.
|
||||
|
||||
E.g. any section that looks like this:
|
||||
>>> import numpy.matlib
|
||||
>>> np.matlib.empty((2, 2)) # filled with random data
|
||||
matrix([[ 6.76425276e-320, 9.79033856e-307], # random
|
||||
[ 7.39337286e-309, 3.22135945e-309]])
|
||||
>>> np.matlib.empty((2, 2), dtype=int)
|
||||
matrix([[ 6600475, 0], # random
|
||||
[ 6586976, 22740995]])
|
||||
"""
|
||||
|
||||
def parse(self, text: str) -> T.Iterable[DocstringMeta]:
|
||||
"""Parse ``DocstringExample`` objects from the body of this section.
|
||||
|
||||
:param text: section body text. Should be cleaned with
|
||||
``inspect.cleandoc`` before parsing.
|
||||
"""
|
||||
lines = dedent(text).strip().splitlines()
|
||||
while lines:
|
||||
snippet_lines = []
|
||||
description_lines = []
|
||||
while lines:
|
||||
if not lines[0].startswith(">>>"):
|
||||
break
|
||||
snippet_lines.append(lines.pop(0))
|
||||
while lines:
|
||||
if lines[0].startswith(">>>"):
|
||||
break
|
||||
description_lines.append(lines.pop(0))
|
||||
yield DocstringExample(
|
||||
[self.key],
|
||||
snippet="\n".join(snippet_lines) if snippet_lines else None,
|
||||
description="\n".join(description_lines),
|
||||
)
|
||||
|
||||
|
||||
DEFAULT_SECTIONS = [
|
||||
ParamSection("Parameters", "param"),
|
||||
ParamSection("Params", "param"),
|
||||
ParamSection("Arguments", "param"),
|
||||
ParamSection("Args", "param"),
|
||||
ParamSection("Other Parameters", "other_param"),
|
||||
ParamSection("Other Params", "other_param"),
|
||||
ParamSection("Other Arguments", "other_param"),
|
||||
ParamSection("Other Args", "other_param"),
|
||||
ParamSection("Receives", "receives"),
|
||||
ParamSection("Receive", "receives"),
|
||||
RaisesSection("Raises", "raises"),
|
||||
RaisesSection("Raise", "raises"),
|
||||
RaisesSection("Warns", "warns"),
|
||||
RaisesSection("Warn", "warns"),
|
||||
ParamSection("Attributes", "attribute"),
|
||||
ParamSection("Attribute", "attribute"),
|
||||
ReturnsSection("Returns", "returns"),
|
||||
ReturnsSection("Return", "returns"),
|
||||
YieldsSection("Yields", "yields"),
|
||||
YieldsSection("Yield", "yields"),
|
||||
ExamplesSection("Examples", "examples"),
|
||||
ExamplesSection("Example", "examples"),
|
||||
Section("Warnings", "warnings"),
|
||||
Section("Warning", "warnings"),
|
||||
Section("See Also", "see_also"),
|
||||
Section("Related", "see_also"),
|
||||
Section("Notes", "notes"),
|
||||
Section("Note", "notes"),
|
||||
Section("References", "references"),
|
||||
Section("Reference", "references"),
|
||||
DeprecationSection("deprecated", "deprecation"),
|
||||
]
|
||||
|
||||
|
||||
class NumpydocParser:
|
||||
"""Parser for numpydoc-style docstrings."""
|
||||
|
||||
def __init__(self, sections: T.Optional[T.Dict[str, Section]] = None):
|
||||
"""Setup sections.
|
||||
|
||||
:param sections: Recognized sections or None to defaults.
|
||||
"""
|
||||
sections = sections or DEFAULT_SECTIONS
|
||||
self.sections = {s.title: s for s in sections}
|
||||
self._setup()
|
||||
|
||||
def _setup(self):
|
||||
self.titles_re = re.compile(
|
||||
r"|".join(s.title_pattern for s in self.sections.values()),
|
||||
flags=re.M,
|
||||
)
|
||||
|
||||
def add_section(self, section: Section):
|
||||
"""Add or replace a section.
|
||||
|
||||
:param section: The new section.
|
||||
"""
|
||||
|
||||
self.sections[section.title] = section
|
||||
self._setup()
|
||||
|
||||
def parse(self, text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the numpy-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
ret = Docstring(style=DocstringStyle.NUMPYDOC)
|
||||
if not text:
|
||||
return ret
|
||||
|
||||
# Clean according to PEP-0257
|
||||
text = inspect.cleandoc(text)
|
||||
|
||||
# Find first title and split on its position
|
||||
match = self.titles_re.search(text)
|
||||
if match:
|
||||
desc_chunk = text[: match.start()]
|
||||
meta_chunk = text[match.start() :]
|
||||
else:
|
||||
desc_chunk = text
|
||||
meta_chunk = ""
|
||||
|
||||
# Break description into short and long parts
|
||||
parts = desc_chunk.split("\n", 1)
|
||||
ret.short_description = parts[0] or None
|
||||
if len(parts) > 1:
|
||||
long_desc_chunk = parts[1] or ""
|
||||
ret.blank_after_short_description = long_desc_chunk.startswith(
|
||||
"\n"
|
||||
)
|
||||
ret.blank_after_long_description = long_desc_chunk.endswith("\n\n")
|
||||
ret.long_description = long_desc_chunk.strip() or None
|
||||
|
||||
for match, nextmatch in _pairwise(self.titles_re.finditer(meta_chunk)):
|
||||
title = next(g for g in match.groups() if g is not None)
|
||||
factory = self.sections[title]
|
||||
|
||||
# section chunk starts after the header,
|
||||
# ends at the start of the next header
|
||||
start = match.end()
|
||||
end = nextmatch.start() if nextmatch is not None else None
|
||||
ret.meta.extend(factory.parse(meta_chunk[start:end]))
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
def parse(text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the numpy-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
return NumpydocParser().parse(text)
|
||||
|
||||
|
||||
def compose(
|
||||
# pylint: disable=W0613
|
||||
docstring: Docstring,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
indent: str = " ",
|
||||
) -> str:
|
||||
"""Render a parsed docstring into docstring text.
|
||||
|
||||
:param docstring: parsed docstring representation
|
||||
:param rendering_style: the style to render docstrings
|
||||
:param indent: the characters used as indentation in the docstring string
|
||||
:returns: docstring text
|
||||
"""
|
||||
|
||||
def process_one(
|
||||
one: T.Union[DocstringParam, DocstringReturns, DocstringRaises],
|
||||
):
|
||||
if isinstance(one, DocstringParam):
|
||||
head = one.arg_name
|
||||
elif isinstance(one, DocstringReturns):
|
||||
head = one.return_name
|
||||
else:
|
||||
head = None
|
||||
|
||||
if one.type_name and head:
|
||||
head += f" : {one.type_name}"
|
||||
elif one.type_name:
|
||||
head = one.type_name
|
||||
elif not head:
|
||||
head = ""
|
||||
|
||||
# If this is a parameter, check if it's optional.
|
||||
# If it is and there's a not-None default, include that in the type
|
||||
# declaration, otherwise just mark it as optional.
|
||||
if isinstance(one, DocstringParam):
|
||||
if one.default not in [None, "None"]:
|
||||
head += f", default={one.default}"
|
||||
elif one.is_optional or one.default == "None":
|
||||
head += ", optional"
|
||||
|
||||
if one.description:
|
||||
body = f"\n{indent}".join([head] + one.description.splitlines())
|
||||
parts.append(body)
|
||||
else:
|
||||
parts.append(head)
|
||||
|
||||
def process_sect(name: str, args: T.List[T.Any]):
|
||||
if args:
|
||||
parts.append("")
|
||||
parts.append(name)
|
||||
parts.append("-" * len(parts[-1]))
|
||||
for arg in args:
|
||||
process_one(arg)
|
||||
|
||||
parts: T.List[str] = []
|
||||
if docstring.short_description:
|
||||
parts.append(docstring.short_description)
|
||||
if docstring.blank_after_short_description:
|
||||
parts.append("")
|
||||
|
||||
if docstring.deprecation:
|
||||
first = ".. deprecated::"
|
||||
if docstring.deprecation.version:
|
||||
first += f" {docstring.deprecation.version}"
|
||||
if docstring.deprecation.description:
|
||||
rest = docstring.deprecation.description.splitlines()
|
||||
else:
|
||||
rest = []
|
||||
sep = f"\n{indent}"
|
||||
parts.append(sep.join([first] + rest))
|
||||
|
||||
if docstring.long_description:
|
||||
parts.append(docstring.long_description)
|
||||
if docstring.blank_after_long_description:
|
||||
parts.append("")
|
||||
|
||||
process_sect(
|
||||
"Parameters",
|
||||
[item for item in docstring.params or [] if item.args[0] == "param"],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Attributes",
|
||||
[
|
||||
item
|
||||
for item in docstring.params or []
|
||||
if item.args[0] == "attribute"
|
||||
],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Returns",
|
||||
[
|
||||
item
|
||||
for item in docstring.many_returns or []
|
||||
if not item.is_generator
|
||||
],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Yields",
|
||||
[item for item in docstring.many_returns or [] if item.is_generator],
|
||||
)
|
||||
|
||||
if docstring.returns and not docstring.many_returns:
|
||||
ret = docstring.returns
|
||||
parts.append("Yields" if ret else "Returns")
|
||||
parts.append("-" * len(parts[-1]))
|
||||
process_one(ret)
|
||||
|
||||
process_sect(
|
||||
"Receives",
|
||||
[
|
||||
item
|
||||
for item in docstring.params or []
|
||||
if item.args[0] == "receives"
|
||||
],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Other Parameters",
|
||||
[
|
||||
item
|
||||
for item in docstring.params or []
|
||||
if item.args[0] == "other_param"
|
||||
],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Raises",
|
||||
[item for item in docstring.raises or [] if item.args[0] == "raises"],
|
||||
)
|
||||
|
||||
process_sect(
|
||||
"Warns",
|
||||
[item for item in docstring.raises or [] if item.args[0] == "warns"],
|
||||
)
|
||||
|
||||
if len(docstring.examples) > 0:
|
||||
parts.append("")
|
||||
parts.append("Examples")
|
||||
parts.append("--------")
|
||||
for example in docstring.examples:
|
||||
if example.snippet:
|
||||
parts.append(example.snippet)
|
||||
if example.description:
|
||||
parts.append(example.description)
|
||||
|
||||
for meta in docstring.meta:
|
||||
if isinstance(
|
||||
meta,
|
||||
(
|
||||
DocstringDeprecated,
|
||||
DocstringParam,
|
||||
DocstringReturns,
|
||||
DocstringRaises,
|
||||
DocstringExample,
|
||||
),
|
||||
):
|
||||
continue # Already handled
|
||||
|
||||
parts.append("")
|
||||
parts.append(meta.args[0].replace("_", "").title())
|
||||
parts.append("-" * len(meta.args[0]))
|
||||
|
||||
if meta.description:
|
||||
parts.append(meta.description)
|
||||
|
||||
return "\n".join(parts)
|
||||
@@ -0,0 +1,100 @@
|
||||
"""The main parsing routine."""
|
||||
|
||||
import inspect
|
||||
import typing as T
|
||||
|
||||
from docstring_parser import epydoc, google, numpydoc, rest
|
||||
from docstring_parser.attrdoc import add_attribute_docstrings
|
||||
from docstring_parser.common import (
|
||||
Docstring,
|
||||
DocstringStyle,
|
||||
ParseError,
|
||||
RenderingStyle,
|
||||
)
|
||||
|
||||
_STYLE_MAP = {
|
||||
DocstringStyle.REST: rest,
|
||||
DocstringStyle.GOOGLE: google,
|
||||
DocstringStyle.NUMPYDOC: numpydoc,
|
||||
DocstringStyle.EPYDOC: epydoc,
|
||||
}
|
||||
|
||||
|
||||
def parse(
|
||||
text: T.Optional[str], style: DocstringStyle = DocstringStyle.AUTO
|
||||
) -> Docstring:
|
||||
"""Parse the docstring into its components.
|
||||
|
||||
:param text: docstring text to parse
|
||||
:param style: docstring style
|
||||
:returns: parsed docstring representation
|
||||
"""
|
||||
if style != DocstringStyle.AUTO:
|
||||
return _STYLE_MAP[style].parse(text)
|
||||
|
||||
exc: T.Optional[Exception] = None
|
||||
rets = []
|
||||
for module in _STYLE_MAP.values():
|
||||
try:
|
||||
ret = module.parse(text)
|
||||
except ParseError as ex:
|
||||
exc = ex
|
||||
else:
|
||||
rets.append(ret)
|
||||
|
||||
if not rets:
|
||||
raise exc
|
||||
|
||||
return sorted(rets, key=lambda d: len(d.meta), reverse=True)[0]
|
||||
|
||||
|
||||
def parse_from_object(
|
||||
obj: T.Any,
|
||||
style: DocstringStyle = DocstringStyle.AUTO,
|
||||
) -> Docstring:
|
||||
"""Parse the object's docstring(s) into its components.
|
||||
|
||||
The object can be anything that has a ``__doc__`` attribute. In contrast to
|
||||
the ``parse`` function, ``parse_from_object`` is able to parse attribute
|
||||
docstrings which are defined in the source code instead of ``__doc__``.
|
||||
|
||||
Currently only attribute docstrings defined at class and module levels are
|
||||
supported. Attribute docstrings defined in ``__init__`` methods are not
|
||||
supported.
|
||||
|
||||
When given a class, only the attribute docstrings of that class are parsed,
|
||||
not its inherited classes. This is a design decision. Separate calls to
|
||||
this function should be performed to get attribute docstrings of parent
|
||||
classes.
|
||||
|
||||
:param obj: object from which to parse the docstring(s)
|
||||
:param style: docstring style
|
||||
:returns: parsed docstring representation
|
||||
"""
|
||||
docstring = parse(obj.__doc__, style=style)
|
||||
|
||||
if inspect.isclass(obj) or inspect.ismodule(obj):
|
||||
add_attribute_docstrings(obj, docstring)
|
||||
|
||||
return docstring
|
||||
|
||||
|
||||
def compose(
|
||||
docstring: Docstring,
|
||||
style: DocstringStyle = DocstringStyle.AUTO,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
indent: str = " ",
|
||||
) -> str:
|
||||
"""Render a parsed docstring into docstring text.
|
||||
|
||||
:param docstring: parsed docstring representation
|
||||
:param style: docstring style to render
|
||||
:param indent: the characters used as indentation in the docstring string
|
||||
:returns: docstring text
|
||||
"""
|
||||
module = _STYLE_MAP[
|
||||
docstring.style if style == DocstringStyle.AUTO else style
|
||||
]
|
||||
return module.compose(
|
||||
docstring, rendering_style=rendering_style, indent=indent
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
# Marker file for PEP 561.
|
||||
@@ -0,0 +1,259 @@
|
||||
"""ReST-style docstring parsing."""
|
||||
|
||||
import inspect
|
||||
import re
|
||||
import typing as T
|
||||
|
||||
from .common import (
|
||||
DEPRECATION_KEYWORDS,
|
||||
PARAM_KEYWORDS,
|
||||
RAISES_KEYWORDS,
|
||||
RETURNS_KEYWORDS,
|
||||
YIELDS_KEYWORDS,
|
||||
Docstring,
|
||||
DocstringDeprecated,
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringRaises,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
ParseError,
|
||||
RenderingStyle,
|
||||
)
|
||||
|
||||
|
||||
def _build_meta(args: T.List[str], desc: str) -> DocstringMeta:
|
||||
key = args[0]
|
||||
|
||||
if key in PARAM_KEYWORDS:
|
||||
if len(args) == 3:
|
||||
key, type_name, arg_name = args
|
||||
if type_name.endswith("?"):
|
||||
is_optional = True
|
||||
type_name = type_name[:-1]
|
||||
else:
|
||||
is_optional = False
|
||||
elif len(args) == 2:
|
||||
key, arg_name = args
|
||||
type_name = None
|
||||
is_optional = None
|
||||
else:
|
||||
raise ParseError(
|
||||
f"Expected one or two arguments for a {key} keyword."
|
||||
)
|
||||
|
||||
match = re.match(r".*defaults to (.+)", desc, flags=re.DOTALL)
|
||||
default = match.group(1).rstrip(".") if match else None
|
||||
|
||||
return DocstringParam(
|
||||
args=args,
|
||||
description=desc,
|
||||
arg_name=arg_name,
|
||||
type_name=type_name,
|
||||
is_optional=is_optional,
|
||||
default=default,
|
||||
)
|
||||
|
||||
if key in RETURNS_KEYWORDS | YIELDS_KEYWORDS:
|
||||
if len(args) == 2:
|
||||
type_name = args[1]
|
||||
elif len(args) == 1:
|
||||
type_name = None
|
||||
else:
|
||||
raise ParseError(
|
||||
f"Expected one or no arguments for a {key} keyword."
|
||||
)
|
||||
|
||||
return DocstringReturns(
|
||||
args=args,
|
||||
description=desc,
|
||||
type_name=type_name,
|
||||
is_generator=key in YIELDS_KEYWORDS,
|
||||
)
|
||||
|
||||
if key in DEPRECATION_KEYWORDS:
|
||||
match = re.search(
|
||||
r"^(?P<version>v?((?:\d+)(?:\.[0-9a-z\.]+))) (?P<desc>.+)",
|
||||
desc,
|
||||
flags=re.I,
|
||||
)
|
||||
return DocstringDeprecated(
|
||||
args=args,
|
||||
version=match.group("version") if match else None,
|
||||
description=match.group("desc") if match else desc,
|
||||
)
|
||||
|
||||
if key in RAISES_KEYWORDS:
|
||||
if len(args) == 2:
|
||||
type_name = args[1]
|
||||
elif len(args) == 1:
|
||||
type_name = None
|
||||
else:
|
||||
raise ParseError(
|
||||
f"Expected one or no arguments for a {key} keyword."
|
||||
)
|
||||
return DocstringRaises(
|
||||
args=args, description=desc, type_name=type_name
|
||||
)
|
||||
|
||||
return DocstringMeta(args=args, description=desc)
|
||||
|
||||
|
||||
def parse(text: T.Optional[str]) -> Docstring:
|
||||
"""Parse the ReST-style docstring into its components.
|
||||
|
||||
:returns: parsed docstring
|
||||
"""
|
||||
ret = Docstring(style=DocstringStyle.REST)
|
||||
if not text:
|
||||
return ret
|
||||
|
||||
text = inspect.cleandoc(text)
|
||||
match = re.search("^:", text, flags=re.M)
|
||||
if match:
|
||||
desc_chunk = text[: match.start()]
|
||||
meta_chunk = text[match.start() :]
|
||||
else:
|
||||
desc_chunk = text
|
||||
meta_chunk = ""
|
||||
|
||||
parts = desc_chunk.split("\n", 1)
|
||||
ret.short_description = parts[0] or None
|
||||
if len(parts) > 1:
|
||||
long_desc_chunk = parts[1] or ""
|
||||
ret.blank_after_short_description = long_desc_chunk.startswith("\n")
|
||||
ret.blank_after_long_description = long_desc_chunk.endswith("\n\n")
|
||||
ret.long_description = long_desc_chunk.strip() or None
|
||||
|
||||
types = {}
|
||||
rtypes = {}
|
||||
for match in re.finditer(
|
||||
r"(^:.*?)(?=^:|\Z)", meta_chunk, flags=re.S | re.M
|
||||
):
|
||||
chunk = match.group(0)
|
||||
if not chunk:
|
||||
continue
|
||||
try:
|
||||
args_chunk, desc_chunk = chunk.lstrip(":").split(":", 1)
|
||||
except ValueError as ex:
|
||||
raise ParseError(
|
||||
f'Error parsing meta information near "{chunk}".'
|
||||
) from ex
|
||||
args = args_chunk.split()
|
||||
desc = desc_chunk.strip()
|
||||
|
||||
if "\n" in desc:
|
||||
first_line, rest = desc.split("\n", 1)
|
||||
desc = first_line + "\n" + inspect.cleandoc(rest)
|
||||
|
||||
# Add special handling for :type a: typename
|
||||
if len(args) == 2 and args[0] == "type":
|
||||
types[args[1]] = desc
|
||||
elif len(args) in [1, 2] and args[0] == "rtype":
|
||||
rtypes[None if len(args) == 1 else args[1]] = desc
|
||||
else:
|
||||
ret.meta.append(_build_meta(args, desc))
|
||||
|
||||
for meta in ret.meta:
|
||||
if isinstance(meta, DocstringParam):
|
||||
meta.type_name = meta.type_name or types.get(meta.arg_name)
|
||||
elif isinstance(meta, DocstringReturns):
|
||||
meta.type_name = meta.type_name or rtypes.get(meta.return_name)
|
||||
|
||||
if not any(isinstance(m, DocstringReturns) for m in ret.meta) and rtypes:
|
||||
for return_name, type_name in rtypes.items():
|
||||
ret.meta.append(
|
||||
DocstringReturns(
|
||||
args=[],
|
||||
type_name=type_name,
|
||||
description=None,
|
||||
is_generator=False,
|
||||
return_name=return_name,
|
||||
)
|
||||
)
|
||||
|
||||
return ret
|
||||
|
||||
|
||||
def compose(
|
||||
docstring: Docstring,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
indent: str = " ",
|
||||
) -> str:
|
||||
"""Render a parsed docstring into docstring text.
|
||||
|
||||
:param docstring: parsed docstring representation
|
||||
:param rendering_style: the style to render docstrings
|
||||
:param indent: the characters used as indentation in the docstring string
|
||||
:returns: docstring text
|
||||
"""
|
||||
|
||||
def process_desc(desc: T.Optional[str]) -> str:
|
||||
if not desc:
|
||||
return ""
|
||||
|
||||
if rendering_style == RenderingStyle.CLEAN:
|
||||
(first, *rest) = desc.splitlines()
|
||||
return "\n".join([" " + first] + [indent + line for line in rest])
|
||||
|
||||
if rendering_style == RenderingStyle.EXPANDED:
|
||||
(first, *rest) = desc.splitlines()
|
||||
return "\n".join(
|
||||
["\n" + indent + first] + [indent + line for line in rest]
|
||||
)
|
||||
|
||||
return " " + desc
|
||||
|
||||
parts: T.List[str] = []
|
||||
if docstring.short_description:
|
||||
parts.append(docstring.short_description)
|
||||
if docstring.blank_after_short_description:
|
||||
parts.append("")
|
||||
if docstring.long_description:
|
||||
parts.append(docstring.long_description)
|
||||
if docstring.blank_after_long_description:
|
||||
parts.append("")
|
||||
|
||||
for meta in docstring.meta:
|
||||
if isinstance(meta, DocstringParam):
|
||||
if meta.type_name:
|
||||
type_text = (
|
||||
f" {meta.type_name}? "
|
||||
if meta.is_optional
|
||||
else f" {meta.type_name} "
|
||||
)
|
||||
else:
|
||||
type_text = " "
|
||||
if rendering_style == RenderingStyle.EXPANDED:
|
||||
text = f":param {meta.arg_name}:"
|
||||
text += process_desc(meta.description)
|
||||
parts.append(text)
|
||||
if type_text[:-1]:
|
||||
parts.append(f":type {meta.arg_name}:{type_text[:-1]}")
|
||||
else:
|
||||
text = f":param{type_text}{meta.arg_name}:"
|
||||
text += process_desc(meta.description)
|
||||
parts.append(text)
|
||||
elif isinstance(meta, DocstringReturns):
|
||||
type_text = f" {meta.type_name}" if meta.type_name else ""
|
||||
key = "yields" if meta.is_generator else "returns"
|
||||
|
||||
if rendering_style == RenderingStyle.EXPANDED:
|
||||
if meta.description:
|
||||
text = f":{key}:"
|
||||
text += process_desc(meta.description)
|
||||
parts.append(text)
|
||||
if type_text:
|
||||
parts.append(f":rtype:{type_text}")
|
||||
else:
|
||||
text = f":{key}{type_text}:"
|
||||
text += process_desc(meta.description)
|
||||
parts.append(text)
|
||||
elif isinstance(meta, DocstringRaises):
|
||||
type_text = f" {meta.type_name} " if meta.type_name else ""
|
||||
text = f":raises{type_text}:" + process_desc(meta.description)
|
||||
parts.append(text)
|
||||
else:
|
||||
text = f':{" ".join(meta.args)}:' + process_desc(meta.description)
|
||||
parts.append(text)
|
||||
return "\n".join(parts)
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Utility functions for working with docstrings."""
|
||||
|
||||
import typing as T
|
||||
from collections import ChainMap
|
||||
from inspect import Signature
|
||||
from itertools import chain
|
||||
|
||||
from .common import (
|
||||
DocstringMeta,
|
||||
DocstringParam,
|
||||
DocstringReturns,
|
||||
DocstringStyle,
|
||||
RenderingStyle,
|
||||
)
|
||||
from .parser import compose, parse
|
||||
|
||||
_Func = T.Callable[..., T.Any]
|
||||
|
||||
assert DocstringReturns # used in docstring
|
||||
|
||||
|
||||
def combine_docstrings(
|
||||
*others: _Func,
|
||||
exclude: T.Iterable[T.Type[DocstringMeta]] = (),
|
||||
style: DocstringStyle = DocstringStyle.AUTO,
|
||||
rendering_style: RenderingStyle = RenderingStyle.COMPACT,
|
||||
) -> _Func:
|
||||
"""A function decorator that parses the docstrings from `others`,
|
||||
programmatically combines them with the parsed docstring of the decorated
|
||||
function, and replaces the docstring of the decorated function with the
|
||||
composed result. Only parameters that are part of the decorated functions
|
||||
signature are included in the combined docstring. When multiple sources for
|
||||
a parameter or docstring metadata exists then the decorator will first
|
||||
default to the wrapped function's value (when available) and otherwise use
|
||||
the rightmost definition from ``others``.
|
||||
|
||||
The following example illustrates its usage:
|
||||
|
||||
>>> def fun1(a, b, c, d):
|
||||
... '''short_description: fun1
|
||||
...
|
||||
... :param a: fun1
|
||||
... :param b: fun1
|
||||
... :return: fun1
|
||||
... '''
|
||||
>>> def fun2(b, c, d, e):
|
||||
... '''short_description: fun2
|
||||
...
|
||||
... long_description: fun2
|
||||
...
|
||||
... :param b: fun2
|
||||
... :param c: fun2
|
||||
... :param e: fun2
|
||||
... '''
|
||||
>>> @combine_docstrings(fun1, fun2)
|
||||
>>> def decorated(a, b, c, d, e, f):
|
||||
... '''
|
||||
... :param e: decorated
|
||||
... :param f: decorated
|
||||
... '''
|
||||
>>> print(decorated.__doc__)
|
||||
short_description: fun2
|
||||
<BLANKLINE>
|
||||
long_description: fun2
|
||||
<BLANKLINE>
|
||||
:param a: fun1
|
||||
:param b: fun1
|
||||
:param c: fun2
|
||||
:param e: fun2
|
||||
:param f: decorated
|
||||
:returns: fun1
|
||||
>>> @combine_docstrings(fun1, fun2, exclude=[DocstringReturns])
|
||||
>>> def decorated(a, b, c, d, e, f): pass
|
||||
>>> print(decorated.__doc__)
|
||||
short_description: fun2
|
||||
<BLANKLINE>
|
||||
long_description: fun2
|
||||
<BLANKLINE>
|
||||
:param a: fun1
|
||||
:param b: fun1
|
||||
:param c: fun2
|
||||
:param e: fun2
|
||||
|
||||
:param others: callables from which to parse docstrings.
|
||||
:param exclude: an iterable of ``DocstringMeta`` subclasses to exclude when
|
||||
combining docstrings.
|
||||
:param style: style composed docstring. The default will infer the style
|
||||
from the decorated function.
|
||||
:param rendering_style: The rendering style used to compose a docstring.
|
||||
:return: the decorated function with a modified docstring.
|
||||
"""
|
||||
|
||||
def wrapper(func: _Func) -> _Func:
|
||||
sig = Signature.from_callable(func)
|
||||
|
||||
comb_doc = parse(func.__doc__ or "")
|
||||
docs = [parse(other.__doc__ or "") for other in others] + [comb_doc]
|
||||
params = dict(
|
||||
ChainMap(
|
||||
*(
|
||||
{param.arg_name: param for param in doc.params}
|
||||
for doc in docs
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
for doc in reversed(docs):
|
||||
if not doc.short_description:
|
||||
continue
|
||||
comb_doc.short_description = doc.short_description
|
||||
comb_doc.blank_after_short_description = (
|
||||
doc.blank_after_short_description
|
||||
)
|
||||
break
|
||||
|
||||
for doc in reversed(docs):
|
||||
if not doc.long_description:
|
||||
continue
|
||||
comb_doc.long_description = doc.long_description
|
||||
comb_doc.blank_after_long_description = (
|
||||
doc.blank_after_long_description
|
||||
)
|
||||
break
|
||||
|
||||
combined = {}
|
||||
for doc in docs:
|
||||
metas = {}
|
||||
for meta in doc.meta:
|
||||
meta_type = type(meta)
|
||||
if meta_type in exclude:
|
||||
continue
|
||||
metas.setdefault(meta_type, []).append(meta)
|
||||
for meta_type, meta in metas.items():
|
||||
combined[meta_type] = meta
|
||||
|
||||
combined[DocstringParam] = [
|
||||
params[name] for name in sig.parameters if name in params
|
||||
]
|
||||
comb_doc.meta = list(chain(*combined.values()))
|
||||
func.__doc__ = compose(
|
||||
comb_doc, style=style, rendering_style=rendering_style
|
||||
)
|
||||
return func
|
||||
|
||||
return wrapper
|
||||
Reference in New Issue
Block a user