spinn_front_end_common.interface.provenance package

Submodules

spinn_front_end_common.interface.provenance.router_prov_mapper module

class spinn_front_end_common.interface.provenance.router_prov_mapper.Plotter(db_filename: str, verbose: bool = False)

Bases: ContextManager[SQLiteDB]

Code to plot provenance data from the database

cmap

get_per_chip_prov_details(info: str) → Tuple[str, int, int, ndarray]

Gets the provenance of a per chip basis

Parameters:

info (str) – The name of the metadata to sum

Returns:

name, max x, max y and data

Return type:

tuple(str, int, int, numpy.ndarray)

get_per_chip_prov_types() → FrozenSet[str]

Get a set of the descriptions available at chip level

Return type:

set(str)

get_per_core_prov_types() → FrozenSet[str]

Get a set of the descriptions available at core level

Return type:

set(str)

get_sum_chip_prov_details(info: str) → Tuple[str, int, int, ndarray]

Gets the sum of the provenance

Parameters:

info (str) – The name of the metadata to sum

Returns:

name, max x, max y and data

Return type:

tuple(str, int, int, numpy.ndarray)

plot_per_chip_data(key: str, output_filename: str)

Plots the metadata for this key/term to the file at a chip level

Parameters:

• key (str) – The name of the metadata to plot, or a unique fragment of it

• output_filename (str)

plot_per_core_data(key: str, output_filename: str)

Plots the metadata for this key/term to the file at a core level

Parameters:

• key (str) – The name of the metadata to plot, or a unique fragment of it

• output_filename (str)

spinn_front_end_common.interface.provenance.router_prov_mapper.main() → None

Generate heat maps from SpiNNaker provenance databases

Module contents

class spinn_front_end_common.interface.provenance.AbstractProvidesLocalProvenanceData

Bases: object

Indicates an object that provides locally obtained provenance data.

GraphProvenanceGatherer will check all Vertices and all Edges in the ApplicationGraph

abstract get_local_provenance_data() → None

Get provenance data items and store them in the provenance DB.

class spinn_front_end_common.interface.provenance.AbstractProvidesProvenanceDataFromMachine

Bases: object

Indicates that an object provides provenance data retrieved from the machine.

abstract get_provenance_data_from_machine(placement: Placement)

Get provenance data items for a placement and store them in the provenance DB.

Parameters:

placement (Placement) – the placement of the object

class spinn_front_end_common.interface.provenance.FecTimer(algorithm: str, work: TimerWork)

Bases: object

Timer.

APPLICATION_RUNNER = 'Application runner'

classmethod end_category(category: TimerCategory)

This method should only be called via the View!

Parameters:

category (TimerCategory) – Stage to end

error(reason: str)

Ends an algorithm timing and records that it failed.

Parameters:

reason (str) – What caused the error

classmethod setup(simulator: AbstractSpinnakerBase)

Checks and saves cfg values so they don’t have to be read each time

Parameters:

simulator (AbstractSpinnakerBase) – Not actually used

skip(reason: str)

Records that the algorithms is being skipped and ends the timer.

Parameters:

reason (str) – Why the algorithm is being skipped

skip_if_cfg_false(section: str, option: str) → bool

Skips if a Boolean cfg values is False.

If this cfg value is True this methods keep the timer running and returns False (did not skip).

If the cfg value is False this method records the reason, ends the timing and returns True (it skipped).

Typically called if the algorithm should run if the cfg value is set True.

Parameters:

• section (str) – Section level to be applied to both options

• option1 (str) – One of the options to check

• option2 (str) – The other option to check

Return type:

bool

skip_if_cfgs_false(section: str, option1: str, option2: str) → bool

Skips if two Boolean cfg values are both False.

If either cfg value is True this methods keep the timer running and returns False (did not skip).

If both cfg values are False this method records the reason, ends the timing and returns True (it skipped).

Typically called if the algorithm should run if either cfg values is set True.

Parameters:

• section (str) – Section level to be applied to both options

• option1 (str) – One of the options to check

• option2 (str) – The other option to check

Return type:

bool

skip_if_empty(value: bool | int | str | Sized | None, name: str) → bool

Skips if the value is one that evaluates to False.

If the value is considered True (if value) this methods keep the timer running and returns False (did not skip).

If the value is False this method records the reason, ends the timing and returns True (it skipped).

Parameters:

• value – Value to check if True

• name (str) – Name to record for that value if skipping

Return type:

bool

skip_if_has_not_run() → bool

Skips if the simulation has not run.

If the simulation has run used this methods keep the timer running and returns False (did not skip).

If there was no run this method records the reason, ends the timing and returns True (it skipped).

Currently not used as a better check is skip_if_empty on the data needed for the algorithm.

Return type:

bool

skip_if_virtual_board() → bool

Skips if a virtual board is being used.

If a real board is being used this methods keep the timer running and returns False (did not skip).

If a virtual board is being used this method records the reason, ends the timing and returns True (it skipped).

Typically called for algorithms that require a real board to run.

Return type:

bool

classmethod start_category(category: TimerCategory, machine_on=None)

This method should only be called via the View!

Parameters:

• category (TimerCategory) – category to switch to

• machine_on (None or bool) – What to change machine on too. Or None to leave as is

classmethod stop_category_timing() → None

Stops all the timing.

Typically only called during simulator shutdown

class spinn_front_end_common.interface.provenance.GlobalProvenance(database_file: str | None = None, memory: bool = False)

Bases: SQLiteDB

Specific implementation of the Database for SQLite 3.

Note

Not thread safe on the same database file. Threads can access different DBs just fine.

Note

This totally relies on the way SQLite’s type affinities function. You can’t port to a different database engine without a lot of work.

Parameters:

• database_file (str or None) – The name of a file that contains (or will contain) an SQLite database holding the data. If omitted, either the default file path or an unshared in-memory database will be used (suitable only for testing).

• memory (bool) – Flag to say unshared in-memory can be used. Otherwise a None file will mean the default should be used

get_category_timer_sum(category: TimerCategory) → int

Get the total runtime for one category of algorithms

Parameters:

category (TimerCategory)

Returns:

total off all run times with this category

Return type:

int

get_category_timer_sums(category: TimerCategory) → Tuple[int, int]

Get the runtime for one category of algorithms split machine on, machine off

Parameters:

category (TimerCategory)

Returns:

total on and off time of instances with this category

Return type:

int

classmethod get_global_provenace_path() → str

Get the path of the current provenance database of the last run

Warning

Calling this method between start/reset and run may result in a path to a database not yet created.

Raises:

ValueError – if the system is in a state where path can’t be retrieved, for example before run is called

get_run_time_of_buffer_extractor() → str

Gets the buffer extractor provenance item(s) from the last run

Returns:

A possibly multi line string with for each row which matches the LIKE %BufferExtractor

Return type:

str

get_run_times() → str

Gets the algorithm running times from the last run. If an algorithm is invoked multiple times in the run, its times are summed.

Returns:

A possibly multi line string with for each row which matches the like a line description_name: time. The times are in seconds.

Return type:

str

get_timer_provenance(algorithm: str) → str

Gets the timer provenance item(s) from the last run

Parameters:

algorithm (str) – The value to LIKE search for in the algorithm column. Can be the full name, or have % and _ wildcards.

Returns:

A possibly multi line string with for each row which matches the like a line algorithm: value

Return type:

str

get_timer_sum_by_algorithm(algorithm: str) → int

Get the total runtime for one algorithm

Parameters:

algorithm (str)

Returns:

total off all run times with this algorithm

Return type:

int

get_timer_sum_by_category(category: TimerCategory) → int

Get the total runtime for one category of algorithms

Parameters:

category (TimerCategory)

Returns:

total of all run times with this category

Return type:

int

get_timer_sum_by_work(work: TimerWork) → int

Get the total runtime for one work type of algorithms

Parameters:

work (TimerWork)

Returns:

total off all run times with this category

Return type:

int

insert_category(category: TimerCategory, machine_on: bool)

Inserts category into the category_timer_provenance returning id

Parameters:

• category (TimerCategory) – Name of Category starting

• machine_on (bool) – If the machine was done during all or some of the time

insert_category_timing(category_id: int, delta: timedelta)

Inserts run time into the category

Parameters:

• category_id (int) – id of the Category finished

• delta (timedelta) – Time to be recorded

insert_timing(category: int, algorithm: str, work: TimerWork, delta: timedelta, skip_reason: str | None)

Inserts algorithms run times into the timer_provenance table

Parameters:

• category (int) – Category Id of the Algorithm

• algorithm (str) – Algorithm name

• work (TimerWork) – Type of work being done

• delta (timedelta) – Time to be recorded

• skip_reason (str or None) – The reason the algorithm was skipped or None if it was not skipped

insert_version(description: str, the_value: str)

Inserts data into the version_provenance table

Parameters:

• description (str) – The package for which the version applies

• the_value (str) – The version to be recorded

retreive_log_messages(min_level: int = 0) → List[Tuple[int, str]]

Retrieves all log messages at or above the min_level

Parameters:

min_level (int)

Return type:

list(tuple(int, str))

run_query(query: str, params: Iterable[str | int | float | None | bytes] = ()) → List[Row]

Opens a connection to the database, runs a query, extracts the results and closes the connection

The return type depends on the use_sqlite_rows parameter. By default this method returns tuples (lookup by index) but the advanced tuple type can be used instead, which supports lookup by name used in the query (use AS name in the query to set).

This method will not allow queries that change the database unless the read_only flag is set to False.

Note

This method is mainly provided as a support method for the later methods that return specific data. For new IntergationTests please add a specific method rather than call this directly.

Parameters:

• query (str) – The SQL query to be run. May include ? wildcards

• params (Iterable(str or int)) – The values to replace the ? wildcards with. The number and types must match what the query expects

Returns:

A list possibly empty of tuples/rows (one for each row in the database) where the number and type of the values corresponds to the where statement

Return type:

list(tuple or Row)

store_log(level: int, message: str, timestamp: datetime | None = None)

Stores log messages into the database

Parameters:

• level (int)

• message (str)

class spinn_front_end_common.interface.provenance.LogStoreDB

Bases: LogStore

Log message storage mechanism that uses a database.

get_location() → str

Retrieves the location of the log store.

Return type:

str

retreive_log_messages(min_level: int = 0) → List[Tuple[int, str]]

Retrieves all log messages at or above the min_level.

Parameters:

min_level (int) – Constraint on the minimum logging level to retrieve.

Returns:

A list of messages that satisfy the constraint.

Return type:

list(tuple(int, str))

store_log(level: int, message: str, timestamp: datetime | None = None)

Writes the log message for later retrieval.

Parameters:

• level (int) – The logging level.

• message (str) – The logged message.

• timestamp (datetime or None) – The time-stamp of the message.

class spinn_front_end_common.interface.provenance.ProvenanceReader(provenance_data_path: str | None = None)

Bases: BaseDatabase

Provides a connection to a database containing provenance for the current run and some convenience methods for extracting provenance data from it.

By default this will wrap around the database used to store the provenance of the last run. The path is not updated so this reader is not effected by a reset or an end.

The assumption is that the database is in the current provenance format. This includes both that DDL statements used to create the database but also the underlying database structure (currently sqlite3)

Warning

This class is only a wrapper around the database file so if the file is deleted the class will no longer work.

Create a wrapper around the database.

The suggested way to call this is without the provenance_data_path parameter, allowing get_last_run_database_path() to find the correct path.

Parameters:

provenance_data_path (None or str) – Path to the provenance database to wrap

cores_with_late_spikes() → List[Tuple[int, int, int, int]]

Gets the x, y, p and count of the cores where late spikes arrived.

Cores that received spikes but where none were late are not included.

Returns:

A list hopefully empty of tuples (x, y, p, count) of cores where their where late arriving spikes.

Return type:

list(tuple(int, int, int, int))

static demo() → None

A demonstration of how to use this class.

See also unittests/interface/provenance/test_provenance_database.py

get_cores_with_provenace() → List[Tuple[int, int, int]]

Gets the cores with provenance.

Returns:

A list tuples (x, y, p)

Return type:

list(tuple(int, int, int))

classmethod get_last_run_database_path() → str

Get the path of the current provenance database of the last run.

Warning

Calling this method between start/reset and run may result in a path to a database not yet created.

raises ValueError:

if the system is in a state where path can’t be retrieved, for example before run is called

get_monitor_by_chip(description: str) → List[Tuple[int, int, str | int | float | bytes | None]]

Gets the monitor values for a specific item.

Parameters:

description (str)

Returns:

list of tuples x, y, value)

Return type:

list(tuple(int, int, float))

get_provenance_for_router(x: int, y: int) → str

Gets the provenance item(s) from the last run relating to a chip.

Parameters:

• x (int) – The X coordinate of the chip

• y (int) – The Y coordinate of the chip

Returns:

A possibly multi-line string with for each row which matches the like a line description_name: value

Return type:

str

get_router_by_chip(description: str) → List[Tuple[int, int, int | float]]

Gets the router values for a specific item.

Parameters:

description (str)

Returns:

list of tuples (x, y, value)

Return type:

list(tuple(int, int, float))

messages() → List[str]

List all the provenance messages.

Returns:

all messages logged or not; order is whatever the DB chooses

Return type:

list(str)

run_query(query: str, params: Iterable[str | int | float | None | bytes] = ()) → List[Sequence[str | int | float | bytes | None]]

Opens a connection to the database, runs a query, extracts the results and closes the connection.

The return type depends on the use_sqlite_rows parameter. By default this method returns tuples (lookup by index) but the advanced tuple type can be used instead, which supports lookup by name used in the query (use AS name in the query to set).

This method will not allow queries that change the database unless the read_only flag is set to False.

Note

This method is mainly provided as a support method for the later methods that return specific data. For new IntergationTests please add a specific method rather than call this directly.

Parameters:

• query (str) – The SQL query to be run. May include ? wildcards

• params (Iterable(str or int)) – The values to replace the ? wildcards with. The number and types must match what the query expects

Returns:

A list possibly empty of tuples/rows (one for each row in the database) where the number and type of the values corresponds to the where statement

Return type:

list(tuple or Row)

class spinn_front_end_common.interface.provenance.ProvenanceWriter(database_file: str | None = None)

Bases: BaseDatabase

Specific implementation of the Database for SQLite 3.

Note

Not thread safe on the same database file. Threads can access different DBs just fine.

Note

This totally relies on the way SQLite’s type affinities function. You can’t port to a different database engine without a lot of work.

Parameters:

• database_file (str or None) – The name of a file that contains (or will contain) an SQLite database holding the data. If omitted, either the default file path or an unshared in-memory database will be used (suitable only for testing).

• memory (bool) – Flag to say unshared in-memory can be used. Otherwise a None file will mean the default should be used

insert_board_provenance(connections: Dict[Tuple[int, int], str] | None)

Write the connection details retrieved from spalloc_client job to the boards_provenance table.

Parameters:

connections (dict((int, int): str) or None) – {(x, y): hostname, …} or None

insert_connector(pre_population: str, post_population: str, the_type: str, description: str, the_value: str | int | float | bytes | None)

Inserts edge data into the connector_provenance

Parameters:

• pre_population (str) – Name of the pre-population / vertex

• post_population (str) – Name of the post-population / vertex

• the_type (str) – Class of the connector

• description (str) – type of value

• the_value (int) – data

insert_core(x: int, y: int, p: int, description: str, the_value: str | int | float | bytes | None)

Inserts data for a specific core into the core_provenance table.

Parameters:

• x (int) – X coordinate of the chip

• y (int) – Y coordinate of the chip

• p (int) – ID of the core

• description (str) – type of value

• the_value (int) – data

insert_gatherer(x: int, y: int, address: int, bytes_read: int, run: int, description: str, the_value: str | int | float | bytes | None)

Records provenance into the gatherer_provenance table.

Parameters:

• x (int) – X coordinate of the chip

• y (int) – Y coordinate of the chip

• address (int) – SDRAM address read from

• bytes_read (int) – number of bytes read

• run (int) – run number

• description (str) – type of value

• the_value (float) – data

insert_monitor(x: int, y: int, description: str, the_value: str | int | float | bytes | None)

Inserts data into the monitor_provenance table.

Parameters:

• x (int) – X coordinate of the chip

• y (int) – Y coordinate of the chip

• description (str) – type of value

• the_value (int) – data

insert_power(description: str, the_value: str | int | float | bytes | None)

Inserts a general power value into the power_provenance table.

Parameters:

• description (str) – Type of value

• the_value (float) – data

insert_report(message: str)

Save and if applicable logs a message to the reports table.

Only logs the messages up to the cut-off set by configuration provenance_report_cutoff

Parameters:

message (str)

insert_router(x: int, y: int, description: str, the_value: int | float, expected: bool = True)

Inserts data into the router_provenance table.

Parameters:

• x (int) – X coordinate of the chip

• y (int) – Y coordinate of the chip

• description (str) – type of value

• the_value (int or float) – data

• expected (bool) – Flag to say this data was expected

class spinn_front_end_common.interface.provenance.ProvidesProvenanceDataFromMachineImpl

Bases: AbstractProvidesProvenanceDataFromMachine

An implementation that gets provenance data from a region of integers on the machine.

N_SYSTEM_PROVENANCE_WORDS = 6

get_provenance_data_from_machine(placement: Placement)

Retrieve the provenance data.

Parameters:

placement (Placement) – Which vertex are we retrieving from, and where was it

classmethod get_provenance_data_size(n_additional_data_items: int) → int

Parameters:

n_additional_data_items (int)

Return type:

int

parse_extra_provenance_items(label: str, x: int, y: int, p: int, provenance_data: Sequence[int])

Convert the remaining provenance words (those not in the standard set) into provenance items.

Called by get_provenance_data_from_machine()

Parameters:

• label (str) – A descriptive label for the vertex (derived from label and placed position) to be used for provenance error reporting to the user.

• x (int) – x coordinate of the chip where this core

• y (int) – y coordinate of the core where this core

• p (int) – virtual id of the core

• provenance_data (list(int)) – The list of words of raw provenance data.

parse_system_provenance_items(label: str, x: int, y: int, p: int, provenance_data: Sequence[int])

Given some words of provenance data, convert the portion of them that describes the system provenance into proper provenance items.

Called by get_provenance_data_from_machine()

Parameters:

• label (str) – A descriptive label for the vertex (derived from label and placed position) to be used for provenance error reporting to the user.

• x (int) – x coordinate of the chip where this core

• y (int) – y coordinate of the core where this core

• p (int) – virtual id of the core

• provenance_data (list(int))

reserve_provenance_data_region(spec) → None

Parameters:

spec (DataSpecificationGenerator) – The data specification being written.

class spinn_front_end_common.interface.provenance.TimerCategory(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

Different Categories a FecTimer can be in

DATA_GENERATION = 6

GET_MACHINE = 4

LOADING = 5

MAPPING = 7

RESETTING = 9

RUN_LOOP = 8

RUN_OTHER = 3

SETTING_UP = 2

SHUTTING_DOWN = 10

WAITING = 1

property category_name: str

The category name as passed into the init.

Return type:

str

class spinn_front_end_common.interface.provenance.TimerWork(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

Different Work types an Algorithm could be doing

BITFIELD = 4

COMPRESSING = 5

CONTROL = 6

EXTRACTING = 9

EXTRACT_DATA = 10

GET_MACHINE = 2

LOADING = 3

OTHER = 1

REPORT = 11

RUNNING = 8

SYNAPSE = 7

property work_name: str

The name for this timer distinction.

Return type:

str