Polkascan Development Update #5

WEB3SCAN’s mission is to make multi-chain data accessible and understandable. With the Polkascan project we are working towards a multi-chain exploration and data analytic platform for the Polkadot-ecosystem.

This development update is part of our work that provides the Polkadot-ecosystem with an enriched Substrate Interface (API) for any Substrate-based blockchain with full decoding context of the chain’s respective runtime. This update covers the delivery of a general purpose enriched Substrate API and underlying Python libraries for general use.

Our organization is on an ongoing mission of building multi-chain exploration and data analytic technologies in general and building the Polkascan multi-chain explorer in particular. Our first wave grant resulted in two Python-libraries:

1. Py-Scale-Codec
   Substrate is agnostic with respect to the data in its runtime and its data is encoded by the SCALE Codec. Our Python SCALE-Codec library is a Python implementation of the lightweight, efficient, binary serialization and deserialization codec. The codec allows its users to work with meaningful data when interacting with JSON-RPC API.
2. Py-Substrate-Interface
   The Python Substrate Interface library provides a python based abstraction layer for repetitive tasks when querying Substrate-nodes for required data.

Our previous work limited the scope of these libraries to match the requirements of our block explorer use-case. This project extends and feature-completes these two libraries for general use beyond the scope of our block explorer use-case. The project’s deliverables make multi-chain data accessible and understandable without the need for a rich database as in our block explorer use-case.

Our team has been active in the Polkadot-ecosystem from day one, we believe Substrate and the Polkadot Runtime Environment will solve many industry-wide problems. Additionally we are highly invested in Python. This project aims to lower the friction of using Python with Substrate-based chains and the completion of the Python-libraries will benefit many teams in the ecosystem. On top of that, this grant work will allow on-boarding of Python developers to integrate with existing or new Python-applications or applications in any other language with existing and new Substrate chains.

1. Getting started

This development update will help you get started by running a release of the enriched Substrate Interface yourself.

1.1. Substrate Interface on Github

Repositories: The source code of the enriched Substrate Interface can be found at our Github organization. This organization consists of a number of distinct repositories which collectively form the enriched Substrate Interface. We apply a number of conventions for branches and versioned releases across these repositories.

Branches: Each repository has our most recent work in the master-branch.
Releases: This development update’s work is released and tagged as ‘v0.9.x’.

1.2. What makes Substrate Interface?

Substrate Interface consists of a number of distinct software artifacts which collectively orchestrate the interface. The Substrate Interface APIprovides JSON-RPC end-points that allow for enriched interaction with Substrate’s standard JSON-RPC interface with the help of Python Substrate Interface Library and the Python SCALE Codec Library. We have included Substrate Interface GUI as a basic user interface to showcase how the enriched interface could be applied in front-end Typescript application. We have chosen to provide full Docker support for all our artifacts, hence all our repositories have Dockerfiles in their root. Although these four distinct components of the enriched Substrate Interface could be applied independently (in other projects), we offer a fifth piece of software, called: Substrate Interface that glues together all these components with Docker Compose.

1.3. Requirements

1. Recommended hardware: Memory: >2GB (more is better), Storage: >5GB (SSD is better), Processor: more and faster cores is better (Intel i3).
2. Software requirements: Git, Docker & Docker Compose.
3. Deployment of the enriched Substrate Interface has been tested on Mac, Linux and Windows.

1.4. Running Substrate Interface

This paragraph provides a step-by-step guide on running our v0.9.* release of the enriched Substrate Interface on your own machine. The instructions outlined in this paragraph can be found in our Github repository.

1. Clone the repository:
   git clone https://github.com/polkascan/substrate-interface.git
2. Go to the new folder:
   cd substrate-interface
3. Check available tagged releases:
   git tag
4. Checkout latest release in the 0.9.x range (replace ‘x’ with highest number):
   git checkout 0.9.x
5. Initialize and update submodules:
   git submodule update --init --recursive
6. Build and initialize the other containers:
   # Default public Kusama CC3 node by Parity Technologies
docker-compose up --build
# OR
# Local Kusama CC3 archive node
docker-compose -f docker-compose.kusama-cc3-local.yml up --build
7. The Substrate Interface API should now be available for use in Postman: http://127.0.0.1:8000
8. The Substrate Interface GUI should now be available in a browser: http://127.0.0.1:8001

1.5. Considerations

Step 6 in section 1.4 builds and runs a Substrate Interface that uses a public Substrate-node for Kusama CC3 serviced by Parity Technologies (wss://kusama-rpc.polkadot.io/). An alternative is listed that runs a full local archive node for Kusama CC3. Running a local node may be useful when a higher performance (network i/o) is required or if you run into firewall issues when setting up a persistent WebSocket connection with the public node.

1.6. Cleaning up

The following [docker] commands should help you clean up.

• Stop all containers of the Docker Compose file.
  docker-compose down
# OR
docker-compose -f docker-compose.kusama-cc3-local.yml down
• Remove all unused containers, volumes and images.
  docker system prune
[confirm] Y
• Remove all containers and images.
  docker system prune -a
[confirm] Y
• Remove all volumes.
  docker volume prune
[confirm] Y

2. Substrate Interface API

Substrate Interface API provides a JSON-RPC interface to a Substrate-node and is fully compatible with JSON-RPC interface of a regular Substrate-node. E.g. the RPC-method chain_getBlock provides the details of a block at the current head of the chain. The RPC-method rpc_methods provides a discovery mechanism of the supported RPC-methods for all known groups, such as: author, chain, state, system, etc.

Substrate Interface API extends the available RPC-methods with an entirely new group called: runtime, which enriches the JSON-RPC interface with all data particular to the current and previous runtimes of the Substrate-node.

The extension allows for discovery-mechanisms of the runtimes’ metadata with various levels of granularity such as: modules, call functions, storage functions, events, constants, errors and type-registry. In turn these discovery-mechanisms enable querying the Substrate-node’s storage with the decoding-context of the runtimes’ metadata and allow for composing transactions with the encoding-context of the runtimes’ metadata. These composed transactions are in turn fully prepared for signing by an external wallet or signer such as Parity Signer.

The current and extended RPC-methods have been fully documented through a Postman Collection and can be found here. The documentation provides tested examples for all current and extended RPC-methods and can easily be imported into your own Postman application by clicking on the ‘Run in Postman’ button at the top of the screen in the documented Postman Collection. After following the getting started instructions, the Substrate Interface API should be available for use in Postman through the following URI: http://127.0.0.1:8000. This URI should be pre-configured in the Postman Collection.

Substrate Interface API provides an enriched JSON-RPC interface to a Substrate-node. The JSON-RPC standard is a general interface envelop that any developer can use regardless of their respective development environment. The documented Postman Collection provides code-snippets for up to 25 distinct development environments (C#, cURL, Go, Java, Javascript, NodeJs, OCaml, PHP, Python, Ruby) for all included RPC-methods. This should illustrate the broad applicability of our grant work.

The example (image 2c) illustrates these ease at which a FreeBalance can be queried for a SS58-encoded account address through Postman.

3. Substrate Interface GUI

After following the getting started instructions, the Substrate Interface GUI should be available in a browser through the following URI: http://127.0.0.1:8001

Substrate Interface GUI provides three generic sections covering: executing raw RPCs, querying chain state and composing transactions. These sections replicate the similar respective functions of Polkadot JS Apps without any need for the Typescipt/Javascript libraries used by the latter. The following table provides the hyperlinks to the Substrate Interface GUI sections and their Polkadot JS Apps counterpart.

| Section      | Substrate Interface GUI | Polkadot JS Apps |              |--------------|-------------------------|------------------|
| Raw RPC      | link                    | link             |
| Chain State  | link                    | link             |
| Compose Call | link                    | link             |

Execute raw remote procedure calls (RPCs)
The method control lists all remote procedure call methods the Substrate Interface offers. This includes all native Substrate-client methods and thus provides full backwards compatibility with a regular Substrate-client. Additionally a large number of new methods are available related to this project’s grant work. The example (image 3b) illustrates these ease at which the TypeRegistry of the Kusama CC3 runtime can be queried at any block height (by BlockNumber or BlockHash).

Query runtime chain state
The storage function control lists all storage functions the runtime offers. The example (image 3c) illustrates these ease at which a FreeBalance can be queried for a SS58-encoded account address at any block height (by BlockNumber or BlockHash).

Compose calls (unsigned transaction payloads)
The call function control lists all call functions the runtime offers. The example (image 3d) illustrates these ease at which a Balances.Transfer transaction payload can be composed for a SS58-encoded account address. Unsigned transaction payloads can be signed by external wallets/signers such as Parity Signer. This requires some further research and requires implementation of Universal Offline Signatures through QR-codes. Hence external signing remains off-scope for this project’s deliverables.

Final note: Substrate Interface GUI has been included primarily as a convenience for it illustrates how to utilize the Substrate Interface API (section 2) in a Typescript (Angular) application framework. That said, this artifact is far from polished and warrants significant enhancement, such as: dynamic input-control parsing for varying types of each of the three sections’ input-parameters. Such feature is however beyond the scope of this project.

4. Python Substrate Interface

The Python Substrate Interface library specializes in interfacing with a Substrate node, providing additional convenience methods to deal with SCALE encoding/decoding (the default output and input format of the Substrate JSON-RPC), metadata parsing, type registry management and versioning of types. The Python Substrate Interface library is utilized by the Substrate Interface (and of course by the broader Polkascan-stack).

The grant work that has been completed with this development update has feature-completed the Python Substrate Interface for general use and can be used as an independent Python library.

Full Python library documentation has been made available through Pdoc.

The documentation illustrates how to use all available methods. These methods should look familiar since feature-wise it repeats the available JSON-RPC-methods that have been made available through the Substrate Interface API (section 2).

The grant work provides a full stack of artifacts to illustrate how this library could be applied. That said, this library could be applied independently in any Python-based application. The readme in the root of the repository provides details on how to get started with using the Pyhton Substrate Interface library in your own Python project.

Initialization
Initialize the interface and get head block hash of Kusama CC3 chain. WebSocket support is still quite limited at the moment as connections are not yet re-used. Until support is improved it is recommended to use HTTP-endpoints with your local Substrate-node (e.g. http://127.0.0.1:9933)

substrate = SubstrateInterface(
url="wss://kusama-rpc.polkadot.io/", 
address_type=2, 
type_registry_preset='kusama'
)
substrate.get_chain_head()

The readme contains additional examples that illustrate how to get extrinsics for a certain block, make a storage call and compose an unsigned transaction payload.

5. Python SCALE Codec

The Python SCALE Codec library is a lightweight, efficient, binary serialization and deserialization codec. The Python SCALE Codec is utilized by the Python Substrate Interface (and of course by the broader Polkascan-stack).

The grant work that has been completed with this development update has feature-completed the Python SCALE Codec for general use and as an independent Python library.

Full Python library documentation has been made available through Pdoc.

Additionally — as part of the grant work — we have provided an extensive set of unit tests to assure quality and conformity of the library’s functionality. Additionally these unit tests should provide a Python developer with sufficient hints on how to utilize the library’s rich features. The Python SCALE Codec’s unit tests include the following sections: metadata, type-encoding, type-registry, core scale-type and compose transaction payload.

The unit-test in the example (image 5.b) illustrates how to compose an unsigned transaction payload for a balance transfer.

The input of the unit test illustrates how to use a SS58-encoded account address or a HEX-formatted public key to identify the account and then illustrates how to use a decimal representation of the amount to be transferred. The unit-test results in an unsigned transaction payload, which could be signed by an external wallet provider or signer, such as Parity Signer (as illustrated in section 2 which covers the Substrate Interface GUI). The unit-tests evaluate the quality of the library’s methods with an expected output. The base-line for the expected outputs for all unit-tests in the compose transaction payload section have been gathered from the Polkadot JS Apps application as a reference client. Since the unit-tests are complete for a recent Kusama CC3 runtime (1040), we are fairly confident that the Python SCALE Codec conforms to features of the available reference clients of SCALE Codec (JSand RUST).

6. Outlook

With this development update and the respective deliverables we have gone beyond the work we outlined in our grant application. This project could result in being the go-to library/application for any non-Typescript application developer that intends to interact with any Substrate-runtime. The artifacts of this project provide a rich API to the Substrate-runtime with full decoding context. Our team will continue to develop and mature the artifacts that have been covered in this development update.

6.1. Immediate future development activities

1. Onboarding networks
   Substrate Interface currently supports Kusama CC3. We aim to provide turn-key support for all major public Substrate-networks.
2. WS-support
   Extension of the currently limited WebSocket support of Substrate Interface.
3. Refactoring block explorer repositories
   Substrate Interface allows for a better separation of concerns in our existing block explorer repositories. A significant amount of refactoring will happen which will improve and clean-up the block explorer repositories code-base.
4. Parity Signer support
   Unsigned transaction payloads can be signed by external wallets/signers such as Parity Signer. This requires some further research and requires implementation of Universal Offline Signatures through QR-codes.
5. Beta release
   A v1.0 beta release of Substrate Interface will be released at some point in the near future.

6.2. Exploration of major future projects

1. Polkadot JS Apps alternative
   The grant work covered in this development update has resulted in a number of low-level feature-complete Python libraries. We will be exploring the development of a full Polkadot JS Apps alternative based solely on our Python stack. The appeal of this approach to the broader community would be to have a full alternative to Polkadot JS Apps without any reliance on Javascript/Typescript based libraries. Polkadot JS Apps is becoming increasingly important to the broader Polkadot/Substrate community and from a risk management/mitigation perspective the community would benefit from a completely independent alternative.
2. Python-based Polkadot Runtime Environment
   The python libraries aligns with our mid to long-term objective of building a full Python-based implementation of the Polkadot Runtime Environment (PRE). Critical components for building a Python PRE are: 1) SCALE-codec, 2) Trie DB, 3) Libp2p Networking, 4) Consensus Babe/Grandpa. 5) WASM interpreter. Time can work in our favor since some components (3 and 5) may be available soon due to work in neighboring communities.

About Polkascan & WEB3SCAN

Polkascan contributes to the Polkadot ecosystem by providing block exploration and data analytic technologies. We will continue providing updates to our multi-chain explorer platform polkascan.io. The following public resources enable tracking of progress of the project: Medium, Twitter & GitHub. We encourage you to reach out if you would like to collaborate especially if you intend to be a Substrate implementer or ecosystem service provider. You can find us on the Riot channels on a daily basis. Come say hello and talk to us on how to get involved.