Introduction to libraries

This article goes over the different libraries available for building blockchains with Substrate.

Core libraries

At a high level, Substrate consists of libraries to build: a client, a runtime and the communication layer between the two.

                            ┌────────────────────────┐
                            │ Client                 │
                            │                        │
                            │                        │
                            │                        │
                            │     ┌──────────────────┤
                            │     │ Primitives       │
                            │     │                  │
                            │     │     ┌────────────┤
                            │     │     │            │
                            │     │     │            │
                            │     │     │   Runtime  │
                            └─────┴─────┴────────────┘

NOTE: Diagram is a rough sketch. Each part is not meant to be interpreted as nested, rather that "primitives" enable communication between the Clients and Runtimes. Need to annotate diagram with below:

Client: Libraries that enable the client and networking layer, including consensus and block execution.

Primitives: Libraries responsible for communicating between the client and the runtime, creating the transaction pool and building blocks for the block executor.

FRAME: Libraries to facilitate building runtime logic and encoding and decoding information passing to and from the runtime.

Each of these components are built from Rust libraries that fall under four categories:

sc_*: Substrate client libraries encapsulate the numerous crates for node and client facing infrastructure, including consensus critical infrastructure, P2P networking, RPC APIs and block execution. For example, sc_service is responsible for building the networking layer for Substrate blockchains, managing the communication between the network, client and transaction pool.
sp_*: Substrate primitives are libraries to facilitate communication between the client and the runtime. For example, sp_std takes useful primitives from Rust's standard library and makes them usable with any code that depends on the runtime.
frame_*: runtime SDK libraries for building use case specific runtime logic and calling to and from a runtime. For example, frame_support enables developers to easily declare runtime storage items, errors and events and frame_system acts as the base layer for other pallets to interact with other Substrate components.
pallet_*: a single FRAME module, of which exists an existing collection created for Polkadot and Kusama. Other pallet libraries exist such as the Open Runtime Module Library (ORML).

SCALE codec

SCALE (Simple Concatenated Aggregate Little-Endian) Codec is a lightweight, efficient, binary serialization and deserialization codec.

It is designed for high-performance, copy-free encoding and decoding of data in resource-constrained execution contexts, like the Substrate runtime. It is not self-describing in any way and assumes the decoding context has all type knowledge about the encoded data.

Substrate uses the parity-scale-codec, a Rust implementation of the SCALE Codec. This library and the SCALE codec are advantageous for Substrate and blockchain systems because:

It is lightweight relative to generic serialization frameworks like serde, which add significant boilerplate that can bloat the size of the binary.
It does not use Rust libstd making it compatible with no_std environments that compile to Wasm, such as the Substrate runtime.
It is built to have great support in Rust for deriving codec logic for new types using: #[derive(Encode, Decode)].

It's important to define the encoding scheme used in Substrate rather than reuse an existing Rust codec library because this codec needs to be re-implemented on other platforms and languages that want to support interoperability among Substrate blockchains.

The table below shows how the Rust implementation of Parity's SCALE codec encodes different types.

SCALE codec examples of different types

Type	Description	Example SCALE encoded value	SCALE decoded value
Fixed-width integers	Basic integers are encoded using a fixed-width little-endian (LE) format.	`signed 8-bit integer 69`	`0x45`
		`unsigned 16-bit integer 42`	`0x2a00`
		`unsigned 32-bit integer 16777215`	`0xffffff00`
Compact/general integers[^1]	A "compact" or general integer encoding is sufficient for encoding large integers (up to 2**536) and is more efficient at encoding most values than the fixed-width version. (Though for single-byte values, the fixed-width integer is never worse.)	`unsigned integer 0`	`0x00`
			`unsigned integer 1`	`0x04`
			`unsigned integer 42`	`0xa8`
			`unsigned integer 69`	`0x1501`
			`unsigned integer 65535`	`0xfeff0300`
			`BigInt(100000000000000)`	`0x0b00407a10f35a`
Boolean	Boolean values are encoded using the least significant bit of a single byte.	`false`	`0x00`
		`true`	`0x01`
Results [^2]	Results are commonly used enumerations which indicate whether certain operations were successful or unsuccessful.	`Ok(42)`	`0x002a`
		`Err(false)`	`0x0100`
Options [^3]	One or zero values of a particular type.	`Some`	`0x01` followed by the encoded value.
		`None`	`0x00`
		`true`	`0x01`
		`false`	`0x02`
Vectors (lists, series, sets)	A collection of same-typed values is encoded, prefixed with a compact encoding of the number of items, followed by each item's encoding concatenated in turn.	Vector of unsigned 16-bit integers: `[4, 8, 15, 16, 23, 42]`	`0x18040008000f00100017002a00`
Strings	Strings are Vectors of bytes (`Vec<u8>`) containing a valid UTF8 sequence.
Tuples	A fixed-size series of values, each with a possibly different but predetermined and fixed type. This is simply the concatenation of each encoded value.	Tuple of compact unsigned integer and boolean: `(3, false)`	`0x0c00`
Structs	For structures, the values are named, but that is irrelevant for the encoding (names are ignored - only order matters). All containers store elements consecutively. The order of the elements is not fixed, depends on the container, and cannot be relied on at decoding. This implicitly means that decoding some byte-array into a specified structure that enforces an order and then re-encoding it could result in a different byte array than the original that was decoded.	A `SortedVecAsc<u8>` structure that always has byte-elements in ascending order: `SortedVecAsc::from([3, 5, 2, 8])`	`[3, 2, 5, 8]`
Enumerations (tagged-unions)	A fixed number of variants, each mutually exclusive and potentially implying a further value or series of values. Encoded as the first byte identifying the index of the variant that the value is. Any further bytes are used to encode any data that the variant implies. Thus, no more than 256 variants are supported.	`Int(42)` and `Bool(true)` where `enum IntOrBool { Int(u8), Bool(bool),}`	`0x002a` and `0x0101`

Footnotes:

[^1]: Compact/general integers are encoded with the two least significant bits denoting the mode:

- `0b00`: single-byte mode; upper six bits are the LE encoding of the value (valid only for values
of 0-63).
- `0b01`: two-byte mode: upper six bits and the following byte is the LE encoding of the value
(valid only for values `64-(2**14-1)`).
- `0b10`: four-byte mode: upper six bits and the following three bytes are the LE encoding of the
value (valid only for values `(2**14)-(2**30-1)`).
- `0b11`: Big-integer mode: The upper six bits are the number of bytes following, plus four. The
value is contained, LE encoded, in the bytes following. The final (most significant) byte must be
non-zero. Valid only for values `(2**30)-(2**536-1)`.

[^2]: Results are encoded as:

- `0x00` if the operation was successful, followed by the encoded value.
- `0x01` if the operation was unsuccessful, followed by the encoded error.

[^3]: Options are encoded as:

- `0x00` if it is `None` ("empty" or "null").
- `0x01` followed by the encoded value if it is `Some`.

As an exception, in the case that the type is a boolean, then it is always one byte.

The Parity SCALE Codec has been implemented in many other languages, including:

Python: polkascan/py-scale-codec
Golang: itering/scale.go
C: MatthewDarnell/cScale
C++: soramitsu/scale-codec-cpp
JavaScript: polkadot-js/api
AssemblyScript: LimeChain/as-scale-codec
Haskell: airalab/hs-web3
Java: emeraldpay/polkaj
Ruby: itering/scale.rb

Other libraries

There are a number of language-specific client libraries that can be used to interact with the Substrate framework. In general, the capabilities that these libraries expose are implemented on top of the Substrate remote procedure call (RPC) API. Although it is possible to build an alternative to FRAME using Substrate primitives, there has not yet been any significant community efforts to do so yet.

Rust

Parity maintains subxt, which is a Rust library specifically designed for submitting extrinsics to Substrate blockchains.

The the Substrate API Client is another Substrate client library written in Rust that is maintained by Supercomputing Systems; its API is more general-purpose than subxt.

JavaScript

The Polkadot JS team maintains a rich set of tools for interacting with Substrate-based blockchains. Refer to the main Polkadot JS page to learn more about that suite of tools.

Parity also maintains txwrapper, which is a Javascript library for offline generation of Substrate transactions.

Go

The Go Substrate RPC Client (GSRPC), is maintained by Centrifuge.

C

Polkadot API DotNet is a Substrate RPC client library for .NET developers. It is maintained by Usetech.

SubstrateNetApi is a .NET Standard API (nuget) allowing full Substrate integration in Unity3D for gaming development, starter template project. It is maintained by DOTMog Team.

C++

Usetech also maintains Polkadot API CPP, which is a C++ library for interacting with the Substrate RPC.

Python

py-substrate-interface is a Python library for interacting with the Substrate RPC. It supports a wide range of capabilities and powers the Polkascan multi-chain block explorer. This library is maintained by Polkascan Foundation.