Translation notice

This page was translated with machine translation and may contain inaccuracies. If you can help improve it, please open an issue or submit a pull request.

shader02 core shader workflow (Part 1)

Xuanyu1725

Review: The work of vertex shaders

Rendering pipeline subsection from previous section

$\text{“}$Various things loaded into the game will send their vertex attributes to specific shader objects, and the .vsh vertex shader will process these vertices.

$\text{“}$The main task of the vertex shader at this stage is position conversion. Taking the entity in the world as an example, the vertex coordinates sent to the shader are all based on the relative coordinates of the camera. However, according to OpenGL convention, the output coordinates of vertices are in the range from (-1.0, -1.0, -1.0)to(1.0, 1.0, 1.0), and the origin is at the center of the screen, and the z-axis is vertical to the outside of the screen. The task of the vertex shader is to use a series of data sent by the game, including camera data, and match the properties of the vertex itself to perform a series of mathematical calculations to move the vertex to the correct position.

GLSL

GLSL, OpenGL Shading Language, is the language used by OpenGLshader, and its syntax is similar to C. Programming language is not the focus of this tutorial. Here we only briefly introduce some of the more important concepts of GLSL.

If necessary, you can check GLSL’s official documentation and other unofficial information

• docs.GL
• LearnOpenGL - GLSL Chapter
• GLSL Built-in Functions
• GLSL Reference Card (PDF)

GLSL is a language that runs on the GPU. Since it cannot access memory, GLSL does not support recursion, but it has stronger parallel computing capabilities. There is no order in which the shaders run at the same stage. They are processed in parallel at the same time.

GLSLversion

Each GLSL program needs to use the #version <version number> keyword at the beginning to declare the version used. Different GLSL versions have different features. If there is no special instructions, we will use 440 version for explanation.

GLSL data types

In addition to the common int, float, double, uintandbool types, GLSL also has its own Vector and Matrix.

GLSL provides a variety of different vectors and matrices, and you can construct the desired vector or matrix through different prefixes and suffixes. The default components of vectors and matrices are of type float. You can construct variables of different component types through different prefixes. The prefix imeans that the components are allint, umeansuint, bmeansbool, and dmeansdouble.

For vectors, you can add 2 3 4at the end to specify the number of components. For example,vec4represents a vector composed of 4floatcomponents. For matrices, only fill in one number to represent the number of columns and rows of the square matrix, such asmat4representing afloatmatrix with 4 rows and 4 columns.mat3x2represents afloat matrix with 3 columns and 2 rows.

A major feature of GLSL is that you can combine vectors and matrices at will to construct new vectors and matrices. Refer to the following code:

vec3 v1 = vec3(1.0, 0.0, 1.0);
vec4 v2 = vec4(v1, 1.0);
mat4 m1 = mat4(
    v1, 2.0,
    v2,
    1.0, 2.0, 0.0, 1.0,
    vec4(1.0)     // 这是一种简写，代表全部分量为1.0的vec4，在低于GLSL 330的环境中不被允许
);

The matrix in GLSL adopts a column-major perspective. Although the code is written in rows, the column data will be filled in internally when constructing the matrix. This feature determines the way the matrix is ​​stored in memory, which affects the performance of the matrix involved in the calculation. When performing matrix transformation, you must pay attention to the transposition problem

Pass variables

Transmitting variables (Ins and Outs) is the main bridge for shader communication. Add inbefore the declaration of a variable, which means that its data is passed in from the outside, andout means that its data is to be passed to the outside.

For the vertex shader, its incoming variables are the vertex attributes mentioned in the previous section, which we will introduce in detail later. The outgoing variable is the data that is calculated at the vertex and sent to the fragment shader for interpolation.

Prior to GLSL 130, incoming variables were passed using the attributekeyword and outgoing variables were passed usingvarying

global quantity

Global variables (Uniforms) are some game preset variables. They remain the same when rendering different vertices and different fragments of the same object, but may be different between different objects.

Global variables are declared using the uniform keyword. As long as they are declared, the game will automatically assign values ​​to them.

Before 1.21.5, global variables need to be configured with a .json file that corresponds to the shader example, and must be explicitly declared in the shader file. The specified initial values ​​have no effect, and the game will reassign them. After 1.21.6, the game uses Uniform block declaration (the declaration is introduced by including the shader, and is also automatically assigned by the game)

Vertex attributes and global variables constitute the entire input of the shader, which also reflects the limited access to shader data in Minecraft. Vertex attributes and global variables are hard-coded, and we cannot obtain more data at will.

Vector and matrix operations

I won’t go into details about linear algebra here, readers who need it can learn it by themselves. [3Blue1Brown]Essential Series of Linear Algebra

Different from mathematics, there are many operations between GLSL vectors, including scalar multiplication, cross multiplication, dot multiplication, four arithmetic operations, etc.

Except for the operations defined in the previous mathematics, other operations are component-wise operations. Please refer to the following code:

5.0 * vec4(1.0, 2.0, 3.0 ,1.0);
            // 标量乘法 == vec(5.0, 10.0, 15.0, 5.0)

cross(vec3(1.0, 2.0, 1.0), vec3(2.0, 3.0, 1.0));
            // 叉乘 == vec3(-1.0, 1.0, -1.0)

dot(vec3(1.0, 2.0, 1.0), vec3(2.0, 3.0, 1.0));
            // 点乘 == 9.0

vec3(1.0, 2.0, 1.0) + vec3(2.0, 3.0, 1.0);
            // 逐分量加法 == vec3(3.0, 5.0, 2.0)
            
abs(vec4(-1.0 ,2.0, -4.0));
            // 逐分量取绝对值 == vec4(1.0, 2.0, 4.0)

vertex shader

Vertex attributes

Vertex Attributes are the main input to the vertex shader. In Minecraft, vertex attributes include:

• Position - the input vertex position. When rendering different objects, the input position may be in different coordinate systems. The coordinate system section will be introduced later.
• Color - The color of the vertex, which generally includes color information such as automatic coloring and lighting that can be processed at the vertex. Complex color effects such as texture and fog are only processed in the fragment shader.
• UV/UV0 - Texture coordinate, but it is not the UVcoordinate defined in the baked model file, but the UVcoordinate of the corresponding texture in the texture atlas after the game combines the textures into a large texture atlas according to the rules under assets/minecraft/atlases.
• UV2 - the texture coordinate of the brightness texture. The horizontal coordinate represents the block brightness, and the vertical coordinate represents the sky brightness. Unlike UV/UV0, it is not normalized (that is, the range is not 0.0 to 1.0), and the range is 0 to 256 (actually only 240 will be taken)
• Normal - The unit normal vector of the vertex.
• Padding - can be considered a placeholder, used only to align vertex data, has no practical effect, and does not appear in most shaders.

In the previous section, we briefly introduced the concept of vertex attributes, which define all the data of the vertices. In each shader, not all available vertex attributes are necessarily entered. Currently, vertex attributes are hard-coded, and other available vertex attributes cannot be obtained in the resource pack. In the following tutorials we will introduce how these vertex attributes work in detail one by one.

Shader configuration file before 1.21.4

{
    "vertex": "minecraft:core/terrain",
    "fragment": "minecraft:core/terrain",
    "samplers": [
        { "name": "Sampler0" },
        { "name": "Sampler2" }
    ],
    "uniforms": [
        { "name": "ModelViewMat", "type": "matrix4x4", "count": 16, "values": [ 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0 ] },
        { "name": "ProjMat", "type": "matrix4x4", "count": 16, "values": [ 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0 ] },
        { "name": "ModelOffset", "type": "float", "count": 3, "values": [ 0.0, 0.0, 0.0 ] },
        { "name": "ColorModulator", "type": "float", "count": 4, "values": [ 1.0, 1.0, 1.0, 1.0 ] },
        { "name": "FogStart", "type": "float", "count": 1, "values": [ 0.0 ] },
        { "name": "FogEnd", "type": "float", "count": 1, "values": [ 1.0 ] },
        { "name": "FogColor", "type": "float", "count": 4, "values": [ 0.0, 0.0, 0.0, 0.0 ] },
        { "name": "FogShape", "type": "int", "count": 1, "values": [ 0 ] }
    ]
}

This is the json configuration file of the core shader rendertype_solid in 1.21.4. It contains 3 main sections.

1.21.5 The json file has been removed, the program called by the shader instance is now hardcoded and opaque. It is recommended that readers first search for the configuration of the shader instance in 1.21.4, and then go to a higher version to find out whether there is a corresponding shader program (Mojang, are you doing this just to disgust me?)

• shader program: The vertexandfragment fields specify the vertex shader and fragment shader used by this shader instance.

• sampler: The samplers field specifies the samplers that this shader instance can call. We will talk about it in the tutorial related to the fragment shader.

• Global quantity: The uniform field specifies the global variables that this shader can use. If the name and type do not match the "special Uniform" specified by the game, then it will not be automatically assigned, but will use the initial value specified by values.

coordinate system

During the rendering process, vertices will undergo transformations in multiple coordinate systems, including:

• Local Space Coordinates: The coordinates of the vertices relative to the model origin, that is, the coordinates defined by the baked model file.

• World Space Coordinates: After the model is placed on the world, the coordinates of the vertices in the world are${16}^3$The chunk size is a repeating unit, and the value of coordinate is also between 0.0 and 16.0. The Position vertex attribute of most block classes is this coordinate. These coordinates have been pre-processed by the game Model Transformation from local space to world space, and there is no need to perform this transformation in vsh.

• View Space Coordinates: The coordinates of the vertices in the space with the camera as the origin, the z-axis behind the camera, and the y-axis above. The Position vertex attribute of most entity classes has this coordinate. The Position of the block class obtains this coordinate after View Transformation. In order to use view transformation, the coordinate we use here is composed of 4 components Homogeneous Coordinate (Homogeneous Coordinate), in the form${\displaystyle (x,y,z,w)}$The coordinate corresponds to the three-dimensional${\displaystyle (\frac{x}{w},\frac{y}{w},\frac{z}{w})}$, we will mention why we need to define such a coordinate later.

_The figure maps xyz to rgb, so the three vertices with negative coordinates appear black, the vertices in the x direction are red, and the vertices in the z direction are blue, and are scaled to 1/64 so that the colors within the four chunks can be distinguished. _

• Clip Space Coordinates: The coordinates obtained by the view space coordinates after Projection Transformation. That is, the coordinate output by vsh. This space only retains vertices within the range from (-1.0, -1.0, -1.0)to(1.0, 1.0, 1.0) in the three-dimensional space. Vertices outside this range will be eliminated.

This picture needs to be understood in conjunction with the projection transformation described later. Please be patient. The tutorial will specifically cover the introduction of this transformation. The "w" in the illustration here is actually the w component of the homogeneous coordinate. We will introduce it later. The text description here shall prevail. "w" can be regarded as "1"

• Normalized Device Coordinates (NDC): The three-dimensional space coordinate obtained after the clipping space coordinate is processed by Perspective Division, that is, the xyz component of the homogeneous coordinate is divided by the w component.

• Screen Space Coordinates: The vertices in NDC are finally mapped to the two-dimensional screen coordinates ranging from (0.0, 0.0)to(1.0, 1.0)through Viewport Trasform,(0.0, 0.0)corresponds to the lower left corner of the screen, and(1.0, 1.0)corresponds to the upper right corner of the screen (if you are not using the screen upside down)

shader tasks

Through the overview of the transformation process above, we can summarize the main work of the vertex shader: transform the vertices from world space or view space into clipping space through model view transformation and projection transformation (collectively called MVP transformation).

So, how does the shader do this?

The following is the core part of a typical shader for a rendering block.

_Note: 1.21.6 Above, the uniform variable is provided by the shaderdynamictransforms.glsl (that is, the global volume block mentioned above). Before this, the uniform was directly declared in the shader program. For the convenience of reading, I gave it directly above. _

uniform vec3 ModelOffset;
uniform mat4 ProjMat;
uniform mat4 ModelViewMat;

void main() {
    vec3 pos = Position + ModelOffset;
    gl_Position = ProjMat * ModelViewMat * vec4(pos, 1.0);
}

Since the terrain is rendered by chunks, the ModelOffset global variable provides a constant offset of the camera to the chunk origin when rendering each chunk.

The model transformation has been done by the game, and the pos in the first line is obtained by adding the Position and the chunk offset. From the previous description, we know that the Position is the world coordinate within the chunk (can also be regarded as a relative coordinate), and the offset is the vector from the camera to the origin of the chunk. The addition of the two is the coordinate of the vertex centered on the camera. This is the first half of the view transformation.

The naming of ModelViewMat below is confusing. In fact, it only completes the second half of the view transformation, rotating the coordinate system so that the back of the camera becomes the positive z-axis direction, and the top of the camera becomes the positive y-axis direction (right-hand system, of course). ProjMat completes the projection transformation. More specifically, it does Perspective ​Projection Transformation .

Note: MVP in Minecraftshader can also be understood from another perspective, that is, chunk is the "model" in the rendering process, and Position is the local coordinate. The first line is the model transformation that moves the local coordinate to the world coordinate, and the second line is to complete the remaining view transformation and model transformation, but omits the step of moving the coordinate to the camera (because the camera itself is at the origin from this perspective)

matrix transformation

In order to understand the above transformation process, some operating rules related to linear algebra must be introduced here.

First, **What is a vector? ** Everyone has learned vectors in middle school. Column vectors are generally used in linear algebra, such as$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]$, this expression is the same as$(x,y,z)$is equivalent, but note that with row vectors$\left[\begin{array}{ccc}x& y& z\end{array}\right]$Distinguish (the representation of vectors and matrices using parentheses and square brackets is the same). Generally used in GLSL are 2-, 3-, and 4-dimensional vectors. What is a matrix? An array of numbers consisting of different columns and rows is a matrix, and a vector can also be thought of as a matrix with only one column. Among them, a matrix with the same number of rows and columns is called a square matrix. Defining these mathematical tools can greatly facilitate operations in shaders.

The process of matrix·vector is called matrix transformation. First of all, it must be noted that matrix multiplication is from right to left, for example$C\cdot B\cdot A$It represents first A, then B and then C. Obviously, the commutative law is not satisfied in general (but the associative law is satisfied).

In terms of numerical operations, the operation process of matrix and vector is to linearly combine the columns of the matrix with each component of the vector as the weight (that is, add multiple vectors according to a certain weight).
Or dot multiply the nth row of the matrix with the vector to obtain the nth component of the result vector. These two operations are equivalent.

The operation of matrix A·matrix B is to use A to perform matrix transformation on each column of B, and then combine them together to form a new matrix.
Or the nth row of A and the mth column of B are dot multiplied to obtain the nth row and m column components of the result matrix.

This operation process is mechanical and easy to remember. I believe students who have studied linear algebra are familiar with it (if you haven’t studied it before, just remember the above operation rules =.=)

Here are examples to help understand the two operations:

$$\begin{array}{rlrl}Ax=\left[\begin{array}{ccc}1& 1& 4\\ 5& 1& 4\\ 1& 9& 1\end{array}\right]\left[\begin{array}{c}1\\ 3\\ 2\end{array}\right]& =1\cdot \left[\begin{array}{c}1\\ 5\\ 1\end{array}\right]+3\cdot \left[\begin{array}{c}1\\ 1\\ 9\end{array}\right]+2\cdot \left[\begin{array}{c}4\\ 4\\ 1\end{array}\right]& & =\left[\begin{array}{c}12\\ 16\\ 30\end{array}\right]\\ & =\left[\begin{array}{c}(1,1,4)\cdot (1,3,2)\\ (5,1,4)\cdot (1,3,2)\\ (1,9,1)\cdot (1,3,2)\end{array}\right]& & =\left[\begin{array}{c}12\\ 16\\ 30\end{array}\right]\end{array}$$

Of course, matrix transformation also has an intuitive geometric understanding, that is, the transformation of space basis vectors. This perspective is especially useful in graphics.

The basis transformation view of matrix transformation

Observe such a matrix transformation process

$$A\cdot \overrightarrow{v}$$

We know that any vector can be expressed as a linear combination of a set of basis in this space, that is

$$\overrightarrow{v}=a\hat{i}+b\hat{j}+c\hat{k}$$

Since the matrix transformation is linear, the original transformation is equivalent to

$$A\cdot (a\hat{i}+b\hat{j}+c\hat{k})=a(A\cdot \hat{i})+b(A\cdot \hat{j})+c(A\cdot \hat{k})$$

That is, the new vector is equivalent to the linear combination of the matrix transformation of each basis vector. In particular, for ease of understanding, we set the basis vectors to be standard orthogonal basis

$$\hat{ı}=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right],\hat{ȷ}=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right],\hat{k}=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$$

So what do these vectors become after being transformed?

Might as well set

$$A=\left[\begin{array}{ccc}a& d& g\\ b& e& h\\ c& f& l\end{array}\right]$$

So

$$A\hat{i}=\left[\begin{array}{c}a\\ b\\ c\end{array}\right],A\hat{j}=\left[\begin{array}{c}d\\ e\\ f\end{array}\right],A\hat{k}=\left[\begin{array}{c}g\\ h\\ l\end{array}\right]$$

Amazing! **The new basis vectors are actually the columns of the matrix! **

Readers with better imagination should have already figured out that the process of matrix transformation is actually to "drag" the standard orthonormal basis to the basis vectors described by each column of the matrix.

Since we are not focusing on mathematics, if there is a term you don’t understand in the above description, it is recommended to look it up before continuing. In the shader, what we need to master is the actual operation process of the matrix and matrix transformation from the perspective of base transformation. We do not need to delve too deeply into concepts such as calculation techniques or eigenvalues. For a simple introduction to linear algebra, you can take a look at the "Essence of Linear Algebra" series of videos from 3Blue1Brown that I posted above. From here on, I will assume that the reader has some basic knowledge of linear algebra. If the reader really cannot grasp the mathematical concepts, he can go directly to the conclusion.

Homogeneous coordinate

The operation process of matrix transformation has been described above. If you pay a little attention, you will find that matrix transformation cannot handle translation transformation (any matrix can only move the origin$(0,0,0)$Transformed to the same origin position, obviously the translation operation cannot be performed). In order to translate in three-dimensional space, we define a homogeneous coordinate of four components$(x,y,z,w)$, the actual position it represents is${\displaystyle (\frac{x}{w},\frac{y}{w},\frac{z}{w})}$. In this way, the following translation process can be achieved:

$$\left[\begin{array}{cccc}1& 0& 0& t_x\\ 0& 1& 0& t_y\\ 0& 0& 1& t_z\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right]=\left[\begin{array}{c}x+t_x\\ y+t_y\\ z+t_z\\ 1\end{array}\right]$$

At the same time, we also agree that the vector w that only represents the direction is 0, and the vector w that represents the point is 1. This makes the point affected by the translation transformation, but the vector representing the direction is not affected.

MVP transformation

Let’s re-examine the MVP transformation from the above matrix transformation perspective.

ModelViewMat derivation

derivation removed

Since Mojang adopted somewhat different assumptions from graphics standards, the mathematical derivation here was removed on 26/03/04 to avoid confusion. Readers are no longer required to understand the derivation process and just look directly at the conclusion.

When you open F3, you will see the player rotation information as shown in the picture above, which corresponds to the Rotation[]part of playernbt. In order to facilitate communication with the data pack, we set the yaw angle to the player'sRotation[0]and the pitch angle to the player'sRotation[1], and name them yawandpitchrespectively. By default, the player looks towards the +z axis of the world. When looking to the right,yaw increases, then the corresponding inverse yaw matrix is ​​(offsetting the rotation of the camera through inverse rotation):

$$A=\left[\begin{array}{cccc}-\cos \theta & 0& \sin \theta & 0\\ 0& 1& 0& 0\\ -\sin \theta & 0& -\cos \theta & 0\\ 0& 0& 0& 1\end{array}\right]$$

When the player looks down, pitch increases, and the corresponding inverse pitch matrix is:

$$B=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& \cos \varphi & -\sin \varphi & 0\\ 0& \sin \varphi & \cos \varphi & 0\\ 0& 0& 0& 1\end{array}\right]$$

Finally we get

$$\text{ModelViewMat}=B\cdot A=\left[\begin{array}{cccc}-\cos \theta & 0& \sin \theta & 0\\ \sin \theta \sin \varphi & \cos \varphi & \cos \theta \sin \varphi & 0\\ -\sin \theta \cos \varphi & \sin \varphi & -\cos \theta \cos \varphi & 0\\ 0& 0& 0& 1\end{array}\right]$$

However, please note that since GLSL internally stores matrices in columns, the matrix that plays the role of ModelViewMat in the code should be declared as

mat4 ModelViewMat = mat4(
    -cos(yaw),  sin(yaw)*sin(pitch), -sin(yaw)*cos(pitch),  0,
        0    ,       cos(pitch)    ,       sin(pitch)    ,  0,
     sin(yaw), -cos(yaw)*sin(pitch), -cos(yaw)*cos(pitch),  0,
        0    ,          0          ,         0           ,  1
);

From this, we can:

• Read the player's rotation angle Yaw and Pitch from each item of ModelViewMat to provide information to the shader.

• Directly specify a specific view transformation to allow the player to look in a specific direction at the client level without causing jitter.

ProjMat derivation

A three-dimensional object in Minecraftworld is represented on a two-dimensional screen by projecting it onto a viewing plane. ProjMat is responsible for such a projection process.

Orthographic projection

Before understanding perspective projection, we can first learn orthogonal projection, which is a relatively simple projection method. Its characteristic is that there is no near-large or far-small effect**. The image size of the same object after projection is the same regardless of the distance.

The principle of orthogonal projection is to directly project objects parallel (usually parallel to the Z-axis) onto a plane. The projected area is limited by the six faces of the rectangle. We can define this area by the coordinates l, r, t, b, n, f projected by each face on the coordinate axis perpendicular to the face.

The function of the orthogonal projection matrix is ​​to map this plane as$(-1,-1,-1)$arrive$(1,1,1)$Within the clipping space, the final clipping space is subjected to perspective division to obtain NDC. There are three steps in total:

• 1. Move the rectangular center of the orthogonal projection area to the origin. This transformation is recorded as$T$

We first calculate the center coordinate of the area$(x,y,z)$, since the coordinates of the six boundaries are known, it is obvious that the coordinate of the midpoint is the mean value of the coordinates of each boundary.${\displaystyle (x,y,z)=(\frac{r+l}{2},\frac{t+b}{2},\frac{f+n}{2})}$

Combined with the knowledge of translation transformation mentioned earlier, if you want to move the midpoint to the origin, then you need to subtract the coordinate of the midpoint from the coordinate of each point in the three-dimensional coordinate system.

$$T=\left[\begin{array}{cccc}1& 0& 0& -\frac{r+l}{2}\\ 0& 1& 0& -\frac{t+b}{2}\\ 0& 0& 1& -\frac{f+n}{2}\\ 0& 0& 0& 1\end{array}\right]$$

• 2. Scale the size of the area to [-1,1]^3. This transformation is recorded as$S$

We first calculate the size of the current area. In this step, we need to pay special attention to the coordinate axis direction of the view space, and subtract the small coordinate value from the large coordinate value. So we can get the x width of the current area as$(r-l)$, the y height is$(t-b)$, the z-depth is$(f-n)$. The length of each edge of the target area is 2 (from -1 to 1). We can easily get the scaling transformation matrix.

Note: The location of the near plane is$z=-n$, the location of the far plane is$z=-f$,and$(-n)-(-f)=f-n$, so here we still use the large coordinate minus the small one.

$$S=\left[\begin{array}{cccc}\frac{2}{r-l}& 0& 0& 0\\ 0& \frac{2}{t-b}& 0& 0\\ 0& 0& \frac{2}{f-n}& 0\\ 0& 0& 0& 1\end{array}\right]$$

• 3. Calculate the composite transformation of two transformations

$$M_{Ortho}=S\cdot T=\left[\begin{array}{cccc}\frac{2}{r-l}& 0& 0& -\frac{r+l}{r-l}\\ 0& \frac{2}{t-b}& 0& -\frac{t+b}{t-b}\\ 0& 0& \frac{2}{n-f}& -\frac{f+n}{f-n}\\ 0& 0& 0& 1\end{array}\right]$$

perspective projection

Different from orthogonal projection, Perspective Projection is a projection method that is closer to the photographic results in the real world. It is also the projection method used in most games including Minecraft.

The principle of perspective projection is to connect the vertex and the camera. This connecting line will have an intersection on a virtual plane. This intersection is the image of the vertex under perspective projection.

We call this plane Near Plane, and slightly further away there is a farthest plane that can be rendered, called Far Plane. The camera and the two planes form a View Cone (Camera Frustum)

The function of ProjMat is to map part of the view frustum consisting of the near plane to the far plane to the clipping space.

To construct a perspective projection matrix, we can first construct a matrix that converts the view frustum into a rectangular area of ​​​​orthogonal projection.$ M_{Pers \to Ortho} $.Then,withtheabovededuced$M_{Ortho}$complex

As shown in the figure below, what this matrix has to do is to transform the prism into a rectangle. In order to obtain the only transformation, we agree on the following two properties:

• All points on the near plane remain unchanged
• The zcoordinate of all points on the far plane remains unchanged

Recall the properties of homogeneous coordinate,$(x,y,z,w)$Represents a point in three-dimensional space$(\frac{x}{w},\frac{y}{w},\frac{z}{w})$, it can be found that after scaling the coordinate, it still represents the same point, that is:

$$k(x,y,z,w)=(kx,ky,kz,kw)\to (\frac{kx}{kw},\frac{ky}{kw},\frac{kz}{kw})=(x,y,z)$$

We will use this in the following derivation.

First, we observe the upper part of the view cone from the side. For any point inside the view cone$(x,y,z)$, its projection on the near plane is$(x^{\prime },y^{\prime },z^{\prime })$, from the similarity relationship of triangles we can get that it exists at every point$x^{\prime }=\frac{n}{z}x，y^{\prime }=\frac{n}{z}y$The relationship (n is the zcoordinate of the near plane).

We know that after changing the view frustum into a rectangle, the x and y coordinates of each point will be the same as its projection, but we don’t know how the zcoordinate will change.

$$\text{M\_{Persp to Ortho}begin{bmatrix} x y z 1 end{bmatrix} = begin{bmatrix} frac{n}{z}x frac{n}{z}y text{unknown} 1 end{bmatrix} to text{text{multiply by } z text{ to eliminate the denominator}} to begin{bmatrix} nx ny text{unknown} z end{bmatrix}}$$

If$M_{Persp\to Ortho}$It is a 4x4 matrix, and we can actually fill in part of it.

$$M_{Persp\to Ortho}=\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ ?& ?& ?& ?\\ 0& 0& 1& 0\end{array}\right]$$

At this time, we need to use the two properties we just agreed on.

• 1. All points on the near plane remain unchanged

We substitute any point on the near plane$(x,y,n,1)$, they will always be mapped to the same location, namely:

$$\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ ?& ?& ?& ?\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}x\\ y\\ n\\ 1\end{array}\right]=\left[\begin{array}{c}x\\ y\\ n\\ 1\end{array}\right]\to \text{multiply by }n\text{ to keep the format consistent}\to \left[\begin{array}{c}nx\\ ny\\ n^2\\ n\end{array}\right]$$

The third component of the observation, which is obtained by dot multiplying the third row of the matrix with the vector

$$\left[\begin{array}{cccc}?& ?& ?& ?\end{array}\right]\left[\begin{array}{c}x\\ y\\ n\\ 1\end{array}\right]=n^2$$

Obviously, the result has nothing to do with x and y, so we know that the first two unknowns must be$0$, you might as well set the last two numbers as$A、B$,Right now

$$M_{Persp\to Ortho}=\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ 0& 0& A& B\\ 0& 0& 1& 0\end{array}\right]$$

and we have

$$\left[\begin{array}{cccc}0& 0& A& B\end{array}\right]\left[\begin{array}{c}x\\ y\\ n\\ 1\end{array}\right]=n^2⟹nA+B=n^2$$

• 2. The zcoordinate of all points on the far plane remains unchanged.

We substitute the midpoint on the far plane, and its z-axis will be mapped to f, which is the original value.

$$\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ 0& 0& A& B\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}x\\ y\\ f\\ 1\end{array}\right]=\left[\begin{array}{c}\frac{n}{f}x\\ \frac{n}{f}y\\ f\\ 1\end{array}\right]\to \text{multiply by }f\text{ to keep the format consistent}\to \left[\begin{array}{c}nx\\ ny\\ f^2\\ f\end{array}\right]$$

The third component of the observation, which is obtained by dot multiplying the third row of the matrix with the vector

$$\left[\begin{array}{cccc}0& 0& A& B\end{array}\right]\left[\begin{array}{c}x\\ y\\ f\\ 1\end{array}\right]=f^2⟹fA+B=f^2$$

now we have

$$\{\begin{array}{l}nA+B=n^2\\ fA+B=f^2\end{array}$$

can be solved

$$\{\begin{array}{l}A=n+f\\ B=-nf\end{array}$$

So we get

$$M_{Persp\to Ortho}=\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ 0& 0& n+f& -nf\\ 0& 0& 1& 0\end{array}\right]$$

Finally, we get

$$\begin{array}{rl}\text{ProjMat}& =M_{Ortho}\cdot M_{Persp\to Ortho}\\ & =\left[\begin{array}{cccc}\frac{2}{r-l}& 0& 0& -\frac{r+l}{r-l}\\ 0& \frac{2}{t-b}& 0& -\frac{t+b}{t-b}\\ 0& 0& \frac{2}{n-f}& -\frac{f+n}{f-n}\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}n& 0& 0& 0\\ 0& n& 0& 0\\ 0& 0& n+f& -nf\\ 0& 0& 1& 0\end{array}\right]\\ & =\left[\begin{array}{cccc}\frac{2n}{r-l}& 0& \frac{l+r}{l-r}& 0\\ 0& \frac{2n}{t-b}& \frac{b+t}{b-t}& 0\\ 0& 0& \frac{n+f}{n-f}& \frac{2nf}{n-f}\\ 0& 0& -1& 0\end{array}\right]\end{array}$$

In addition to l, r, t, b, n, f describing parameters, more often we use FOV (field of view) and Aspect (aspect ratio), n, f to describe. The conversion relationship is as follows:

$$\{\begin{array}{l}t=\tan (\frac{\text{FOV}}{2})\times n\\ b=-t\\ r=t\times \text{Aspect}\\ l=-r\end{array}$$

If described by FOV, Aspect, n, f, then

$$\text{ProjMat}=\left[\begin{array}{cccc}\frac{1}{\tan \frac{\text{FOV}}{2}\times \text{Aspect}}& 0& 0& 0\\ 0& \frac{1}{\tan \frac{\text{FOV}}{2}}& 0& 0\\ 0& 0& \frac{n+f}{n-f}& \frac{2nf}{n-f}\\ 0& 0& -1& 0\end{array}\right]$$

Back to the shader, from the above derivation we can know that we can obtain the field of view, aspect ratio, near plane and far plane information from ProjMat to control the shader process.

As with ModelViewMat, if we want to construct a matrix that functions as a ProjMat, we need to fill it in column-wise:

mat4 ProjMat = mat4(
    1/(tan(FOV/2)*Aspect)  ,        0      ,       0      , 0,
                0          , 1/(tan(FOV/2)),       0      , 0,
                0          ,        0      ,  (n+f)/(n-f) , -1,
                0          ,        0      , (2*n*f)/(n-f), 0
);

From this, we can not only read the player's field of view, aspect ratio and other information from each item of ProjMat. We can also control and modify the projection process, such as using a fixed FOV or switching to orthographic projection.

perspective division

The coordinates of the vertices enter the clipping space after MVP transformation and are output to the special variable gl_Position of OpenGL. These vertex coordinates will undergo perspective division, that is, dividing the x, y, and z components by the w component to obtain the corresponding three-dimensional coordinates. At this time, the vertices beyond the clipping range will be discarded, and the remaining vertices will enter the NDC coordinate system, undergo viewport transformation, and output to the screen.

At this time, the z value in NDC will be mapped to$[0,1]$Within the range, write to Depth Buffer, which we will cover in detail in future tutorials.

The matrix of the viewport transformation is not given here. Firstly, we do not care about the transformation that is not operated by the shader at this step. Secondly, if the derivation of perspective projection is understood, the derivation of this matrix is ​​not difficult for readers.

Summarize

This tutorial starts from the Position attribute in vsh, and explains the whole process of ModelViewMat and ProjMat participating in the transformation to the final output through mathematical derivation. If you overcome the mathematical difficulties, you will have a deeper understanding of the coordinate system conversion and matrix transformation in the shader after reading this tutorial.

After reading this tutorial, readers should be able to know some of the important data that can be obtained in the shader:

• Vertex coordinate: read Position directly
• Yaw angle: atan(ModelViewMat[2][0] / ModelViewMat[0][0])
• Pitch angle: atan(ModelViewMat[1][2] / ModelViewMat[1][1])
• Field of view (FOV): Angle value output 114.591559 * atan(1 / ProjMat[1][1])(one or four quadrants) oratan(1.0, ProjMat[1][1]) * 114.591559 (full quadrant)Note: 114.591559 is the radian angle value coefficient * 2, because ProjMat uses the half-angle of FOV
• View frustum aspect ratio: ProjMat[1][1] / ProjMat[0][0]
• Near plane distance: ProjMat[3][2]/(1-ProjMat[2][2])
• Far plane distance: `

• ProjMat[3][2]/(1+ProjMat[2][2])`

• Whether it is a GUI: GUI performs orthogonal projection transformation, and ProjMat is derived from the previous article.$M_{\text{Ortho}}$. Of course, in GLSL we have to swap rows and columns. We can directly detect ProjMat[2][3] == 0.0

In future practice, we will often calculate and operate under a specific coordinate system, so this section serves as the foundation of the vertex shader and requires mastering more mathematical content. I strongly recommend readers who have no foundation in linear algebra to watch "The Essence of Linear Algebra" by 3b1b, "Painless Line Generation" by Manshi, especially Lecture03 and 04 of GAMES101. These tutorials are not limited in length and are more in-depth and detailed than what I have taught you. GAMES101 is an introductory course in modern computer graphics, which covers the principles and derivation of various transformation matrices in detail.

Of course, the derivation process does not require shader writers to master it. The focus here is still to distinguish the data in the coordinate system and matrix involved in each stage of the code. As we said in the previous section, Minecraftshader's data acquisition is very limited, so we must obtain all the data we need from various global quantities. This includes data such as the perspective in ModelViewMat and the FOV in ProjMat.

The content of this section has reached the most difficult level in the entire tutorial. Future tutorials may be simpler than this one. For example, in the next section we will cover the basic workflow of the fragment shader and explain the remaining Color, UV/UV0, UV2 and Normal vertex attributes with it. These simple contents involve less mathematics and can be completed in one section.

Appendix - Confusion between column major and row major order

The linear algebra content we talk about is generally described in row-major order, but in GLSL it is described in column-major order, that is, the matrix is ​​filled into the GPU memory in columns. If readers try to construct MVP transformations in GLSL code in practice, they will encounter such problems:

If you construct the yaw matrix directly in row-major order$M_{\text{Yaw}}$and pitch matrix$M_{\text{Pitch}}$, and used for MVP transformation. (Calculated from right to left)

$$M_{\text{Pitch}}\cdot M_{\text{Yaw}}\cdot P$$

This is fine from a row-major perspective, but from a column-major perspective in GLSL, this operation is actually in linear algebra:

$$M_{\text{Pitch}}^T\cdot M_{\text{Yaw}}^T\cdot P$$

symbol here$M^T$Represents Transpose , that is, exchanging rows and columns. In particular, our rotation matrix happens to be an Orthogonal Matrix, that is, a matrix in which the columns are orthogonal to each other. It has a property that the transpose of an orthogonal matrix is ​​equal to its inverse.

Meaning that the operation we just performed is actually:

$$M_{\text{Pitch}}^{-1}\cdot M_{\text{Yaw}}^{-1}\cdot P$$

It still yaws first and then pitches, but the direction of each rotation is opposite. This does not cause much visual error, and if you do not check the direction of rotation, you may mistakenly think that the results are as expected.

But if at this time, we want to use a matrix to represent$ M_\text{Pitch} \cdot M_\text{Yaw} $Problems will arise.

We calculate in column major order$M_{\text{view}}=M_{\text{Pitch}}\cdot M_{\text{Yaw}}$, and replace the transformation in the code. What is actually running?

$$M_{\text{view}}^T\cdot P=(M_{\text{Pitch}}\cdot M_{\text{Yaw}}{)}^T\cdot P$$

According to the definition of transposition,$(A\cdot B{)}^T=B^T\cdot A^T$, then our calculation becomes

$$M_{\text{Yaw}}^T\cdot M_{\text{Pitch}}^T\cdot P=M_{\text{Yaw}}^{-1}\cdot M_{\text{Pitch}}^{-1}\cdot P$$

This becomes pitch first, then yaw, and the two rotation directions are also opposite.

Since these operations rotate the object's own coordinate system, the order is important. If you don't understand that the order of two transformations is so important, you can turn your head. If you yaw first and then pitch, you shake your head left and right first, then up and down. At this time, the world will not "slant" in your eyes. But if you change the order, first pitch, then yaw, that is, first nod up and down, and then shake your head left and right along the axis from your chin to the top of your head, then the world will "tilt over." Obviously not as expected.

In particular, when the pitch executed first reaches 90°, the yaw executed later will be directly equivalent to a Roll operation. The problem involved behind this is called "universal joint deadlock" or "Eulerian angle deadlock". Interested readers can check it out by themselves.

So when you derive matrices from a row-major perspective, be sure to remember to swap rows and columns when filling in GLSL.

mat3 m = mat3(
    1.0, 2.0, 3.0,   // 第一列
    4.0, 5.0, 6.0,   // 第二列
    7.0, 8.0, 9.0    // 第三列
);