superterrainplus - Bug Fix and Improvement

Release 0.16.7

Map type refactor

Instead of using primitive type like unsigned short, use fixed-size integer type such as std::uint16_t for terrain maps such as biomemap and heightmap. This provides a stronger guarantee when exchanging data between OpenGL (which specifies data in exact bit-width like GL_R16) and host.

This release adds the following map types to STPWorldMapPixelFormat:

• STPHeightFloat_t
• STPHeightFixed_t
• STPSample_t
• STPRegion_t
• STPSeed_t

STPDeviceLaunchSetup

This release adds a device launch configuration utility. This helps refactor calculation of device grid and block size during device launch, and calculate the thread index in kernel. The launch configuration can determine the best grid and block size to minimise the number of idling thread, although some justifications are required to prove that the calculation yields the optimal configuration.

General fixes and improvement

• Add STPPlatform.h with platform detection and compiler warning utilities.
• Fix a UB in STPWorldPipeline that reinterpret casting std::unique_ptr<void*[]> to arbitrary type, such as float** without lifetime startup.
• Replace unsigned long long with STPSeed_t whenever applicable, i.e., when the variable is used as a 64-bit seed value.
• For SuperTest+, remove use of using namespace to bring all SuperTerrain+ names into scope, and only do using for names needed in the current test source code.
• Improve CUDA compiler flags, make sure all host flags are forwarded correctly to NVCC.
• Fix some compiler warning about structure padding in STPSimplexNoise.
• Make creator in STPObjectPool public.
• Remove test for STPSmartDeviceMemory because of triviality.
• Add a few more TODOs for C++20.
• The test reporter now prints the random seed used to run the test.
• Remove unused compiler warning suppression. Fix some warning rather than suppressing.
  • Reorder fields for STPFractalSimplexInformation in STPSimplexNoise to avoid padding.
• Change validation for STPAtmosphereSetting: view altitude does not have to always be greater than planet radius.

Placement new

Use placement new instead of cast in the following places to avoid undefined behaviour.

• Distribute erosion brush shared memory in STPRainDrop
• STPCamera
• STPCascadedShadowMap
• STPScenePipeline
• STPSceneLight
• STPMaterialLibrary

Generator

• Merge STPHeightfieldSetting with STPRainDropSetting.
• Simplify shared memory loading in STPRainDrop.
  • Remove use of shared memory to compute base and range index of hydraulic erosion boundary; this is premature optimisation.
• Refactor renamed some types in STPHeightfieldKernel.

STPRealism

• Refactor STPRandomTextureGenerator and put generation utilities inside this source rather than STPHeightfieldGenerator.
• Use 64-bit seed instead of 32-bit for STPStarfield.
• Move seed from STPStarfieldSetting to STPStarfield function.

- C++
Published by stephen-hqxu about 3 years ago

superterrainplus - Project Restructure and Refactor

Release 0.16.6

Project directory restructure

The most noticeable changes in this release is the better organised project directory structure and cleaner build system. Despite this fact, the API is untouched so there aren't many major changes to be made to the codebase.

• Tidy everything up with CMake's source_group.
• Separate SuperAlgorithm+ host and device library into two distinct file trees. This would require a tiny bit of modification to the include paths.
• Refactor the build script a lot.
• Refactor the config file generation function.
• Make sure source and header file subdirectories are consistent.
• Suppress unwanted nvcc warnings.

General fixes and improvement

• In STPTextureDatabase, use std::string_view directly rather than std::optional<std::string_view> for less verbosity.
• Improve STPCamera subscriber, it can now automatically unsubscribe to avoid holding dangling pointer, and no longer requires listener to write user-defined copy/move constructor.
• Remove STPListenerError as being unused.
• Simplify exception message formatting implementation.
• nanobench is no longer downloaded automatically, and use CMake's install target instead, for the sake of compilation time. With that said, you will need to install this library manually on your computer.
• Put STPLayerCache in header because there is no need to use PIMPL for such as small class.
• Simplify naming for various objects in STPSmartDeviceMemory.
• Use type brace initialisation instead of explicit value initialisation in STPSmartDeviceObject.
• Rename functions in STPObjectPool for better simplicity.

Biome

• Replace use of variadic template with std::initializer_list.
• Make behaviour of ascendant access when index is out of bound undefined.
• Refactor biome generator in the demo application. This includes:
  • Move definitions from header to a separate source file.
  • Rename variables for style consistency.
  • Remove using from headers.
• Pass ascendant as reference rather than pointer for standardised representation.
• Rather than storing layers as an array of std::unique_ptr of layer, we initialise all layers at once and store them in a structure, then store this structure as an array member. This significantly reducing the number of memory allocation and avoid memory fragmentation.

STPShaderManager

• Can now automatically scan the shader include directory and load all shader source files as required, thanks to std::filesystem.
• Remove redundant functions.

STPSingleHistogramFilter

• Improve signed integer arithmetic to avoid overflow.
• Replace bit casting implementation with compiler built-in. This change acts as a transition before C++20 thus is temporary.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Improvement to SHF

Release 0.16.4

STPThreadPool

This release adds automatic loop parallelisation for thread pool, includes splitting a number of loop iteartion into equal-size blocks, and dispatch each block in parallel.

STPSingleHistogramFilter (#54)

This release brings some improvements to our SHF and it is now more efficient.

• Now intermediate filter working memory is stored along with the user-managed filter buffer rather than keeping it within an object pool in the internal filter implementation.
• Remove use of allocator traits for STPArrayList, and make it simple to use either default memory allocator or pinned. This reduces the memory consumption by more than a half because std::function is very big.
• Remove use of union in the data structure, and use template instead. So we don't need an access modifier for that any more.
• Replace most std::copy with std::memcpy; it is about 10% faster.
• Improve internal dynamic array memory expansion. Now it no longer doubles the size when resized or inserting N members based on the new size, but chosen smartly between this and the original capacity. This helps to cut-down over-allocation.
• Simplify the implementation significantly because the data structure no longer has private members.
• Make all fields in STPSingleHistogram to use const T* instead of T* because they are not intended to be modified by the user.
• General refactor.
• Update documentation.

Single-thread

To make the filter suitable for a wider variety of problem size, it now supports running in serial mode. Whether the filter is run on serial or parallel mode depends on the type of filter buffer user pass to the filter. A serial filter buffer can only be run on serial filter, while parallel filter buffer can be run on both version, and will be forced to serial version if the problem size is too small.

General fixes and improvement

• Remove unnecessary typename from STPTestNearestNeighbour.
• Make all global variable in the configuration files (those .h.in files) inline rather than static.
• Add a new configuration file containing version information.
  • Add version printing for the demo program.
• Use std::optional as return match length for STPRegularLanguage.
• STPSmartDeviceMemory: Now allows array type specialisation. This is useful for memory accessible on host, e.g., pinned memory. Although the deleter is the same, it allows using the array access operator. Also refactor a little bit.
• Use heap memory to store the log to avoid blowing up the stack in STPCommonCompiler.
• CMake downloaded external resources now are placed in a directory with version number encoded, which allows CMake to automatically detect and re-download whenever necessary if any of the 3rd-party library is updated.
• Add nanobench as a new dependency for benchmarking, whose code will be integrated in SuperTest+.
• Bump CMake version to 3.20 minimum.
• Bump stb_image version to 2.28.

STPCommandLineParser

• Nested name refactor for STPCommandLineParser.inl.
• Allow nested type for argument conversion, and user-defined argument conversion function.
• Use move when returning option table.
• Fix a bug for allowing using short name when a long name option control is used.

STPDemo

• The demo application now uses a unified RNG seed value for simplicity.
• Add more command line options.
• Make demo program version information more concise.
• STPBiomefieldGenerator now selects appropriate filter execution type based on the map dimension.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Command Line Parser

Release 0.16.2

STPCommandLineParser (#51)

This release brings a flexible command line parser framework that allows application to define how to read command line inputs and convert those into specific type then stored to variables.

Obviously this will be nowhere as good as the existing mature command line parsers out there in the open source community, but it suffices for our application, and for fun.

Key features: - Standard Unix-like command line syntax - Configurable - Positional argument - Variable binding to option - Group and subcommand - Pre/Post validation - Help message printer

STPDemo

• Add command line options to the demo application. Run -h to get the help message for more detail.
• Split STPMasterRenderer from STPStart.cpp because it takes too long to compile this large file.
• Fix a bug at the start of the demo program for not setting the viewport size along with the window size.
• Now support full screen mode.

General fixes and improvements

• Bump license date.
• Change some exceptions from syntax error to semantics error in TDL parser.
• Simplify test assertions with the new library features.
• Fix STPTestNearestNeighbour for forgetting to check the return value of a test function.

STPLexer

• Allow caching lexical state index to avoid repetitive calculation.
• Now lexical state can also be assigned by the lexical state object rather than its ID only.
• Fix a lexer error when the last few tokens in the input are taking discard action. Now the lexer checks if the remaining string is empty every loop cycle.
• Allow configuring the lexer by supplying a lexical behaviour structure as an constructor argument.
• Make all members in STPToken mutable so it can be copy assigned properly.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Lexer Library

Release 0.16.0

STPLexer

We scrap out the lexer from STPTextureDefinitionLanguage and make it a more general-purpose lexer that allows defining custom tokens and matching functions. This makes writing parsers in the project in the future easier. The lexer uses a state-based approach that allows grouping token matching functions in a number of states.

The following new class are added to SuperAlgorithm+Host. A test code STPTestLexer is also added to the test program. - STPBasicStringAdaptor, is intended as an replacement to STPINIBasicString. - STPLexer

STPTextureDefinitionLanguage

• Rewrite TDL parser to make use of STPLexer.
• TDL parser is now stateless.
• Change the TDL specification a little bit. It is no longer required to write semicolon at the end of the texture directive, and texture variable can contain underscore.

STPINIParser

• Remove STPINIBasicString, it should be replaced by STPBasicStringAdaptor.
• Remove implementation for writing to an INI string because it is so flawed.
• Merge STPINIReader and STPINIWriter into STPINIParser.
• Merge STPINIStorage and STPINISection into STPINIData.
• INI parser now preserves order of each item declared in the parsed input.

STPRegularLanguage

The regex library in C++ standard library is terrible, it is a good idea to implement our own. Basically it is a regex parser, but it uses template as matching terms rather than regex string. DFA/NFA are constructed at compile-time with no dynamic memory allocation in runtime.

The regular language parser STPRegularLanguage, resides in the algorithm+ package. Test is implemented as STPTestRegLang.cpp.

General fixes and improvement

• Add file filters for SuperAlgorithm+ and SuperDemo+ built targets.
• Do some clean up to STPTextureDatabase.
• Reduce the printed horizontal separator width in the exception error message.
• Simplify error message for STPParserError.
• Make sure all use of functions from header cctype have the input argument cast to unsigned char to avoid UB.
• Make global constexpr variable to use external linkage rather than internal (change from static to inline) for:
  • STPAlgebraCommon
• Simplify naming for nested namespace for:
  • STPHashCombine
  • STPSmartDeviceMemory
  • STPSmartDeviceObject
• Remove an unused variable in STPWorldPipeline.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Improve Exception Library

Release 0.15.10

STPException (#47)

• Reclassify exception categories including breaking vague exceptions into a smaller, more clearer exceptions, as well as removing unused exceptions.
• All STPExceptions now provide basic source information to aid with debugging.
• Exception details are broken down based on exception types and separate each piece of information as member variable in the exception class.
• Most exceptions now provide a macro for easier creation.
• Support assertion exception that is similar to C assert but works in release mode as well.

General fixes and improvement

• Fixing compiler errors in test program.
• Add documentation for single histogram filter.
• Add information for citation.

STPWorldPipeline

• Incorrect use of neighbour local chunk index coordinate offset as world coordinate offset for splatmap generation.
• Incorrect assumption about main chunk request response and force heightmap reload response that they are subset of each other, in fact they might be disjoint. This causes some chunks are not correctly loaded.
• A mistake for not waiting for the last batch of work to complete and causes a race condition at the end of the erosion work launcher.
• Fix a case where chunks are not updated when the camera is moving too fast, such that the old loading request is discarded.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Improve World Generator

Release 0.15.8

Heightfield generator

To disambiguate and for clarity, we renamed a few free-slip related stuff to nearest neighbour. This decision is made because free-slip is a concept used in hydraulic erosion, but it is overused in the codebase. Things like biome smoothing and splatmap generation which take also neighbour chunks around the centre chunk during generation should not be referred as and confused with free-slip generation.

Therefore, the following files (and corresponded class) are renamed:

• STPFreeSlipTextureBuffer -> STPNearestNeighbourTextureBuffer
• STPFreeSlipInformation -> STPNearestNeighbourInformation
• STPTestFreeSlip -> STPTestNearestNeighbour

A quick summary of the major changes:

• In STPChunkSetting, split NN setting for diversity generation and erosion. We also renamed a few settings and improve documentation for clarity.
• Rewrite and re-document STPNearestNeighbourTextureBuffer. Now memory mode is part of the template parameter.
• Split generation procedure in STPHeightfieldGenerator into 2 separate functions, heightfield generation and erosion. Also eliminate dynamic memory allocation by passing terrain maps as raw pointers.
• STPHeightfieldGenerator no longer calls cudaSetDevice(), it gives user more freedom to choose their device context if they are working with a multi-GPU system (for example rendering on GPU 0 and computing on GPU 1).
• It is now application's responsibility to ensure the number of nearest neighbour chunk is the same as the value passed during initialisation of the NN texture buffer and generator.
• Rewrite test for STPNearestNeighbourTextureBuffer.

There are way too many changes involving heightfield generators to summarise them briefly, I have rewritten the documentation in code so take a look to find out more.

STPWorldPipeline

We have moved the terrain generation from real-time streaming, i.e., send chunk to renderer as soon as one is done, to the way that returns only when all chunks in a render distance are available. This is done because STPWorldPipeline already has a double buffer system that is swapped only when all generations are done, so it is no longer necessary to stream chunks immediately to back buffer.

Generator manager in STPWorldPipeline has been revised with added implementation for efficient neighbour condition checking and scheduling to avoid race condition.

STPChunk

• Change the function calcChunkNeighbour to calcChunkNeighbourOffset. The difference is the prior one returns chunk world coordinate and the latter one returns relative offset, which can be converted to world coordinate by adding the coordinate of the centre chunk. This can help the world pipeline to cache neighbour offsets and reuse to avoid repetitive small memory allocation.
• Remove all internal states tracking such as occupancy status, it only holds texture data and completion status.
• Fix an incorrect calculation in calcLocalChunkCoordinate().
• Add a new function calcLocalChunkOrigin().
• Refactor some class like STPNearestNeighbourTextureBuffer and heightfield renderer to make use of the chunk utility functions instead of calculating from scratch.
• Improve testing robustness for chunk calculations in STPTest2DChunk.

General fixes and improvement

• Remove the constructor overload for taking C-style array in STPProgramManager.
• Qualify all function arguments as const whenever sensible.
• Add inline to all separate template definitions to ensure ODR.
• Fix an erroneous surface 3D write addressing in STPRandomTextureGenerator kernel, surface meant to be addressed using byte rather than index.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Major Class Refactor

Release 0.15.6

Class refactor (#52)

Based on the suggestion in the issue, the following class has been rewritten:

• STPPermutationGenerator
• STPErosionBrushGenerator, merged with STPHeightfieldGenerator.
• Refactor STPBindlessTexture to use the bindless texture handle utility (in STPSmartDeviceObject).
• Merge STPBindlessBuffer with STPBuffer and remove the wrapper because not making the buffer address non-resident will not cause any damage as long as the buffer itself is deleted properly.
• STPShaderMangar
• STPProgramManager and STPPipelineManager are now stateless.

Robust memory management

STPSmartDeviceObject

• Add managed GL texture and bindless texture memory management.
• Suffix keyword Resource to makeGLBuffer and makeGLImage function to disambiguate.
• Remove default parameters when creating managed stream and event.
• GL texture memory in STPWorldPipeline and STPTextureFactory are now managed by smart pointer.

STPTextureFactory

• Creates bindless texture handle for every texture object.
  • Because texture sampler state cannot be changed once bindless handle is created, sampler states are not changed via a function passed during construction of STPTextureFactory.
• STPHeightfieldTerrain now no longer needs to create bindless texture separately.

General fixes and improvement

• Simplify initial permutation table generation in STPPermutationGenerator.
• Now you can use left shift key to move twice as fast in the demo program.
• STPHeightfieldTerrain now generates random noise texture during constructor call.
• Make some functions in STPSingleHistogramFilter and STPLightSpectrum noexcept.
• Fix some spelling errors in documentation.

SuperRealism+

• Make all GL object wrappers noexcept because C API never throws exception, instead we use debug callback to capture any error.
• Reduce dynamic memory allocation by replacing std::vector with std::initialiser_list in the following function:
  • drawBuffers() function in STPFrameBuffer.
• setData() function in STPLightSpectrum now takes raw data parameters rather than using template, it is too much work to deduce pixel types information from input type.
• Add a validate function in STPFrameBuffer to help decoding the framebuffer status flag into an exception.
• Improve STPLogHandler by using function to set the active log handler rather than using global variable.
• Add program active state manager to prevent state overrides to program pipeline.
• Refactor STPDebugCallback, now user is responsible for handling the debug log.
  • Now the demo program will exit if an error GL message is encountered.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - General Refactor

Release 0.15.3

SuperAlgorithm+Device

• Remove OOP designs for most SuperAlgorithm+Device source code because they are too trivial.
• Remove STPKernelMath because it is too trivial.
• Make implementation of STPSingleHitogramWrapper inline.
• Revamp STPTextureSplatRuleWrapper, decouple the process of finding splatmap registry to avoid repetitive computation.
• simplex2DFractal() function now by default returns normalised noise value. The fractal information now is taken as a reference to constant value.
• Use unsigned int in simplex2D whenever applicable.

STPCamera

• Remove STPOrthographicProject because we will never need this in this particular application; merge STPPerspectiveProject with STPCamera.
  • We will add more camera model in the future.
• Remove STPCameraInformationMemory from STPScenePipeline. Instead, the camera class now manages its own memory. User only needs to attach the camera to the scene pipeline.
• Now camera callback no longer uses a bunch of virtual functions. As camera events are queued and require an explicit event polling by the application, camera update events are now reported as flags.
  • This improve performance. If user needs to update many camera attributes every frame, we don't need to flush buffer and recompute transform matrices for every small changes, instead it is done once all small changes are consolidated.

STPEnvironment

• Remove STPSetting base class because we don't need polymorphic behaviour.
• All STPEnvironment are now default constructed, this makes initialising parameters using brace initialisation much easier.
• Change specification of validate() function. Instead of returning a bool indicating the validation status, it throws exception directly when it fails.
• Remove STPTestEnvironment because it is so trivial.
• Remove STPConfiguration because it is useless in the core engine.
• Refactor STPHightfieldGenerator slightly, it only copies erosion settings (rather than the whole heightfield settings) to the device.
  • Also for STPHeightfieldSetting, compositing STPRainDropSetting instead of inheriting.

Pass by value

• STPWorldPipeline now copies the chunk setting into the object rather than taking a reference.
• STPHeightfieldGenerator does the same as above for STPHeightfieldSetting.
• STPDemo::STPWorldManager no longer needs to store those settings.

General fixes and improvement

• Remove STPLayerTree. Now the user application is responsible for managing memory to layer tree structure provided to STPBiomeFactory.
• Replace some use of std::accumulate with std::reduce.
• Remove a unused variable from STPTestFreeSlip.
• Fix a compilation error in STPTestUtility.
• Avoid doing floating point division in a massive loop in STPSingleHistogramFilter. However this causes a tiny loss in FP accuracy, and we have changed the tolerance in test program to accommodate this tiny error.
• Offset parameter in STPSimplexNoiseSetting now by default takes a unit in radians, was in degree, for consistency with our API.
• Make string object composed in STPINIBasicString rather than inherited.
• Refactor constructor arguments for STPHeightfieldGenerator into a structure.
• Refactor heightmap texel unit calculation in terrain fragment shader.
• Correct many typos.

Distance settings in SuperRealism+

Now the following shader parameters are specified in terms of maximum viewing distance multiplier rather than an absolute distance. This make tweaking camera far and terrain size much easier. - Terrain texture triple scaling far distance. - Terrain and water mesh tessellation distance. - Water wave height (relative to water plane altitude). - Cascaded shadow map light frustum settings.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Update Environment and Refactor

Release 0.15.1

Update CUDA

• Bump CUDA version to 11.7 minimum (main development is carried on 11.8). Remember to regenerate your build tree. Some features from CUDA beyond 11.3 that are going to be integrated shortly:
  • 11.4: link time optimisation for NVRTC with NVVM interop.
  • 11.6: NVRTC automatic includes directory where source file resides.
  • 11.7: optix-ir output for NVCC.
  • In general, a huge reduction on compilation time is observed.
• Add support to NVVM compiler. STPDeviceRuntimeBinary and STPDeviceRuntimeProgram now accept NVVM input.
  • Enable NVVM mode under Release build for STPCommonCompiler; this change is experimental.
• Change cuRAND generator in STPRandomTextureGenerator to curandStatePhilox43210_t because the previous one takes a massive number of register (somewhere around 250). This also reduces the binary size of SuperRealism+ significantly.

There isn't much to change really, CUDA is pretty good built for backward compatibility.

GLSL shader refactor (#49)

• Replace all unsigned int with uint.
• Make sure the lod parameter in all use of textureLod function is a float.
• Remove space between layout and the open parenthesis.
• Remove camera function selection because they are too redundant.
• Remove inappropriate restrict qualifier to pointers. Qualify underlying value of pointer as const whenever applicable.

Biome generator refactor

The biome utilities have not been patched for years and there are a few bad programming practice I want to eliminate.

• Merge STPLayerCache into STPLayer.
• Move all private static functions in STPLayerCache to the source file and assign them with internal linkage.
• Remove 3D generation option for STPBiomeFactory.

STPLayer

• Make constructor and destructor public.
• Add layer cache size at initialisation.
• Use std::unique_ptr rather than std::vector to store layer ascendant because the size is fixed.
• Rename STPLocalRNG to STPLocalSampler.
• Refactor function visibilities for STPLocalSampler.
• Rename a few more functions.

Sorry for breaking the API completely...

STPLayerTree

• Rename STPLayerManager to STPLayerTree.
• Inline all definitions because they are too trivial.
• Layer cache size is now specified from function argument rather than non-type template parameter. Unfortunately this also breaks the API.

General fixes and improvement

• Add Zc:lambda flag to MSVC compiler to prevent spurious compilation error on lambda expression.
  • Change STPThreadPool to make use of the standard-conforming lambda.
  • Fix a compilation error on MSVC in STPFile.inl by moving the lambda outside the conditional statement.
• Bump CMake project version to 0.15.0.
• Split the generate filename utility from STPFile and put it into a new file STPStringUtility.h.
• Replace all size_t literal from ull to u.
• Simplify STPEngineInitialiser.
• Add noexcept to certain functions around the engine.
• Make recNeighbourChecking function in STPWorldPipeline template regarding recursion pass.
• Remove option to toggle AVX instruction usage. The minimum CPU requirement in the README already guarantees availability of AVX, and I don't see the reason for not using it whenever applicable (and of course I don't want to maintain 2 different code paths).
• Fix a compiler error on some compilers when compiling STPTestMain.cpp.
• Change [[nodiscard]] usage. Only add this attribute when discarding return value will cause serious problems such as memory leak, or hard-to-spot bug.
• Move ID counters in STPTextureDatabase to the source file.
• Fix some typos.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Ray Tracing Framework

Release 0.15.0

This big release brings a shiny new framework for ray tracing to improve the rendering quality; it is so new such that we don't have time to test it properly. The new feature is currently detached from the main rendering pipeline, and the remaining works is expected to be finished in the next major release. Plus many more bug fix and optimisation...

The main thing about this release is not about this new framework because it is not usuable yet, but all critical fixes and improvement to the current engine.

What's new

STPExtendedScenePipeline, as an extension to STPScenePipeline, focuses on ray traced rendering as opposed to rasterisation thanks to OptiX. This pipeline currently only does ray casting rather than tracing for testing intersection of reflection and transmission ray from water renderer; more amazing rendering algorithms with ray tracing are coming in the future.

See issue #50 for more details regarding the framework.

Quick summary of major changes

The major changes are here because they are essential before we can implement the new framework. There are too many, you may want to read the release notes from the preview versions for full detail.

SuperTerrain+

• Overhaul to NVRTC runtime compiler framework with better user-control and memory management. NVRTC compiler and CUDA driver module management are separated into STPDeviceRuntimeBinary and STPDeviceRuntimeProgram respectively, and they now use a stateless model.
• Overhaul to configure template including renaming and namespace enclosure.
• Overhaul to API error handler, STPGenericErrorHandlerBlueprint is included for easier future integration to other APIs.
• Overhaul to smartly managed CUDA objects and put them to STPSmartDeviceObject.
• Overhaul to STPThreadPool for more robust destruction control.
• Remove STPMemoryPool.

SuperRealism+

• Add more GL types to STPOpenGL.
• Refactor all GL object class to be consistent with GL official API signature.
• Simplify abstract rendering component class STPSceneObject by merging shadow and non-shadow variant of STPOpaqueObject and removing STPDepthRenderGroup. Add a new type of object STPAnimatedObject which is currently used by STPWater and all environment objects.
• Change the logic for STPScreen and STPAurora and application is no longer forced to derive from them.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Finishing up Ray Tracing Framework

Release 0.15.0-Preview.3

SSRI shader

• Reduce primitive vertex memory and payload register usage by replacing vertex position with ray time to calculate the intersection position.
• Refactor geometry attribute for vertex and index data into 2 structures for easier data accessing. This is placed in a new file STPGeometryAttributeFormat.hpp.
• Fix some OptiX shader compiler flags, it should be using a physical architecture.
• Add a new public GLSL header STPRayTracedIntersectionData.glsl for sharing SSRI data with each renderers.
• Add check for the scene memory limit in STPExtendedScenePipeline to make sure it does not exceed the maximum allowance of each shader.

General fixes and improvement

• Deprecate and remove STPMemoryPool to save memory.
• Add a new smart device memory object STPPinnedMemory for management of memory allocated by cudaMallocHost.
• Improve documentation for STPSingleHistogramFilter.
• Don't need to check for last CUDA error when launching a kernel via cuLaunchKernel.
• Fix use of explicit template specialisation without declaration in the header, which is UB, in:
  • STPTexture
  • STPFrameBuffer
• Exclude structure definitions for types used in shaders only in STPTextureInformation for compatibility between NVRTC and other compilers.
• using cleans up for various scene components.
• For NVRTC used in STPTestRTC and STPCommonCompiler, zero-init the log outputs for them to avoid printing garbage string.
• Remove linking against CUBIN in STPDeviceRuntimeProgram because CUDA does not support.
• Disable representative fragment testing during normal rendering because this is not how it is supposed to be used.
• Add a new documentation entry regarding SSBO index usage.

Compiler-specific warnings and errors

• Avoid a signed-unsigned comparison in STPWorldPipeline.
• Ensure enum type for shader memory type in STPExtendedScenePipeline provided is valid.
• Add extra parentheses in STPScenePipeline for clarity.
• Add dependent template specifier in STPNullablePrimitive.
• Remove TypeCount from STPTextureType because it is no longer useful.

STPThreadPool

• Fix the thread pool for proper tasks completion before pool is destroyed.
• Fix a UB test case on thread pool when the pool is used after is has been destroyed.
• Improve test case for the thread pool.
• Remove STPDeadThreadPool exception.
• Add a new license regarding our new thread pool implementation.

GL API consistency

• Add more GL compatibility types to STPOpenGL.
• Add GL vector types, which stay in a new file STPGLVector.hpp under STPRealism.
• Improve consistency for GL objects wrapper with GL specification, for:
  • STPBuffer
  • STPFrameBuffer
  • STPImageParameter
  • STPRenderBuffer
  • STPSampler
  • STPTexture

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Implement Extended Scene Pipeline

Release 0.15.0-Preview.2

STPExtendedScenePipeline

• Add STPExtendedScenePipeline and STPExtendedSceneObject for rendering with ray tracing.
• Add STPAsyncAccelBuilder as a helper for double buffered async AS builder.
• Setup rendering pipeline for screen-space ray intersection testing.

SSRI shader

• Suffix the structure name in shader STPScreenSpaceRayIntersection.cuh with Data to avoid name collision.
• Change the specification of primitive data. Now it uses object ID to locate an array of instances with that object, and instance ID to locate an array of primitive data, to make support for multiple objects with multiple instances better.
  • Both IDs are encoded into the user ID provided during IAS build.
• Remove empty SBT data fields to save memory.
• Enclose all texture types in SSRIData inside a structure for easy update.
• Change SSStencil to unsigned char* from cudaSurfaceObject_t because CUDA does not support mapping a stencil texture.
• Change environment ray ID to 127 to avoid collision with object ID 0.

STPSceneObject

• Add a new type of abstract object, STPAnimatedObject, in STPSceneObject.
• Modify STPAurora, STPStarfield, STPWater and STPSun to inherit from STPAnimatedObject to allow user to control the animation timer explicitly.
  • This is done to make sure these objects use the same animation time when they are rendered again during ray tracing pipeline so the ray traced scene match that of rasterised.
  • Note that STPSun now takes an absolute time in second rather than previously a delta time.
• Add a helper function in STPEnvironmentObject for checking visibility.

General fixes and improvement

• Loop fusion for texture object build in STPTextureFactory.
• Make function copyToBuffer() template regarding value normalisation in STPSingleHistogramFilter.
• Fix an incorrect floating point modulo in camera rotate function.
• Add a sampler binding state manager to ensure sampler is unbound from a texture unit once finished to avoid state leakage (sampler overrides texture).
  • Deprecate and remove the unsafe sampler binding function.
• STPWorldPipeline::STPMemoryManager: instead of taking the reference to the whole world pipeline, now it only takes a CUDA stream because that's the only thing it uses.
• Improve documentation and CMake setup to indicate the minimum OptiX version (currently being 7.3).
• Add a new function getTextureImage() to STPTexture.
• Add STPSmartDeviceObject::STPGraphicsResource for managing GL-CUDA interop buffer and texture smartly.
• Fix an incorrect assumption about column-major matrix in STPAmbientOcclusion.frag and STPAurora.frag.
• Fix an incorrect addressing related to surf2Dread and surf2Dwrite in STPSplatmapGeneratior and SSRI shader.
• Remove explicit enum value declaration for TotalType in STPTextureType.
• Add more details about coding style in the contribution guide.
• Make sure STPRandomTextureGenerator waits for kernel execution before freeing all memories.

STPNullablePrimitive

• Add STPUniqueResource alias as a refactor for long typename.
  • Replace all use of std::unique_ptr<std::remove_pointer_t<T>, Del> with STPUniqueResource<T, nullptr, Del> for consistency.
• Improve to make sure it obeys strictly to NullablePointer requirement to avoid superfluous surprises.

STPLogHandler

• Remove inline static specifier for two global variables in STPLogHandler to use external linkage. To ensure unique address across DLL on Windows platform, they are defined explicitly in a source file.
• STPLogHandler now takes a std::string_view. This is to avoid passing extra and possibly garbage data to the log handler since STPExtendedScenePipeline uses an over-allocated string and retains the previous log content if not overwritten.

Runtime compiler

• Abandon OOP design for STPDeviceRuntimeBinary and STPDeviceRuntimeProgram.
• Add 2 handy function in STPDeviceRuntimeBinary to quickly retrieve program PTX and CUBIN.

STPOpaqueObject

• Now both shadow and non-shadow variant reside in the same base class for simplicity. This is done to also make future use of STPExtendedSceneObject on a STPOpaqueObject easier to reduce complexity of permutation.
• Remove implementations for STPDepthRenderGroup because they are too trivial, instead the type aliases are provided.

STPScreen and STPSkybox

• Add STPScreen::drawScreenFromExecutor() function to use RAII for state management.
  • The original version now manages states automatically instead of asking user to bind the buffer and use the program.
• Now user is not forced to derive from STPScreen and STPSkybox, they can be used as member variable.
  • This is done because in the future when ray tracing is added to the main rendering pipeline, environment object needs to derived from both STPScreen (to resolve G-Buffer output from ray tracer) and STPSkybox (the rasterisation environment rendering), which is ugly.
• Remove redundant accessor functions such as getColorHandle from STPSimpleScreenBindlessFrameBuffer because all fields are public.

- C++
Published by stephen-hqxu over 3 years ago

superterrainplus - Setup Ray Tracing Environment

Release 0.15.0-Preview.1

Setup OptiX

• Add a CMake module finder for OptiX.
• Add STPRendererErrorHandler to handle API errors from OptiX.

Setup ray tracing shader

• Implement simple ray tracing shaders for ray-primitive intersection test.
• Refactor STPCameraInformation.glsl slightly for code reuse.

Runtime compiler

• Split STPRuntimeCompilable into STPDeviceRuntimeBinary and STPDeviceRuntimeProgram. This is done to make future use of NVRTC for OptiX compilation easier.
  • STPDeviceRuntimeBinary is a wrapper over NVRTC and outputs nvrtcProgram.
  • STPDeviceRuntimeProgram is a wrapper over CUDA driver API module manager and outputs CUmodule.
• The new runtime compilers are now stateless.
• Explicitly manage memory of all const char* using std::string.
• Move device compute capability query to STPEngineInitialiser.

General fixes and improvement

• Simplify environment setup for SQLite and CUDA to use find_package instead of include.
• Use NDEBUG instead of _DEBUG to check for build mode in source code to maintain C++ standard.
• Remove flag --fmad=false from runtime compilers.

Config file template

• Refactor STPAlgorithmDeviceInfo and STPRealismInfo; remove redundant naming prefixes, enclose variables into appropriate namespaces.
• Add information regarding OptiX directory to STPRealismInfo.
• Add STPCoreInfo.h.in containing information regarding the core engine and include directory for libraries.

Error handler

• Add STPGenericErrorHandlerBlueprint as an extendable C API error handler.
• All definitions are now inlined. Use of template specialisation previously without header declaration leads to undefined behaviour.
• Split the old STPDeviceErrorHandler into STPDeviceErrorHandler for CUDA call and STPDatabaseErrorHandler for SQLite database call.
• Capitalise all macros and rename to reduce verbosity, for example STPcudaCheckErr is renamed to STPCHECKCUDA. However it creates a huge diff since all uses of such macros needs to be renamed.

STPSmartDeviceObject

• Merge STPSmartStream and STPSmartEvent into one file, called STPSmartDeviceObject.
• Add STPMemPool, STPTexture and STPSurface as two new managed CUDA objects for CUDA memory pool, texture and surface objects respectively.
• Remove tests for these CUDA objects because they are too trivial.

- C++
Published by stephen-hqxu almost 4 years ago

superterrainplus - Improve Precision

Release 0.14.5

Reversed depth buffer (#45)

• Modify projection matrices for main camera and shadow camera to use zero-to-one convention.
• Change clip volume to DirectX convention.
• Flip near and far plane for projection.
• Clear depth of main camera and shadow camera to zero, as well as border colours of shadow texture / samplers.
• Use greater than for depth comparison logic.
• Update implementation for depth reconstruction in shader to match up reversed depth logic.
• Modify shadow bias to use addition instead of subtraction on the fragment depth.
• Add a documentation regarding reversed depth buffer as it is not trivial for developers.

General fixes and improvement

• Avoid updating the light direction for directional light (and hence re-render the shadow map) when the sun is invisible (and hence no light).
• Add a few flags to STPLightShadow to allow user to mask out shadow map update.
• STPSun now calculates the initial sun direction at initialisation.
• Clarify some behaviours in STPDirectionalLight.
• Refactor deferred lighting shader, view direction is computed outside the loop to avoid repetitive calculation.
• Make several colour spectrums to use SRGB format.
• Improve the water opacity function to make the edge connecting the terrain smoother.
• Refactor STPFile, STPSmartDeviceMemory and STPHashCombine to use namespace instead of class.
• Improve GL context robustness.
• Enable no error context on release build.
• Disable depth and stencil buffer on default framebuffer.
• Inline definition for STPNullablePrimitive; declare all functions as noexcept.
• Refactor GL objects management class in STPRealism+.
• Do not quote a variable with ${} when using it as an if condition for CMake.
• Improve documentations for sqlite build script.

- C++
Published by stephen-hqxu almost 4 years ago

superterrainplus - Add Aurora Night Effect

Release 0.14.3

STPAurora

To improve realism of night time, we also add a procedurally generated and rendered aurora and enable its rendering in SuperDemo+. Aurora renderer can be found in STPAurora.

General fixes and improvement

• Disambiguate conversion of some types in STPTerrainParaLoader.
• Add a new function to clear texture image for STPTexture.
• Formalise and clarify copyright notices for several 3rd-party licenses.

STPScenePipeline

• Improve robustness of light capacity checking when add a new light.
• For performance consideration, the first environment object is not forced to be drawn even when visibility is zero.
  • Instead, the environment area is cleared to black before drawing, or user-defiend clear colour if no environment object is available.
  • Post process buffer colour attachment is no longer cleared every frame because this already guarantees to draw over the whole screen every frame, even when there is no environment object.
  • Aerial perspective now works even when there is no environment object.
  • Environment clear colour is set to black when there is at least one environment object presented in the scene pipeline, or user-defined value if none.
• Refactor some function calls.
• Fix a bug that the ambient occlusion output framebuffer does not have stencil attachment causing computation to be done to the whole screen.
• Remove inline definitions for add() function in STPScenePipeline.
  • Now, if you are going to add a shadow opaque object, you need to call the add() function with the same opaque object twice, each time it will be cast to non-shadow opaque object and shadow opaque object implicitly.

STPSun

• Add a new setting to allow specifying the start of the year.
• Change the timing specification from the discrete tick to continuous second.
• Simplify sun orbiting calculation.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Introduce Night

Release 0.14.0

STPStarfield

The highlight in this release is this procedurally rendered and animated starfield. Like most others renderers, it comes with the main renderer class STPStarfield, a setting class STPStarfieldSetting, and finally a shader STPStarfield.frag.

To allow simultaneously rendering multiple environments, some improvements have been done to STPScenePipeline:

• Add an extinction environment rendering cache for holding temporary environment colours before blended with other scene objects.
• Add a visibility switch for STPEnvironmentObject.
• All environment colours are now summed together instead of overwriting.
• Replace small range-based for-loop with std::for_each for clarity.
• Add ambient light at night.

General fixes and improvement

• Fix a mistake in STPGaussianFilter for forgetting to assign uniform for image depth sampler.
• Remove use of element buffer in STPScreen and use triangle fan instead.
• Avoid explicit call to bindless texture destructor during random texture generation in STPHeightfieldTerrain which causes the destructor to be called twice at move construction.
• Add STPSimpleScreenBindlessFrameBuffer in STPScreen to allow using screen colour texture attachment on a framebuffer as a bindless target.
• Refactor STPBidirectionalScattering with the newly introduced STPSimpleScreenBindlessFrameBuffer.
• Fix a unspecified argument evaluation order in erosion kernel in STPHeightfieldKernel.
• Ambient occlusion blending now only enables at output to avoid doing the useless 1.0 x anything = anything stuff.
• Don't clear G-Buffer every frame because each pixel is read only by exactly one of the responsible shading invocation.
• Fix some incorrect code style.
• Improve STPLightSpectrum to allow supporting different channel types.

Refactor environment object rendering

• Add STPSkybox; similar to STPScreen, it is used by environment renderers that require an axis-aligned origin-centred skybox.
• Refactor STPSun to make it benefits from the new skybox renderer.
• Add projection view rotation matrix in STPCameraInformation.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - New Blur Filter

Release 0.13.14

Bilateral filter (#44)

On top of the current implementation of STPGaussianFilter, a depth-aware bilateral filter is added for more robust blurring for ambient occulusion noise. For now the blur filter supports two different variants:

• Naive Gaussian filter
• Bilateral filter

Also refactor Gaussian kernel generation a little bit.

STPWorldPipeline

• Remove copy local chunk function because it is useless.
• Remove the local chunk complete flag, and check for memory completeness with the old method std::all_of for robustness.
• Fix a bug where there is a chance splatmap of the centre chunk is not updated (because the we had only checked the neighbours).
• When a chunk is reloaded, all its neighbours will also have the splatmap recomputed; previously only the reloading chunk will have it recomputed.
• Wait for stream to finish at the end of loader thread to avoid stalling the rendering thread.
• Add STPConcurrentStreamManager in the world pipeline implementation.
• CUDA async function calls are now sent to multiple streams to maximise the ability of parallelism.

Free-slip erosion

• Separate STPErosionBrushGenerator from STPRainDropSetting.
• Add STPErosionBrush to store information about generated erosion brush.
• Make STPRainDropSetting as a POD-type.
• Remove extra declarations for default functions in STPRainDropSetting, STPHeightfieldSetting and STPConfiguration.
• Refactor rename member variables for consistent style in STPRainDrop.

General fixes and improvement

• Fix a grammar mistake in bug report template.
• Improve STPSingleHistogramFilter with pointer to implementation.
• Add STPSmartEvent as a smartly managed CUDA event object.
• Add STPObjectPool which reuses objects to avoid expensive recreation.
  • Refactor STPHeightfieldGenerator, STPBiomefieldGenerator, STPBiomeFactory and STPSingleHistogramFilter with the new object pool.
• Add test for STPSmartEvent, STPObjectPool and STPSmartDeviceMemory::STPPitchedDeviceMemory.
• Replace an outdated information in STPTestRTC.
• Correct some code style.
• For all use of std::optional, replace unnecessary value() with dereferencing operator to avoid runtime check.
• Fix some typos.
• Remove an unused variable from STPDeferredShading.frag.
• Add a 3rd-party license for water wave shader.
• Add a credit for normal map blending.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Resolve Issues

Release 0.13.11

World pipeline double buffering (#37)

• Implement a double buffering mechanism for STPWorldPipeline. Generation is done in the back buffer while the renderer can still read from the front buffer. Therefore sync before rendering is no longer necessary.
• Fix the splatmap border seams by delaying splatmap generation until all chunks are loaded.
• Add a memory manager for STPWorldPipeline.
• Revamp the specification of STPWorldPipeline. See the documentation in source code for more details.

Configurable specular power (#38)

Now the scene pipeline allows user to specify specularity for Phong shading model. As Phong does not model roughness very well, our implementation simply linear interpolates between a user-specified range of shininess based on the roughness read from G-Buffer.

To reserve spaces for supporting more shading model in the future, the scene pipeline now allows specifying shading model at initialisation.

Automatic resource download (#42)

• Simplify the process of downloading additional resources not included in the commit, and provide a better way of tracking resource changes.
• Update README build instruction to reflect this.
• Remove unused functions from GLAD and only retain ARB and NV extensions.
• Bump stb_image.h to https://github.com/nothings/stb/commit/5ba0baaa269b3fd681828e0e3b3ac0f1472eaf40.

General fixes and improvement

• Add two issue templates. (#46)
• Add version checking for SQLite library.
• Remove some unnecessary ${} for variables in if condition in CMake.
• Add a default constructor for STPBindlessTexture and STPBindlessBuffer.
• Replace use of std::optional<STPBindlessTexture> with STPBindlessTexture as now it can be constructed by default.
  • The same applies to use of STPBindlessBuffer.
• Make bindless texture and bindless buffer object default constructible.
• Make STPSceneLight and STPLightShadow default move constructible.
• STPDirectionalLight now take the shadow instance as an optional rather than a unique pointer, to avoid dynamic memory allocation.
• Improve STPCascadedShadowMap for better handling of move construction.
  • STPCascadedShadowMap can now be moved, and it will properly deregister the old instance from camera status callback and register the new one.
• Remove unnecessary buffer unmap call in destructor since GL does it automatically upon deletion of object.
• Add a clang-format document.
• Break some super super long codes into two lines.
• Change variable BaseVertex in STPIndirectCommand::STPDrawElement from uint to int to match the GL specification.
• Avoid repetitive evaluation of end iteration in STPWorldPipeline.
• Remove unused typedef STPLightIdentifier from STPScenePipeline.
• Add STPPitchedDeviceMemory as another smart device memory object.
• Convert encoding for README to UTF-8.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Implement Parser and Fix

Release 0.13.7

Parser library

• Migrate STPTextureDefinitionLanguage to build target STPAlgorithm+Host.
• Modify the specification of the TDL parser, it now never owns the raw TDL string object, and instead initialised with std::string_view.
• Implement INI utility for parsing MS INI document. It currently supports simple INI read and write.
• Implement test for the new INI utility.
• Completely abandoned library SIMPLE.

Compiler warning fix

• Remove redundant use of ; at the end of a namespace block.
• Fix a few signed-unsigned integer comparison.
• Declare explicit instantiation of template class with nested namespace specifier.
• Reorganise order of initialisation in constructors to ensure it is the same as the declaration order in class.
• Use attribute to suppress certain compiler warnings, such as switch case fall through and unused variable.
• Fix invalid scoping and nested typename specification in STPTestFreeSlip which causes some conpilers to complain.

General fixes and improvement

• Fix a few spelling mistakes in the source code documentation.
• Bump Catch2 version to v3.0.1. Make certain updates to STPConsoleReporter to address the API changes.

As the API of the test library becomes stable, it should work with any v3.0 version onwards in the future without the need to update the source code every time.

• Resolve a double unresidence of texture handle in STPBidirectionalScattering which causes GL to throw error.
• Add checking during creation of STPBindlessTexture and STPBindlessBuffer to prevent multiple instances of the same texture handle / buffer address are being resident, which implies that it is prohibited to create multiple instances such objects, i.e., uniqueness gurantee.
• Make declaration of Buffer variable in STPTestHistogram.cpp non-static as the destructor is called after CUDA context is destroyed.
  • Remove a redundant test case.
• STPTestRTC.cpp can now detect GPU compute capability automatically in runtime.
• Remove redundant getter in STPRainDropSetting.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - General Quality Improvement

Release 0.13.5

Due to recent change to the build script, you may need to clean and recompile the whole project.

Quality update

• Improve project source directory structure for SuperTerrain+ and SuperRealism+ to make them more consistent with the structure of their headers.
• Change the normalmap specification used by the terrain renderer from DX (negative y) to GL (positive y).
• Resolve a potential link-time error by inlining definition for STPSmartDeviceMemory instead of including the definition file separately in the source. (#40)
  • Error handling for detecting if stream is assigned is redundant thus removed, since the custom constructor always assigns a stream; if memory moved, the destructor is not called by std::unique_ptr.
• Change variable useOrtho in STPCameraInformation buffer from bool to unsigned int to match up GL specification.
• Fix an error for stencil buffer in the scene rendering pipeline where the first bit of the stencil is not used.
• Clean up some unused include.
• Rename STPSimplexNoiseSetting.cpp to STPAlgorithmEnvironment.cpp for consistency.
• Avoid dynamic memory allocation when converting std::string_view to a numeric value in STPTextureDefinitionLanguage.
• STPCommonCompiler can now detect CUDA architecture automatically. Remove redundant compiler option.

Build script improvement

• Refactor SetupEnvironment.cmake a little bit. Remove redundant environment setup such as assigning CMAKE_CUDA_RUNTIME_LIBRARY.
• Add new option STP_DEVELOPMENT_BUILD. Previous option STP_BUILD_TEST and STP_CUDA_VERBOSE_PTX and a new option STP_ENABLE_WARNING depend on this setting.
  • Now compiler warning can be enabled on any configuration; previously it is only enabled on debug build.
• CUDA runtime library is default to use shared library.
• Make build target GLAD as a sub-project.

SIMD algebra

• For performance consideration all definitions are exposed in the header and inlined.
• Declare both classes to be copiable explicitly.
• All camera classes in the renderer now uses STPMatrix4x4d directly.
• Add function to cast to glm::vec4 and glm::mat4 directly from STPVector4d and STPMatrix4x4d.

Namespace refactor

• Completely remove namespace STPCompute. This was introduced very early on in the project to separate host and device code. It seems more than verbose at this stage thus it is removed.
• All previous object in the core build target under STPCompute are moved to namespace SuperTerrainPlus.
• Introduce new namespace for SuperAlgorithm+ target, STPAlgorithm.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Improve Vector Maths

Release 0.13.2

Linear algebra library

This release introduce an initial implementations of hand-optimised SIMD linear algebra libraries, STPVector4d and STPMatrix4x4d. Currently it supports:

• Matrix transpose and inverse
• Matrix-matrix and matrix-vector multiplication
• Vector dot product
• Vector broadcast
• Vector addition and division
• Represent an equivalent 3-by-3 matrix

Tests are implemented and the linear algebra library passes all 103 assertions.

The new library is integrated into STPCascadedShadowMap and STPScenePipeline::STPCameraInformation.

General fixes and improvement

• Fulfilling license requirements for all redistributed external projects.
• Improve documentation on build instructions.
• Improve visuals for the project README document.
• Fix a bug in the SuperDemo+ build script which did not copy STPBiomeProperty.hpp to the binary directory to be used by NVRTC.
• Improve build script so it can group headers when opening from an IDE, i.e., MSVC.
• Attempt to group some source files to make it more developer-friendly.

Biome generator in STPBiome

• For simplification, remove the old STPBiome, rename STPBiomeSettings to STPBiome.
• Refactor all biome variable names to avoid using all capital letters (which is supposed to be used by macros).
• Replace use of std::vector with std::array in STPBaseBiomeLayer.
• Change STPBiomeRegistry from class to namespace.
• Make all declarations of functions and variables in STPBiomeRegistry as inline to enforce auto-initialised external linkage.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Introduce Water

Release 0.13.0

Please refer to pull request #40 for more development discussions.

Water system

The major highlight in this release is the simple water rendering system with procedural animation and basic reflection / refraction. STPWater is added as a new rendering component for rendering water geometry. STPBidirectionalScattering is also added as a new rendering component for rendering water visual effects using bidirectional scattering distribution function. The current implementation is far from being photorealistic, though.

We may decide to use global illumination in the future for improving rendering quality of water, such as adding caustics and underwater effect, as screen-space reflection produces underwhelming quality.

Water shader shares a lot in common with the terrain shader so all vertex-related shaders co-exist with the terrain one.

• Add STPWaterSetting to configure the behaviour of water renderer.
• Add STPMaterialLibrary to configure material properties for water geometry. It assumes that the water has uniform material to reduce memory cost.

Free-slip logic

We have also improved the very old system introduced in #15 and bring significant performance improvement to the particle-based hydraulic erosion. Due to the fact that the old index lookup table degrades performance dramatically, we ended up removing this entirely from the engine.

To ensure data safety, modification to the heightmap is made to be atomic during erosion. For performance consideration this is not enabled in debug mode as atomics perform poorly without turning on optimisation.

General fixes and improvement

• STPAlphaCulling now supports using double culling expressions.
• Add a new G-Buffer to STPScenePipeline for storing material ID.
• Improve STPPlaneGeometry to fully utilise the power of index buffer and remove all repeated vertices.
• Simplify terrain shader. Remove redundant pass-through of normal vector. Remove useless geometry shader during regular rendering. Remove unused stuff from the shader.
• Improve terrain tessellation control shader to fully exploit the power of invocation parallelism.
• Separate STPTessellationSetting from STPMeshSetting.
• Remove texture type Specular.
• Fix some bugs in the HBAO implementation which causes the occlusion to be weaken.
• Fix an error related to use of char* on string literal in a number of places like STPWorldManager in the demo program and STPDebugCallback in the rendering engine.
• Make argument std::string to const std::string& for a number of functions in STPRuntimeCompilable.
• Make include directory template file STPAlgorithmDeviceInfo.h.in to use std::string_view instead of const char[].
• STPCommonCompiler in the demo program now generates include directory in compile-time.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Water Shading

Release 0.13.0-Preview.3

BSDF

In this release we implemented a BSDF shader dedicated for rendering reflection and refraction of transparent object such as water. We have also added a material system to make rendering special effect flexibly and be easier to add supports for more objects in the future.

Currently BSDF is implemented using screen-space reflection/refraction therefore certain limitations apply. We will try to improve the visual quality and reduce artefacts brought by this approach, as well as introducing better algorithms, in the future.

We also update and refactor STPScenePipeline to support the newly added transparent objects and post effect.

• Add a new fragment shader that implements BSDF for water reflection and refraction using screen-space technique.
• Add STPBidirectionalScattering as a new rendering component for rendering reflective and refractive water (for now).
• Add STPBidirectionalScatteringSetting.
• Add STPMaterialLibrary which allows specifying material properties for special rendering components like water.
• STPWater is now allowed to specify material property with a material ID.
• Introduce a new G-Buffer, material buffer. This buffer is a unsigned int format texture which stores material ID. This G-Buffer is enabled and has memory allocated only when material library is attached to the scene pipeline.

General fixes and improvement

• Improve STPAlphaCulling and now it supports use of dual boolean expressions.
• Replace unique light space size record in STPScenePipeline from std::vector to std::unordered_set.
• Improve ambient occlusion shader to reuse the casted projection matrix in every iteration to avoid repetitive casting.
• Change noise texture in STPHeightfieldTerrain to a bindless sampler.
• Improve the HBAO algorithm a bit to make the occlusion more noticeable, i.e., stronger.
• Refactor view to NDC conversion function from STPAmbientOcclusion.glsl into STPCameraInformation.glsl.
• Add a new function for camera to linearise depth.
• STPScreen now passes shared vertex buffer around via a std::weak_ptr rather than a pointer to a std::shared_ptr.
• STPTexture and STPSampler now allows to set integer and unsigned integer format of border colour.
• Remove setData(vec3) function in STPLightSpectrum because it is not useless.
• STPSun now copies STPSunSetting under its instance instead of taking a reference.

### STPWater

• Fix a mistake in water tessellation shader which uses a variable before it is declared.
• Simplify the tessellation shader slightly.
• Fix a few silly mistakes in the shader which causes error during compilation.
• Fix some errors in the wave function.
• Improve water plane culling algorithm to reduce the chance of false positive.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Animated Water Geometry

Release 0.13.0-Preview.2

STPWater

In this release we have added a rendering target for water. The water plane is placed as defined by user using biome map and a lookup table that returns water height given biome ID.

• Add a new type of scene object STPTransparentObject for objects with reflective and refractive materials.
• Implement water tessellation shader on top of the current heightfield terrain shader.
  • Water plane culling allows water to be placed only when necessary.
  • Water plane is also procedurally animated to give a wavy visual effect.
• Implement water fragment shader for computing water surface normal.
• Add STPWaterSetting.

General fixes and improvement

• Refactor render() function for base scene objects into a new base class STPRenderable.
• STPTessellationSetting is separated out from STPMeshSetting.
• Refactor shader built-in struct into a separable common source header STPSeparableShaderPredefine.glsl.
• Make splatmapGenerator function in STPWorldPipeline returns a const reference.
• STPScenePipeline no longer manages memory for each renderinig component. User is responsible for retaining the lifetime of components added to the scene.
• Remove Specular texture type because it is useless in PBR model.
• initScreenShader() function in STPScreen now takes a STPShaderManager rather than STPShaderManager::STPShaderSource as input for logical consistency.
• STPProgramManager now detaches all shaders upon successful program linking.

STPHeightfieldTerrain

• Separate terrain vertex shader into a standalone program so it can be shared with water shader.
• Remove biomemap binding because it is not used.
• Remove pass-through of normal vector in the shader because it is a constant vector facing upward for every vertex.
• Remove geometry shader stage for regular terrain rendering because it is useless and GS is slow. Now geometry shader is only used for shadow pass.
  • Camera space conversion from world to clip is now performed in the tessellation evaluation shader.
• Improve level-of-detail computation in tessellation control shader to exploit invocation parallelism.

STPPlaneGeometry

• Improve plane generation algorithm such that no vertex overlaps.
• Remove Y component from the position vector because it is a constant.
• With both optimisations, the memory usage has reduced to only 1/5 of the original.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Improvement to Free-Slip Erosion

Release 0.13.0-Preview.1

Free-slip logic

Previously it has been believed that 2D memory copy function is slower than linear copy, i.e., cudaMemcpy2D() is supposed to be slower than cudaMemcpy(). However after spending some time on benchmarking they in fact perform exactly the same, and 2D copy is even marginally faster than linear copy; perhaps that's due to some internal optimisation done by CUDA driver.

Hence, it is sensible to pack the free-slip buffer using 2D copy than the old "linear copy + global-local index table" combo. Therefore:

• Deprecate and remove completely the index table logic from all generation pipeline stages which use free-slip logic, such as single histogram filter and hydraulic erosion.
• Modify STPFreeSlipTextureBuffer to use 2D copy for merging free-slip texture.
• Remove STPFreeSlipGenerator and global-local index table generation kernel in STPHeightfieldKernel.
• Remove STPFreeSlipManager.
• Remove global-local index table entry from STPFreeSlipInformation (renamed from STPFreeSlipData).
• Merge STPFreeSlipLocation into STPFreeSlipTextureBuffer.
• Put all the rests of the free-slip classes to Chunk directory and remove FreeSlip directory.

This has improved the performance in all free-slip-related heightfield generators, especially for erosion, by at most 10 times. It has shown that the index lookup table approach is slow.

General fixes and improvement

• For hydraulic erosion, enable atomic instruction on release build considering for both data safety and performance. On debug build it is too slow and make the program non-debuggable hence we trade data safety with speed.
• Deprecate and remove multi-channel texture support for STPFreeSlipTextureBuffer because it is useless.
• Fix an error on some compilers during compilation of STPSingleHistogramFilter which report std::max() has not found overload for size_t and unsigned long long.
• Merge STPSeedMixer with STPLayer.

- C++
Published by stephen-hqxu about 4 years ago

superterrainplus - Bug Fix and Optimisation

Release 0.12.13

General fixes and improvement

• Rearrange GLAD build target file structure so it matches the default file structure downloaded via GLAD web generator. You may need to clean and re-compile the entire build tree.
• Replace all uses of std::reduce with std::accumulate for the following class because it is not neccessary to use so when std::execution is not used, also some compilers have yet support this function:
  • STPSingleHistogramFilter
  • STPCascadedShadowMap
  • STPGaussianFilter

Chunk coordinate

Re-specification of chunk coordinate system. Chunks are now located using an integer vector of chunk coordinate in world position, the chunk coordinate will be a multiple of chunk size.

The benefit of using integer values effectively avoids floating point errors when looking up chunks in the hash table using chunk coordinate as key.

Terrain erosion

• Make get erosion brush function in STPRainDropSetting returns pointer to const value, was pointer to value.
• Improve erosion implementation in general.
• Potentially resolve #39 and fix a race condition in the random number generator by assigning a different RNG to each generation invocation and pool the RNG.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Patch and Optimisation

Release 0.12.11

General fixes and improvement

• Update SIMPLE to version 3.0.
• Fix an incorrect use of comma operator during initialisation of const SIStorage& in STPStart which creates unnecessary data copies.
• Fix an incorrect use of inline in the following files which causes link-time undefined reference when compile under Release build:
  • STPSmartDeviceMemory
  • STPScreen
  • STPScenePipeline
• For STPShaderManager, improve shader macro defining with std::string_view. Add support for detecting #define with leading and in-between white space.
• All use of std::queue are changed to use std::deque as container instead of std::list.
• Deprecate and remove STPChunkStorage because it is just a useless thin wrapper over std::unordered_map.
• Change the declaration order for STPThreadPool in STPWorldPipeline and STPSingleHistogramFilter to ensure the correct destruction order and avoid illegal memory access.
• For STPOpenGL.h, STPuint64 is now expressed as uint64_t to make it compatible with system which uses different specification to define GLuint64.
• Fix an error on some compilers where:
  • sqrt() is not found in STPErosionBrushGenerator by including correspond math header.
  • memset is not found in STPLayerCache by including cstring.
  • std::shuffle is not found in STPPermutationGenerator by including algorithm.
  • std::list is not found in STPShaderManager and STPThreadPool.
• Remove useless typename declaration which causes errors for some compilers, from:
  • STPSmartDeviceMemory.tpp
  • STPMemoryPool.h
  • STPFreeSlipTextureBuffer.h and STPFreeSlipTextureBuffer.cpp
  • STPSingleHistogramFilter.cpp
  • STPSceneObject.h
• Remove extra namespace qualification for some functions in STPScenePipeline.
• Launch buffer size in STPBiomefieldGenerator and STPSplatmapGenerator now uses sizeof to determine rather than using hardcoded values because pointer size on different platform might not be always 8 byte.

STPWorldPipeline

• Fix an undefined behaviour where a temporay returned vector is used in a range-based for loop.
• Fix a race condition where chunk might be used while it is being occupied.

To avoid running into the nasty race condition again, we introduce STPChunk::STPMapVisitor which has two operation modes: shared and unique. It works like a shared/unique mutex in the standard library, the only difference is rather than stalling threads when mutex condition is unmet, exception will be generated. This attempts to prevent developers from making this mistake in the future again.

STPTextureDefinitionLanguage

• Fix a bug where a std::string is created from calling the data() function in std::string_view which basically copies the entire chunk of memory for every number parsed and results in significant memory overhead.
• Simplify readNumber() and readString() functions. Remove repeated string read.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Ambient Occlusion

Release 0.12.8

We introduced SSAO in release 0.11.8 as a fundamental technique for ambient occlusion. In this release a more modern ambient occlusion technique is brought to SuperTerrain+.

Horizon-based ambient occlusion

• Implement HBAO based on the current implementation of ambinet occlusion.
• Enable switching between different AO algorithms in STPAmbientOcclusion.
• Rename STPScreenSpaceAmbientOcclusion.glsl to STPAmbienOcclusion.glsl for generality.
• Refactor and optimise view space geometry calculation.

General fixes and improvement

• Explicit initialisation to sqlite database during SuperTerrain+ engine initialisation.
• Fix a undefined behaviour when naming the texture database STPTextureDatabase where operator+ is used between a const char* and unsigned int. Surround the integer with to_string to fix the issue.
• Add STPProjectionCategory in STPCamera to identify the type of projection of a camera instance.
• Use offsetof operator to identify variable offset instead of using hardcoded offset in STPScenePipeline::STPCameraInformationMemory.
• Add support for GL extension GLNVrepresentativefragmenttest.
• Refactor heightfield triple texture scale blending system to cut down the scale of branching and divergence.
• Deprecate and remove support for logarithmic depth buffer because it is not friendly to depth reconstruction. We will consider using reversed depth buffer in the future release.
• Add ambient occlusion texture for the demo application.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Simplify Lighting System

Release 0.12.6

The major focus of this release is to replace the previously complicated uniform light list with simple pointers to buffer. A pointer can be used instead of having all different types of arrays and locate light and shadow information using indices.

Shadow

STPLightShadow receives some improvement. Now STPScenePipeline no longer manges shadow memory and instead the light shadow class holds depth texture and framebuffer, and more importantly, a buffer object where a buffer address can be extracted and send to the shader.

The pointer approach significantly reduces the amount of data shared between light shadow instance and the scene pipeline because most shadow data can be sent directly to GPU via the address.

Light

STPSceneLight receives a major overhaul. Different types of lights are broken down as children of STPSceneLight. We removed STPEnvironmentLight and replaced it with two new types of lights, STPAmbientLight and STPDirectionalLight. In addition, STPSun is no longer a child of any light source but STPEnvironmentObject, which is a new type of scene opaque object.

Likewise, light data are now shared with the shader using buffer address. This significantly reduces the complexity of our application. Now all light settings are associated with each light and it is no longer necessary to set light properties such as light strength and light direction via the scene pipeline.

General fixes and improvement

• Fix various spelling errors in the source code documentation.
• Each shadow-casting light now initialises a shadow by moving a unique_ptr enclosed shadow instance rather than taking many constructor arguments.
• Add a new function for STPBuffer which allows copying sub-data from another STPBuffer instance.
• STPLightSpectrum is now a concrete class, all functionalities from all previous children are merged to the base class.
• Merge STPSun::STPSunSpectrum with STPSun.
• Replace texture lookup function texture() with textureLod(..., 0) for texture that are not intended to have level-of-details in shaders.

Renderer logging system

• Remove log storage instances from all rendering components.
• Replace STPLogStorage with STPLogHandle which allows user to provide a pointer to implementation which defines how a log string should be handled.
• STPShaderManager, STPProgramManager and STPPipelineManager no longer returns a string of log. Instead all logs will be directed to the pointer provided by STPLogHandle::ActiveLogHandler.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Optimisation to Renderer

Release 0.12.3

STPHeightfieldTerrain

• Addresses issue #35 by computing terrain mesh using double precision before casting to single precision and storing results.
• Introduce STPPlaneGeometry as a plane generator that allows speficiation of number of subdivision on a plane and optimised for fast and high-precision generation of high quality mesh.
• Refactor terrain shader to allow making use of the new plane generator and avoid precision problem.
• Simplify terrain texture splat region lookup using shader pointer. Remove registry dictionary from the terrain fragment shader, now region registry holds uses pointers to splat texture and null pointer to identify regions with no texture data.

General fixes and improvement

• Introduce STPBindlessBuffer as a thin wrapper over buffer address. This requires extension GLNVshaderbufferload, however.
• Remove extension check in STPBindlessTexture because this is mandatory anyway.
• Refactor academic report reference to a single file.
• Add a new GLSL common header STPNullPointer.glsl to help better interacting with use of pointers in shading language.
• STPLightSpectrum now manages its own texture handle instead of keeping an array of spectrum handle in STPScenePipeline.
• Add documentation for texture definition language.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Multi-Scale Texturing

Release 0.12.0

Improve texturing system

STPTextureDatabase

Introduce a new texture group, texture view group, which specifies the UV scale of a particular texture. Remember a texture is a collection of many texture maps, such that the view group settings will be applied to all texture maps under texture as well.

The new system allows specification of 3 different UV scales, and the scales will be chosen by the renderer based on user-specified distance to the texel.

• Remove multi-add functions such as addTextures() because they seem to be useless after the introduction of texture definition language.
• addTexture() now allows assigning, optionally, a user-specified name to the texture.
• Restructure texture group logic, rename texture group to map group to disambiguate.
• Refactor database record manupulation functions a little bit.

STPTextureDefinitionLanguage

The specification of TDL has been updated to add support for the new texture view group system. The view group is specified using a new directive, #group and is identified by group type view.

For more information about the new syntax, check out TDL specification in the project documentation.

General fixes and improvement

• Resolve issue #36.
• Simplify STPSimplexNoise calculation, remove redundant arithmetic operations.
• All camera-related calculations (viewer camera, light-space camera etc.) are now all done with double on host side before converting to float and sending to device.

STPHeightfieldTerrain

• Fix an incorrect texture region dictionary boundary check logic in the fragment shader which could result in out-of-bound array access.
• Noise scale used for stratified sampling now uses unsigned int rather than float, because a real number will make the uv scale non-aligned with tile boundaries and the whole texture tiles move when switching centre chunk.
• Refactor texture smoothing using histogram bins instead of fetching texture in every iteration during splatmap smoothing to greatly reduce memory bandwidth overhead.
• Remove UVScaleFactor from STPMeshSetting, add STPTextureScaleDistanceSetting to control texture scale to be made active based on texel distance.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Aerial Perspective

Release 0.11.10

Horizon extinction effect

To avoid having the edge of the terrain getting cut out with a very sharp border as it approaches camera far clipping plane, we proposed a fix by blending the atmosphere with the extinction area on the terrain.

This effect is achieved by using alpha blending on the extinction area with the sky. The alpha channel on the geometry is determined by the distance from fragment to camera.

Instead of perform a full-screen ray marching to do the sky blending, STPAlphaCulling is implemented to modify stencil buffer such that only geometries in the extinction region are included in the environment rendering.

The extinction parameter can be adjusted in STPScenePipeline.

General fixes and improvement

• STPPostProcess now supports alpha channel framebuffer.
• Fixed the alpha channel of the default clear color in scene pipeline to 1.0, was 0.0 previously.
• The program now uses real timer to calculate, for example sun position, rather than using frametime. Such that the timestep is independent of FPS.
• Declare all static-declared lambda expression variables as constexpr as well, in the entire project .
• Make the destructor of STPLocalRNG in STPLayer as default in the header.
• Adjusted chunk scale to make sure the camera cannot see the border of rendered chunks.
• Making slight adjustment to the shadow bias to reduce shadow acne flickering.
• Enable robust context when the demo program is compiled in debug mode.
• Improve permutation table generation in STPPermutationGenerator. Replace raw array with std::array and initial table is generated in compile time instead of hand-typed. Rename inappropriate variable name.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Ambient Occlusion

Release 0.11.8

Screen-space ambient occlusion

The focus in this release is the newly added STPAmbientOcclusion for performing SSAO which allows us to see the details on the terrain even when it is in shadow. The shader implementation can be found in STPScreenSpaceAmbientOcclusion.glsl. As always, the effect is configurable and the settings are located in STPOcclusionKernelSetting.

Also we introduced STPGaussianFilter which is a separable Gaussian blur kernel. It features a configurable Gaussian function. This class is currently used by the ambient occlusion calculator for blurring the final output.

Improve off-screen rendering

We refactor STPScreen into two sub-classes, STPScreenVertexShader and STPScreenVertexBuffer. This allows sharing vertex shader during initialisation among all off-screen renderers and vertex buffer during rendering to safe memory and reduce number of binding operations.

Additionally each STPScreen base instance contains a STPProgramManager with some helper functions to simplify the process for setting up off-screen rendering. Please keep in mind that drawScreen() function no longer binds vertex buffer automatically, binding and program usage needs to be done by the user externally.

Also add STPSimpleScreenFrameBuffer under STPScreen as a refactor class for single color attachment off-screen rendering.

General fixes and improvement

• Add a texture barrier call in the main rendering loop before deferred light pass to ensure data safety since there might still be a potential feedback loop problem.
• Add extra static_assert for camera buffer struct to ensure the alignment is consistent with OpenGL std430 alignment rule.
• For loading INI into the program, all setting names are made inlined to make it more readable. Add INI loader function for ambinent occlusion settings.
• Simplify spectrum generation shader. Remove explicit uniform for spectrum size since it can be queried by calling imageSize.
• Change memory barrier function post spectrum generation from image access barrier to texture fetch barrier.
• Fix a bug in STPSunSpectrum for forgetting to assign a sampler binding to the shader. This did not lead to error as OpenGL assigns sampler binding 0 by default.
• STPPostProcess now holds its own memory rather than taking it from the scene pipeline.
• Add new function in STPFrameBuffer to allow detaching a target.
• All color buffers are explicitly cleared instead of relying on the fact that they are all overdrawn to avoid running into bugs in the future.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Smooth Shadow

Release 0.11.5

Shadow map filtering

• Implement the following shadow map filters:
  • Percentage-closer filtering
  • Variance shadow map
• Improve shadow biasing system. Add normal biasing and far plane bias scaling for directional shadow.
• Refactor shadow map filter selection in the scene pipeline. Different shadow map filters take different initialisation parameters.
• Add a new argument to addDepthConfiguration for opaque object, the new argument takes a pointer to depth fragment shader. This function will be called by the scene pipeline automatically.
• Introduce cross-cascade blending which resolves an ugly line when applying shadow filters at the cascade border. To resolve light bleeding introduced by the blending system, cascade plane can be biased when initialising STPCascadedShadowMap, this bias value is called CascadeBandRadius.
• Fine-tune some shadow settings in the demo program to make it looks better.

General fixes and improvement

• Remove virtual declaration on the destructor on STPCascadedShadowMap because this class is no longer intended to be a base class.
• Simplify cascaded shadow map layer lookup process.
• Uniform locations of all light properties are now cached with std::unordered_map using light identifier and light type instead of performing string lookup every time.
• STPProgramManager now allows sending uniform data using uniform location.
• Add more overloads for setLight() in STPScenePipeline.
• Terrain renderer now allows using two different tessellation level of detail controls for normal rendering and depth rendering.
• Add support for logarithmic depth buffer, realted settings can be found under STPCameraSetting.

The log depth buffer currently does not work with shadow map because shadow calculations are not in log scale.

• Improve error handling for scene pipeline initialisation. Now all values set for the shadow parameters are checked for validity.
• Refactor STPDeferredShading.glsl to make it more organised.
• Force the shadow map to be a square texture. The resolution of all shadow maps is now specified by a single unsigned integer, namely extent length.

Bug discovered

• Fix a bad sampling pattern when performing texture splatting smoothing during terrain rendering, which incorrectly assumes the range of coordinate causing most of the samples biased towards negative axis.
• Fix an issue where in the texture management system mipmap is used however only a single level of detail is allocated, causing no mipmap is actually being generated.
  • STPTextureDatabase now allows user to specify the number of mipmap level to allocate when creating a new texture group.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Dynamic Scene Pipeline

Release 0.11.2

Improve level of abstraction for STPRealism

This release makes STPScenePipeline dynamic, meaning all rendering components like opaque objects and lights, can be added to the scene pipeline after construction in runtime.

Due to the dynamic nature, all memory in the deferred shader needs to be declared prior to initialisation, and the number of objects added must not exceed this limit. This limit is set by the user. The limit will all be stored in STPSceneShaderCapacity.

• Include directory restructured. Renderer has been moved under Scene and renamed to Component. STPScenepipeline has been moved to the root.
• Shadow map texture is no longer shared because I don't find it necessary, each shadow-casting light is now given its own texture memory, such that the shadow map resolution can be adjusted on per-light basis. Each light is assigned with one framebuffer object.
• Add a new pure virtual function to STPLightShadow for getting shadow map resolution.
• Terrain renderer now allows adding shadow-casting light with different depth configuration dynamically, instead of passing STPShadowInformation in the constructor.
• Add lightDirection() virtual function in STPEnvironmentLight<false> so the scene pipeline can get the light direction automatically without asking inputs from the user.

General fixes and improvement

• Address #34 that the demo program incorrectly assumes glfwSwapBuffer() will do implicit synchronisation (in fact it will not), which causes data racing.
• Fix a potential feedback loop (therefore undefined behaviour) for a bound depth buffer while reading it in the shader. It is fixed by only binding stencil attachment to the post process framebuffer as we don't need depth buffer for off-screen rendering.
• Make all shader storage buffer block named to avoid variable collision.
• Add result of projection * view matrix and its inverse to camera information buffer to further reduce the amount of runtime computation on GPU.
• Add a new member in light space information shader storage block to locate a light space matrix for a particular light.
• Rename STPDepthRendererGroup to STPDepthRenderGroup in STPSceneObject.
• Remove currently unused explicit instantiation for STPRandomTextureGenerator in STPRealism.
• With the new concept of depth render group, make light space shader storage buffer as a dynamic array and STPShadowInformation is no longer required because no information is required to shared with any opaque object.
• Remove log storage and status flags in STPShaderManager, STPProgramManager and STPPipelineManager. Also remove redundant validity check in various rendering components because those managers throw exception before validity check when there is error.
• STPShaderManager now allows setting a mandatory source name which will be prepended to the compilation log (if any) for easier identification of error. All rendering components now set this source name to the filename of shader.
• The shadow-casting version of terrian renderer now has an internal queue for holding up logs from compilation of depth renderer.
• STPMemoryPool in the main engine now uses std::unordered_map instead of sorted std::vector to search for memory chunk with given size because benchmark shows hash table is still faster than binary search even when the entry size is very small.
• Optimise view frustum corners calculate in STPCascadedShadowMap. Use SIMD for shadow frustum calculations.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Introduce Deferred Shading

Release 0.11.0

We perform an overhaul to the master scene pipeline and all rendering components for deferred shading. Deferred shading makes designing an abstract engine much easier as light calculation can be concentrated into one shader instead of spreading around and distributing by shader include and macros. Also it is inevitable for more advanced lighting effect which will be added to the engine later such as SSAO, SSR and adding support for 1000+ lights.

Deferred shading

• Add STPGeometryBufferResolution as a G-buffer manager and lighting processor. It current captures the following geometry data with data format of:
  • Albedo (RGB8)
  • Normal (RGB16_SNORM)
  • Specular (R8)
  • Ambient Occlusion (R8)
  • Depth Stencil (DEPTH32F_STENCIL8)

Geometry position is reconstructed from depth buffer.

• Add a new common shader STPGeometryBufferWriter for writing geometry data into G-buffer from various rendering components.
• Rename STPDirectionalLight back to STPCascadedShadowMap to make it sounds more fancy😏 .
• Remove all lighting calculation, light-related uniforms and constructor parameters from STPHeightfieldTerrain. Remove light settings from STPMeshSetting.
• For easy initialisation, sun spectrum is now a member of STPSun, so the sun acts as both a light source and a rendering component.

Renderer abstraction

• Add STPSceneObject acting as a base abstract renderable objects. STPHeightfieldTerrain current inherits from it. This new base class also uses the shadow-casting logic and provide two different versions of objects.
• Add STPSceneLight basically for the same purpose as the scene object. There are many derivations of different types of light to be added in the future. Currently only STPEnvironmentLight is supported. STPSun now inherits from this derivation.
• Add STPLightShadow as a base class for all shadow implementations. STPCascadedShadowMap now inherits from this.
• Add STPScreen as a base off-screen renderer. The screen vertex data can be shared by various off-screen rendering targets such as STPPostProcess.
• Introduce the idea of depth renderer group that allows rendering opaque objects to depth texture with different light space size. The basic idea is by compiling GL shaders with all possible declared light space sizes.

STPScenePipeline

The rendering pipeline has received the most amount of attention. It's now the master rendering controller and memory repository for deferred rendering, and shares memory buffer among rendering components.

To initialise a scene pipeline, one needs to use STPScenePipelineInitialiser to first add shadow-casting light to get shadow information, then the rests of the object. To reduce efforts for user to manage rendering component, all rendering components are held by the scene pipeline and user will get a pointer to the objects added.

STPPostProcess no longer hold its own texture and framebuffer, it now reads texture from arguments when process() function is called. This helps concentrating memory in the scene pipeline for the sake of deferred shading.

General fixes and improvement

• Buffer resolution in STPPostProcess is checked for non-zero.
• For STPTexture, Target member variable is no longer declared as const to allow use of move assignment operator.
  • Regarding this modification, STPPostProcess now recreate screen buffer using move assignment rather than std::optional.
• Add a new function in STPFrameBuffer for specifying mutiple framebuffer draw buffers as an array.
• The texture splatting system no longer supports displacement and emissive mapping. Add support for roughness mapping.
• Remove unused shader variables such as position_clip passed between each shader in terrain rendering pipeline.
• STPLightSpectrum now allows specifying the number of color in the spectrum, as well as the channel format.
  • Now STPStaticLightSpectrum and STPArrayLightSpectrum all use monotonic color (GLTEXTURE1D) with GL_RGB8 format.
  • STPSunSpectrum still remains the specification unchanged. using bitonic color (GLTEXTURE1DARRAY) with GLRGBA16F format.
• Declare STPSetting with virtual destructor. Remember to recompile all CUDA codes in SuperTerrain+ module.
• Remove texture type Displacement and Emissive, add Roughness.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Improvement to Shadow System

Release 0.10.6

Better shadow system

• Shadow map filter can now be chosen in runtime. It current supports nearest and bilinear filter. More shadow smoothing techniques will be implemented in the future.
• Rename STPCascadedShadowMap to STPDirectionalLight to make it more general.
• Move shader storage buffer for shadow mapping from STPDirectionalLight to STPShadowPipeline which is a new class.
• STPSun is now also given choices for casting or not casting shadows.
• Add STPShadowInformation for sharing shadow map setting from STPShadowPipeline to various renderers.
• STPDirectionalLight now uses camera callback to determine if light space matrices are outdated and only recompute them (and flush the buffer) whenever necessary.

With the new system, the correct order for setting up a rendering pipeline is:

• Create light that casts shadows.
• Create STPShadowPipeline with an array of pointers to light that just created.
• Create all the rest of the scene including light that does not cast shadow and opaque object.
• Create STPScenePipeline with a pointer to STPShadowPipeline.

As now lights hold memory owned by the scene pipeline, make sure lights are not writting to the buffer after the dependent scene pipeline is destroyed.

General fixes and improvement

• Fix a potential undefined behaviour in terrain fragment shader for accessing array of bindless samplers in a non-uniform manner.
  • GL extension NV_gpu_shader5 is now required in order to run the program.
• Remove unnecessary extension directives from some shader codes.
• Move bindless shadow handle from individual shader to shader storage buffer to be shared among all renderers.
• STPCamera now allows registering as many listeners as user wants.
• STPScenePipeline now deregisters itself from camera callback at destruction.
• Fix an incorrect return value semantics in shadow map sampling shader. The return value of that function represents the light intensity instead of shadow intensity, or light intensity = 1.0 - shadow intensity.
• Rename rendering function name from operator() to some more concrete name like render().

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Real-time Shadow

Release 0.10.4

Terrain shadow

• Add STPOrthographicCamera as an ortho projection camera, along with STPOrthographicCameraSetting to store settings.
• Add another type of flag to determine which component will contribute to shadow map to traverse() function in STPScenePipeline.
• Add STPBindlessTexture for handling automatic unresident of bindless texture handle.

STPCascadedShadowMap

This is a shadow manager that implements CSM technique. It allows rendering shadow on a large scene using directional light with adaptive shadow quality based on distance from camera to the main scene.

STPSun now derives from this class, STPHeightfieldTerrain has provided users with two options, to cast or not cast shadow.

General fixes and improvement

• Change exception type thrown when validating STPSunSetting and STPAtmosphereSetting in STPSun to STPInvalidEnvironment.
• Refactor Lottes and Uncharted2 tone mapping calculation.
• Move Near and Far setting from STPPerspectiveCameraSetting to STPCameraSetting as these settings are shaded by both types of projection.
• Make all functions in STPImageParameter public.
• Add a new pure virtual function in STPCamera that allows overriding the near and far plane to calculate the projection matrix.
• Add compareFunction() and compareMode to STPImageParameter as well as implementations for all derived classes.
• Add drawBuffer() and readBuffer() for STPFrameBuffer.
• Fix incorrectly calculated tangent and bitangent for terrain unit plane that causes the normal Y direction to be flipped. Also change the unit plane winding order from CW to CCW for consistency reason.
• Fix a repetitive use of RNG in STPLayerCache test program that causes warning due to generation of the same RNG sequence to be generated during test session.
• Change the specification of all cameras. Now they all use callback functions to determine if the camera state has changed rather than having them returned from the camera matrix retrieval call.
• Instead of using an exponential function, terrain LoD shifting now uses linear. Remove ShiftFactor setting from STPMeshSetting.
• Buffer stream in STPWorldPipeline is now initialised as non blocking.
• supply() function in STPBiomeFactory now returns an instance rather than a pointer.
• Remove redundant storage of list iterator in STPSingleHistogramFilter, now the queue returns a moved memory block directly.

Shader compilation

• Add some wrappers function over shader source code in STPShaderManager that allows pre-processing the source string directly rather than messing up other functionalities with the shader class.
• Remove all unnecessary shader include paths from all renderers (STPSun etc.) because all shader source code use absolute path.
• Fix incorrect semantics in STPShaderManager, shader include path should be an relative directory rather than the full name of the include source.
• STPShaderManager now requires explicit initialisation for system shader include. For such reason, added STPRendererInitialiser dedicated for initialisations.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - HDR Rendering

Release 0.10.0

STPRealism

• Introduce STPFrameBuffer and STPRenderBuffer as smart managed OpenGL objects.
• Add textureStorageMultisample() method is STPTexture.
• Add a new managed object STPSampler.
• Refactor STPTexture and extract functions which set texture parameters to a new interface class STPImageParameter. STPSampler also derives from this interface class.

STPPostProcessing

An off-screen renderer that captures rendering buffer from other renderers and process the buffer before drawing it onto the screen.

• Allow attaching multisampled texture.
• The shader provides the following functionalities:
  • HDR tone mapping, specifically it supports the following tone mapping functions:
  • Gran Turismo
  • Lottes
  • Uncharted2
  • Gamma correction.

Also enable HDR tone mapping and gamma correction in the demo program.

General fixes and improvement

• Update Catch2 to v3.0.0-preview4.
• Fix a few typos in the README document.
• Fix a bug for which GL reports invalid image format when unbindImage() is called in STPTexture.
• Replace std::shared_mutex with std::mutex in STPSingleHistogramFilter. Also eliminate unused function arguments.
• Improve shader source macro definition, add a helper class to convert other data type to string automatically.
• For the demo program, albedo texture are now corrected to linear space, default framebuffer has multisampling disabled.
• Change traverse() in STPScenePipeline to a template function and use bit flag to determine components to be rendered in compile time to avoid having a lot of branches in runtime.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Dynamic Lighting

Release 0.9.8

Terrain light color

• Add 2 configurable light color in the terrain shader:
  • Indirect: ambient
  • Direct: diffuse
• Allow changing individual lighting component color.
• Introduce the idea of spectrum lighting that allows looking up light color from a pre-computed light spectrum.
• Wrap the atmosphere scattering algorithm such that it can be used by both the sun renderer and spectrum emulator (which is a compute shader).

STPLightSpectrum

A utility that allows storing terrain light color(s) in an array for looking up in runtime. It currently provides: - Static spectrum that defines a monotonic color. - Array spectrum that defines user-specified colors. - Sun spectrum that uses STPSun to calculate the light spectrum based on the sun elevation. - Enable sun spectrum lighting in the demo program.

General fixes and improvement

• Allow loading displacement map in the demo program. Refactor texture loading and texture type checking.
• Fix an incorrect return type in the terrain shader.
• Improve sun direction calculation, remove member variable LocalSolarTime and change Day from unsigned long long to double. Now both time in a day and a year can be calculated using one variable.
• Replace some size_t with unsigned long long because the underlying type of size_t is platform-dependent.
• Change the specification of day-night cycle offset. Instead of using angle it now uses sun elevation vector.
• Add a function in STPProgramManager to allow to query work group size for compute shader program.
• Add shading language include extension check in STPShaderManager if any include path is specified.
• STPTexture now requires specifying which GL function to be called explicitly for texture storage and sub-image function.
• finalise() in STPProgramManager now returns pointer to log.
• Fix a bug in STPProgramManager that messes up shader type with shader renference ID.
• Remove the following settings from STPAtmosphereSetting:
  • LowerElevation
  • UpperElevation
  • CycleElevationOffset
• Improve exception handling in STPLayerManager to avoid memory leak on exception.
• Improve STPLayerCache caching mechanism. Instead of using a callback function, it now splits caching operation into 3 steps, locate address, read and write.
• Set sample() function to private in STPLayer.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Terrain Shading

Release 0.9.5

Basic lighting

• Fix a problem in terrain normalmap generation where normals are not normalised. Also fix the facing of the normalmap so that it is in terrain world space rather than tangent space.
• Add lighting to the terrain, light direction is based on the sun.
• Attach texture normalmap to the project. The engine currently supports:
  • Colormap, a.k.a., albedomap
  • Normalmap
  • Specularmap
  • Ambient-occlusionmap
• Implement 5 normalmap blending algorithms and they can be chosen during heightfield renderer initialisation.
• Move light settings to INI file.
• Improve terrain shader to handle all cases when each type of texture is not enabled.

General fixes and improvement

• Improve level of abstraction for the scene pipeline and each rendering component. STPScenePipeline no longer changes component states, such as camera position, automatically before rendering. Internal buffers like shader storage buffers that are not exposed to the external will be managed automatically.
• Move renderer uniform update functions out of rendering loop. They are now updated explicitly using corresponding setters rather than doing it automatically during rendering loop. This helps reducing unnecessary GPU communication when not needed.
• Fix some hard-coded texture splatting region lookup in the shader.
• Add a function in world pipeline to retrieve the last central chunk position. Also made some changes to the return specification of the load() function, which now returns whehter the chunk position has been changed.
• Further eliminate unnecessary GL uniform.
• Fix the terrain normal direction that causes incorrect lighting by flipping x and y components.
• Improve region searching algorithm to avoid repetitive texture look-up in the terrain fragment shader.
• Turned on level-4 compiler warning at debug mode. Also modify all source codes to ensure no warning is generated during compilation.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Procedural Sky Rendering

Release 0.9.3

• Add a function in STPTerrainParaLoader to load sun and atmosphere settings from the INI.
• Fix some maths error in the sun shader.
• Fix more maths error in sun direction calculation function.
• Enable procedural sky rendering in the demo program. Enjoy the realism 😄
• Simplify calculation for scattering to improve performance.

SuperRealism+

• Add STPTexture as a GL texture object manager.
• Add STPRandomTextureGenerator as a utility of noise texture generation for GLSL shader. It is done using cuRAND device library.
• Perform smoothing to texture splatmap using volumetric noise, now the edge of texture region blends more natrually to adjacent regions. Algorithm is buit based on Efficient Soft-Edged Shadows Using Pixel Shader Branching.

General fixes and improvement

• Improve error handling in STPStart and catches exception thrown during main engine setup and rendering.
• Attach resource folder to the repo.
• Fix a typo atomsphere to atmosphere.
• Keep improving STPWorldManager, now it copies buffer from cache only when the chunk status has been marked as true previously. This avoids problem of chunk not getting refreshed when chunk is loaded buffer the central chunk switches.
• Update the camera rotation specification, instead of taking an absolute cartesian position, it now takes a relative direction offset. This addresses the problem of having sudden camera rotation when the camera moved for the first time because the lastOffsetPosition was set to zero initially by default.
• Remove some types from STPTextureType that are probably never going to be used.
• Move most hard-coded settings in the shader to INI, following a suggestion in #32.
• Remove normal, tangent and bitangent attributes from heightfield renderer.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Photorealistic Rendering Engine

Release 0.9.0

Please check out #31 for more development notes :+1:

SuperRealism+

This is the newly introduced rendering engine dedicated for provding photorealistic experience. All rendering tasks previouly exists in the demo program have all been moved here.

It provides simple utilities for managing GL objects such as shader and buffer in a RAII manner. Also provides helper functions to manipulate such objects in an object-oriented way.

The realism engine provides built-in support for debug callback and shading language include, also allows usage in a multi-context environment.

Other utilities such as camera and GL error handling system are also be found.

STPSun

A photorealisitc sun and sky rendering. Sun is simulated like how we can observe the Sun on the Earth with day-night cycle and physical effects. Sky is rendered based on the location of the sun with atomshperic scattering using two-phase sphere-ray marching.

STPHeightfieldTerrain

Aims to replace the old STPProcedural2DINF located in the demo program. Currently it functions in the exact same way as it predecessor. As the purpose of the demo program is to provide a minial working example for users who wish to use our library, doing this reduces user efforts to implement shading code.

SuperDemo+

In this release, we have removed all renderers from the demo program and restructure STPStart to improve general style and programming practice. In addition, as SuperRealism+ is now a self-contained rendering engine, we decided to abandon SglToolkit.

General fixes and improvement

• Add STPFile as a simple file manipulator that reads all lines from a file. It also provides simple utilities for processing filenames.
• Further reduced the amount of memory transaction between host and device in STPWorldPipeline.
• CMake target include and link refactoring. remove redundant codes.
• CMake config file outputs are now concentrated in a single directory with different names rather than assigning different filenames based on configuration.
• Add a new exception STPGLError.
• Move STPMeshSetting to the realism engine. Remove all getters from STPConfiguration.
• Add fractal noise function in STPSimplexNoise.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Setup Scene Pipeline

Release 0.9.0-Preview.3

SuperRealism+

• Remove STPContextStateManager as it is being redundant and unhelpful.
• Add STPScenePipeline as a master rendering pipeline for the entire procedural terrain.
• Setup STPHeightfieldTerrain as a heightfield-based terrain renderer. It aims to replace STPProcedural2DINF in the demo program.
• Improve STPShaderManager, add a helpful function to allow to define macros in the shader source before compilation.
• To reduce verbosity, remove all error handling in STPDebugCallback except important functions.
• Also implement a shader include internal cache in STPShaderManager for auto shader include and cross GL context support.
• Fix incorrect shader include directory.
• Fix illegam memory access in STPHeightfieldTerrain where reading source code as a reference from a temporarily created STPFile.
• Fix incorrect condition check during shader include and keeps throwing exception for not found include source.
• Fix issues in the camera class for mixing up degrees and radians.
• Move camera setting structs to separate files under STPEnvironment namespace.

SuperDemo+

• Remove SglToolkit as dependencies, the demo program is now self-contained as a rendering engine.
• Remove all renderers because SuperRealism+ will do the job instead.
• Restructure the starter function STPStart.
• Improve general coding practice and readability.

General fixes and improvement

• Fix the extension directive for GLSL shader, prefixed with GL_.
• Remove state record in STPDebugCallback and instead using GL query functions. This allows better support to multiple GL contexts.
• Add a simple helper function in STPFile for build filename.
• Replace shader filename processing in STPSun with compile-time functions instead of using std::string to allocate memory.
• Remove deprecated notes in the Readme, add more references.
• Change returned value in STPTextureFactory, instead of throwing exception when type is unregistered, it returns a flag.
• Remove all getters from STPConfiguration as it is very unecessary.
• Fixed an issue in STPShaderIncludeManager that incorrectly deletes an array with scalar deleter.
• Move STPMeshSetting to SuperRealism+.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Setup Sky Renderer

Release 0.9.0-Preview.2

SuperRealism+

• Add two utilities, STPCamera and an inheritied class STPPerspectiveCamera.
• Refactor the directory structure and group classes into folders.
• Refactor three shader managers in SuperRealism+ using pure RAII given greater flexibility.
• Add a simple wrapper STPDebugCallback for handling OpenGL debug output automatically.
• Add a simple context manager STPContextManager for storing current context state and issues context state change functions whenever necessary.
• Add STPIndirectCommand as an indirect rendering command wrapper according to GL specification.

All wrappers are trivally implemented and only functionalities used within this project are written, so they are not complete for the sake of development time.

GL object management

• Add more GL object wrappers to SuperRealism+ library:
  • STPBuffer for GL buffer object such as vertex buffer and element buffer.
  • STPVertexArray for vertex array buffer object.
• Add more GL compatibility types to STPOpenGL.

Sky renderer

• Add a new setting STPAtomsphereSetting for storing atomsphere rendering parameters.
• Setup shader STPSun for physically-based atomsphere scattering.
• Setup rendering pass for STPSun.

General fixes and improvement

• Add a new exception STPGLError to the main engine.
• CMake config file include directory overhaul. Config files are now all placed into different directories distinguished by configuration such that #include in source code can remain unchanged when build configuration changes.
  • As a refactorisation, config template is placed into a separate interface target.
• Add a simple file IO utility STPFile for reading all contents in the file quickly.
• Fix an issue where STPNullablePrimitive cannot be assigned to nullptr.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Rendering Toolkit

Release 0.9.0-Preview.1

Note: You may need to recompile the entire application for this release.

SuperRealism+

We introduce a new build target (or library) dedicated for shading and rendering. - STPShaderManager is a smart GLSL shader object manager that takes source code and compile it with error handling. - STPProgramManager is a GLSL program object manager that takes some shader managers with link and validation error handling. - STPPipelineManager is a GLSL program pipeline object manager that recombines program objects into a new program.

STPSun

We introduce STPSun as an utility for global light source on the procedural terrain, with STPSunSetting for storing the settings.

STPSun is a physically-based sun position calculator that simulates how the Sun rotates around the Earth in real life with seasonal effects.

General fixes and improvement

• Reduce the number of memory operations further in STPWorldPipeline. Now the front rendering buffer is only cleared as per-chunk basis when there is no old buffer to be used instead of clearing the entire buffer at the beginning.
  • Also clear buffer is now a pitched device memory rather than a pinned host memory.
• Texture anisotropy levels can now be adjusted via Engine.ini instead of being hard-coded in STPWorldManager.
• Add a fractal utility in STPSimplexNoise for fast fractal simplex noise generation.
• Fix an issue in algorithm device library that causes warning during compilation due to use of non-const in a const function.
• Simplify CMake library include and link structure, remove redundant codes.
• Rename shared library exports macros for consistency.
• Introduce STPNullablePrimitive in the main engine to allow using a primitive type in std::unique_ptr without allocating dynamic memory.
• Move terrain normalmap generation to fragment shader for higher level-of-detail and accuracy.
• Terrain splatmap genertaion now uses fractal simplex noise instead of a single one.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Improvement to Texture System

Release 0.8.13

SuperTest+

• Apply unit and converage testing to STPTextureDefinitionLanguage.
• Apply unit testing to new functions in STPChunk.

STPWorldPipeline

• Major optimisation to STPWorldPipeline. It now caches rendering buffer when the central chunk changes, and only load chunk map from STPChunkStorage and compute splatmap when old chunk map is not available to be reused.
• Type of rendering buffer and cache are all now identified by indices instead of some variable names. This greatly reduced the coding effort such that to update buffers for every type only a single for loop is required.
• Refactor some functions to cutdown line-of-code.

General fixes and improvement

• Removed function getChunkCoordinate() in STPChunk as it is not being used.
• Add a UnuseTextureType identifier in STPTextureFactory for indicating the value when texture type is not used in the texture type dictionary.
  • Also passes this new identifier to the shader and check for unused texture type before searching for texture data in the array.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Multi-Biome Texture Splatting

Release 0.8.11

Generator pipeline overhaul

This release provides solution to #30. STPChunkManager and STPChunkProvider are deprecated and removed from the engine, and replaced by a single STPWorldPipeline that functions exactly the same as them. However there are a few changes.

• Rename some functions to make them more compact.
• Chunk can be retrieved in one function call, instead of previous two (check first then get).

Apply texture to splatmap

• Allow retrieval of texture in STPTextureFactory; allow retrieving texture object using group ID.
• Add STPSplatTextureDatabase for storing pointers to texture data in STPTextureInformation.
• Setup texture parameters for splatting texture objects. This is currently done in STPWorldManager.
• Setup texture system in STPProcedural2DINF.
• SuperDemo+ can now render terrain with texture using splatmap. Improve UV coordinate system to make sure texture is stable when chunk position is updated.

Improvement to splatmap generator

• All splatmap generator parameters are put inside STPSplatmapGenerator.cu as constants instead of importing from INI file.
• Add simplex noise to terrain height to make the region boundary more natural.
• Change texture coordinate during splatmap generation from normalised UV to original pixel coordinate. Normalised UV causes floating point precision issues such that splatmap is unstable when rendered chunk changes world position. It also improves generation efficiency significantly.

We will be optimising the texture system later.

General fixes and improvement

• Fix a potential illegal memory access where texture and surface objects used during splatmap generation are deleted before kernel execution is finished.
• Change the way to retrieve permutation table in STPPermutationGenerator from calling operator() to operator*.
• Fix an incorrect bound when looping through gradient registry in the device splat rule wrapper.
• Simplex noise permutation table is now shared across all runtime compiled program, located in STPCommonCompiler.
• Add const functions in STPChunk for retrieving various maps.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Texture Splatmap Generation

Release 0.8.9

Enabling splatmap generation in SuperDemo+

In this release we enable the pipeline for texture splatmap generation and it can be visualised during runtime. We will be focusing on applying the texture according to the splatmap and parameter tweaking later.

• Setup external texture loading which takes place in a new sub-class STPWorldSplattingAgent in STPWorldManager in the demo program.
  • Setup texture loading and texture database in this new sub-class.
• Augment STPTextureStorage in the demo program for smart texture memory management.
• Enable parallel texture loading from file system in STPSkyRenderer and STPWorldManager.
• Add STPCommonGenerator.cu in the demo program as a data holder for RTC, allowing all generator scripts to be linked as a single program.

SuperTerrain+ Texture Definition Language

• Introduce TDL as a handy custom script for defining texture splatting rules.
• Refactor the lexer and parser with error handling improvement.
• Add a new exception type STPInvalidSyntax for any lexing and parsing error.
• Fix a problem of calling front() from an empty std::string_view. This is because neither std::string nor std::string_view includes null terminator.

General fixes and improvement

• Eliminate unused texture type when generating texture information in STPTextureFactory.
• Remove __constant__ qualifier for contant values in STPSimplexNoise device kernel.
• Refactor all runtime compilers and merge all of them into a single linkable program in STPCommonCompiler.
• Change rehash() to reserve() for std::unordered_map used in STPTextureFactory.
• Improve robustness of STPSingleHistogramFilter, the last working thread will guarantee to finish all remaining tasks if the texture dimension is not divisible by degree of concurrency. Change the threading strategy so STPSingleHistogramFilter is now thread safe and results in better concurrency with less waiting. Remove critical section in the demo program when calling histogram filter.
• Move Exception to SuperTerrain+ root directory.
• Refactor functions to attach inherited object and remove unused functions from STPWorldManager.
• Fix an incorrectly set launch configuration in STPSplatmapGenerator.
• Resolve #28 by completely removing stream callback to deallocate host memory for STPFreeSlipTextureBuffer and STPBiomefieldGenerator. It's now simply done using a stream sync call.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Setup Splatmap Generation

Release 0.8.5

Splatmap generator

• Move nested structure STPSplatDatabase from STPTextureFactory to STPTextureInformation and rename it to STPSplatRuleDatabase.
• Add lower_bound() function in STPKernelMath similar to the standard library version of this function.
• Setup STPTextureSplatRuleWrapper in SuperAlgorithm+Device as a helper class to access splat rules.

Setup SuperDemo+ splatmap generation

• Add STPCommonCompiler as a reusable compiler options loader. Remove compiler options setup in STPBiomefieldGenerator.
• Move all runtime-compiled scripts to Script directory in SuperDemo+ root. Rename filename suffix to .cu for consistency.
• Create STPSplatmapGenerator and setup splatmap generation routines. Also setup runtime script for device generation.

General fixes and improvement

• Add new static utility functions getChunkCoordinate(), calcChunkMapOffset() and getLocalChunkCoordinate() to retrieve chunk coordinate and map offset in STPChunk.
• Move smart memory and memory pool classes to a subdirectory Memory.
• Change the way to retrieve cudaSteam_t in STPSmartStream, instead of using implicit cast it now uses dereference operator.
  • Declared noexcept for that function.
• Change implicit cast to free-slip location to a function where() in STPFreeSlipTextureBuffer.
• Add OpenGL compatibility checking in STPEngineInitialiser.
• Replace memory block container in STPMemoryPool from std::unordered_map to a sorted array, because we expect the number of entry within the memory pool to be very small (no more than 10) and hash table is unnecessary.
• Remove unnecessary cast to uintptr_t when passing runtime compiler flags.
• Add invalid data checking during construction of STPTextureFactory and before any memory is allocated.
• Remove default value from STPPermutation so CUDA can initialise it in constant memory.
• Add some contents to README.md.

STPChunkManager

• Fix an undefined behaviour, change register flag for cudaGraphicsGLRegisterImage() from cudaGraphicsRegisterFlagsWriteDiscard to cudaGraphicsRegisterFlagsNone, because our pipeline only updates a portion of the whole texture each time rather than asking CUDA to discard the entire content.
  • A resource mapping flag cudaGraphicsResourceSetMapFlags() is also added with cudaGraphicsMapFlagsNone set.
• Use unsigned int rather than int to represent chunk ID.
• Refactor setting up texture in the constructor.
• Group cudaArray_t into a structure to reduce the number of function argument.
• Refactor copying buffer to GL texture using lambda functions.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Device Code Refactoring

Release 0.8.1

Refactoring

• SuperDemo+ now terminates when shader compilation fails.

Remove normalmap from rendering buffer

• Deprecate selective edge copy algorithm introduced in #17. Remove normalmap generation from heightfield generator pipeline. Remove normalmap from rendering buffer.

Now, heightfield rendering only contains a single channel of formatted heightmap of unsigned int, i.e., R16.

• Normal calculation in geometry shader in SuperDemo+.
• Change STPChunkManager and STPHeightfieldGenerator as well as shaders to address the changes.
• Move Strength from STPHeightfieldSetting to STPMeshSetting since normalmap will no longer be generated using our engine. Now mesh surface normal calculation can be done in either geometry shader or fragment shader.
• Remove prototype testing for Sobel filter from SuperTest+ since normalmap generation is no longer needed.

Device functions refactorisation

• Move the following CUDA global functions to a new file STPHeightfieldKernel, and use a host function wrapper to launch the device kernel. Note that all host function wrapper comes without the KERNEL suffix.
  • From STPHeightfieldGenerator
  • curandInitKERNEL()
  • performErosionKERNEL() renamed to hydraulicErosionKERNEL()
  • generateRenderingBufferKERNEL() renamed to texture32Fto16KERNEL()
  • From STPFreeSlipGenerator
  • initGlobalLocalIndexKERNEL()
• Hence, all instances of global functions in the original classes mentioned above are removed. This greatly reduce the code complexity.
• Relocate the following files from /GPGPU directory to /World/Chunk:
  • STPHeightfieldGenerator
  • STPDiversityGenerator
  • All files under /FreeSlip
• Rename the suffix of the following files from .cuh/.cu to .h/.cpp
  • STPHeightfieldGenerator
  • STPFreeSlipGenerator

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Texture Splating Utility

Release 0.8.0

In this release we introduces various tools for texturing which will be implemented in the later release, as well as some general quality-of-life improvements to our engine.

See developer notes in #29 to learn more about rule-based biome-dependent texture splatting

Texture

STPTextureDatabase

STPTextureDatabase is a non-owning storage of collection of texture data and information.

As discussed in the related PR, textures are managed into texture, group and map. Each texture contains multiple maps of different types, and one map belongs to a group which contains properties of all maps.

STPTextureSplatBuilder

Part of STPTextureDatabase. A simple utility that allows each biome to have different texture. Texture can have different splatting rules within the biome on the terrain, and currently the builder allows altitude and gradient splatting. Each rule maps to a texture which allow implementation to render the texture for that active region.

STPDatabaseView

Part of STPTextureDatabase. It allows querying large result sets from database for efficient data processing by STPTextureFactory.

STPTextureFactory

A generator class that takes a texture database, making some batch queries and convert all texture and rules into new data structures such that device and shader can make use of those information efficiently to generate terrain texture splatmap.

New

• Introduce STPHashCombine for mixing up hash values. It was previously located in STPChunkStorage.
• Introduce STPRuntimeCompilable as a high-level wrapper to NVRTC APIs, it was previously part of STPDiversityGeneratorRTC which has been removed in this update. To keep using JIT compilation feature for diversity generator client must inherit both STPDiversityGenerator and STPRuntimeCompilable.
• SQLite is now one of the middleware of SuperTerrain+. Introduce STPSQLite.h as a compatibility include header.
• Introduce STPOpenGL as a compatibility header to glad.h.
• New exception type STPDatabaseError.
• New error handling functionality for errors from SQLite.

General fixes and improvements

• Update unit test system to https://github.com/catchorg/Catch2/commit/48a889859bca45ee2c5e5064199c1e5b4b3e00cb
• Replace std::string usage in STPConsoleReporter with std::string_view.
• Refine reallocation condition, array insertion function and memory operations in STPSingleHistogramFilter, making it more efficient.
• Replace static-only classes with namespaces.
• Update README and LICENSE.
• Declare noexcept for all copy/move constructors/assignment operators.
• Improve CUDA stream sync. behavio in STPChunkManager.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Setup Texture Splatmap Generator

Release 0.8.0-Preview.3

Texture utilities

We plan to split texture splating into two stages: splatmap generation which will be done with CUDA before rendering, and terrain texture splatting using the splatmap generated in GLSL shader.

STPTextureFactory

• Refactor to make code more readable and maintainable.
• Cut down the class size by moving most standard library containers inside the function as local variable.

STPTextureDatabase

• Allow accessing splat rule data with getSplatDatabase().
• Rename some of the texture semantics:
  • TextureCollection -> Texture : contains multiple maps of different types.
  • TextureData -> Map : individual texture data of a specific type.
• Add some remove functions to allow dropping texture from the database.
• Allow addTexture() function to take a parameter count to insert count number of texture at once, and return an array of all texture IDs.
• STPTextureDatabase and STPTextureSplatBuilder are now move-constructable and move-assignable.
• Improve code usability for batch queries.

Texture utility testing

• Add STPTestTexture to SuperTest+.
• Fix potential exception throw when prepared statement sqlite3_stmt is finalised, since finalize() may return error codes from the previous operation. This has been mitigated by resetting prepare statement if error is detected.
• Fix syntactic errors in the templates addAltitudes() and addGradients() that cause failed compilation.
• Fixing incorrect use of blob data type.

General improvement and fixes

• Update the concurrency calculator in STPChunkProvider so it matches the latest free-slip algorithm.
• Provide solutions for #25 and #26.
• Refactor STPVoronoiLayer and move namespace alias declaration inside the function.
• Separate the runtime compiler from STPDiversityGeneratorRTC, and now it's presented as a standalone function STPRuntimeCompilable.
• Remove redundant multithreading in STPMasterRenderer.

In order to use runtime compilation feature for STPDiversityGenerator, one must inherit both STPDiversityGenerator and STPRuntimeCompilable, a.k.a., multiple inheritance.

• Link OpenGL as an interface target with our custom GLAD build target in CMake, because GLAD is usually used in conjunction with OpenGL, it doesn't make sense to take them apart.
• Move STPSQLite to the SuperTerrain+ root directory.
• Add a OpenGL compatibility header STPOpenGL and replace all GL type presence exposed in headers with such. This can be used to reduce unnecessary library include and linking.
• Improvement to README.
• Update LICENSE.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Texture Data Structure Generation

Release 0.8.0-Preview.2

Texture utilities

• Update the hashing algorithm for key pair using the newly introduced STPHashCombine.
• Split texture data structures into a separate file STPTextureInformation.
• Refactor STPTextureSplatBuilder with STPTextureInformation.
• Move STPTextureSplatBuilder into STPTextureDatabase.
• Simplify texture group logic in STPTextureDatabase. Now texture group only stores information about the texture format, texture data is mapped to texture ID type mapping table.
• Texture ID is now assigned by the system rather than the user. Also rename various functions to make sure all operations make sense.
• STPTextureFactory can now convert texture database into arrays that can be used by OpenGL.
• Create a STPDatabaseView which allows querying large result sets from database.

Database system

• Setup SQLite3 database. Add database initialiser in STPEngineInitialiser. Add helper header STPSQLite.
• Overhaul STPTextureDatabase and STPTextureSplatBuilder with in-memory private SQL database.
• Add a new exception type STPDatabaseError.
• Allow STPDeviceErrorHandler to handle database error.

General fixes and improvement

• Simplify STPHashCombine, it now contains only the variadic version of combine().
• Latest stable version of test framework has been updated to https://github.com/catchorg/Catch2/commit/48a889859bca45ee2c5e5064199c1e5b4b3e00cb
• Use namespace instead of class for which only contain static function and object creation is not intended.

Improvement to STPSingleHistogramFilter

• Refine the algorithm for removing empty bin. Instead of unnecessarily looping through the whole dictionary, only dictionary entries affected due to removal of bin will be visited and updated. Approximately 20% speed up compared to the old algorithm.

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Data Structure for Texture Utility

Release 0.8.0-Preview.1

Biome-specific texture building system

STPTextureDatabase

STPTextureDatabase is a non-owning storage of collection of texture data and information.

As a whole, texture ID and texture type acts as a compound primary key to uniquely identify the texture data. Internally, texture data with the same property will be grouped into a structure STPTextureGroup. This is beneficial for later packing texture with the same format into the same OpenGL texture array.

STPTextureSplatBuilder

A simple utility that allows each biome to have different texture. Texture can have different splatting rules within the biome on the terrain, and currently the builder allows altitude and gradient splatting.

STPTextureFactory (WIP)

A generator class that takes texture database and splat builder, combine them in new data structures such that device and shader can make use of those information efficiently to generate terrain texture splatmap.

We have the data structure ready for use, data generation will be coming later...

General fixes and improvement

• Update SuperTest+ to support the latest release of Catch2:
  • As of https://github.com/catchorg/Catch2/commit/3f8cae8025f4f5e804383f44bb393a63bcef90a4 and https://github.com/catchorg/Catch2/commit/f02c2678a1a891f9577712fe160c1f6f3baef3a8
  • Update STPConsoleReporter and replace most std::string with std::string_view to eliminate unnecessary memory operations.
  • Allow reporter to use Catch::StringRef and convert it to std::string_view.
  • Reporter now indents the test section name based on the nested depth.
  • As of https://github.com/catchorg/Catch2/commit/426954032f263754d2fff4cffce8552e7371965e
  • Make some modification to STPTestRTC to support the renamed matcher.
  • Change floating point comparison in STPTestPermutation with Catch build-in floating point matchers.
• Replace most emplace() for map containers with try_emplace().
• Declare all move constructor as noexcept.
• Change type of PV matrix mapping from char* to unsigned char* in STPMasterRenderer.
• STPChunkManager now sync cudaStream_t when cudaGraphicsUnmapResources() instead of doing it explicitly during async loading. Also stream is now no longer sync after each rendering buffer clear operation. This should result in better device workflow overlap.
• Extract the hash combine algorithm from STPChunkStorage into a new class STPHashCombine.

Improvement to STPSingleHistogramFilter

• Simplify loops with functions from standard library algorithm header.
• Change the reallocation condition check from greater or equal than the last element to greater.
• Add a return type to insert_back_n(), which now returns the iterator to the last inserted element to eliminate the need of indexing the element.
• Replace C standard library functions with C++ version, for example memmove() to std::copy().

- C++
Published by stephen-hqxu over 4 years ago

superterrainplus - Finalise Test Program

Release 0.7.17

Changes made in STPRainDropSetting

• Add a new class STPErosionBrushGenerator which is responsible for generating the erosion brush and managing the memory space.
• Implement a PIMPL for erosion brush generation. The implementation is defined in STPErosionBrushGenerator. This is for avoiding sending useless objects like std::vector (takes up 40 bytes each) to the device.
• Explicitly declare the following classes as non-copiable to make sure the erosion brush cache on host side is unique:
  • STPRainDropSetting
  • STPHeightfieldSetting
  • STPConfiguration
• Replace raw device pointer with STPSmartDeviceMemory.
• Change the usage in SuperDemo+ so setting objects are moved instead of copied.

Development on SuperTest+

• Add STPTestInformation to store common data for testing.
• Add stream-ordered memory allocation testing for STPSmartDeviceMemory.
• Add device memory test for STPFreeSlipTextureBuffer and STPFreeSlipGenerator.
• Add STPTestEnvironment for testing class under namespace STPEnvironment.

By now, all classes except the following are tested with theoretical full coverage: - STPChunkManager - STPChunkProvider - STPHeightfieldGenerator (only tested using prototype) - STPRainDrop - STPSimplexNoiseGenerator

The objects above are mostly in our generator pipeline stages, testing involves a large amount of work that can even surpass the effort to program the engine itself. We are going to either do the test using different methods (e.g. static or user acceptance test), or do it in the unit test manner but defer it to later. There are still awaiting goals to be completed by deadline.

General fixes and improvement

• Declare constexpr and static or both for lambda expression instantiation whenever necessary.
• Remove some unnecessary pass by reference, for example glm vector types which are usually small and cheap to copy than dereferencing.
• Simplify formula in STPVoronoiLayer.
• Update cache load algorithm in rendering buffer generator to optimised for coalesced memory access.
• Remove inheritance from STPFreeSlipData in STPFreeSlipGenerator because it doesn't make sense.
• Improve STPChunkSetting validation.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Coverage and Prototype Testing

Release 0.7.14

Development on SuperTest+

• Add more test cases to STPDiversityGeneratorRTC to cover more scenarios.
• Add missing test scenatrios in STPFreeSlipTextureBuffer.
• Add missing tests for, in the exisiting test source code:
  • STPChunkStorage in STPTestChunk.
  • STPSmartDeviceMemory in STPTestUtility.

Prototype test

STPPrototypeCachedConvolution for rendering buffer generation in STPHeightfieldGenerator. It aims to test if the rendering buffer is the same as before and after shared memory optimisation.

Coverage test

We perform coverage test for the program to ensure our test cases cover every line of code.

Functionalities added to test to ensure full coverage: - Rendering buffer retrieval in STPChunk. - Exception when asking for volumetric biomemap and memory pool in STPBiomeFactory. - Exception when requested memory size is zero in STPMemoryPool. - Exception when inserting into a dead thread pool in STPThreadPool. - Exception when asking for memory location when tetxture buffer is unmerged in STPFreeSlipTextureBuffer. - Storage clear in STPChunkStorage. - Individual data option in STPDiversityGeneratorRTC. - Stream created with priority in STPSmartStream.

General fixes and improvement

• Rename member functions in STPChunkStorage for more consistent style. Add more auxiliary functions.
• Elaborate the documentation in STPFreeSlipTextureBuffer to notify the user about the memory behaviour of host memory when the texture buffer is in device mode.
• Improve index clammping in STPHeightfieldGenerator with glm functions and refactor filter kernel.
• Remove conditional instantiation for rendering buffer in STPFreeSlipGenerator, instead a static assertion is added.
• Improve documentation for STPSmartStream about the stream priority, and flip the output priority to [greatest, least] instead of using [low, high] as CUDA defines low as greatest, which was not the program was assuming previously.

Engine breaking bug in STPLayer

• Fix an incorrect use of local static variable in STPLocalRNG which causes the local seed for every RNG instance to be the same, as well as data racing.
• Remove floorMod() function in nextVal() and use modulo operator instead, they work the same way when the input data have the same sign.

The same changes have been applied to STPVoronoiLayer as well.

• STPLocalRNG takes in a layer seed in the constructor, and uses layer seed to mix the local seed.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Test for Algorithm and Runtime Compiler

Release 0.7.12

Development on SuperTest+

• Add the following test code
  • STPTestPermutation for STPPermutationGenerator.
  • STPTestHistogram for STPSingleHistogramGenerator.
  • STPTestRTC for STPDiversityGeneratorRTC.
• Update STPConsoleReporter to address some important bug fixes in the latest version of Catch2:

See https://github.com/catchorg/Catch2/commit/e5938007f7dab5c9886ca561d09106ee4d7f2301 and https://github.com/catchorg/Catch2/commit/290c1b60e6c6d9fd4dde1b28959a6d673caad937 for the details of changes.

General fixes and improvement

• Replace all double with float in STPPermutationGenerator in host algorithm library and STPSimplexNoise in device algorithm library.
• Simplify permutation table generation with glm.
• Make the constructor for STPFreeSlipManager public and remove STPFreeSlipGenerator as its friend class.
• Fix an incorrect pre-launch argument check in STPSingleHistogramFilter.
• Add a cudaFree(0) call in STPEngineInitialiser to enforce context creation at startup.
• Fix an incorrect cache loading operation during rendering buffer generation which causes int to be implicitly-casted to unsigned int and underflow to UINT32_MAX instead of -1 as expected.

Visually it's unaffected when free-slip erosion is turned on because selective edge copy prevents rendering buffer generator from modifying the edge pixel.

• Adjust the working directory for the test target.
• Case valued variable to uintptr_t before casting to void* when passing compiler flags.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Project Restructure and Exception Handling

Release 0.7.8

Resolution for issues

Issue #23

• Add STPSmartDeviceMemory in SuperTerrain+ Utility. Replace all memory that are previously freed manually with cudaFree() with the new smart device memory.
• Move pointer assignment to STPSmartDeviceMemory to right after cudaMalloc(), if memory allocation fails no free is required, or if future operation throws an exception the memory will be freed automatically.
• Implement std::unique_ptr for nvrtcProgram, CUmodule and CUlinkState in STPDiversityGeneratorRTC, and replace all occurences with managed memory.
• Improve exception handling in STPChunkProvider. All exceptions from async compute threads are captured into an exception queue and will be merged together and rethrow to STPChunkManager.
• Remove try-catch blocks in STPHeightfieldGenerator, STPFreeSlipGenerator and STPFreeSlipTextureBuffer. Use std::unique_ptr with custom deleter to recycle memory so no memory leak is introduced when exception is thrown.

Issue #24

Files that were in CoreInterface are now moved into a newly created folder CoreInterface/SuperTerrain+. This is a more standard way of serving libraries to allow user to distinguishing between a mixture of different libraries and avoid include name collision.

Change in include directory structure has been addressed in all sources and headers.

• Simplify STPDeviceErrorHandler. Remove error severity switch due to its unnecessity. It now throws exception by default and un-caught exception or exception thrown in the dtor will simply terminate the program.
• Change setting macro for STPDeviceErrorHandler from STP_SUPPRESS_MESSAGE to STP_DEVICE_ERROR_SUPPRESS_CERR. This will prevent error handler from writing exception message to std::cerr stream.

General fixes and improvement

• Replace the following external libraries with imported package, and remove them from the External diectory:
  • SglToolkit
• Simplify template instantiation model in STPMemoryPool.
• Remove using and unnecessary include from STPFreeSlipManager.inl.
• Declare STPEngineInitialiser as final.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Error Handling and Build System Change

Release 0.7.5

SuperTest+

• Fix an integer underflows issue when line is wider than the console width in STPConsoleReporter.
• Add a smart text wrapper for the consoel reporter.
• Add test for free-slip utilities.

General fixes and improvement

Exception handling

• Simplify exception handling in some classes to ensure there's only one try-catch block in one function. If exception can be thrown out safely without the need to clear memory, try-catch block is removed.
• Avoid throwing exception in the destructor, and instead terminate the program.
• Replace exit(-1) with std::terminate().
• Add a new exception class STPInvalidEnvironment which is thrown when STPEnvironment is not validated at the constructors of STPChunkProvider, STPHeightfieldGenerator and STPermutationGenerator.
• Add numeric range check for functions in STPChunk.
• Add size check in STPMemoryPool and STPFreeSlipGenerator to make sure the numbers provided are positive.
• Add input value range check for STPFreeSlipTextureBuffer.

File system change

• Move free-slip utilites into a separate folder GPGPU/FreeSlip.
• Re-style include directories. Header should prefer using double quotes while sources prefer using angle brackets. Replace system header includes with angled brackets.
• Replace the following external libraries with imported package, and remove them from the External diectory:
  • glfw3
  • glm
  • OpenGL
  • SIMPLE

Out final goal is to replace all explicit include directories with CMake imported targets via CMake find_package(). - Change GLAD build type to share library. This is beneficial for sharing OpenGL contexts between the main engine and demo application.

Others

• Remove checker for disabled free-slip hydraulic erosion and return nullptr if so. It's generally un-safe and adds more overhead during run-time to check for null pointer.
• Use emplace_back() instead of push_back() for getRegion() in STPChunk.
• Address changes brought by SIMPLE v2.0.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Testing and Improvement

Release 0.7.2

More test units in SuperTest+

• Complete STPTestUtility, the test for utilities.
• Refine on the test cases to make it more clear.
• Add new test STPTest2DBiome for biome generation testing.

General fixes and improvement

A number of bugs are found during writing and running the test program.

STPDeviceErrorHandler

• Remove unnecessary semi-colon for the macro defined in STPDeviceErrorHandler.
• Refactor STPDeviceErrorHandler and STPFreeSlipGenerator to eliminate repetitive coding during template explicit instantiation with macros.
• Add a define symbol STP_SUPPRESS_ERROR_MESSAGE to STPDeviceErrorHandler to suppress printing of any error message.

Diversity

• Remove unnecessary destructor implementation in STPLayerCache.
• The clear value for key has been changed to UINT64_MAX (equivalent to -1 for signed integer system) instead of 0 when layer cache is initialised and cleared in STPLayerCache.
• Remove layer cache for the starting layer in SuperDemo+ since STPBiomeFactory only calls the uncached sampling method for the starting layer and it's unnecessary to cache data that only uses once.
• Fix a bug when layer manager is not reused by STPBiomeFactory.

Multi-threading

• Fix an issue in thread pool that causes program deadlock on program exit. This is caused by the looped check in STPChunk destructor which will not return until in-use flag is false, and thread pool is killed as soon as it finishes the current task even if the task queue is not empty. If any thread left in the task queue is responsible for releasing the flag for the chunk, it will be a deadlock. A simple fix is don't terminate the thread poll until kill signal is sent and the task queue is not empty.
  • Add additional protection (also good programming practice in general) in STPMasterRenderer to make sure thread pool is the last object to be destroyed so other stuff can be finished.
  • Add more protection to STPChunkProvider to make sure it synchronises and waits for its internal thread pool to finish before getting destroyed.
  • Add even more protection in STPWorldManager to make sure STPChunkProvider is the first member to be destroyed before destroying other generators.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - New Unit Test System

Release 0.7.0

SuperTest+

• Upgrade test framework to catch2 v3.0.
• Setup test utils with CTest and catch2 integration.
• Re-program the default test reporter, STPConsoleReporter.
• Add new test units:
  • STPTest2DChunk
  • STPTestUtility

Custom exception class

• Add a new namespace STPException, located in CoreInterface/Utility/Exception.
• Add new exception classes:
  • STPBadNumericRange
  • STPCompilationError
  • STPCUDAError
  • STPDeadThreadPool
  • STPInvalidArgument
  • STPMemoryError
  • STPSerialisationError
  • STPUnsupportedFunctionality
• Replace all standard-library exception in the engine with custom exception. This makes catching more specific exception during unit testing easier.

General fixes, improvement and refactoring

• Simplify formula for generating the global-local index table.
• Replace all CUDA vector functions (uint2, float3, etc.) with glm vectors (uvec2, vec3, etc.) and simplify calculations using said vector functions.
• Remove all unecessary use of floor() on unsigned integer division.
• Simplify codes with glm functions where applicable.
• Refactor STPThreadPool to cut down the number of unecessary code.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Better Interpolation and Project

Release 0.6.7

Better smooth biome transition

• Simplify the data structure model for SuperAlgorithm+Device. Include data structure headers in each main class header to eliminate the need of declaring opaque pointers in runtime device script.

Pros: It has cut down register usage on GPU and significantly reduced the programming difficulty. Cons: NVRTC needs to know the include directory of SuperTerrain+ Core engine due to the use of STPBiomeDefine in STPSingleHistogram.

• Simplify single histogram data being sent to device, instead of sending a pointers wrapped over a number of pointers, the struct which contains those pointers is sent directly.
• Add custom allocator for STPArrayList.
• Fix error thrown during pinned memory allocation in histogram buffer.

Build system change

• Remove deprecated CMAKE options:
  • STPCUDAARCH: now use CMAKE built-in CUDA arch variable.
  • STPENGINEBUILD_SHARED: SuperTerrain+ will now always be built into shared library
  • STPDEVICELINKSTATIC: no longer useful when the engine is always built into shared.
• Move STPCoreDefine into directory CoreInterface from Template.
• Merge SuperError+ into SuperTerrain+ Utility folder, and remove SuperError+ build target.

General improvement and fixes

• Add interpolationRadius in Biome.ini for dynamic configuration.
• Add core include directory information into STPAlgorithmDeviceInfo.
• Use placement new to initialise data allocated from memory pool instead of using memset.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Biome Edge Interpolation

Release 0.6.5

Data structure in STPSingleHistogramFilter

Profiling shows the implementation using std::vector has led to significant performance issue on MSVC compiler when debug runtime library (/MDd and /MTd) is enabled. Although the final performance in release mode is not affected, the slowness makes debugging during development impossible.

Result of a run with std::vector: | Compiler | -O0/ms | -O3/ms | | --------- | ---- | ---- | | MSVC 2019 | >2000 | ~60 | | GCC 10 | ~200 | ~30 |

We also notice time taken in release mode on MSVC is twice as long compared to GCC.

Improvement

• Implement a custom version of std::vector, namely STPArrayList. It's a minimal data structure that only serves to STPSingleHistogramFilter. The goal is make it as simple as possible.
• Remove all uses of back_inserter and instead pre-resize the array list and do a simple copy at once.

Result of a run with STPArrayList: | Compiler | -O0/ms | -O3/ms | | --------- | ---- | ---- | | MSVC 2019 && GCC 10 | ~140 | ~25 |

Data sets used in all benchmarks

| Item | Value | | ----- | ----- | | Dimension | 512 * 512 | | Free-slip range | 1536 * 1536 | | Radius | 32 | | Sample range | RNG, 0 to 14, both inclusive |

Implement smooth biome transition

• Implement procedures in STPBiomefieldGenerator to enable smooth biome transition with STPSingleHistogramFilter. This includes:
  • A CUDA memory pool.
  • A histogram buffer pool for reusing memory.
  • A host memory pool for releasing histogram buffer after execution.
  • Critical sections for thread safety.
  • Stream-ordered operations.
• Making changes in the runtime compiled script to utilises histogram being sent to GPU.
• Pinned memory allocation for histogram buffer is temporarily disabled due to some context issues.

General fixes and improvement

• Inline definitions of STPFreeSlipManager since device functions cannot be exported as shared library.
• Declare default on move constructor/assignment operators for certain class to make sure compiler does the right thing.
• Replace custom deleter for STPPinnedHistogramBuffer with default-constructed struct so unique_ptr can be default constructed.
• Improve documentation on the memory behaviour about the texture buffer.
• Fix an issue causes uncatched comilation error in SuperDemo+ when using CUDA runtime compiler due to the changes made to the exception throw system previously.
• Remove const specifier for variables in STPSingleHistogram.
• Change the function parameter from taking a value to a rvalue reference in operator() in STPSingleHistogramWrapper

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Prepare for Biome Interpolation

Release 0.6.3

Major changes to STPSingleHistogramFilter

• Move default declared destroctor for STPSingleHistogramFilter into source code so compiler won't complain about using incomplete type of Pimpl.
• Remove constructor of STPSingleHistogramFilter to predict the size of the texture because it won't make any difference to the performance. Now STPSingleHistogramFilter is trivally constructable.
• Augment the data structure, now the output histogram is no longer bound to a particular STPSingleHistogramFilter instance, but rather, user can ask for a single histogram buffer where the output will then be stored.
• Split STPSingleHistogramFilter::STPHistogramBuffer with two versions, being default-allocated and page-locked-allocated. The buffer passed to external will always be using page-locked allocator for fast device memory transfer.

General fix and improvement

• Refactor code base using new features in C++17 like conditional initialiser and structured binding,
• Enable move constructor/operator in STPChunk and STPChunkStorage.
• Simplify class construction using initialiser list, for STPRainDrop, STPSingleHistogramFilter and STPFreeSlipGenerator and various STPEnvironment classes.
• STPChunk will now always allocate memory for internal texture during construction.
• Fix some typos.
• Fix a detrimental bug that causes host memory to be freed before CUDA stream has finished execution. To resolve that we use CUDA stream host callback.
• Augment STPPinnedMemoryPool such that it can be used on pagable memory. To avoid confusion it has been renamed to STPMemoryPool and use template parameter to denote the memory type of it.

Partial resolution to issue #21 - Use remove_pointer to determine the base type for cudaStream_t instead of hard coding. - Change the storage type in STPChunkStorage from unique_ptr<STPChunk> to STPChunk. - Simplify chained function call to set compiler and linker flag in STPDiversityGeneratorRTC.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Tools for Smooth Biome Transition

Release 0.6.0

In this release we introduces a lot of new algorithms and utilities that are aimed to resolve the issue of strong cut-off between biome edges. The actual implementation for smooth biome transition will be coming later. More development details have been discussed and documented in #22.

STPSingleHistogramFilter

• Fast execution to generate historams for every pixel on a discrete texture like biomemap within a given radius. The runtime of the algorithm is loosely depended on the size of the radius which is useful for large scale smooth biome interpolaiton.
• Shipped with a histogram wrapper for accessing histogram on GPU.
• It's implemented on CPU, located in SuperAlgorithm+Host library. The output matrix of histograms is jaggedly flatten and arranged in row-major.

STPFreeSlipTextureBuffer

It is a wrapper over free-slip texture. Free-slip texture buffer will do the free-slip texture merging and un-merging automatically. User can choose to obtain free-slip texture which is in local-index order, in different memory space, either host or device, flexibly with relatively low overhead.

STPSmartStream

The smart stream is essentially a wrapper of cudaStream_t and unique_ptr. A CUDA stream will be created at construction of the smart stream object and auto destroyed at destruction of the smart stream. Like unique_ptr, smart stream is only movable.

STPPinnedMemoryPool

We deprecate the old STPMemoryPool, and introduces a more specialised memory pool for pinned memory exclusively with more friendly function call. It also makes use of standard library to make the code more readable.

The new memory pool is thread safe already, and implement a memory header which contains information about the allocation so user doesn't need to tell the memory pool, for example the size of the allocation, when memory is returned.

General fixes and improvement

• Refactor all classes and make use of the new utilities introduced in this release.
• Improve general programming styles and practice.
• Improvement to hashing algorithm for uvec2 in STPChunkStorage.
• Remove shared library exports for unnecessary classes.
• Make project include directories more consistent such that including each other won't cause file path issues.
• STPFreeSlipGenerator can now generate STPFreeSlipManagerAdaptor which will generate different STPFreeSlipManager for host or/and device usage.
• Pre-cache STPFreeSlipData in STPFreeSlipGenerator so it can strategically select the data based on chosen memory space.
• Improve varioud class constructors.
• Replace some throwing-string exceptions with classes inherited from std::exception.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - New Utilities

Release 0.6.0-Preview.3

STPSmartStream

• Add a smart CUDA stream manager that utilises RAII techniques -- create a stream at construction, destroy the stream at destruction.
• Implement smart stream from unique_ptr such that the object can only be moved.
• Implement custom cast operator to cast STPSmartStream to cudaStream_t implicitly so no codebase needs to be changed.

STPPinnedMemoryPool

• Implement a new memory pool system dedicated for pinned memory.
• Memory is managed by smart pointer.
• Implement memory header which contains information about the allocated memory while external user is completely unaware of that, which makes returning memory back to the pool easier.
• Make pinned memory pool thread safe.

General fixes and improvement

• Deprecate STPMemoryPool in SuperTerrain+ Utility.
• STPFreeSlipTextureBuffer now uses STPPinnedMemoryPool for pinned memory allocation.
• Change the argument in filter execution call in STPSingleHistogramFilter to STPFreeSlipManager for data safety.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Free-Slip Texture Buffering

Release 0.6.0-Preview.2

STPFreeSlipTextureBuffer

STPFreeSlipTextureBuffer is a memory manager dedicated for managing texture from free-slip neighbour chunks and merging them into a large local-indexed buffer for free-slip computing; like what STPHeightfieldGenerator does previously.

• STPFreeSlipTextureBuffer can now merge buffer to different memory space (host and device) depends on caller.
• Deprecate memory management in STPHeightfieldGenerator.
• Improve level of abstraction in STPFreeSlipGenerator, which now takes a pointer to STPFreeSlipTextureBuffer instead of a pointer to texture.
• Improve documentation in STPFreeSlipGenerator::STPFreeSlipAdaptor.
• Improve behaviour for STPFreeSlipTextureBuffer when doing repetitive calls to retrieve texture buffer. See documentation for more details.

Future work

• Allocation of pinned memory in STPFreeSlipManagerAdaptor is not pooled currently.

Development on STPSingleHistogramFilter

• We finally decice to swap the order of two separate filter stages in STPSingleHistogramFilter as benchmark shows there's negligible performance difference as the original one. The output histogram from calling STPSingleHistogramFilter is now in row-major order.
• Correct the documentation in STPSingleHistogramFilter so it matches the new algorithm.
• Extract STPSingleHistogramFilter::STPFilterReport as a separate struct and rename it to STPSingleHistogram.
• Add STPSingleHistogramWrapper in SuperAlgorithm+Device for easy STPSingleHistogram access on GPU.

General fixes and improvement

• Rename destroy() in STPSingleHistogramFilter to destroyReport() to avoid confusion.
• Inline more in-source functions in STPSingleHistogramFilter.
• Change the argument in filter execution call in STPSingleHistogramFilter to STPFreeSlipManagerAdaptor for better abstraction.
• Make STPFreeSlipManagerAdaptor into a template class to improve type safety.
• Improve hashing algorithm for vec2 type in STPChunkStorage.
• Remove shared library export symbols for enum and POD-struct.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Single Histogram Filter

Release 0.6.0-Preview.1

STPSingleHistogramFilter

Implement a filter utility in SuperAlgorithm+Host library, which is now linked with SuperTerrain+.

Single histogram filter generates an analytical histogram for every pixel in Sample texture format within a given radius. This is useful for retrieving weights of all biomes presented in a given radius during biome edge interpolation.

More development details will be released along with finalised version.

Issues

The current implementation of STPSingleHistogramFilter has been tested to output correct result. However the output histogram is a flatten, jagged column-major matrix, and will cause problem if use in conjunction with other maps generated by out engine, which are all aligned in row-major. There're two alternatives:

• Transpose the output matrix. It's an easy solution but needs to loop through outputs and since the array is jagged, it might be difficult to optimise it.
• Perform two separable kernels in reverse order, i.e., do vertical pass first then horizontal. Require some structure overhauls, but does not require additional operations. One major drawback is the cache spatial locality is broken as we need to read row-major input sample map in column order.

General improvement and fixes

• Refine on STPFreeSlipGenerator such that it can generate global-local index table for host usage.
• Make STPFreeSlipManager template so it can deal with different types of texture.
• STPFreeSlipGenerator now pre-caches STPFreeSlipData in host and device memory to reduce the amount of memory copy done.
• Add STPFreeSlipGenerator::STPFreeSlipManagerAdaptor which cache texture data inside and can be extracted in different memory space without exposing STPFreeSlipGenerator.
• Replace pointer type with reference type where applicable.
• Remove redundant configure file (STPAlgorithmDefine.h), and include in the header include directory instead.
• Simplify CMake include directory models, making them more consistent.
• Explicitly delete classes that are not supposed to use copy/move constructor/assignment operator.
• Explicitly delete constructor for classes that are only used statically.
• Improve exeception handling such that an execption instead of string is thrown.
• Replace C style cast with C++ style in CUDA kernel in STPHeightfieldGenerator.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Miscellaneous Improvement

Release 0.5.8

Restructure

• Enable free-slip neighbour logic on heightmap generation. SuperTerrain+ will generate neighbour biomemap before generating the central heightmap, so free-slip biomemap can be passed to heightmap generator for biome interpolation.
• Overhaul the structure of checkChunk() in STPChunkProvider for better nested neighbour logic.
• Split STPFreeSlipManager into three different files, being STPFreeSlipGenerator and STPFreeSlipData in addition to that. This aims to separate data with implementation.
• Move CUDA context parameter setting from STPMasterRenderer to STPEngineInitialiser.
• Add STPKernelMath in SuperAlgorithm+Device for easier programming.

General fixes and improvement

• Add some checkers in SuperTerrain+ to prevent user from including internal headers like STPRainDrop.cuh which has no exported symbol.
• Remove unused function parameter in STPChunkProvider.
• Improvement to documentation.
• Fix a bug when generating biomemap in column-major order which breaks spatial locality of cache.
• Make STPThreadPool constructed as an object rather than a dynamic pointer in some classes.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Runtime Multi-Biome Generation

Release 0.5.6

Runtime multi-biome heightfield generation in demo program

• Implement runtime-compiled heightfield generation in SuperDemo+ with STPDiversityGeneratorRTC.
• Separate STPBiomeSettings into another struct STPBiomeProperty which is dedicated to be used by runtime compiler to minimise include directories and compile time.
• Split STPAlgorithmDefine into another file STPAlgorithmDeviceInfo.... The former one contains shared library export symbols while the latter one contains path to SuperAlgorithm+Dervice library and include. This reduces file conflicts and remove any need to hard-code the filename based on different configuration.
• Implement multi-biome heightfield generation, generation settings are picked from a biome lookup table indexed by biomemap.

There are 2 biomes implemented in the demo program, being ocean and plains, and the parameters are not the best. The overall terrain looks coarse, we will be looking at biome edge interpolation later.

General fixes and improvement

• Add guard to check for duplicate source name when compiling source with compileSource().
• Fix a case when calling discardSource() the actual nvrtcProgram is not deleted, causing memory leaks.
• Improve error handling on STPDiversityGeneratorRTC and throw exception when necessary.
• Name expressions are now cached into complication database in STPDiversityGeneratorRTC so they can be retrieved later without need to re-typing everything.
• Change the template STPAlgorithmDefine.h.in to output the path to device include and library.
• Add a utility in STPDiversityGeneratorRTC for reading source code from local file.
• Move CMake configure_file output to binary directory (#20).
• Remove redundant gradient table shuffle. Note that the generation result will be different from the previous implementaion.
• Increase the camera height on the demo application.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Better Algorithm Library for Runtime Compilation

Release 0.5.4

SuperDemo+ is temporarily disabled for this release due to the high amount of updates to the main engine, we will make it work in the next release.

Runtime compiler improvement

• Separate SuperAlgorithm+ into two libraries, being SuperAlgorithm+Host for host initialisation, and SuperAlgorithm+Device for device execution.

For static complication, both libraries need to be included with the application. For runtime complication, only SuperAlgorithm+Host is required, and add SuperAlgorithm+Device to library dependencies of JIT linker.

• Make constants compile-time initialised to eliminate the need of calling initialisation function in STPSimplexNoise.

Shared library

• SuperTerrain+ engine can now support shared library build.
• Add STPEngineInitialiser. If shared library is used or there's no other OpenGL contexts bounded to GLAD being created in the application regardlessly, underlying functions must be called before calling other STP APIs to initialise OpenGL and CUDA context.
• Add CMake option to enable device link on core engine static library build.

General improvement and fixes

• Change all namespaces in SuperDemo+ to STPDemo for consistent style.
• Add STPAlgorithmDefine.h and move contents from STPCoreDefine.h.
• Suffix template name with .in to avoid confusion.
• Rename STPRainDropSetting.hpp to STPRainDropSetting.h to avoid confusion.
• Pre-compile glad.c and link to avoid duplicate complications.
• Further reduces compile time by moving definitions to source files instead of inlining.
• Remove some redundant CMake codes.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - File System Overhaul

Release 0.5.0

This release aims to replace VS 2019 build system with CMAKE 3 due to some unforeseen bugs related to Visual Studio, and improve cross-platform compatibility. Also remap the current file hierarchy to separate the actual engine with the demo program.

Before building system with CMake, unzip STPExtras.zip in the root project directory.

Overhaul to build system

• Deprecate VS 2019 build system, instead SuperTerrain+ will use CMake 3.18. VS 2019 solution can however be generated.
• Separate source code and header.
• Split STPDeviceErrorHandler into a separate shared library target SuperError+.
• Split STPSimplexNoise and STPPermutationsGenerator into a separate static library target SuperAlgorithm+ to allow the newly introduced runtime compiled heightmap generator script to load noise library directly during runtime.
• Completely split SuperTerrain+ main engine from the demo program SuperDemo+.
• Add header files to build dependencies so IDE like Visual Studio can see it.
• Add a template file STPCoreDefine for easier runtime filename referencing.

Rename

• Change Helpers directory in SuperTerrain+ to Utility.
• Change Settings directory in SuperTerrain+ to Environment.
• Change Biome directory in SuperTerrain+ to Diversity.
• Change namespace STPSettings to STPEnvironment to avoid name conflict .
• Change STPPermutationsGenerator to STPPermutationGenerator.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Improve Level of Generality

Release 0.4.4

In this release we aim to remove heightmap generation stage from the engine, and instead we take in the user-defined STPDiversityGenerator and call the virtual function for generation.

Programmable pipeline stage

• Rename STPDiversityGenerator to STPDiversityGeneratorRTC. And add a new class STPDiversityGenerator. For runtime-compiled heightmap generation script STPDiversityGeneratorRTC can be used; for static-compiled heightmap generation script STPDiversityGenerator can be use instead. This can help cutting down the class size if runtime complication is not required, or not supported on target machine.
• Allow input type to be customised in linkProgram() in STPDiversityGeneratorRTC for better control over complier and linker flags.
• Allow attaching archive to STPDiversityGeneratorRTC.
• Main generator STPHeightfieldGenerator no longer takes simplex noise to generate heightmap, instead it uses STPDiversityGenerator implemented by user.
• Overhaul to STPSettings and INI to reduce redundancy for the new system.
• Move STPBiome and STPBiomeSettings into namespace STPDemo. Also they are moved to SuperTerrain+\World\Biome\Biomes.
• Move all heightmap generation algorithms to STPDemo, and it's located in STPMultiHeightGenerator whereas STPBiomefieldGenerator implements STPDiversityGenerator. The demo generation uses static complication.

Improvement and fixes

Suggestions mentioned in #19

• [x] Correct the include guard for STPBiomeFactory so the symbol is consistent with the filename.
• [x] Include function info for STPDeviceErrorHandler.
• [x] Catch and rethrow exception thrown in STPDiversityGeneratorRTC instead of letting the program to crash.
• [ ] ~Put in-class struct declarations in a separate inline file in STPDiversityGeneratorRTC.~

General

• Rename STPSettings to STPSetting to resolve naming conflict with namespace STPSettings.
• Remove the following settings from STPHeightfieldSettings, to address the fact that heightmap generation is now user-defined. cpp float Scale; unsigned int Octave; float Persistence; float Lacunarity;
• Improve exception catching and handling.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - New Biome and Height Generator

Release 0.4.0

Biomemap generator 🏭

• Abandon linked list structure of STPLayer. Add a separate master manager STPLayerManager.
• Re-implement STPBiomeFactory with a better programmable interface, as well as concurrent heightmap generation and memory management.
• Integrate demo biome generator into the generator pipeline.
• Import settings from Biome.ini.
• Biomemap can now be stored in STPChunk and loaded into OpenGL rendering buffer.

Generalised heightmap generator

• Add STPDiversityGenerator which contains some high-level helper functions for simple runtime complication.
• Allow programmable heightmap generation, such that user can develop their biome-specific algorithms.
• Allow runtime-compiled script for heightmap generator. Current heightmap generation pipeline will become deprecated and change to programmable pipeline in future release.

Other fixes and improvement 🔧

• Move some include into source code.
• Change some names for consistent programming style and conflict resolution.
• Deprecate INI entry mapOffsetY.

Also note that all heightmap generator parameters like octave and persistence will be depreacted in future release.

• Separate chunk management engine and heightfield generator, so creating renderer won't lead to a chained creation of STPChunkManager, STPChunkProvider and STPHeightfieldGenerator. Instead, individual parts need to be created by user and linked to the dependee program by reference. This gives user better controls on whether they want to share it with other pipeline, for example.
• Add a high-level management unit STPWorldManager in case developer doesn't need such control mentioned above and wish to init the generator with ease.

For more technical details please refer to the pull request page 👉🏼 #19.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Generalised Heightmap Generator

Release 0.4.0-Preview.3

In this release we aim to separate heightmap generator from STPHeightfieldGenerator and allow user and developer to create their own algorithms and parameter sets for each biome. Scripts for heightmap generator will be compiled in runtime.

Highlights

• Add a high-level wrapper class STPDiversityGenerator which includes high-level functions for easy runtime-compiled CUDA scripts.
• Add various helper functions in the class above for script complication, linking and management.
• Extend STPDeviceErrorHandler so it can handle all APIs used (cuda, cudart and nvrtc) within our program.

NVRTC is required in order to compile and execute runtime script.

General fixes and improvement

• Fix some documentation typos.
• Move some header includes into source file to reduce compile time, and remove unnecessary headers.
• Fix incorrectly defined copy/move constructor/assignment operator in various classes.
• Add virtual to destructor of classes containing pure virtual function(s).

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Enable Biomemap Generation

Release 0.4.0-Preview.2

In this release our heightfield generator can generate biome map and store the map for each chunk. Later we will integrate heightmap generation with biomemap to create a multi-biome procedural world.

Highlights

• Break down chunk management engines into individuals, and remove automatica implicit constructions. For example previously instantiating STPChunkProvider will have STPHeightfieldGenerator instantiated automatically, now STPHeightfieldGenerator is passed as a reference into the contructor call of STPChunkProvider.
• All parameters passed to above said class are now by reference instead of copy, such that if one wishes to keep using low-level API call (constructing each component individually), all components and settings need to be kept alive until the engine is destroied.
• Add STPWorldManager as a high-level API for easy initialisation of all generation engine components.

The purpose of doing this is to create some level of abstraction for each component, such that specific functions from each stage of the pipeline can be called directly from external environment. This will be particularly useful for setting up biome generator.

• Allow attaching a concrete instance of STPBiomeFactory to STPWorldManager.
• Allow heightfield generator linked with biome factory for biomemap generation.
• Enable non-blocking biomemap generation.
• Load biomemap into OpenGL buffer and enable biomemap to be visualised in fragment shader.

General improvement and fixes

• Add const to some member functions of chunk engine components so they can be easily called by the returning reference from STPWorldManager.
• Rename sample_cached() in STPLayer to retrieve() to avoid confusion and accidental call to sample() when writing concrete classes of STPLayer.
• Rename create() in STPLayerManager to insert().
• Remove seed and salt argument in the insert() above, allowing more abstract constructor of derived STPLayer

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Setup Biomemap Generator

Release 0.4.0-Preview.1

This release mainly focuses on refining implementations for biomemap generator so it can be integrated into our heightfield generator later.

Biome implementation

• Enhance memory management for STPLayer, remove explicit memory allocation and deletion, instead it's now automatically managed by a new class STPLayerManager.
• Enable loading biome settings from Biome.ini.
• Add biome map storage into STPChunk.
• Reimplement STPBiomeFactory for safe parallel biome map generation.
• Implement biome layer chain for demo purpose.
• Integrate layer implementation supplier into STPBiomeFactory.
• Add linker function in STPHeightfieldGenerator and allow linking STPBiomeFactory with it for biome map generation.
• Add biome map generation options to STPHeightfieldGenerator, however STPHeightfieldGenerator is still not able to generate biome map, integration will yet be implemented until further notice.

Changes

• Remove using in headers to avoid naming conflict.
• Remove mapOffsetY from INI entry.
• Change namespace STPBiome to STPDiversity to resolve naming conflict with class STPBiome.
• Rename some files to make our programming style more consistent, such as from STPBiome_def.h to STPBiomeDefine.h.
• Remove meaningless const qualifier for value return type.
• Implement device pointer deleter, and all device memory are now managed by std::unique_ptr.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Simplification

Release 0.3.5

Highlights

There're not many major functional changes introduced in this release, but fixes and simplification to our program so less setup work is required before launching the terrain generator.

• Remove STPHeightfieldSettings from CUDA cosntant memory due to some performance concern about random access, especially during the phase of hydraulic erosion. Now STPHeightfieldSettings is initialised statefully, bounded to STPHeightfieldGenerator class.
• Remove the necessity to call setErosionIterationCUDA() separately, erosion iteration count is now stored in STPRainDropSettings and terrain generator will initialise erosion generator automatically during object construction.
• Cache ErosionBrushIndices and ErosionBrushWeights in STPRainDropSettings on host memory using std::vector so the code base is more neat without the need to implement all copy/move constructor/assignment operator explicitly.
• Add function makeDeviceAvailable() and omitDeviceAvailable() in STPRainDropSettings to give user control when to transfer host cache to device memory and invalidate device memory.
• Separate header and source code for STPRainDropSettings to cut down coompile time.

Fixes and improvement

• Move the definition of some private class/structure to source file so they are opaque.
• Fix incorrectly set chunk rendering buffer loading checker due to the removal of some previously deprecated features, causing all rendering chunks to be mapped and reloaded every frame even if all of them have been loaded.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Texture Seams Elimination

Release 0.3.4

What's new

• Merging layered rendering buffer into a single rendering buffer, such that texture filtering and edge interpolation that fixes terrain seams are now automatically performed by OpenGL. (#17)
• Completely elimiate normal map edge artifact by selectively pre-copying the edge from previous rendering buffers selectively.
• Post erosion interpolation has been removed completely.
• Implementing built-in memory pool introduced in CUDA 11.2.
• Group rendering buffer updates for all required chunks into a single API call/kernel launch.
• Adding error checking for all CUDA API calls.
• CUDA kernel launch configurations are now determined dynamically, also grid size calculation has been improved to address the case when texture size or raindrop count etc. are not a whole number multiple of block size. (#16)

Fixes and improvements

• Fix an incorrect indexing calculation during rendering buffer generation.
• Remove some redundant const keyword when pass by value.
• Factorise cudaStream_t usage, program reuses stream more efficiently.
• Combine some initialisation functions into the class constructor call so they don't need to be called explicitly.
• Relace std::list with std::vector where applicable.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Introducing Free-slip Hydraulic Erosion

Release 0.3.0

This is a update for #15 to address suggestion pointed out in #1.

New feature

• Water droplet can now freely move to their respective neighbour chunks instead of ending its lifetime early.
• Adding two INI lines freeSlipX and freeSlipZ, denotes the number of chunk in X and Z direction that will be used as neighbours for free-slip synchronisation.
• Implementing local to global index lookup table to reference neighbour chunks efficiently.
• Implementing interpolation index lookup table to correct edge seams efficiently.
• Update CUDA toolkit version to 11.3 for bug fixe stated in #14, improvement in performance and new features, this will be the minimum requirement in all future releases.

Other improvement

• A major overhaul to kernel launch and host multithread model.
• Remove dedicated normal map cache in memory.
• Integreate heightmap and normalmap into a single texture in rendering buffer.
• Reduce rendering buffer clear buffer size by 20%.
• Merge formatting kernel and normalmap generation kernel into one new kenel, rendering buffer generation kernel, thus cutting down the number of kernel launch and copy both by 20%.
• Precache heightmap into shared memory during normalmap generation.
• Fixing over-allocated memory, improve memory use efficiency.
• cudaStream_t is now pooled.
• General programming practice and style improvement.
• Documentation for free-slip erosion.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Various Optimisation to Free-slip Hydraulic Erosion

Release 0.3.0-Preview.3

Optimisation to free-slip hydraulic erosion (#1)

• Heightmap is now precached into shared memory before normal map calculation for reduced memory latency when the same pixel is accessed upto 9 times.
• Caching from global to shared memory is now done in parallel to utilise as many threads as possible. Improvement to exisiting caching algorithm to make it smarter such that it can determine if the number of thread in a block is sufficient to cache all data in parallel, and reuse threads whenever necessary.
• Adding checker whenever loadChunksAsync() is called and only proceed when there is at least one chunk is not loaded or rendering buffer needs be updated, this avoids unecessary calls to map rendering buffer and chunk checking.
• Removing FP32 normalmap storage, integrating normalmap and heightmap into one rendering buffer, RGB channel being normalmap and A channel being heightmap. Cut down the number of kernel launch and memory copy both by around 20%.
• Integrating FP32-INT16 converter into normalmap generator, and renamed into generateRenderingBufferKERNEL().
• No longer generate and use global-local index and interpolation table when free-slip hydraulic erosion is disabled.

General fixes and improvement

• Fixing syntax error from compiler related to built-in threadpool when emplace_void() function is called.
• Changing new to make_unique for better practice.
• Cut down rendering buffer clear buffer size by 20%.
• Update to address changes in dependencies.
• Fixing incorrect serialisation size.
• Remove some redundant codes.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Improving Free-slip Hydraulic Erosion

Release 0.3.0-Preview.2

CUDA compiler version has been updated to 11.3, future release of SuperTerrain+ will have CUDA 11.3 as minimum requirement due to compatibility restriction

Improvement to free-slip hydraulic erosion algorithm (#1)

• Erosion brush indices are now correctly computed with free-slip range instead of heightmap size.
• Re-program the structure of generateHeightmap() function (now is operator()) to make it more structured and execution-efficient.
• Normal map generation and formation are now done to all neighbours after erosion such that the rendering buffer is correct.
• Seam between chunks is now fixed with post-erosion edge interpolation with interpolation lookup table which is generated during class initialisation.
• A major overhaul to the thread model in host side.

Plan

• Cutting down kernel launch by using device subrountine, i.e., group edge interpolation, normal map generation and formation into one kernel launch, and use flag to identify which operation to perform.
• Change heightfield generator operator() so it can be more flexible and adaptive. For example in current build function forces user to compute normal map and formation in a free-slip manner if free-slip dimension has been defined when initialising the class. A potential solution is to let user input free-slip dimension at function call instead of storing the parameters in the class object.
• Adding support for async device memory management (cudaMallocAsync and cudaFreeAsync) and built-in memory pool introduced in CUDA 11.3
• Cache repeated global memory access on device into shared memory for further performance enhancement.
• Simpify multithread model.

Please refer to the documentation in source code and issue link (https://github.com/stephen-hqxu/superterrainplus/issues/1#issuecomment-868819599) for more developer discussions regarding this lookup table.

Other improvements

• Fully addressed issue identified as in #14, mititgated by updating nvcc to 11.3.
• Reducing the number of memory pool, memory usage now is more efficient.
• Fixing over-allocted device and page-locked memory.
• Refactoring memory copy between device and host.
• Optimise CUDA memory R/W access.
• Constant numbers are computed inside kernel and stored in shared memory to improve performance.
• Erosion brush indices and weights are now cached into shared memory.
• CUDA kernel streams are pooled and reused to reduce stream creation/destroy overhead.
• Fixing data incoherence in multithreaded environment on host by force reloading updated chunks.

There is noticeable performance gain by applying those improvements, which is good.

- C++
Published by stephen-hqxu almost 5 years ago

superterrainplus - Basic Free-slip Hydraulic Erosion

Release 0.3.0-Preview.1

Basic feature with free-slip hydraulic erosion (#1)

• Implementing global-local index converter. We are using a pre-computed index table based on the INI parameters.
• As revised in the previous version, heightmap generation is now split into configurable stages. Chunk compute and loading process are now separated into different functions.
• Improve performance of sparse data copy between host and device.

For more development discussions, please refer to the issue page :point_right: https://github.com/stephen-hqxu/superterrainplus/issues/1#issuecomment-814930153

---

Issues and "not yet implemented" with the current preview

• Data races identified, the same chunk is modified by different threads at the same time, causing some chunks not computed correctly. For a temporary workaround, multithreading is disabled.
• Erosion brush calculation is not correct with free-slip range.
• Normal map and formatting are not updated after eroding the central chunk, such that chunks are not seamless.
• Though not noticable on the generated terrain, erosion brush indices are no longer correct under neighbour chunk logic.
• Normal map is not seamless.

Possible Improvement

• User should be given an option to disable free-slip hydraulic erosion. To disable this in the current build, set free slip chunk range to (1,1). However this doesn't effectively prevent the generation of global-local index table, even though global and local index are the same.
• Thread model looks excessively complicated.

Other improvements

• Improving documentation of how to configure and use the free-slip erosion system.

- C++
Published by stephen-hqxu about 5 years ago

superterrainplus - Implementing Suggested Features

Release 0.2.5

Highlights

• Refactor STPChunkProvider and STPHeightfieldGenerator. STPImageConverter has been removed. Now all heightfield generations are performed step-wise using flags, meaning programmers can have their own combination of generation procedures instead of forcing them to generate everything that they may not need. This also provide benefits to break the system down into modules and improve maintainability.

Chunk serialisation (#7)

• STPChunk can output and input chunk data into and from stream.
• Test program implemented.
• Currently the program doesn't make use of this functionality. Later the program will selectively serialise chunks to secondary storage when it reaches the upper memory bound.

Free-slip hydraulic erosion (#1)

• We are working on this suggested feature right now, and the pre-release version should be published soon.

Fixes and improvements

• Fixing precision problems for the random number generator in biome layer.
• Fixing GPU side crash due to the removal of floor function after some updates of Visial Studio 2019, as being reported in issue #14.
• Reducing usage of raw pointers.
• Reducing unnecessary thread usage.
• Improving error handling, program will throw errors when chunk compute cannot be completed.
• Removing deprecated documentation.
• Fixing illegal memory access.

Plan

In version 1.0.0, Super Terrain + will be capable of generating multi-biome terrain, and will be more stable than the current version. It should be coming soon, we are looking forward to releasing alpha version in the near future. Later, we will implement more terrain generation algorithm, see repo project page.

- C++
Published by stephen-hqxu about 5 years ago

superterrainplus - Preparing for Biome Generator

Release 0.2.3

Highlights

• Reduce the dynamic memory usage.
• More semantics improvement from the last updates.
• More test cases implemented.

Updates to biome generator

• Implement multi-threaded biome map generator with reusable caching. > I finally found a way to implmenet biome generator more efficiently in multi-threaded environement by spliting generations into different threads and assigning each worker its own layer cache.
• Implement more biome layers for demo.
• Engine-breaking bug fixes.

Memory pool (#9)

• Memory pool has fully customisable allocator and deallocator, and user-defined arguments.
• Implement memory pool for GPU compute cache on heightfield generator to improve performance.
• Implement memory pool for biome map generator to reuse biome cache for its multithreaded environment.

Fixes

• Fixing some potential race conditions for the memory pool.

- C++
Published by stephen-hqxu over 5 years ago

superterrainplus - Cache Optimisation and Test Framework

Release 0.2.0

Release highlight

• More types of constructors are implemented accroding to #3. Some functional classes such as rendering classes that do not expect to be created from existing instance will be given delete modifiers, utility classes like settings have been given explicit implementations.

More constructors will be coming in the future, after the program comes out from alpha state. Now my priority is to finish biome generator.

• Improving programming practice, especially adding const for certain functions such that it can be used in constant semantics.
• GPU memory and cache optimisation for constant values and random-read-only lookup tables.
• Improve compiler settings for better debug experience and performance under relase mode.
• Adding debug preprocessor _STP_DEBUG_.
• Updating minimum system requirement, see README.
• Minor bug fixes for biome generators.

Introduction to test framework

• Adding testing project with test framework.
• Implementing a dedicated test report format for Super Terrain +.
• Implementing tests along with bug fixes (that's why tests are important).

More test cases are coming...

Bug fix

• Providing solution for #2, a copy of host class will be kept to retain the underlying device pointers.
• Providing solution for #10, it's yet implemented in the current release but it can be easily workarounded by simply reduce the number of thread per block.
• Fixing typo for README file.

We are working on biome generator to our best. Source code for biomes are released together for public debugging and internal testing purposes only.

Proposed features

• #11 requetsed a terrain collision first-person camera.

- C++
Published by stephen-hqxu over 5 years ago

superterrainplus - Fixes and Improvements

Release 0.1.2

Release highlights

• Biome generation progressed, it's not currently implemented by the engine, the interface however can be used. > Biome generator framework and work-in-progress demo programs are included in the source code, however it is untested, unoptimised, unimplemented and unstable and therefore not considered as a release. Use at your own risk.
• Multithreading feature for biome generation has been removed due to bad performance.
• Implementation of various constructor and assignment operator for primary interfaced classes.
• Fix a bug that prevent from compiling the solution, due to the incorrect chosen compiler version.

- C++
Published by stephen-hqxu over 5 years ago

superterrainplus - Initial 2D Generator

Release 0.1.0

Release highlight - 3D mesh terrain using 2D heightmap approach

• Fully adjustable terrain generation, highly parameterised.
• Physics-based hydraulic erosion using particles.
• Improved (by myself) simplex noise algorithm with more customisable gradients and random number generator.
• Terrain generated using CUDA, make sure you have a CUDA compatible GPU, with compute capability > 7.5.
• OpenGL 4.5 features are used, make sure your GPU can handle v4.5. > You will need the OpenGL library to run the program, see glad.c attached.
• Multi-threading optimised.
• Infinite terrain with tile-based mesh (instead of regular chunk-based), with massively reduced bandwidth overhead.

Future Plan

• Add cmake support so it can be compiled on different platform and doesn't need to rely on visual studio.
• Mutli-biome generation with minecraft biome generation algorithm (so-called "scaling algorithm"), with different parameters in different biomes. Currently in progress.
• Volumetric terrain generation using marching-cubes.
• Procedural planet generation.
• Ray-traced shadow and reflection with optix7.
• Volumetric cloud with ray-marching algorithm and voroni noise.

All source codes are clearly documented.

- C++
Published by stephen-hqxu over 5 years ago